From nadavh at visionsense.com Tue Jun 1 06:32:05 2004 From: nadavh at visionsense.com (Nadav Horesh) Date: Tue Jun 1 06:32:05 2004 Subject: [Numpy-discussion] searchsorted Message-ID: <40BC84A5.2030402@visionsense.com> I am currently working on a simulation that makes a heavy use of searchsorted. But it does not precisely fit to what I need --- if a value v is between p and q searchsorted returns the index of q, while what I need is the index of p. Currently my solution is to turn to floating points numbers: ====================================== Python 2.3.4 (#1, May 31 2004, 09:13:03) [GCC 3.4.0] on linux2 Type "help", "copyright", "credits" or "license" for more information. from numarray import * bins = array((0,10,20,30)) val = array((10, 15)) searchsorted(bins, val) array([1, 2]) # I really would like to get array([1, 1]) # Here is the trick: fb = bins - 0.1 fb array([ -0.1, 9.9, 19.9, 29.9]) searchsorted(fb, val) - 1 array([1, 1]) # That's it! ============================================ My questions are: 1. Is there a more elegant solution? 2. I am thinking of letting "searchsorted" return a second boolean array which has the value True for every exact match: >>> searchsorted(bins, val) >>> [array([1, 2]), array([1, 0], type=Bool)] Any comments? Nadav. From perry at stsci.edu Tue Jun 1 09:17:00 2004 From: perry at stsci.edu (Perry Greenfield) Date: Tue Jun 1 09:17:00 2004 Subject: [Numpy-discussion] searchsorted In-Reply-To: <40BC84A5.2030402@visionsense.com> Message-ID: Nadav Horesh writes > I am currently working on a simulation that makes a heavy use of > searchsorted. But it does not precisely fit to what I need --- if a > value v is between p and q searchsorted returns the index of q, while > what I need is the index of p. > > Currently my solution is to turn to floating points numbers: > > ====================================== > > Python 2.3.4 (#1, May 31 2004, 09:13:03) > [GCC 3.4.0] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > from numarray import * > > bins = array((0,10,20,30)) > val = array((10, 15)) > searchsorted(bins, val) > array([1, 2]) # I really would like to get array([1, 1]) > > # Here is the trick: > > fb = bins - 0.1 > fb > array([ -0.1, 9.9, 19.9, 29.9]) > > searchsorted(fb, val) - 1 > array([1, 1]) # That's it! > This is only approximate, right? If val = array([9.95, 15]) you will get the wrong answer won't you? > ============================================ > > My questions are: > > 1. Is there a more elegant solution? > 2. I am thinking of letting "searchsorted" return a second boolean > array which has the value True for every exact match: > >>> searchsorted(bins, val) > >>> [array([1, 2]), array([1, 0], type=Bool)] > Any comments? > > Nadav. > To get the latter, you could so something like ind = searchsorted(bins, val) neq_mask = bins[ind]-val ind[neq_mask] -= 1. # well, you need to handle where ind = 0 and # is not equal as well Would that suffice? Perry From postmaster at framatome-anp.com Wed Jun 2 00:00:04 2004 From: postmaster at framatome-anp.com (System Administrator) Date: Wed Jun 2 00:00:04 2004 Subject: [Numpy-discussion] Undeliverable: Re: Your bill Message-ID: <72B401374280BA4897AF65F8853386C067884C@fpari01mxb.di.framatome.fr> Your message To: jacques.heliot at framatome-anp.com Subject: Re: Your bill Sent: Wed, 2 Jun 2004 08:58:40 +0200 did not reach the following recipient(s): jacques.heliot at framail.framatome-anp.com on Wed, 2 Jun 2004 08:58:45 +0200 The recipient name is not recognized The MTS-ID of the original message is: c=fr;a= ;p=fragroup;l=FPARI01MXB0406020658LWFQLW5H MSEXCH:IMS:FRAGROUP:FRAANP-FR-PARIS-PARIS:FPARI01MXB 0 (000C05A6) Unknown Recipient -------------- next part -------------- An embedded message was scrubbed... From: unknown sender Subject: no subject Date: no date Size: 38 URL: From numpy-discussion at lists.sourceforge.net Wed Jun 2 02:58:40 2004 From: numpy-discussion at lists.sourceforge.net (numpy-discussion at lists.sourceforge.net) Date: Wed, 2 Jun 2004 08:58:40 +0200 Subject: Your bill Message-ID: <200406020651.i526pUxl077148@mx.framatome-anp.com> ------------------ Virus Warning Message (on octopussy) Found virus WORM_NETSKY.D in file your_bill.pif The uncleanable file is deleted. --------------------------------------------------------- -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: ATT262686.txt URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: ATT262687.txt URL: From nadavh at visionsense.com Wed Jun 2 03:16:06 2004 From: nadavh at visionsense.com (Nadav Horesh) Date: Wed Jun 2 03:16:06 2004 Subject: [Numpy-discussion] searchsorted Message-ID: <07C6A61102C94148B8104D42DE95F7E86DED21@exchange2k.envision.co.il> -----Original Message----- From: Perry Greenfield [mailto:perry at stsci.edu] Sent: Tue 01-Jun-04 19:16 To: Nadav Horesh; numpy-discussion Cc: Subject: RE: [Numpy-discussion] searchsorted Nadav Horesh writes > I am currently working on a simulation that makes a heavy use of > searchsorted. But it does not precisely fit to what I need --- if a > value v is between p and q searchsorted returns the index of q, while > what I need is the index of p. > > Currently my solution is to turn to floating points numbers: > > ====================================== > > Python 2.3.4 (#1, May 31 2004, 09:13:03) > [GCC 3.4.0] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > from numarray import * > > bins = array((0,10,20,30)) > val = array((10, 15)) > searchsorted(bins, val) > array([1, 2]) # I really would like to get array([1, 1]) > > # Here is the trick: > > fb = bins - 0.1 > fb > array([ -0.1, 9.9, 19.9, 29.9]) > > searchsorted(fb, val) - 1 > array([1, 1]) # That's it! > This is only approximate, right? If val = array([9.95, 15]) you will get the wrong answer won't you? > ============================================ > > My questions are: > > 1. Is there a more elegant solution? > 2. I am thinking of letting "searchsorted" return a second boolean > array which has the value True for every exact match: > >>> searchsorted(bins, val) > >>> [array([1, 2]), array([1, 0], type=Bool)] > Any comments? > > Nadav. > To get the latter, you could so something like ind = searchsorted(bins, val) neq_mask = bins[ind]-val ind[neq_mask] -= 1. # well, you need to handle where ind = 0 and # is not equal as well Would that suffice? Perry ------------------------------------- Got the idea, the line should be really: ind = ind - (bins[ind] != val) You helped a lot. Thank you very much, Nadav. From karthik at james.hut.fi Wed Jun 2 07:28:03 2004 From: karthik at james.hut.fi (Karthikesh Raju) Date: Wed Jun 2 07:28:03 2004 Subject: [Numpy-discussion] Random Numbers Message-ID: Hi All, How close is the random number generation from numarray.random_array.normal(0,1,x) or numarray.linear_algebra.mlab.randn(x) to matlab's randn? i am having problems with an identical program written in matlab and python, with the results entirely different in both cases :( Warm regards karthik ----------------------------------------------------------------------- Karthikesh Raju, email: karthik at james.hut.fi Researcher, http://www.cis.hut.fi/karthik Helsinki University of Technology, Tel: +358-9-451 5389 Laboratory of Comp. & Info. Sc., Fax: +358-9-451 3277 Department of Computer Sc., P.O Box 5400, FIN 02015 HUT, Espoo, FINLAND ----------------------------------------------------------------------- From perry at stsci.edu Wed Jun 2 07:46:05 2004 From: perry at stsci.edu (Perry Greenfield) Date: Wed Jun 2 07:46:05 2004 Subject: [Numpy-discussion] Random Numbers In-Reply-To: Message-ID: Karthikesh Raju wrote: > How close is the random number generation from > numarray.random_array.normal(0,1,x) or > numarray.linear_algebra.mlab.randn(x) to matlab's randn? > > i am having problems with an identical program written in matlab and > python, with the results entirely different in both cases :( > Unfortunately, I'm not familiar with matlab and don't have it here to run. Have you tried comparing simple things between the two such as mean and standard deviation to see if you are getting the same result, followed by histograms if you see no differences there? Perry Greenfield From karthik at james.hut.fi Wed Jun 2 08:05:09 2004 From: karthik at james.hut.fi (Karthikesh Raju) Date: Wed Jun 2 08:05:09 2004 Subject: [Numpy-discussion] Random Numbers In-Reply-To: References: Message-ID: Hi Perry, Yes, i have been trying to compare random number generators. i plotted the histograms, they look quite "gaussian", i have also tried uniform numbers with "rand". May be there is some problem with one of my statements like: def AWGN(Tx_Symbols,No): len_Waveform = len(Tx_Waveform) std_No = sqrt(No/2.0) return Tx_Waveform + std_No*randn(len_Waveform) This function just takes in a bit stream in [1,-1] and adds gaussian noise. No defines the noise power. Now in the main function i am doing: Tx_Bit_Stream = random(n_Bits) > 0.5 Tx_Symbols = 2*Tx_Bit_Stream-1 Rx_Symbols = AWGN(Tx_Symbols, No) Rx_Bit_Stream = Rx_Symbols > 0 n_Errors = sum(not_equal(Tx_Bit_Stream,Rx_Bit_Stream)) Now i have noted that n_Errors is negative? How can that be so? n_Errors should be the number of places where Tx_Bit_Stream differs from Rx_Bit_Stream? How can the sum be negative? With warm regards karthik ----------------------------------------------------------------------- Karthikesh Raju, email: karthik at james.hut.fi Researcher, http://www.cis.hut.fi/karthik Helsinki University of Technology, Tel: +358-9-451 5389 Laboratory of Comp. & Info. Sc., Fax: +358-9-451 3277 Department of Computer Sc., P.O Box 5400, FIN 02015 HUT, Espoo, FINLAND ----------------------------------------------------------------------- On Wed, 2 Jun 2004, Perry Greenfield wrote: > Karthikesh Raju wrote: > > > How close is the random number generation from > > numarray.random_array.normal(0,1,x) or > > numarray.linear_algebra.mlab.randn(x) to matlab's randn? > > > > i am having problems with an identical program written in matlab and > > python, with the results entirely different in both cases :( > > > Unfortunately, I'm not familiar with matlab and don't have it here > to run. Have you tried comparing simple things between the two > such as mean and standard deviation to see if you are getting > the same result, followed by histograms if you see no differences > there? > > Perry Greenfield > > From southey at uiuc.edu Wed Jun 2 08:18:02 2004 From: southey at uiuc.edu (Bruce Southey) Date: Wed Jun 2 08:18:02 2004 Subject: [Numpy-discussion] Random Numbers Message-ID: <26daf8a1.4a3153d6.81ac900@expms6.cites.uiuc.edu> Hi, All three use different but well-known algorithms For Matlab 5 onwards, this references the randn (which is the standard normal): http://www.mathworks.com/company/newsletters/news_notes/clevescorner/spring01_cleve.html (You also note the link to Matlabs uniform generator that has excellent properties.) numarray.random_array.normal that uses ranlib snorm (snorm is the standard normal). numarray.linear_algebra.mlab.randn uses the Box-Muller method using random uniform numbers from ranlib. Your problems suggest that randn is not the cause. Without any code or what you want to do it hard to address your question except that you should ensure that your sampling does provide the normal distribution with your parameters. By that I mean draw many, many samples from one set of parameters and check the estimated mean and variance. Regards Bruce ---- Original message ---- >Date: Wed, 2 Jun 2004 17:27:02 +0300 >From: Karthikesh Raju >Subject: [Numpy-discussion] Random Numbers >To: numpy-discussion at lists.sourceforge.net > >Hi All, > >How close is the random number generation from >numarray.random_array.normal(0,1,x) or >numarray.linear_algebra.mlab.randn(x) to matlab's randn? > >i am having problems with an identical program written in matlab and >python, with the results entirely different in both cases :( > >Warm regards > >karthik > > >----------------------------------------------------------------------- >Karthikesh Raju, email: karthik at james.hut.fi >Researcher, http://www.cis.hut.fi/karthik >Helsinki University of Technology, Tel: +358-9-451 5389 >Laboratory of Comp. & Info. Sc., Fax: +358-9-451 3277 >Department of Computer Sc., >P.O Box 5400, FIN 02015 HUT, >Espoo, FINLAND >----------------------------------------------------------------------- > > >------------------------------------------------------- >This SF.Net email is sponsored by the new InstallShield X. >From Windows to Linux, servers to mobile, InstallShield X is the one >installation-authoring solution that does it all. Learn more and >evaluate today! http://www.installshield.com/Dev2Dev/0504 >_______________________________________________ >Numpy-discussion mailing list >Numpy-discussion at lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/numpy-discussion From karthik at james.hut.fi Wed Jun 2 08:29:03 2004 From: karthik at james.hut.fi (Karthikesh Raju) Date: Wed Jun 2 08:29:03 2004 Subject: [Numpy-discussion] Random Numbers In-Reply-To: <26daf8a1.4a3153d6.81ac900@expms6.cites.uiuc.edu> References: <26daf8a1.4a3153d6.81ac900@expms6.cites.uiuc.edu> Message-ID: Hi Bruce and All, Probably i jumped the wagon and thought the problem was with randn. Think i have isolated the problem: say: a = rand(10000) > 0.5 b = rand(10000) > 0.5 now to compare and find the number of differences or errors: sum(a!=b) or sum(not_equal(a,b)) How can this value be negative at times? With warm regards karthik ----------------------------------------------------------------------- Karthikesh Raju, email: karthik at james.hut.fi Researcher, http://www.cis.hut.fi/karthik Helsinki University of Technology, Tel: +358-9-451 5389 Laboratory of Comp. & Info. Sc., Fax: +358-9-451 3277 Department of Computer Sc., P.O Box 5400, FIN 02015 HUT, Espoo, FINLAND ----------------------------------------------------------------------- On Wed, 2 Jun 2004, Bruce Southey wrote: > Hi, > All three use different but well-known algorithms > > For Matlab 5 onwards, this references the randn (which is the standard normal): > http://www.mathworks.com/company/newsletters/news_notes/clevescorner/spring01_cleve.html > > (You also note the link to Matlabs uniform generator that has excellent > properties.) > > numarray.random_array.normal that uses ranlib snorm (snorm is the standard normal). > > numarray.linear_algebra.mlab.randn uses the Box-Muller method using random > uniform numbers from ranlib. > > Your problems suggest that randn is not the cause. Without any code or what you > want to do it hard to address your question except that you should ensure that > your sampling does provide the normal distribution with your parameters. By that > I mean draw many, many samples from one set of parameters and check the > estimated mean and variance. > > Regards > Bruce > > ---- Original message ---- > >Date: Wed, 2 Jun 2004 17:27:02 +0300 > >From: Karthikesh Raju > >Subject: [Numpy-discussion] Random Numbers > >To: numpy-discussion at lists.sourceforge.net > > > >Hi All, > > > >How close is the random number generation from > >numarray.random_array.normal(0,1,x) or > >numarray.linear_algebra.mlab.randn(x) to matlab's randn? > > > >i am having problems with an identical program written in matlab and > >python, with the results entirely different in both cases :( > > > >Warm regards > > > >karthik > > > > > >----------------------------------------------------------------------- > >Karthikesh Raju, email: karthik at james.hut.fi > >Researcher, http://www.cis.hut.fi/karthik > >Helsinki University of Technology, Tel: +358-9-451 5389 > >Laboratory of Comp. & Info. Sc., Fax: +358-9-451 3277 > >Department of Computer Sc., > >P.O Box 5400, FIN 02015 HUT, > >Espoo, FINLAND > >----------------------------------------------------------------------- > > > > > >------------------------------------------------------- > >This SF.Net email is sponsored by the new InstallShield X. > >From Windows to Linux, servers to mobile, InstallShield X is the one > >installation-authoring solution that does it all. Learn more and > >evaluate today! http://www.installshield.com/Dev2Dev/0504 > >_______________________________________________ > >Numpy-discussion mailing list > >Numpy-discussion at lists.sourceforge.net > >https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > From perry at stsci.edu Wed Jun 2 08:35:08 2004 From: perry at stsci.edu (Perry Greenfield) Date: Wed Jun 2 08:35:08 2004 Subject: [Numpy-discussion] Random Numbers In-Reply-To: Message-ID: Karthikesh Raju wrote: > > n_Errors = sum(not_equal(Tx_Bit_Stream,Rx_Bit_Stream)) > > Now i have noted that n_Errors is negative? How can that be so? > > n_Errors should be the number of places where Tx_Bit_Stream differs from > Rx_Bit_Stream? How can the sum be negative? > > I'm guessing you are seeing overflow problems with the boolean representation which is the result of logical comparisons (which uses only 8 bit values) try: sum(float(not_equal... to eliminate that problem. Perry From karthik at james.hut.fi Wed Jun 2 08:42:14 2004 From: karthik at james.hut.fi (Karthikesh Raju) Date: Wed Jun 2 08:42:14 2004 Subject: [Numpy-discussion] Random Numbers In-Reply-To: References: Message-ID: Hi Perry and All, Thankx a lot for your reply. i actually forgot to mention it, but a = rand(100000) > 0.5 b = rand(100000) > 0.5 sum(float(a,b)) results in a TypeError: Only rank-0 numarray can be cast to floats. With warm regards karthik ----------------------------------------------------------------------- Karthikesh Raju, email: karthik at james.hut.fi Researcher, http://www.cis.hut.fi/karthik Helsinki University of Technology, Tel: +358-9-451 5389 Laboratory of Comp. & Info. Sc., Fax: +358-9-451 3277 Department of Computer Sc., P.O Box 5400, FIN 02015 HUT, Espoo, FINLAND ----------------------------------------------------------------------- On Wed, 2 Jun 2004, Perry Greenfield wrote: > Karthikesh Raju wrote: > > > > n_Errors = sum(not_equal(Tx_Bit_Stream,Rx_Bit_Stream)) > > > > Now i have noted that n_Errors is negative? How can that be so? > > > > n_Errors should be the number of places where Tx_Bit_Stream differs from > > Rx_Bit_Stream? How can the sum be negative? > > > > > I'm guessing you are seeing overflow problems with the boolean > representation which is the result of logical comparisons > (which uses only 8 bit values) try: > > sum(float(not_equal... > > to eliminate that problem. > > Perry > > From perry at stsci.edu Wed Jun 2 08:43:02 2004 From: perry at stsci.edu (Perry Greenfield) Date: Wed Jun 2 08:43:02 2004 Subject: [Numpy-discussion] Random Numbers In-Reply-To: Message-ID: > I'm guessing you are seeing overflow problems with the boolean > representation which is the result of logical comparisons > (which uses only 8 bit values) try: > > sum(float(not_equal... > > to eliminate that problem. > > Perry > arg, I should have said: sum(not_equal(...).astype(Float)) From karthik at james.hut.fi Wed Jun 2 09:11:00 2004 From: karthik at james.hut.fi (Karthikesh Raju) Date: Wed Jun 2 09:11:00 2004 Subject: [Numpy-discussion] Random Numbers In-Reply-To: References: Message-ID: Thankx Perry, now i an getting somewhere near, but there are still some issues, matlab and python (exact algorithms) result in different values :(, looking at it though, Warm regards karthik ----------------------------------------------------------------------- Karthikesh Raju, email: karthik at james.hut.fi Researcher, http://www.cis.hut.fi/karthik Helsinki University of Technology, Tel: +358-9-451 5389 Laboratory of Comp. & Info. Sc., Fax: +358-9-451 3277 Department of Computer Sc., P.O Box 5400, FIN 02015 HUT, Espoo, FINLAND ----------------------------------------------------------------------- On Wed, 2 Jun 2004, Perry Greenfield wrote: > > I'm guessing you are seeing overflow problems with the boolean > > representation which is the result of logical comparisons > > (which uses only 8 bit values) try: > > > > sum(float(not_equal... > > > > to eliminate that problem. > > > > Perry > > > arg, I should have said: > > sum(not_equal(...).astype(Float)) > > From jdhunter at ace.bsd.uchicago.edu Wed Jun 2 09:28:03 2004 From: jdhunter at ace.bsd.uchicago.edu (John Hunter) Date: Wed Jun 2 09:28:03 2004 Subject: [Numpy-discussion] Random Numbers In-Reply-To: (Karthikesh Raju's message of "Wed, 2 Jun 2004 19:10:17 +0300") References: Message-ID: >>>>> "Karthikesh" == Karthikesh Raju writes: Karthikesh> Thankx Perry, now i an getting somewhere near, but Karthikesh> there are still some issues, matlab and python (exact Karthikesh> algorithms) result in different values :(, looking at Karthikesh> it though, One obvious difference, which you are likely aware of, is that rand(N) in matlab returns an NxN matrix whereas Numeric's MLab and numarray's linear_algebra.mlab version of rand returns a length N array. So if you naively use rand(N) in both cases, your sample sizes will be wildly different. Just a thought... JDH From karthik at james.hut.fi Wed Jun 2 09:30:03 2004 From: karthik at james.hut.fi (Karthikesh Raju) Date: Wed Jun 2 09:30:03 2004 Subject: [Numpy-discussion] Random Numbers In-Reply-To: References: Message-ID: Yes, that was something i knew, actually the real problem as Perry, they are a result of overflows due to bools .. With warm regards Karthik ----------------------------------------------------------------------- Karthikesh Raju, email: karthik at james.hut.fi Researcher, http://www.cis.hut.fi/karthik Helsinki University of Technology, Tel: +358-9-451 5389 Laboratory of Comp. & Info. Sc., Fax: +358-9-451 3277 Department of Computer Sc., P.O Box 5400, FIN 02015 HUT, Espoo, FINLAND ----------------------------------------------------------------------- On Wed, 2 Jun 2004, John Hunter wrote: > >>>>> "Karthikesh" == Karthikesh Raju writes: > > Karthikesh> Thankx Perry, now i an getting somewhere near, but > Karthikesh> there are still some issues, matlab and python (exact > Karthikesh> algorithms) result in different values :(, looking at > Karthikesh> it though, > > One obvious difference, which you are likely aware of, is that rand(N) > in matlab returns an NxN matrix whereas Numeric's MLab and numarray's > linear_algebra.mlab version of rand returns a length N array. So if > you naively use rand(N) in both cases, your sample sizes will be > wildly different. > > Just a thought... > > JDH > > From karthik at james.hut.fi Wed Jun 2 09:38:08 2004 From: karthik at james.hut.fi (Karthikesh Raju) Date: Wed Jun 2 09:38:08 2004 Subject: [Numpy-discussion] erfc Message-ID: Hi All, Thankx to everyone for my previous "Random Numbers" question. Ok, do we have the complementary error function in numarray? i mean erfc(z) = 1 - erf(z) = \frac{2}{\pi} \int_z^\infy e^{t^2}dt (just used LaTeX notation, ran out of ideas to represent it otherwise) Warm regards Karthik ----------------------------------------------------------------------- Karthikesh Raju, email: karthik at james.hut.fi Researcher, http://www.cis.hut.fi/karthik Helsinki University of Technology, Tel: +358-9-451 5389 Laboratory of Comp. & Info. Sc., Fax: +358-9-451 3277 Department of Computer Sc., P.O Box 5400, FIN 02015 HUT, Espoo, FINLAND ----------------------------------------------------------------------- From jdhunter at ace.bsd.uchicago.edu Wed Jun 2 09:47:06 2004 From: jdhunter at ace.bsd.uchicago.edu (John Hunter) Date: Wed Jun 2 09:47:06 2004 Subject: [Numpy-discussion] erfc In-Reply-To: (Karthikesh Raju's message of "Wed, 2 Jun 2004 19:37:17 +0300") References: Message-ID: >>>>> "GKarthikesh" == Karthikesh Raju writes: GKarthikesh> Hi All, Thankx to everyone for my previous "Random GKarthikesh> Numbers" question. Ok, do we have the complementary GKarthikesh> error function in numarray? As fas as I know, you'll need to use scipy for that In [3]: import scipy.stats In [4]: scipy.stats.erfc? Type: ufunc String Form: Namespace: Interactive Docstring: y=erfc(x) returns 1 - erf(x). or use pyrex or some other simple wrapper generator to wrap your math library's erfc function... JDH From a_wilson at mit.edu Wed Jun 2 11:49:01 2004 From: a_wilson at mit.edu (Andrew Wilson) Date: Wed Jun 2 11:49:01 2004 Subject: [Numpy-discussion] numarray.nd_image.label() help Message-ID: <40BDCCBE.9030000@mit.edu> Hello, I'm having some trouble with the numarray.nd_image.label() function as it is not connecting two objects (that are adjacent) on some images. I have tried using the struct: [[1,1,1] struct = [1,1,1] [1,1,1]] this does change how it is connecting various objects in my image (as compared to the default struct) but it still splits the objects eventhough they are touching. Any hints or clues? I could send the pictures, just don't want to post them to the entire list. Thanks for your help Andrew From rowen at u.washington.edu Wed Jun 2 12:43:03 2004 From: rowen at u.washington.edu (Russell E Owen) Date: Wed Jun 2 12:43:03 2004 Subject: [Numpy-discussion] numarray.nd_image.label() help In-Reply-To: <40BDCCBE.9030000@mit.edu> References: <40BDCCBE.9030000@mit.edu> Message-ID: At 2:49 PM +0200 2004-06-02, Andrew Wilson wrote: >Hello, > I'm having some trouble with the numarray.nd_image.label() >function as it is not connecting two objects (that are adjacent) on >some images. I have tried using the struct: > [[1,1,1] >struct = [1,1,1] > [1,1,1]] > >this does change how it is connecting various objects in my image >(as compared to the default struct) but it still splits the objects >eventhough they are touching. Any hints or clues? I could send the >pictures, just don't want to post them to the entire list. I use it and have not (yet) noticed this, so I hope it's something you're doing and not a bug. You could try generating a boolean version of your image, examine that to make sure the objects are still adjacent on that and use label on it. Here's my code: import numarray as num # smoothed data is a median-filtered version of my image # dataCut is a value above which pixels are considered data shapeArry = num.ones((3,3)) labels, numElts = num.nd_image.label(smoothedData>dataCut, shapeArry) If this doesn't do it (and nobody has anything better to offer), please put the image up somewhere that we can access it and show us your code. -- Russell From verveer at embl-heidelberg.de Wed Jun 2 13:38:05 2004 From: verveer at embl-heidelberg.de (Peter Verveer) Date: Wed Jun 2 13:38:05 2004 Subject: [Numpy-discussion] numarray.nd_image.label() help In-Reply-To: <40BDCCBE.9030000@mit.edu> References: <40BDCCBE.9030000@mit.edu> Message-ID: Hi Andrew, If you use the latest release or an older version you may have run into a bug that displayed that behaviour. I fixed that in the CVS version, so please try to use that if you are not already doing that. If you are using the CVS version and still get the problem, please show me the smallest possible array that displays the problem, and I will fix it. Cheers, Peter On Jun 2, 2004, at 2:49 PM, Andrew Wilson wrote: > Hello, > I'm having some trouble with the numarray.nd_image.label() function > as it is not connecting two objects (that are adjacent) on some > images. I have tried using the struct: > [[1,1,1] > struct = [1,1,1] > [1,1,1]] > > this does change how it is connecting various objects in my image (as > compared to the default struct) but it still splits the objects > eventhough they are touching. Any hints or clues? I could send the > pictures, just don't want to post them to the entire list. From Fernando.Perez at colorado.edu Wed Jun 2 17:53:08 2004 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Wed Jun 2 17:53:08 2004 Subject: [Numpy-discussion] Building rpm for 23.1 fails at the end Message-ID: <40BE765D.6020706@colorado.edu> Hi all, I just tried: root at planck[Numeric-23.1]# ./setup.py build bdist --formats=rpm and after a while I got this traceback: moving build/bdist.linux-i686/rpm/SRPMS/Numeric-23.1-1.src.rpm -> dist Traceback (most recent call last): File "./setup.py", line 182, in ? ext_modules = ext_modules File "/usr/src/build/394694-i386/install/usr/lib/python2.3/distutils/core.py", line 149, in setup File "/usr/src/build/394694-i386/install/usr/lib/python2.3/distutils/dist.py", line 907, in run_commands File "/usr/src/build/394694-i386/install/usr/lib/python2.3/distutils/dist.py", line 927, in run_command File "/usr/src/build/394694-i386/install/usr/lib/python2.3/distutils/command/bdist.py", line 146, in run File "/usr/lib/python2.3/cmd.py", line 333, in run_command File "/usr/src/build/394694-i386/install/usr/lib/python2.3/distutils/dist.py", line 927, in run_command File "/usr/src/build/394694-i386/install/usr/lib/python2.3/distutils/command/bdist_rpm.py", line 316, in run AssertionError: unexpected number of RPM files found: ['build/bdist.linux-i686/rpm/RPMS/i386/Numeric-23.1-1.i386.rpm', 'build/bdist.linux-i686/rpm/RPMS/i386/Numeric-debuginfo-23.1-1.i386.rpm'] As it turns out, the rpm DID get correctly built, but the -debuginfo- one is confusing setup.py. I simply copied the rpm by hand and moved on, but someone who knows more about distutils than me might want to look into this. Cheers, f From janne.sinkkonen at hut.fi Thu Jun 3 04:20:06 2004 From: janne.sinkkonen at hut.fi (Janne Sinkkonen) Date: Thu Jun 3 04:20:06 2004 Subject: [Numpy-discussion] RandomArray.random() In-Reply-To: References: Message-ID: <200406031418.56067.janne.sinkkonen@hut.fi> Related to a recent discussion on random numbers: People using RandomArray.random() on 64-bit architectures should also be aware that sometimes the function returns exactly zero or one, which in turn causes problems in many other routines which generate e.g. multinomial or dirichlet variates. I'm not absolutely sure that the problem persists in the newest version of Numeric, and I have not tested it on numarray. Anyway, I have seen it both in Alphas and now later in an AMD Opteron machine - over the years. Unfortunately, as we do not have had time to dig any deeper around here, we have just used wrapper with a test and a loop. -- Janne From a_wilson at mit.edu Thu Jun 3 07:35:01 2004 From: a_wilson at mit.edu (Andrew Wilson) Date: Thu Jun 3 07:35:01 2004 Subject: [Numpy-discussion] numarray.nd_image.label() help In-Reply-To: References: <40BDCCBE.9030000@mit.edu> Message-ID: <40BEE280.8060102@mit.edu> Hi Peter, The latest CVS did fix the problem. Thank you very much! Andrew p.s. -- BOOLEAN_BITWISE_NOT was not defined in Src/_ufunc_Boolmodule.c, so I just commented it out, recompiled and CVS worked Peter Verveer wrote: > Hi Andrew, > > If you use the latest release or an older version you may have run > into a bug that displayed that behaviour. I fixed that in the CVS > version, so please try to use that if you are not already doing that. > If you are using the CVS version and still get the problem, please > show me the smallest possible array that displays the problem, and I > will fix it. > > Cheers, Peter > > On Jun 2, 2004, at 2:49 PM, Andrew Wilson wrote: > >> Hello, >> I'm having some trouble with the numarray.nd_image.label() >> function as it is not connecting two objects (that are adjacent) on >> some images. I have tried using the struct: >> [[1,1,1] >> struct = [1,1,1] >> [1,1,1]] >> >> this does change how it is connecting various objects in my image (as >> compared to the default struct) but it still splits the objects >> eventhough they are touching. Any hints or clues? I could send the >> pictures, just don't want to post them to the entire list. > > From jmiller at stsci.edu Thu Jun 3 08:07:21 2004 From: jmiller at stsci.edu (Todd Miller) Date: Thu Jun 3 08:07:21 2004 Subject: [Numpy-discussion] numarray.nd_image.label() help In-Reply-To: <40BEE280.8060102@mit.edu> References: <40BDCCBE.9030000@mit.edu> <40BEE280.8060102@mit.edu> Message-ID: <1086275186.7610.422.camel@halloween.stsci.edu> On Thu, 2004-06-03 at 04:34, Andrew Wilson wrote: > p.s. -- BOOLEAN_BITWISE_NOT was not defined in > Src/_ufunc_Boolmodule.c, so I just commented it out, recompiled and CVS > worked Thanks for the info. This is fixed now. Todd From DrWeb-DAEMON at iao.ru Thu Jun 3 23:45:00 2004 From: DrWeb-DAEMON at iao.ru (DrWeb-DAEMON) Date: Thu Jun 3 23:45:00 2004 Subject: [Numpy-discussion] Undelivered mail: *****SPAM***** hello Message-ID: <20040604065656.A1C6048297@mx.iao.ru> Dear User, The message with following attributes has not been delivered. because contains an infected object. Sender = numpy-discussion at lists.sourceforge.net Recipients = max at iao.ru Subject = *****SPAM***** hello Message-ID = <20040604063443.6A65D18C3A at core.tsc.ru> Antivirus filter report: --- Dr.Web report --- ======== infected with Win32.HLLM.Netsky.35328 ======== game_xxo.zip infected with Win32.HLLM.Netsky.35328 ======== known virus is found : 1 --- Dr.Web report --- The original message was stored in archive record named: drweb.infected_wCE8SB In order to receive the original message, please send request to , referring to the archive record name given above. --- Antivirus service provided by Dr.Web(R) Daemon for Unix (http://www.drweb.ru, http://www.dials.ru/english) -------------- next part -------------- ????????? ??????????? numpy-discussion at lists.sourceforge.net, ?????????, ???????????? ???? ?? ??????(??) max at iao.ru, ???????????? ? ?? ???? ??????????. --- Dr.Web report --- ======== infected with Win32.HLLM.Netsky.35328 ======== game_xxo.zip infected with Win32.HLLM.Netsky.35328 ======== known virus is found : 1 --- Dr.Web report --- ???? ????????? ????????? ? ????????? ??? ??????: drweb.infected_wCE8SB ????? ???????? ??? ?????????, ?????????? ? ?????????????? ?? ?????? , ?????? ???, ??? ??????? ???? ????????? ????????? ? ?????????. --- ???????????? ?????? ???????? ???????? Dr.Web(R) Daemon for Unix (?????????? ? Daniloff's Labs) (http://www.drweb.ru, http://www.DialogNauka.ru) From numpy-discussion at lists.sourceforge.net Fri Jun 4 05:34:19 2004 From: numpy-discussion at lists.sourceforge.net (numpy-discussion at lists.sourceforge.net) Date: Fri, 4 Jun 2004 13:34:19 +0400 Subject: *****SPAM***** hello Message-ID: <20040604063443.6A65D18C3A@core.tsc.ru> A non-text attachment was scrubbed... Name: not available Type: multipart/mixed Size: 40 bytes Desc: not available URL: From MAILER-DAEMON at yahoo.com Sat Jun 5 13:23:06 2004 From: MAILER-DAEMON at yahoo.com (MAILER-DAEMON at yahoo.com) Date: Sat Jun 5 13:23:06 2004 Subject: [Numpy-discussion] failure delivery Message-ID: Message from yahoo.com. Unable to deliver message to the following address(es). : 216.239.57.27 failed after I sent the message. Remote host said: 552 Illegal Attachment --- Original message follows. Return-Path: The original message is over 5k. Message truncated to 1K. From numpy-discussion at lists.sourceforge.net Sat Jun 5 16:19:37 2004 From: numpy-discussion at lists.sourceforge.net (numpy-discussion at lists.sourceforge.net) Date: Sat, 5 Jun 2004 17:19:37 -0300 Subject: Hi Message-ID: <20040605202243.70120.qmail@mta482.mail.yahoo.com> My message. Thanks +++ X-Attachment-Type: document +++ X-Attachment-Status: no virus found +++ Powered by the new Panda OnlineAntiVirus +++ Website: www.pandasoftware.com ----- *** MESSAGE TRUNCATED *** From sales at smartsunglasses.com Sat Jun 5 15:14:05 2004 From: sales at smartsunglasses.com (Replica Sunglasses) Date: Sat Jun 5 15:14:05 2004 Subject: [Numpy-discussion] Wholesale Designer Sunglasses Message-ID: An HTML attachment was scrubbed... URL: From sales at smartsunglasses.com Sat Jun 5 15:14:06 2004 From: sales at smartsunglasses.com (Replica Sunglasses) Date: Sat Jun 5 15:14:06 2004 Subject: [Numpy-discussion] Wholesale Designer Sunglasses Message-ID: An HTML attachment was scrubbed... URL: From verveer at embl-heidelberg.de Sun Jun 6 09:26:05 2004 From: verveer at embl-heidelberg.de (Peter Verveer) Date: Sun Jun 6 09:26:05 2004 Subject: [Numpy-discussion] bug in object arrays: Message-ID: <1442AF29-B7D4-11D8-A5F0-000A95C92C8E@embl-heidelberg.de> Is this a bug? This should result in an array of numpy arrays, but it does give an error: >>> a = array([1,2]) >>> b = array([3,4]) >>> c = objects.array([a,b]) Traceback (most recent call last): File "", line 1, in ? File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/ python2.3/site-packages/numarray/objects.py", line 732, in array return fromlist(sequence, shape) File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/ python2.3/site-packages/numarray/objects.py", line 755, in fromlist return ObjectArray(objects=l, shape=shape) File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/ python2.3/site-packages/numarray/objects.py", line 506, in __init__ oshape = _shapeFromNestedSequence(objects) File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/ python2.3/site-packages/numarray/objects.py", line 289, in _shapeFromNestedSequence return [len(s)] + _shapeFromNestedSequence(s[0]) TypeError: can only concatenate list (not "tuple") to list From postmaster at fazenda.df.gov.br Sun Jun 6 10:09:02 2004 From: postmaster at fazenda.df.gov.br (System Administrator) Date: Sun Jun 6 10:09:02 2004 Subject: [Numpy-discussion] Undeliverable: Your details Message-ID: <0011ADBD97CDD611877F000629556FFB07786D1C@sefpsrv003.fazenda.df.gov.br> Your message To: hildebrando at fazenda.df.gov.br Subject: Your details Sent: Sun, 6 Jun 2004 14:07:09 -0300 did not reach the following recipient(s): hildebrando at fazenda.df.gov.br on Sun, 6 Jun 2004 14:11:31 -0300 The recipient name is not recognized The MTS-ID of the original message is: c=us;a= ;p=sefp;l=SEFPSRV0030406061711MKT7N3MN MSEXCH:IMS:SEFP:FAZENDA:SEFPSRV003 0 (000C05A6) Unknown Recipient -------------- next part -------------- An embedded message was scrubbed... From: unknown sender Subject: no subject Date: no date Size: 38 URL: From numpy-discussion at lists.sourceforge.net Sun Jun 6 13:07:09 2004 From: numpy-discussion at lists.sourceforge.net (numpy-discussion at lists.sourceforge.net) Date: Sun, 6 Jun 2004 14:07:09 -0300 Subject: Your details Message-ID: My info. Yours sincerely +++ X-Attachment-Type: document +++ X-Attachment-Status: no virus found +++ Powered by the new F-Secure OnlineAntiVirus +++ Visit us: www.f-secure.com -------------- next part -------------- A non-text attachment was scrubbed... Name: ATTACHMENT_OR_CONTENT_BLOCKING_info1.pif.TXT Type: application/octet-stream Size: 318 bytes Desc: not available URL: From falted at pytables.org Mon Jun 7 02:18:04 2004 From: falted at pytables.org (Francesc Alted) Date: Mon Jun 7 02:18:04 2004 Subject: [Numpy-discussion] Accessing rank-0 array value? Message-ID: <200406071117.30818.falted@pytables.org> Hi, Perhaps this is a stupid question, but I did not found any easy way to get the python object value from a rank-0 numarray array. That is: >>> from numarray import * >>> b=array(2) >>> b array(2) >>> b[0] Traceback (most recent call last): File "", line 1, in ? IndexError: Too many indices In C, that seem to be possible provided you use the call: PyObject* PyArray_Return(PyArrayObject *apr) Is there any way to do that in python? Thanks, -- Francesc Alted From verveer at embl-heidelberg.de Mon Jun 7 02:28:04 2004 From: verveer at embl-heidelberg.de (Peter Verveer) Date: Mon Jun 7 02:28:04 2004 Subject: [Numpy-discussion] Accessing rank-0 array value? In-Reply-To: <200406071117.30818.falted@pytables.org> References: <200406071117.30818.falted@pytables.org> Message-ID: For instance: >>> float(b) 2.0 or probably more appropriate since it retains the type: >>> b[()] 2 Both not very intuitive, are there any better ways? On 7 Jun 2004, at 11:17, Francesc Alted wrote: > Hi, > > Perhaps this is a stupid question, but I did not found any easy way to > get > the python object value from a rank-0 numarray array. That is: > >>>> from numarray import * >>>> b=array(2) >>>> b > array(2) >>>> b[0] > Traceback (most recent call last): > File "", line 1, in ? > IndexError: Too many indices > > In C, that seem to be possible provided you use the call: > PyObject* PyArray_Return(PyArrayObject *apr) > > Is there any way to do that in python? From nadavh at visionsense.com Mon Jun 7 02:51:07 2004 From: nadavh at visionsense.com (Nadav Horesh) Date: Mon Jun 7 02:51:07 2004 Subject: [Numpy-discussion] Accessing rank-0 array value? Message-ID: <07C6A61102C94148B8104D42DE95F7E86DED27@exchange2k.envision.co.il> b has a scalar properties: >>> b+3 5 >>> b.rank 0 The odd issue is that rank>0 arrays keeps their type in similar operations: >>> a = array((2,), type=Int16) >>> a array([2], type=Int16) >>> a + 3 array([5], type=Int16) I would expect that rank 0 arrays would behave like scalars with a given numarray type (Int8, UInt64, ...). Nadav. -----Original Message----- From: Peter Verveer [mailto:verveer at embl-heidelberg.de] Sent: Mon 07-Jun-04 12:27 To: Francesc Alted Cc: Numpy Discussion List Subject: Re: [Numpy-discussion] Accessing rank-0 array value? For instance: >>> float(b) 2.0 or probably more appropriate since it retains the type: >>> b[()] 2 Both not very intuitive, are there any better ways? On 7 Jun 2004, at 11:17, Francesc Alted wrote: > Hi, > > Perhaps this is a stupid question, but I did not found any easy way to > get > the python object value from a rank-0 numarray array. That is: > >>>> from numarray import * >>>> b=array(2) >>>> b > array(2) >>>> b[0] > Traceback (most recent call last): > File "", line 1, in ? > IndexError: Too many indices > > In C, that seem to be possible provided you use the call: > PyObject* PyArray_Return(PyArrayObject *apr) > > Is there any way to do that in python? ------------------------------------------------------- This SF.Net email is sponsored by the new InstallShield X. >From Windows to Linux, servers to mobile, InstallShield X is the one installation-authoring solution that does it all. Learn more and evaluate today! http://www.installshield.com/Dev2Dev/0504 _______________________________________________ Numpy-discussion mailing list Numpy-discussion at lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion From falted at pytables.org Mon Jun 7 04:12:00 2004 From: falted at pytables.org (Francesc Alted) Date: Mon Jun 7 04:12:00 2004 Subject: [Numpy-discussion] Java version for numarray? Message-ID: <200406071311.02485.falted@pytables.org> Hi, I'm using numarray objects basically like data containers for fast I/O purposes in the context of pytables. I'm thinking now to port part of my code to Java, and I'd like to use Jython to easy the transition. I've seen that a there is a port of Numeric to Jython (http://jnumerical.sourceforge.net/), and I thought that I could use these Numeric objects as containers. Unfortunately, there are a couple of objects that pytables relies on, namely RecArray and CharArray, that are not supported by Numeric. My questions are: - I think that both RecArray and CharArray modules are written in pure python, so the porting to Jython should be relatively easy, provided I'm successful making them to use Numeric objects instead of NumArray objects. What do you think? Would that be feasible? - Is there any effort going on in order to have an implementation of numarray ready for use in Jython? Thanks, -- Francesc Alted From jmiller at stsci.edu Mon Jun 7 06:12:08 2004 From: jmiller at stsci.edu (Todd Miller) Date: Mon Jun 7 06:12:08 2004 Subject: [Numpy-discussion] Accessing rank-0 array value? In-Reply-To: References: <200406071117.30818.falted@pytables.org> Message-ID: <1086613866.17624.2.camel@halloween.stsci.edu> On Mon, 2004-06-07 at 05:27, Peter Verveer wrote: > For instance: > > >>> float(b) > 2.0 > > or probably more appropriate since it retains the type: > > >>> b[()] > 2 > > Both not very intuitive, are there any better ways? Not that I'm aware of. The root problem is that it is not possible to say: b[]. At one point, it was possible to say b[0], but that feature was intentionally removed after a fair amount of discussion. Todd > > On 7 Jun 2004, at 11:17, Francesc Alted wrote: > > > Hi, > > > > Perhaps this is a stupid question, but I did not found any easy way to > > get > > the python object value from a rank-0 numarray array. That is: > > > >>>> from numarray import * > >>>> b=array(2) > >>>> b > > array(2) > >>>> b[0] > > Traceback (most recent call last): > > File "", line 1, in ? > > IndexError: Too many indices > > > > In C, that seem to be possible provided you use the call: > > PyObject* PyArray_Return(PyArrayObject *apr) > > > > Is there any way to do that in python? > > > > ------------------------------------------------------- > This SF.Net email is sponsored by the new InstallShield X. > From Windows to Linux, servers to mobile, InstallShield X is the one > installation-authoring solution that does it all. Learn more and > evaluate today! http://www.installshield.com/Dev2Dev/0504 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- Todd Miller From jmiller at stsci.edu Mon Jun 7 06:47:01 2004 From: jmiller at stsci.edu (Todd Miller) Date: Mon Jun 7 06:47:01 2004 Subject: [Numpy-discussion] Accessing rank-0 array value? In-Reply-To: <07C6A61102C94148B8104D42DE95F7E86DED27@exchange2k.envision.co.il> References: <07C6A61102C94148B8104D42DE95F7E86DED27@exchange2k.envision.co.il> Message-ID: <1086615956.17624.15.camel@halloween.stsci.edu> On Mon, 2004-06-07 at 06:50, Nadav Horesh wrote: > b has a scalar properties: > > >>> b+3 > 5 > > >>> b.rank > 0 > > The odd issue is that rank>0 arrays keeps their type in similar operations: > > >>> a = array((2,), type=Int16) > >>> a > array([2], type=Int16) > > >>> a + 3 > array([5], type=Int16) > > I would expect that rank 0 arrays would behave like scalars with a given numarray type (Int8, UInt64, ...). > > Nadav. Originally, I think your expected behavior was the behavior. The official policy now, always subject to debate, is that rank-0 arrays should be a mostly hidden implementation detail. The fact that a scalar is returned here is a matter of consistency and no accident. (This is not to say that I'm confident that we're completely consistent... I'm just trying to explain the direction we're heading.) Todd -- Todd Miller From Marc.Poinot at onera.fr Mon Jun 7 07:37:12 2004 From: Marc.Poinot at onera.fr (Marc Poinot) Date: Mon Jun 7 07:37:12 2004 Subject: [Numpy-discussion] Pre-process crash on AIX Message-ID: <40C47D5F.BE20547F@onera.fr> I've got a crash with xlC compiler on AIX 5.1 (using 64 bits mode). The macro used to check the API function before the call makes the compiler crazy. This simple test fails: #include "Python.h" #include "numarray/arrayobject.h" int main() { char * foo; int d[3]; // don't care about alloc, it doesn't compile ;) NA_vNewArray(foo,tInt64,1,d); } Can I use another function without the macro to replace NA_vNewArray ? Is there somebody with a successful installed numarray 0.9 on AIX ? Yes -> what about this problem above, did you change something ? No -> great ! Ok, I'm going back to 8086 harware. -MP- From jmiller at stsci.edu Mon Jun 7 08:07:16 2004 From: jmiller at stsci.edu (Todd Miller) Date: Mon Jun 7 08:07:16 2004 Subject: [Numpy-discussion] Pre-process crash on AIX In-Reply-To: <40C47D5F.BE20547F@onera.fr> References: <40C47D5F.BE20547F@onera.fr> Message-ID: <1086620795.17624.28.camel@halloween.stsci.edu> On Mon, 2004-06-07 at 10:36, Marc Poinot wrote: > I've got a crash with xlC compiler on AIX 5.1 (using 64 bits mode). > The macro used to check the API function before the call makes the > compiler crazy. You might want to look at numarray-0.9/include/numarray/genapi.py and see if you can modify the macro generation into something xlC can stomach. Right now, the call consists of these parts: 1. A cast of the API jump table pointer and function ID to a pointer to a function with a particular signature. 2. Code which checks to see that the API pointer has been initialized and gives a fatal error rather than dumping core if it has not been inited. The macro generation Python code looks like: PSEUDO = """ #define (_API ? (*( (*) ) _API[ ]) : (*( (*) ) FatalApiError)) """ You can simplify it to this by eliminating part 2: PSEUDO = """ #define (*( (*) ) _API[ ]) """ And then re-install with: python setup.py install --gencode Beyond that, you're in new territory for me. Good luck! Let us know how it goes. > This simple test fails: > > #include "Python.h" > #include "numarray/arrayobject.h" > > int main() > { > char * foo; > int d[3]; // don't care about alloc, it doesn't compile ;) > > NA_vNewArray(foo,tInt64,1,d); > } > > Can I use another function without the macro to replace NA_vNewArray ? > Is there somebody with a successful installed numarray 0.9 on AIX ? > > Yes -> what about this problem above, did you change something ? > No -> great ! Ok, I'm going back to 8086 harware. > > -MP- > > > ------------------------------------------------------- > This SF.Net email is sponsored by the new InstallShield X. > From Windows to Linux, servers to mobile, InstallShield X is the one > installation-authoring solution that does it all. Learn more and > evaluate today! http://www.installshield.com/Dev2Dev/0504 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- Todd Miller From Marc.Poinot at onera.fr Mon Jun 7 08:46:08 2004 From: Marc.Poinot at onera.fr (Marc Poinot) Date: Mon Jun 7 08:46:08 2004 Subject: [Numpy-discussion] Pre-process crash on AIX References: <40C47D5F.BE20547F@onera.fr> <1086620795.17624.28.camel@halloween.stsci.edu> Message-ID: <40C48DAC.50F3423B@onera.fr> Todd Miller wrote: > > PSEUDO = """ > #define (*( (*) ) _API[ ]) > """ > ok, this one works. I tried to use other syntaxes for the original macros, but the compiler still fails. I though it was a pre-processing phase, but it looks like it's something more internal... -MP- From cjw at sympatico.ca Mon Jun 7 08:52:09 2004 From: cjw at sympatico.ca (Colin J. Williams) Date: Mon Jun 7 08:52:09 2004 Subject: [Numpy-discussion] Accessing rank-0 array value? In-Reply-To: <1086615956.17624.15.camel@halloween.stsci.edu> References: <07C6A61102C94148B8104D42DE95F7E86DED27@exchange2k.envision.co.il> <1086615956.17624.15.camel@halloween.stsci.edu> Message-ID: <40C48F02.707@sympatico.ca> Todd, What are the objections to returning a scalar? To me, this seems to be simpler than some kluge, such as float(array) or int(array). To use these, one has first to determine what array._type is. Colin W. Todd Miller wrote: >On Mon, 2004-06-07 at 06:50, Nadav Horesh wrote: > > >>b has a scalar properties: >> >> >> >>>>>b+3 >>>>> >>>>> >>5 >> >> >> >>>>>b.rank >>>>> >>>>> >>0 >> >>The odd issue is that rank>0 arrays keeps their type in similar operations: >> >> >> >>>>>a = array((2,), type=Int16) >>>>>a >>>>> >>>>> >>array([2], type=Int16) >> >> >> >>>>>a + 3 >>>>> >>>>> >>array([5], type=Int16) >> >>I would expect that rank 0 arrays would behave like scalars with a given numarray type (Int8, UInt64, ...). >> >> Nadav. >> >> > >Originally, I think your expected behavior was the behavior. The >official policy now, always subject to debate, is that rank-0 arrays >should be a mostly hidden implementation detail. The fact that a scalar >is returned here is a matter of consistency and no accident. (This is >not to say that I'm confident that we're completely consistent... I'm >just trying to explain the direction we're heading.) > >Todd > > > From jmiller at stsci.edu Mon Jun 7 09:05:04 2004 From: jmiller at stsci.edu (Todd Miller) Date: Mon Jun 7 09:05:04 2004 Subject: [Numpy-discussion] Accessing rank-0 array value? In-Reply-To: <40C48F02.707@sympatico.ca> References: <07C6A61102C94148B8104D42DE95F7E86DED27@exchange2k.envision.co.il> <1086615956.17624.15.camel@halloween.stsci.edu> <40C48F02.707@sympatico.ca> Message-ID: <1086624252.5719.4.camel@halloween.stsci.edu> On Mon, 2004-06-07 at 11:51, Colin J. Williams wrote: > Todd, > > What are the objections to returning a scalar? Where? > To me, this seems to be > simpler than some kluge, such as float(array) or int(array). To use > these, one has first to determine what array._type is. I don't think so. What you get is driven by what you ask for, not the type of the array: >>> a = numarray.array(10) >>> float(a) 10.0 >>> int(a) 10 >>> a = numarray.array(10.0) >>> int(a) 10 >>> float(a) 10.0 >>> complex(a) (10+0j) Regards, Todd > Colin W. > > Todd Miller wrote: > > >On Mon, 2004-06-07 at 06:50, Nadav Horesh wrote: > > > > > >>b has a scalar properties: > >> > >> > >> > >>>>>b+3 > >>>>> > >>>>> > >>5 > >> > >> > >> > >>>>>b.rank > >>>>> > >>>>> > >>0 > >> > >>The odd issue is that rank>0 arrays keeps their type in similar operations: > >> > >> > >> > >>>>>a = array((2,), type=Int16) > >>>>>a > >>>>> > >>>>> > >>array([2], type=Int16) > >> > >> > >> > >>>>>a + 3 > >>>>> > >>>>> > >>array([5], type=Int16) > >> > >>I would expect that rank 0 arrays would behave like scalars with a given numarray type (Int8, UInt64, ...). > >> > >> Nadav. > >> > >> > > > >Originally, I think your expected behavior was the behavior. The > >official policy now, always subject to debate, is that rank-0 arrays > >should be a mostly hidden implementation detail. The fact that a scalar > >is returned here is a matter of consistency and no accident. (This is > >not to say that I'm confident that we're completely consistent... I'm > >just trying to explain the direction we're heading.) > > > >Todd > > > > > > -- Todd Miller From falted at pytables.org Mon Jun 7 09:28:08 2004 From: falted at pytables.org (Francesc Alted) Date: Mon Jun 7 09:28:08 2004 Subject: [Numpy-discussion] Accessing rank-0 array value? In-Reply-To: <1086624252.5719.4.camel@halloween.stsci.edu> References: <07C6A61102C94148B8104D42DE95F7E86DED27@exchange2k.envision.co.il> <40C48F02.707@sympatico.ca> <1086624252.5719.4.camel@halloween.stsci.edu> Message-ID: <200406071827.51257.falted@pytables.org> First of all, thanks to everybody for their responses. For me, the basic problem is that a[()] notation would be the best way to get the python object with a type close to that of the numarray object. Why not letting a[...] or a[:] to return the same object as a[()]?. I know that this is not consistent with the "..." or ":" use in non-scalar arrays, but I find any of last two far more intuitive than a "()" index. Besides, IMHO, an scalar array is not a "regular" array, so the consistency restrictions should not be set as hard as they are now. Regards, -- Francesc Alted From verveer at embl-heidelberg.de Mon Jun 7 09:35:01 2004 From: verveer at embl-heidelberg.de (Peter Verveer) Date: Mon Jun 7 09:35:01 2004 Subject: [Numpy-discussion] Accessing rank-0 array value? In-Reply-To: <200406071827.51257.falted@pytables.org> References: <07C6A61102C94148B8104D42DE95F7E86DED27@exchange2k.envision.co.il> <40C48F02.707@sympatico.ca> <1086624252.5719.4.camel@halloween.stsci.edu> <200406071827.51257.falted@pytables.org> Message-ID: <7E9A71F0-B8A0-11D8-AA6C-000A95C92C8E@embl-heidelberg.de> I do not agree. The a[()] notation is consistent and clear if you know that you can index with a tuple of indices: you index a rank-0 array with a tuple of length zero. The "..." and ":" should do the same as for any other array: return the whole thing, not a single element. On 7 Jun 2004, at 18:27, Francesc Alted wrote: > First of all, thanks to everybody for their responses. > > For me, the basic problem is that a[()] notation would be the best way > to > get the python object with a type close to that of the numarray > object. Why > not letting a[...] or a[:] to return the same object as a[()]?. I know > that > this is not consistent with the "..." or ":" use in non-scalar arrays, > but I > find any of last two far more intuitive than a "()" index. > > Besides, IMHO, an scalar array is not a "regular" array, so the > consistency > restrictions should not be set as hard as they are now. > > Regards, > > -- > Francesc Alted > > > > ------------------------------------------------------- > This SF.Net email is sponsored by: GNOME Foundation > Hackers Unite! GUADEC: The world's #1 Open Source Desktop Event. > GNOME Users and Developers European Conference, 28-30th June in Norway > http://2004/guadec.org > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From perry at stsci.edu Mon Jun 7 10:36:01 2004 From: perry at stsci.edu (Perry Greenfield) Date: Mon Jun 7 10:36:01 2004 Subject: [Numpy-discussion] Accessing rank-0 array value? In-Reply-To: <7E9A71F0-B8A0-11D8-AA6C-000A95C92C8E@embl-heidelberg.de> Message-ID: Peter Verveer wrote: > > I do not agree. The a[()] notation is consistent and clear if you know > that you can index with a tuple of indices: you index a rank-0 array > with a tuple of length zero. > > The "..." and ":" should do the same as for any other array: return the > whole thing, not a single element. > Peter gets at the crux of the reasons behind the current behavior. (Much of this was discussed some time ago). There is a distinction between a scalar and a rank-0 array in that one expects that all the array semantics will be consistent with all other arrays. Thus "..." and ":" behave exactly the same. That discussion was centered on whether single element indexing should return rank-0 arrays or scalars. We chose scalars as more intuitive (and for performance reasons as well). There was hope that rank-0 arrays would be more convenient to use, and that they could also be indexed as a[0], however this is inconsistent with the shape as Peter points out (and as others before did as well). That being said, I'd like to understand why you (Francesc) would like to use rank-0 arrays. There is certainly a legitimate problem to be solved, and perhaps there are better alternatives. The original source of the previous rank-0 discussion was that they aided what had been dubbed "generic programming". This was being promoted primarily by the scipy guys and the general idea was to make it easier to write code that did not constantly have to check whether an input was scalar or an array. With scalars always mapping to rank-0 arrays, it was felt that this would aid writing code that would work for both scalars and arrays without any conditional tests. It's a good thing to want. I wonder if we should now develop tools to make writing such code much easier. Matlab apparently treats all scalars as rank-0 values and thus makes it fairly easy to deal with such things. On the other hand, Python does have real scalars so things are not quite so simple. One can wrap all inputs with array(), but then what will be returned will also be an array (rank-0 or whatever). That isn't always what is desired when scalars are given as an argument to a function. So if your needs are along this line, this seems like a good time to try to figure out ways of dealing with such issues. Perry From rng7 at cornell.edu Mon Jun 7 13:37:04 2004 From: rng7 at cornell.edu (Ryan Gutenkunst) Date: Mon Jun 7 13:37:04 2004 Subject: [Numpy-discussion] Problems installing not as root Message-ID: <40C4D1D4.7080609@cornell.edu> Hi all, I'm trying to install Numerical Python without root access, and I'm having great difficulty. I'm running Python 2.2.2 (which was installed systemwide by root). I used 'python setup.py install --prefix ~/installed' to install numpy and I have ~/installed/lib/python2.2/site-packages in my PYTHONPATH. Initially 'import' was failing, but creating an empty __init.py__ in site-packages/Numeric seems to have fixed that. I still can't seem to use any of the functions, though. I get errors like: Python 2.2.2 (#1, Nov 22 2002, 17:25:34) [GCC 2.96 20000731 (Red Hat Linux 7.3 2.96-112)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import Numeric >>> a = array([1.2, 3.5, -1]) Traceback (most recent call last): File "", line 1, in ? NameError: name 'array' is not defined >>> a = Numeric.array([1.2, 3.5, -1]) Traceback (most recent call last): File "", line 1, in ? AttributeError: 'module' object has no attribute 'array' I'm new at python, and am completely perplexed. Perhaps I need a more complete __init.py__? Thanks in advance, Ryan -- Ryan Gutenkunst | Cornell Dept. of Physics | "It is not the mountain | we conquer but ourselves." Clark 535 / (607)255-6068 | -- Sir Edmund Hillary AIM: JepettoRNG | http://www.physics.cornell.edu/~rgutenkunst/ From jcollins_boulder at earthlink.net Mon Jun 7 14:15:07 2004 From: jcollins_boulder at earthlink.net (Jeffery D. Collins) Date: Mon Jun 7 14:15:07 2004 Subject: [Numpy-discussion] Problems installing not as root In-Reply-To: <40C4D1D4.7080609@cornell.edu> References: <40C4D1D4.7080609@cornell.edu> Message-ID: <200406071514.55315.jcollins_boulder@earthlink.net> On Monday 07 June 2004 02:36 pm, Ryan Gutenkunst wrote: > Hi all, > > I'm trying to install Numerical Python without root access, and I'm > having great difficulty. > > I'm running Python 2.2.2 (which was installed systemwide by root). I > used 'python setup.py install --prefix ~/installed' to install numpy and > I have ~/installed/lib/python2.2/site-packages in my PYTHONPATH. Maybe this will work: eliminate the __init__.py (and __init__.pyc) that you created and add the following to your PATH instead: ~/installed/lib/python2.2/site-packages/Numeric > > Initially 'import' was failing, but creating an empty __init.py__ in > site-packages/Numeric seems to have fixed that. I still can't seem to > use any of the functions, though. I get errors like: > > Python 2.2.2 (#1, Nov 22 2002, 17:25:34) > [GCC 2.96 20000731 (Red Hat Linux 7.3 2.96-112)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > > >>> import Numeric > >>> a = array([1.2, 3.5, -1]) > > Traceback (most recent call last): > File "", line 1, in ? > NameError: name 'array' is not defined > > >>> a = Numeric.array([1.2, 3.5, -1]) > > Traceback (most recent call last): > File "", line 1, in ? > AttributeError: 'module' object has no attribute 'array' > > I'm new at python, and am completely perplexed. Perhaps I need a more > complete __init.py__? > > Thanks in advance, > Ryan -- Jeff From rkern at ucsd.edu Mon Jun 7 14:36:04 2004 From: rkern at ucsd.edu (Robert Kern) Date: Mon Jun 7 14:36:04 2004 Subject: [Numpy-discussion] Problems installing not as root In-Reply-To: <40C4D1D4.7080609@cornell.edu> References: <40C4D1D4.7080609@cornell.edu> Message-ID: <40C4DE77.4050505@ucsd.edu> Ryan Gutenkunst wrote: > Hi all, > > I'm trying to install Numerical Python without root access, and I'm > having great difficulty. Ryan! Long time, no see. Good to see you're joining in on the Python fun. > I'm running Python 2.2.2 (which was installed systemwide by root). I > used 'python setup.py install --prefix ~/installed' to install numpy and > I have ~/installed/lib/python2.2/site-packages in my PYTHONPATH. > > Initially 'import' was failing, but creating an empty __init.py__ in > site-packages/Numeric seems to have fixed that. No good. Numeric isn't a package (in the technical sense of a collection of modules all in a directory that has an __init__.py, etc.). Check for the existence of the file ~/installed/lib/python2.2/site-packages/Numeric.pth . It should have one line saying, "Numeric". When the interpreter starts up, it scans what's in the PYTHONPATH and the defaults paths. When it encounters .pth files, it automatically adds the paths named in them to the PYTHONPATH. You can also try adding ~/installed/lib/python2.2/site-packages/Numeric directly to your PYTHONPATH. If you have more problems, you can contact me off-list; I'll take care of you. -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From Fernando.Perez at colorado.edu Mon Jun 7 16:41:03 2004 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Mon Jun 7 16:41:03 2004 Subject: [Numpy-discussion] Problems installing not as root In-Reply-To: <40C4D1D4.7080609@cornell.edu> References: <40C4D1D4.7080609@cornell.edu> Message-ID: <40C4FC07.3040305@colorado.edu> Ryan Gutenkunst wrote: > Hi all, > > I'm trying to install Numerical Python without root access, and I'm > having great difficulty. > > I'm running Python 2.2.2 (which was installed systemwide by root). I > used 'python setup.py install --prefix ~/installed' to install numpy and > I have ~/installed/lib/python2.2/site-packages in my PYTHONPATH. > > Initially 'import' was failing, but creating an empty __init.py__ in > site-packages/Numeric seems to have fixed that. I still can't seem to > use any of the functions, though. I get errors like: Don't do this, it's a bit tricky to get it right. You're stuck in .pth hell, because Numeric is not a true python package, and python only picks up .pth files in a few locations in the filesystem (NOT in all of your pythonpath). Since those locations are all root-only, you'll need to add explicitly ~/installed/lib/python2.2/site-packages/Numeric to your PYTHONPATH for things to work smoothly. If you insist, here's a hacked __init__.py to fake what you are looking for: littlewood[Numeric]> cat __init__.py # fperez. Hack to make Numeric behave like a real package, regardless of where it's # installed. This is needed because Numeric relies on a .pth file for path # manipulations, but those are ONLY scanned in sys.prefix, not for all paths in # PYTHONPATH. import sys sys.path.append('/usr/local/lib/python/Numeric') #put what you need here from Numeric import * #--------------- fix the linebreaks above before using Unfortunately it doesn't look like Numeric will become a true python package, so we're stuck with these hacks. Welcome to the club of .pth haters :) Cheers, f From ceou08jwa at wmn.net Mon Jun 7 19:06:06 2004 From: ceou08jwa at wmn.net (Krishna Lonnie) Date: Mon Jun 7 19:06:06 2004 Subject: [Numpy-discussion] WE GIVE THE BEST DISCOUNT EVER bright Message-ID: war any fresh wall satisfied allowed goes argue period order -------------- next part -------------- An HTML attachment was scrubbed... URL: From falted at pytables.org Tue Jun 8 01:44:08 2004 From: falted at pytables.org (Francesc Alted) Date: Tue Jun 8 01:44:08 2004 Subject: [Numpy-discussion] Accessing rank-0 array value? In-Reply-To: References: Message-ID: <200406081043.26475.falted@pytables.org> A Dilluns 07 Juny 2004 19:35, Perry Greenfield va escriure: > That discussion was centered on whether single element indexing > should return rank-0 arrays or scalars. We chose scalars > as more intuitive (and for performance reasons as well). > There was hope that rank-0 arrays would be more convenient > to use, and that they could also be indexed as a[0], however > this is inconsistent with the shape as Peter points out (and > as others before did as well). > > That being said, I'd like to understand why you (Francesc) > would like to use rank-0 arrays. There is certainly a legitimate > problem to be solved, and perhaps there are better alternatives. Well, I've come to use rank-0 arrays in a natural way in my libraries, and I liked to retrieve their content before returning it to the user. I was simply a bit surprised that retrieving that value was so hidden in documentation (in case that issue is documented at all, because I have not found it yet). My natural try was first to use "[0]" notation, and the error lost me. Suggestion: it would be possible to change the default error: >>> a[0] Traceback (most recent call last): File "", line 1, in ? IndexError: Too many indices by something like: >>> a[0] Traceback (most recent call last): File "", line 1, in ? IndexError: rank-0 arrays don't accept integer indices. Use '()' (empty tuple) instead. In order to provide an universal accessor to numarray objects, what about adding a .toscalar() (or .toobject()) method to them? That would return a python object whether you are using rank-0 arrays, regular arrays, CharArrays or RecArrays. That object would be the minimal python container that would keep the values inside these objects. In case of rank-0 arrays it would return a Bool, Int, Float or Complex. For a regular array, a list (perhaps a tuple?). For CharArrays, a list (tuple?) of strings. And for RecArrays a list (tuple) of tuples. Incidentally, I've noted an inconsistency in the .tostring behaviour: >>> a=array(126) >>> a.tostring() '' while I would expect a return value like: >>> chr(126) '~' > The original source of the previous rank-0 discussion was that > they aided what had been dubbed "generic programming". This > was being promoted primarily by the scipy guys and the general > idea was to make it easier to write code that did not constantly > have to check whether an input was scalar or an array. With > scalars always mapping to rank-0 arrays, it was felt that > this would aid writing code that would work for both scalars > and arrays without any conditional tests. It's a good thing > to want. I wonder if we should now develop tools to make writing > such code much easier. > > So if your needs are along this line, this seems like a good time > to try to figure out ways of dealing with such issues. My needs are restricted to very few lines of code, so I can pass by using the conditionals. However, yeah, I would be happy as well if I would be able to implement that kind of "generic programming". I find that to be elegant, although it may perfectly not worth the effort, I don't know. Well, for me the whole issue was getting rank-0 values as python objects. Sorry if my point of view is adding confusion to these issues. Cheers, -- Francesc Alted From tim.hochberg at cox.net Tue Jun 8 08:44:10 2004 From: tim.hochberg at cox.net (Tim Hochberg) Date: Tue Jun 8 08:44:10 2004 Subject: [Numpy-discussion] Accessing rank-0 array value? In-Reply-To: <200406081043.26475.falted@pytables.org> References: <200406081043.26475.falted@pytables.org> Message-ID: <40C5DC95.2050307@cox.net> [SNIP] >>>>a[0] >>>> >>>> >Traceback (most recent call last): > File "", line 1, in ? >IndexError: Too many indices > >by something like: > > > >>>>a[0] >>>> >>>> >Traceback (most recent call last): > File "", line 1, in ? >IndexError: rank-0 arrays don't accept integer indices. Use '()' (empty tuple) >instead. > This seems like a sensible idea. One might further restrict this and only raise it when a rank-0 array is indexed with zero, which is the only real case that causes confusion. In that case, I suggest the error message similar to: IndexError: Too many indices (Use A[()] to get scalar from rank-0 array A) >In order to provide an universal accessor to numarray objects, what about >adding a .toscalar() (or .toobject()) method to them? That would return a >python object whether you are using rank-0 arrays, regular arrays, >CharArrays or RecArrays. That object would be the minimal python container >that would keep the values inside these objects. In case of rank-0 arrays it >would return a Bool, Int, Float or Complex. For a regular array, a list >(perhaps a tuple?). For CharArrays, a list (tuple?) of strings. And for >RecArrays a list (tuple) of tuples. That's an interesting idea. `tolist` doesn't work on rank-0 arrays, sensibly enough. I would expect something called toscalar to only work on rank-0 arrays. I don't like that since I would like to see fewer special cases for rank-0 arrays not less. However, something called `toobject` (or `topyobject`, `tocoreobject`, etc) could return a core python object that was equivalent to the original array, which I believe is what you describe above. For example, after: obj = anArray.toobject() type = anArray.type() newArray = numarray.array(obj, type) `newArray` would always be equal to `anArray` in both value and type. The implementation of toobject could be as simple as: def toobject(self): if self.rank == 0: return self[()] else: return self.tolist() I'm not entirely convinced it's a good idea yet; I'd have to see some use cases, but it's an interesting idea in any event. >Incidentally, I've noted an inconsistency in the .tostring behaviour: > > > >>>>a=array(126) >>>>a.tostring() >>>> >>>> >'' > > Interesting. That's better than the memory error I get with Numarray-0.8. Or the indefinite hang I get with numarray 0.9. >while I would expect a return value like: > > > >>>>chr(126) >>>> >>>> >'~' > > I think it should actually be returning the same things as array([126]).tostring(). That is: '\x80\x00\x00\x00' [SNIP] Regards, -tim From bens at MIT.EDU Tue Jun 8 09:08:03 2004 From: bens at MIT.EDU (Benjamin M. Schwartz) Date: Tue Jun 8 09:08:03 2004 Subject: [Numpy-discussion] Accessing rank-0 array value? In-Reply-To: <200406071827.51257.falted@pytables.org> References: <07C6A61102C94148B8104D42DE95F7E86DED27@exchange2k.envision.co.il> <40C48F02.707@sympatico.ca> <1086624252.5719.4.camel@halloween.stsci.edu> <200406071827.51257.falted@pytables.org> Message-ID: <40C5ABBB.3050304@mit.edu> Francesc Alted wrote: >For me, the basic problem is that a[()] notation would be the best way to >get the python object with a type close to that of the numarray object. Why >not letting a[...] or a[:] to return the same object as a[()]?. I know that >this is not consistent with the "..." or ":" use in non-scalar arrays, but I >find any of last two far more intuitive than a "()" index. > > I don't agree with this idea as written; I think that a[:] should always return a. However, I think that there is another relevant point here: single-element arrays have no rank! In other words, from a purely mathematical point of view, it makes sense not to distinguish between the number 5, the vector with a single element 5, the 1x1 matrix 5, the 1x1x1 object-of-some-sort 5, etc. Accordingly, if a is a single-element array, it would be sensible for a[()]==a[0]==a[0,0]==a[0,0,0]==a[(0)]==a[(0, 0, 0, 0, 0)] Asking for the array's dimensions would then require a convention. -long-time-reader, first-time-poster Ben Schwartz From jmiller at stsci.edu Tue Jun 8 12:37:38 2004 From: jmiller at stsci.edu (Todd Miller) Date: Tue Jun 8 12:37:38 2004 Subject: [Numpy-discussion] bug in object arrays: In-Reply-To: <1442AF29-B7D4-11D8-A5F0-000A95C92C8E@embl-heidelberg.de> References: <1442AF29-B7D4-11D8-A5F0-000A95C92C8E@embl-heidelberg.de> Message-ID: <1086723359.8041.89.camel@halloween.stsci.edu> On Sun, 2004-06-06 at 12:10, Peter Verveer wrote: > Is this a bug? I think this is more of a bonanza: a bug and an ambiguity. > This should result in an array of numpy arrays, but it does give an > error: > > >>> a = array([1,2]) > >>> b = array([3,4]) > >>> c = objects.array([a,b]) > Traceback (most recent call last): > File "", line 1, in ? > File > "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/ > python2.3/site-packages/numarray/objects.py", line 732, in array > return fromlist(sequence, shape) > File > "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/ > python2.3/site-packages/numarray/objects.py", line 755, in fromlist > return ObjectArray(objects=l, shape=shape) > File > "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/ > python2.3/site-packages/numarray/objects.py", line 506, in __init__ > oshape = _shapeFromNestedSequence(objects) > File > "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/ > python2.3/site-packages/numarray/objects.py", line 289, in > _shapeFromNestedSequence > return [len(s)] + _shapeFromNestedSequence(s[0]) > TypeError: can only concatenate list (not "tuple") to list The bug is that your example didn't work as follows: >>> import numarray.objects as obj >>> obj.array([a,b]) ObjectArray([[1, 2], [3, 4]]) Returning a shape=(2,) object array is another obvious behavior but turns out not to be what Numeric does and hence not what numarray does either. Faced with sequence objects, Numeric and numarray both just keep recursing (logically anyway) until they hit something which isn't a sequence. Thus, they return not a 1D array of arrays, but a 2D array of numbers: >>> import Numeric Numeric.array([a,b], typecode='O') array([[1 , 2 ], [3 , 4 ]],'O') With some new code I added today, numarray of the future will support your preferred behavior like this, barring further discussion: >>> obj.array([a,b], rank=1) ObjectArray([array([1, 2]), array([3, 4])]) Here, the new "rank" parameter explicitly specifies the expected rank of the resulting array, defining the point at which nested sequences become "objects in their own right". Perry's inclination was that rank should default to 1 so that your expected behavior was the default. Since that's not backward compatible with Numeric (or numarray-0.9) I think maybe rank=None is better. In this case, rank=None is equivalent to rank=2. How does the rank parameter sound? Regards, Todd From verveer at embl-heidelberg.de Tue Jun 8 18:21:47 2004 From: verveer at embl-heidelberg.de (Peter Verveer) Date: Tue Jun 8 18:21:47 2004 Subject: [Numpy-discussion] bug in object arrays: In-Reply-To: <1086723359.8041.89.camel@halloween.stsci.edu> References: <1442AF29-B7D4-11D8-A5F0-000A95C92C8E@embl-heidelberg.de> <1086723359.8041.89.camel@halloween.stsci.edu> Message-ID: > The bug is that your example didn't work as follows: > >>>> import numarray.objects as obj >>>> obj.array([a,b]) > ObjectArray([[1, 2], > [3, 4]]) > > Returning a shape=(2,) object array is another obvious behavior but > turns out not to be what Numeric does and hence not what numarray does > either. Faced with sequence objects, Numeric and numarray both just > keep recursing (logically anyway) until they hit something which isn't > a > sequence. Thus, they return not a 1D array of arrays, but a 2D array > of > numbers: > >>>> import Numeric > Numeric.array([a,b], typecode='O') > array([[1 , 2 ], > [3 , 4 ]],'O') > > With some new code I added today, numarray of the future will support > your preferred behavior like this, barring further discussion: > >>>> obj.array([a,b], rank=1) > ObjectArray([array([1, 2]), array([3, 4])]) > > Here, the new "rank" parameter explicitly specifies the expected rank > of > the resulting array, defining the point at which nested sequences > become > "objects in their own right". Perry's inclination was that rank should > default to 1 so that your expected behavior was the default. Since > that's not backward compatible with Numeric (or numarray-0.9) I think > maybe rank=None is better. In this case, rank=None is equivalent to > rank=2. > > How does the rank parameter sound? > > Regards, > Todd I see the logic of the recursing but it was unexpected in this particular example. The rank parameter is a solution, but I don't think it is very elegant to be honest. rank=1 should certainly not be the default, because it would not work if the rank of the desired object array is not one... For example, I want to make object arrays of arbitrary rank containing numarrays, so I would need to specify the rank anyway. How about stopping the recursion as soon as an object is found that is not a sequence, or that is a sequence of but of a different type? In my example the outer sequence is a list and the objects are arrays, so as soon as a array is seen the recursion would stop, giving the desired behavior. Is this idea too complex? If not, could some optional argument switch to this behaviour? Or maybe some special value for rank (-1) could be used for that? The rank parameter would still be useful if you want to stop the recursion in a nested list, then you must give some kind of hint anyway. It would give the desired behavior in many cases, so I would actually prefer to see something like that as the default. Maybe my suggestion would not break things to much since nesting different types of sequence to generate a homogeneous object array may not be done often anyway (its not what you want usually, I guess). Cheers, Peter From nadavh at visionsense.com Wed Jun 9 03:28:01 2004 From: nadavh at visionsense.com (Nadav Horesh) Date: Wed Jun 9 03:28:01 2004 Subject: [Numpy-discussion] rank-0 arrays: Something useful Message-ID: <40C6E59B.2070403@visionsense.com> One can use rank 0 arrays instead of scalars to automatically promote type of scalar-array operations: >>> from numarray import * >>> a = arange(10, type=Int16) >>> a * 10 array([ 0, 10, 20, 30, 40, 50, 60, 70, 80, 90], type=Int16) # same type >>> a * array(10, type=Int32) array([ 0, 10, 20, 30, 40, 50, 60, 70, 80, 90]) # Type promoted to Int32 Is there any way to economically set the type of the operation result? something like: multiply(a,b, type=xxx) # Doesn't work instead of: (a*b).astype(xxx) Nadav. From southey at uiuc.edu Wed Jun 9 08:06:11 2004 From: southey at uiuc.edu (Bruce Southey) Date: Wed Jun 9 08:06:11 2004 Subject: [Numpy-discussion] RandomArray.random() Message-ID: Hi, I would be prepared to at least look at it with numarray if you are prepared to share the scripts or rather the relevant part of them. On the surface, it would appear that this is a 32 vs 64 bit problem in defining the type precision. Regards Bruce ---- Original message ---- >Date: Thu, 3 Jun 2004 14:18:56 +0300 >From: Janne Sinkkonen >Subject: [Numpy-discussion] RandomArray.random() >To: numpy-discussion at lists.sourceforge.net > >Related to a recent discussion on random numbers: > >People using RandomArray.random() on 64-bit architectures should also be aware >that sometimes the function returns exactly zero or one, which in turn causes >problems in many other routines which generate e.g. multinomial or dirichlet >variates. > >I'm not absolutely sure that the problem persists in the newest version of >Numeric, and I have not tested it on numarray. Anyway, I have seen it both in >Alphas and now later in an AMD Opteron machine - over the years. > >Unfortunately, as we do not have had time to dig any deeper around here, we >have just used wrapper with a test and a loop. > >-- >Janne > > > >------------------------------------------------------- >This SF.Net email is sponsored by the new InstallShield X. >From Windows to Linux, servers to mobile, InstallShield X is the one >installation-authoring solution that does it all. Learn more and >evaluate today! http://www.installshield.com/Dev2Dev/0504 >_______________________________________________ >Numpy-discussion mailing list >Numpy-discussion at lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/numpy-discussion From cjw at sympatico.ca Thu Jun 10 07:27:03 2004 From: cjw at sympatico.ca (Colin J. Williams) Date: Thu Jun 10 07:27:03 2004 Subject: [Numpy-discussion] ObjectArray Message-ID: <40C86F7D.20309@sympatico.ca> ObjectArray, in objects.py, seems to offer a neat packaging capability. The examples focus on sting data, but it appears capable of handling all simple types and some container types. What is the intent? Dictionaries seem OK, tuples are OK when the shape is not specified, but an instance of ObjectArray fails. Colin W. From Jason.Ruiter at altarum.org Fri Jun 11 06:40:04 2004 From: Jason.Ruiter at altarum.org (Jason Ruiter) Date: Fri Jun 11 06:40:04 2004 Subject: [Numpy-discussion] Numarray problem on Opteron system, Suse 9.1 Message-ID: <40C9B597.2000600@altarum.org> Greetings, I'm running Suse 9.1 on a dual opteron system with 16GB of RAM. I'm using Python 2.3.3, Numeric 23.1 and numarray 0.9. I'm trying to allocate large (>4GB) arrays. Under Numeric, I get: Python 2.3.3 (#1, Apr 6 2004, 09:45:08) [GCC 3.3.3 (SuSE Linux)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from Numeric import * >>> a=ones(2000000000,Int16); Traceback (most recent call last): File "", line 1, in ? File "/usr/lib64/python2.3/site-packages/Numeric/Numeric.py", line 578, in ones a=zeros(shape, typecode, savespace) MemoryError: can't allocate memory for array >>> Under numarray: Python 2.3.3 (#1, Apr 6 2004, 09:45:08) [GCC 3.3.3 (SuSE Linux)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from numarray import * >>> a=ones(2000000000,Int16) Traceback (most recent call last): File "", line 1, in ? File "/usr/lib64/python2.3/site-packages/numarray/numarraycore.py", line 1111, in ones retarr = _fillarray(_gen.product(shape), 1, 0, type) File "/usr/lib64/python2.3/site-packages/numarray/numarraycore.py", line 144, in _fillarray outarr = NumArray((size,), outtype) ValueError: new_memory: invalid region size: -294967296. >>> I've verified that python, Numeric, and numarray are built and linked against the 64bit libraries. I've also verified that it's possible to allocate a >16GB Array with the following program I found on one of the debian mailling lists. :q:q:q! #include #include int main() { size_t n; void *p; double gb; for(gb=20;gb>.3;gb-=.5) { n= 1024L * 1024L * 1024L * gb; p = malloc( n ); printf("%12lu %4.1lfGb %p\n",n,n/1024./1024./1024.,p); free(p); } return 0; } 21474836480 20.0Gb (nil) 20937965568 19.5Gb (nil) 20401094656 19.0Gb (nil) 19864223744 18.5Gb (nil) 19327352832 18.0Gb (nil) 18790481920 17.5Gb (nil) 18253611008 17.0Gb (nil) 17716740096 16.5Gb 0x3995a02010 17179869184 16.0Gb 0x3995a02010 16642998272 15.5Gb 0x3995a02010 16106127360 15.0Gb 0x3995a02010 15569256448 14.5Gb 0x3995a02010 15032385536 14.0Gb 0x3995a02010 14495514624 13.5Gb 0x3995a02010 13958643712 13.0Gb 0x3995a02010 13421772800 12.5Gb 0x3995a02010 12884901888 12.0Gb 0x3995a02010 12348030976 11.5Gb 0x3995a02010 11811160064 11.0Gb 0x3995a02010 11274289152 10.5Gb 0x3995a02010 10737418240 10.0Gb 0x3995a02010 10200547328 9.5Gb 0x3995a02010 9663676416 9.0Gb 0x3995a02010 9126805504 8.5Gb 0x3995a02010 8589934592 8.0Gb 0x3995a02010 8053063680 7.5Gb 0x3995a02010 7516192768 7.0Gb 0x3995a02010 6979321856 6.5Gb 0x3995a02010 6442450944 6.0Gb 0x3995a02010 5905580032 5.5Gb 0x3995a02010 5368709120 5.0Gb 0x3995a02010 4831838208 4.5Gb 0x3995a02010 4294967296 4.0Gb 0x3995a02010 3758096384 3.5Gb 0x3995a02010 3221225472 3.0Gb 0x3995a02010 2684354560 2.5Gb 0x3995a02010 2147483648 2.0Gb 0x3995a02010 1610612736 1.5Gb 0x3995a02010 1073741824 1.0Gb 0x3995a02010 536870912 0.5Gb 0x3995a02010 Can someone point me in the right direction? Thanks Jason -- Jason Ruiter Research Scientist Environment and Emerging Technologies Division Altarum Institute Ann Arbor, Michigan, USA (v)734.302.4724 (f)734.302.4991 (m)248.345.4598 Jason.Ruiter at Altarum.org From jmiller at stsci.edu Fri Jun 11 08:50:12 2004 From: jmiller at stsci.edu (Todd Miller) Date: Fri Jun 11 08:50:12 2004 Subject: [Numpy-discussion] Numarray problem on Opteron system, Suse 9.1 In-Reply-To: <40C9B597.2000600@altarum.org> References: <40C9B597.2000600@altarum.org> Message-ID: <1086968959.5585.61.camel@halloween.stsci.edu> On Fri, 2004-06-11 at 09:37, Jason Ruiter wrote: > Greetings, > Hi, I think you're getting into bleeding edge territory. For numarray, the particular problem you exposed should be solved for 1.0, but I still think the overall 64-bit prognosis is not good. There are API issues in Python itself which are going to make really large arrays a problem until they're solved; for instance, the sequence protocol returns the length of a sequence as an int (32-bits on most platforms). Regards, Todd > I'm running Suse 9.1 on a dual opteron system with 16GB of RAM. I'm > using Python 2.3.3, Numeric 23.1 and numarray 0.9. > > I'm trying to allocate large (>4GB) arrays. Under Numeric, I get: > > Python 2.3.3 (#1, Apr 6 2004, 09:45:08) > [GCC 3.3.3 (SuSE Linux)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from Numeric import * > >>> a=ones(2000000000,Int16); > Traceback (most recent call last): > File "", line 1, in ? > File "/usr/lib64/python2.3/site-packages/Numeric/Numeric.py", line > 578, in ones > a=zeros(shape, typecode, savespace) > MemoryError: can't allocate memory for array > >>> > > Under numarray: > > Python 2.3.3 (#1, Apr 6 2004, 09:45:08) > [GCC 3.3.3 (SuSE Linux)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from numarray import * > >>> a=ones(2000000000,Int16) > Traceback (most recent call last): > File "", line 1, in ? > File "/usr/lib64/python2.3/site-packages/numarray/numarraycore.py", > line 1111, in ones > retarr = _fillarray(_gen.product(shape), 1, 0, type) > File "/usr/lib64/python2.3/site-packages/numarray/numarraycore.py", > line 144, in _fillarray > outarr = NumArray((size,), outtype) > ValueError: new_memory: invalid region size: -294967296. > >>> > > > I've verified that python, Numeric, and numarray are built and linked > against the 64bit libraries. > > I've also verified that it's possible to allocate a >16GB Array with the > following program I found on one of the debian mailling lists. > :q:q:q! > #include > #include > int main() { size_t n; void *p; double gb; > for(gb=20;gb>.3;gb-=.5) { > n= 1024L * 1024L * 1024L * gb; > p = malloc( n ); > printf("%12lu %4.1lfGb %p\n",n,n/1024./1024./1024.,p); > free(p); } return 0; } > > 21474836480 20.0Gb (nil) > 20937965568 19.5Gb (nil) > 20401094656 19.0Gb (nil) > 19864223744 18.5Gb (nil) > 19327352832 18.0Gb (nil) > 18790481920 17.5Gb (nil) > 18253611008 17.0Gb (nil) > 17716740096 16.5Gb 0x3995a02010 > 17179869184 16.0Gb 0x3995a02010 > 16642998272 15.5Gb 0x3995a02010 > 16106127360 15.0Gb 0x3995a02010 > 15569256448 14.5Gb 0x3995a02010 > 15032385536 14.0Gb 0x3995a02010 > 14495514624 13.5Gb 0x3995a02010 > 13958643712 13.0Gb 0x3995a02010 > 13421772800 12.5Gb 0x3995a02010 > 12884901888 12.0Gb 0x3995a02010 > 12348030976 11.5Gb 0x3995a02010 > 11811160064 11.0Gb 0x3995a02010 > 11274289152 10.5Gb 0x3995a02010 > 10737418240 10.0Gb 0x3995a02010 > 10200547328 9.5Gb 0x3995a02010 > 9663676416 9.0Gb 0x3995a02010 > 9126805504 8.5Gb 0x3995a02010 > 8589934592 8.0Gb 0x3995a02010 > 8053063680 7.5Gb 0x3995a02010 > 7516192768 7.0Gb 0x3995a02010 > 6979321856 6.5Gb 0x3995a02010 > 6442450944 6.0Gb 0x3995a02010 > 5905580032 5.5Gb 0x3995a02010 > 5368709120 5.0Gb 0x3995a02010 > 4831838208 4.5Gb 0x3995a02010 > 4294967296 4.0Gb 0x3995a02010 > 3758096384 3.5Gb 0x3995a02010 > 3221225472 3.0Gb 0x3995a02010 > 2684354560 2.5Gb 0x3995a02010 > 2147483648 2.0Gb 0x3995a02010 > 1610612736 1.5Gb 0x3995a02010 > 1073741824 1.0Gb 0x3995a02010 > 536870912 0.5Gb 0x3995a02010 > > > Can someone point me in the right direction? > > Thanks > Jason -- Todd Miller From jmiller at stsci.edu Fri Jun 11 13:56:03 2004 From: jmiller at stsci.edu (Todd Miller) Date: Fri Jun 11 13:56:03 2004 Subject: [Numpy-discussion] ObjectArray In-Reply-To: <40C9C779.6090700@sympatico.ca> References: <40C86F7D.20309@sympatico.ca> <1086899001.4044.24.camel@halloween.stsci.edu> <40C9C779.6090700@sympatico.ca> Message-ID: <1086987202.5585.366.camel@halloween.stsci.edu> On Fri, 2004-06-11 at 10:53, Colin J. Williams wrote: > Todd, > > I'm trying out objects.py from the CVS as of June 11, with PROTOTYPE= > 1 > > Suggestions and comments based on that. > > 1. change the order in ObjectsArray so the __init__ becomes: > def __init__(self, objects=None, shape=None, > rank=None): > The thinking is that, in the absence of others, objects is the > argument most likely to be specified. I think this can go either way. For people that care, there are already array() and fromlist(). Subclasses can override the order. Am I missing something? > 2. add a check in __init__ to ensure that the shape is in fact > shape-like. Right now it raises a TypeError, and although it's ugly, it does at least trap the error and give a clue in the right direction. > 3. real doc strings would be helpful. Currently, the doc strings are > monopolized by the doctest stuff, > which was not the original intent of doc strings. True enough. Maybe for 1.1. > 4. I wonder about the need for both rank and shape. NumArray gets > along fine with just shape. This is a new feature, and is useful in the context of object arrays where a sequence object is an ambiguous thing: should the sequence be seen as containing elements of an object array, or itself an element of the object array. rank lets you specify the rank of the resulting object array explicitly. It is also computed implicitly if not specified based on the first object which doesn't match the type of the original sequence. This is new for 1.0. > 5. It is assumed that objects (in __init__) is a sequence, but no > check to make sure that it is. Actually, for 1.0 I don't even think it has to be a sequence anymore. It's possible to make a rank-0 object array. > 6. _byteorder is specified for NumArray, but not for ObjectArray, > this upsets a utility which > looks at array characteristics. _byteorder is a private attribute which doesn't make sense for ObjectArray. Add exception handling to the utility if it must be used with ObjectArrays. > 7. How is an ObjectArray better than a nested list of objects? It > seems to provide speedier access > to object pointers for larger n. It may offer the opportunity > to present data in a more organized > tabular manner. Dictionaries are better, in that they provide > access by means of an arbitrary key, > for the cost of hashing. > > NumArray appears to offer storage saving, in addition to the > above, but not here. > > Am I missing something? ObjectArrays can be indexed, manipulated, and printed like numarrays. ObjectArrays also work with object array ufuncs, so for instance, adding two ObjectArrays together implicitly applies the add builtin to each pair of elements. These ufuncs even support reductions and accumulations. This *is* a good question, but I still think ObjectArrays are useful in some contexts. Thanks for the feedback. Regards, Todd From nadavh at visionsense.com Mon Jun 14 03:41:01 2004 From: nadavh at visionsense.com (Nadav Horesh) Date: Mon Jun 14 03:41:01 2004 Subject: [Numpy-discussion] Subclassing NumArray: _PROTOTYPE flag issue Message-ID: <40CD8055.7050209@visionsense.com> For a simulation project I am working on I've subclasses ArrayType. I was able to do much of my intentions until in one place when I tried to make an array from a list of arrays I got an error message: . . . File "/usr/local/lib/python2.3/site-packages/numarray/numarraycore.py", line 325, in array return fromlist(sequence, type, shape) File "/usr/local/lib/python2.3/site-packages/numarray/numarraycore.py", line 212, in fromlist a = a.astype(type) File "/usr/local/lib/python2.3/site-packages/numarray/numarraycore.py", line 630, in astype retarr = self.__class__(buffer=None, shape=self._shape, type=type) TypeError: __init__() got an unexpected keyword argument 'buffer' The analysis of the code showed that: 1. The NumArray class method definitions depends on the _PROTOTYPE flag 2. The post-mortem debugging showed that when the error flagged, the value of the variable _PROTOTYPE was 0 In a stand alone script there was no problem to do the list-> array conversion: >>> import numarray as N >>> import NumImage as NI # My module with the derived class >>> a = N.arange(4) >>> ia = NI.Cimage(N.arange(4)) # CImage is a derivative of NumImage >>> a array([0, 1, 2, 3]) >>> ia Cimage([0, 1, 2, 3]) >>> N.array([a+i for i in range(3)]) array([[0, 1, 2, 3], [1, 2, 3, 4], [2, 3, 4, 5]]) >>> N.array([ia+i for i in range(3)]) # OK here, but failed as a part of a complex script Cimage([[0, 1, 2, 3], [1, 2, 3, 4], [2, 3, 4, 5]]) My questions are: 1. Is this flag is in use? If I set it to 0 will I be able to derive a class from the "C code"? 2. Any intelligent solution? Nadav. From cjw at sympatico.ca Mon Jun 14 09:46:46 2004 From: cjw at sympatico.ca (Colin J. Williams) Date: Mon Jun 14 09:46:46 2004 Subject: [Numpy-discussion] ObjectArray In-Reply-To: <1086987202.5585.366.camel@halloween.stsci.edu> References: <40C86F7D.20309@sympatico.ca> <1086899001.4044.24.camel@halloween.stsci.edu> <40C9C779.6090700@sympatico.ca> <1086987202.5585.366.camel@halloween.stsci.edu> Message-ID: <40CDD617.2010605@sympatico.ca> Todd Miller wrote: >On Fri, 2004-06-11 at 10:53, Colin J. Williams wrote: > > >>Todd, >> >>I'm trying out objects.py from the CVS as of June 11, with PROTOTYPE= >>1 >> >>Suggestions and comments based on that. >> >>1. change the order in ObjectsArray so the __init__ becomes: >> def __init__(self, objects=None, shape=None, >>rank=None): >> The thinking is that, in the absence of others, objects is the >>argument most likely to be specified. >> >> > >I think this can go either way. For people that care, there are >already array() and fromlist(). Subclasses can override the order. Am >I missing something? > The convenience of being able to call the ObjectArray constructor without using keywords. For this purpose, I suggest that it's best to order the parameters in the most likely frequency of usage. I've found that it's not good to have a subclass with a different signature. A generic call to a constructor is as likely to call the subclass as the class. >>2. add a check in __init__ to ensure that the shape is in fact >>shape-like. >> >> > >Right now it raises a TypeError, and although it's ugly, it does at >least trap the error and give a clue in the right direction. > The error traceback is rather deep and doesn't directly point to the source of the problem. >>3. real doc strings would be helpful. Currently, the doc strings are >>monopolized by the doctest stuff, >> which was not the original intent of doc strings. >> >> > >True enough. Maybe for 1.1. > > This is particularly important for objects which are not elsewhere documented, including private functions. >>4. I wonder about the need for both rank and shape. NumArray gets >>along fine with just shape. >> >> > >This is a new feature, and is useful in the context of object arrays >where a sequence object is an ambiguous thing: should the sequence be >seen as containing elements of an object array, or itself an element of >the object array. rank lets you specify the rank of the resulting >object array explicitly. It is also computed implicitly if not >specified based on the first object which doesn't match the type of the >original sequence. This is new for 1.0. > I hadn't spotted that rank gives one the opportunity to specify the dimensionality of objects, but shape provides another way of doing it. >>5. It is assumed that objects (in __init__) is a sequence, but no >>check to make sure that it is. >> >> > >Actually, for 1.0 I don't even think it has to be a sequence anymore. >It's possible to make a rank-0 object array. > > Yes, it is, this adds to the generality. >>6. _byteorder is specified for NumArray, but not for ObjectArray, >>this upsets a utility which >> looks at array characteristics. >> >> > >_byteorder is a private attribute which doesn't make sense for >ObjectArray. Add exception handling to the utility if it must be used >with ObjectArrays. > I'll do that. > > > >>7. How is an ObjectArray better than a nested list of objects? It >>seems to provide speedier access >> to object pointers for larger n. It may offer the opportunity >>to present data in a more organized >> tabular manner. Dictionaries are better, in that they provide >>access by means of an arbitrary key, >> for the cost of hashing. >> >> NumArray appears to offer storage saving, in addition to the >>above, but not here. >> >> Am I missing something? >> >> > >ObjectArrays can be indexed, manipulated, and printed like numarrays. >ObjectArrays also work with object array ufuncs, so for instance, adding >two ObjectArrays together implicitly applies the add builtin to each >pair of elements. These ufuncs even support reductions and >accumulations. This *is* a good question, but I still think >ObjectArrays are useful in some contexts. > The ability to use the ufuncs is potentially good, but presumably assumes a homogeneous ObjectArray. We can't mix strings and dictionaries. > >Thanks for the feedback. > >Regards, >Todd > > > Many thanks for your response. ObjectArray provides a flexible addition to numarray. Colin W > > > From perry at stsci.edu Mon Jun 14 11:28:04 2004 From: perry at stsci.edu (Perry Greenfield) Date: Mon Jun 14 11:28:04 2004 Subject: [Numpy-discussion] Subclassing NumArray: _PROTOTYPE flag issue In-Reply-To: <40CD8055.7050209@visionsense.com> Message-ID: Unfortunately Todd is away this week. I can try to help if you can illustrate what is being tried in the more complex script that you are referring to that is failing. Offhand, I didn't believe that _PROTOTYPE should ever be set to 0 unless it was done so explicitly for testing or debugging purposes (but perhaps I misremember). Thanks, Perry Nadav Horesh wrote: > For a simulation project I am working on I've subclasses ArrayType. I > was able to do much of my intentions until in one place when I tried to > make an array from a list of arrays I got an error message: > > . > . > . > File > "/usr/local/lib/python2.3/site-packages/numarray/numarraycore.py", line > 325, in array > return fromlist(sequence, type, shape) > File > "/usr/local/lib/python2.3/site-packages/numarray/numarraycore.py", line > 212, in fromlist > a = a.astype(type) > File > "/usr/local/lib/python2.3/site-packages/numarray/numarraycore.py", line > 630, in astype > retarr = self.__class__(buffer=None, shape=self._shape, type=type) > TypeError: __init__() got an unexpected keyword argument 'buffer' > > The analysis of the code showed that: > > 1. The NumArray class method definitions depends on the _PROTOTYPE flag > 2. The post-mortem debugging showed that when the error flagged, the > value of the variable _PROTOTYPE was 0 > > In a stand alone script there was no problem to do the list-> array > conversion: > > >>> import numarray as N > >>> import NumImage as NI # My module with the derived class > >>> a = N.arange(4) > >>> ia = NI.Cimage(N.arange(4)) # CImage is a derivative of NumImage > >>> a > array([0, 1, 2, 3]) > >>> ia > Cimage([0, 1, 2, 3]) > >>> N.array([a+i for i in range(3)]) > array([[0, 1, 2, 3], > [1, 2, 3, 4], > [2, 3, 4, 5]]) > >>> N.array([ia+i for i in range(3)]) # OK here, but failed as a part > of a complex script > Cimage([[0, 1, 2, 3], > [1, 2, 3, 4], > [2, 3, 4, 5]]) > > > My questions are: > > 1. Is this flag is in use? If I set it to 0 will I be able to derive > a class from the "C code"? > 2. Any intelligent solution? > > Nadav. > > > ------------------------------------------------------- > This SF.Net email is sponsored by the new InstallShield X. > From Windows to Linux, servers to mobile, InstallShield X is the > one installation-authoring solution that does it all. Learn more and > evaluate today! http://www.installshield.com/Dev2Dev/0504 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From haase at msg.ucsf.edu Mon Jun 14 16:03:02 2004 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Mon Jun 14 16:03:02 2004 Subject: [Numpy-discussion] numarray.sum should raise exception for wrong axis argument Message-ID: <200406141602.07650.haase@msg.ucsf.edu> Hi, please take a look at this: >>> na.sum( na.zeros((2,6)) ) [0 0 0 0 0 0] >>> na.sum( na.zeros((2,6)) , 0) [0 0 0 0 0 0] >>> na.sum( na.zeros((2,6)) , 1) [0 0] >>> na.sum( na.zeros((2,6)) , 2) [0 0] >>> na.sum( na.zeros((2,6)) , 3) [0 0] >>> na.sum( na.zeros((2,6)) , 4) [0 0] >>> na.sum( na.zeros((2,6)) , -1) [0 0] >>> na.sum( na.zeros((2,6)) , -2) [0 0 0 0 0 0] >>> na.sum( na.zeros((2,6)) , -3) [0 0] >>> na.sum( na.zeros((2,6)) , -4) [0 0] >>> I think here should be a ValueError exception thrown rather than defaulting to the '-1'-axis. Comments ? Thanks, Sebastian Haase From nadavh at visionsense.com Wed Jun 16 08:43:14 2004 From: nadavh at visionsense.com (Nadav Horesh) Date: Wed Jun 16 08:43:14 2004 Subject: [Numpy-discussion] Subclassing NumArray: _PROTOTYPE flag issue Message-ID: <07C6A61102C94148B8104D42DE95F7E86DED3D@exchange2k.envision.co.il> OK it is my mistake, I did not implement the buffer argument in the constructor of my derived class. Still I have some minor problems, but nothing to bother about for the time being. It is very compelling to subclass NumArray, I wish that it will be documented when version 1.0 will be released. Nadav -----Original Message----- From: Perry Greenfield [mailto:perry at stsci.edu] Sent: Mon 14-Jun-04 21:27 To: Nadav Horesh; numpy Cc: Subject: RE: [Numpy-discussion] Subclassing NumArray: _PROTOTYPE flag issue Unfortunately Todd is away this week. I can try to help if you can illustrate what is being tried in the more complex script that you are referring to that is failing. Offhand, I didn't believe that _PROTOTYPE should ever be set to 0 unless it was done so explicitly for testing or debugging purposes (but perhaps I misremember). Thanks, Perry Nadav Horesh wrote: > For a simulation project I am working on I've subclasses ArrayType. I > was able to do much of my intentions until in one place when I tried to > make an array from a list of arrays I got an error message: > > . > . > . > File > "/usr/local/lib/python2.3/site-packages/numarray/numarraycore.py", line > 325, in array > return fromlist(sequence, type, shape) > File > "/usr/local/lib/python2.3/site-packages/numarray/numarraycore.py", line > 212, in fromlist > a = a.astype(type) > File > "/usr/local/lib/python2.3/site-packages/numarray/numarraycore.py", line > 630, in astype > retarr = self.__class__(buffer=None, shape=self._shape, type=type) > TypeError: __init__() got an unexpected keyword argument 'buffer' > > The analysis of the code showed that: > > 1. The NumArray class method definitions depends on the _PROTOTYPE flag > 2. The post-mortem debugging showed that when the error flagged, the > value of the variable _PROTOTYPE was 0 > > In a stand alone script there was no problem to do the list-> array > conversion: > > >>> import numarray as N > >>> import NumImage as NI # My module with the derived class > >>> a = N.arange(4) > >>> ia = NI.Cimage(N.arange(4)) # CImage is a derivative of NumImage > >>> a > array([0, 1, 2, 3]) > >>> ia > Cimage([0, 1, 2, 3]) > >>> N.array([a+i for i in range(3)]) > array([[0, 1, 2, 3], > [1, 2, 3, 4], > [2, 3, 4, 5]]) > >>> N.array([ia+i for i in range(3)]) # OK here, but failed as a part > of a complex script > Cimage([[0, 1, 2, 3], > [1, 2, 3, 4], > [2, 3, 4, 5]]) > > > My questions are: > > 1. Is this flag is in use? If I set it to 0 will I be able to derive > a class from the "C code"? > 2. Any intelligent solution? > > Nadav. > > > ------------------------------------------------------- > This SF.Net email is sponsored by the new InstallShield X. > From Windows to Linux, servers to mobile, InstallShield X is the > one installation-authoring solution that does it all. Learn more and > evaluate today! http://www.installshield.com/Dev2Dev/0504 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From jbaddor at physics.mcgill.ca Thu Jun 17 11:47:03 2004 From: jbaddor at physics.mcgill.ca (Jean-Bernard Addor) Date: Thu Jun 17 11:47:03 2004 Subject: [Numpy-discussion] arrayfns.histogram does it exist in numarray? Message-ID: Hello ! I am analysing data in hdf files with Numeric. For different reasons, including better support of hdf files (huge datasets), I try to switch to numarray. I uses a lot the arrayfns.histogram function of Numeric, does something similar exist in nummarray? See you, Jean-Bernard From perry at stsci.edu Thu Jun 17 13:28:10 2004 From: perry at stsci.edu (Perry Greenfield) Date: Thu Jun 17 13:28:10 2004 Subject: [Numpy-discussion] arrayfns.histogram does it exist in numarray? In-Reply-To: Message-ID: Jean-Bernard Addor wrote: > I am analysing data in hdf files with Numeric. For different reasons, > including better support of hdf files (huge datasets), I try to switch to > numarray. I uses a lot the arrayfns.histogram function of Numeric, does > something similar exist in nummarray? > Not as such yet (we have our own more specialized histogram function for internal projects, but hadn't gotten around to generalizing it for general use). I think we'll take a quick look at seeing if the Numeric arrayfns extension can be recompiled to use numarray with no or few changes. Since Todd is away, this won't be done until next week. I also have a simple Python function to do this, albeit more slowly (but not hugely slower) if you need something quickly. Perry From Fernando.Perez at colorado.edu Thu Jun 17 16:28:00 2004 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Thu Jun 17 16:28:00 2004 Subject: [Numpy-discussion] Using dotblas Message-ID: <40D228F0.2050209@colorado.edu> Hi all, I've built NumPy 23.1 with dotblas enabled, and as per the readme I'm starting to pepper my code with things like: # External packages import Numeric as N # Use BLAS-optimized versions when available try: import dotblas N.dot = dotblas._dotblas.matrixproduct N.multiarray.innerproduct = dotblas._dotblas.innerproduct except ImportError: print '*** dotblas not available, using regular Numeric functions' I wonder if this is really necessary, and if so, why. Is there any reason _not_ to use the dotblas versions when available? If not, shouldn't Numeric itself pick them up internally so that it always exposes the fastest possible version? I can always send in a patch to do this ( a trivial 5-liner like the above code), but I may just be missing something, as it seems too obvious not to be already the default. Regards, f. From Fernando.Perez at colorado.edu Thu Jun 17 16:50:06 2004 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Thu Jun 17 16:50:06 2004 Subject: [Numpy-discussion] Using dotblas In-Reply-To: <40D22C18.4040804@ucsd.edu> References: <40D228F0.2050209@colorado.edu> <40D22C18.4040804@ucsd.edu> Message-ID: <40D22DF1.5020306@colorado.edu> Robert Kern wrote: > Check the last few lines of Numeric.py . Argh. Could've saved myself the embarrasment :) In my meager defense, the README.html file for doblas then needs to be updated, since that's what I followed: # quote Finally, if you don't know whether dotblas will be available on all the machines where you whish to run your code, you can simply use a try-except statement like this: import Numeric try: import dotblas Numeric.dot = dotblas.dot if we can't find the fast dotblas, just use the normal dot except ImportError: pass # /quote Thanks, though. I'll clean up my codes f From shawnemch at hotmail.com Sat Jun 19 21:56:01 2004 From: shawnemch at hotmail.com (OEMcd--Cheap--software--Inc.) Date: Sat Jun 19 21:56:01 2004 Subject: [Numpy-discussion] Review your account 555418212 Message-ID: <36149808074438614442897@hotmail.com> An HTML attachment was scrubbed... URL: From jmiller at stsci.edu Mon Jun 21 07:47:02 2004 From: jmiller at stsci.edu (Todd Miller) Date: Mon Jun 21 07:47:02 2004 Subject: [Numpy-discussion] arrayfns.histogram does it exist in numarray? In-Reply-To: References: Message-ID: <1087829108.2619.66.camel@localhost.localdomain> On Thu, 2004-06-17 at 16:27, Perry Greenfield wrote: > Jean-Bernard Addor wrote: > > I am analysing data in hdf files with Numeric. For different reasons, > > including better support of hdf files (huge datasets), I try to switch to > > numarray. I uses a lot the arrayfns.histogram function of Numeric, does > > something similar exist in nummarray? > > > Not as such yet (we have our own more specialized histogram > function for internal projects, but hadn't gotten around to > generalizing it for general use). I think we'll take a quick > look at seeing if the Numeric arrayfns extension can be recompiled > to use numarray with no or few changes. Since Todd is away, > this won't be done until next week. I also have a simple > Python function to do this, albeit more slowly (but not hugely > slower) if you need something quickly. I ported histogram() this morning to numarray.numeric. It's in CVS now and will be part of numarray-1.0 in the next week or so. Regards, Todd From jmiller at stsci.edu Mon Jun 21 08:00:05 2004 From: jmiller at stsci.edu (Todd Miller) Date: Mon Jun 21 08:00:05 2004 Subject: [Numpy-discussion] Subclassing NumArray: _PROTOTYPE flag issue In-Reply-To: <40CD8055.7050209@visionsense.com> References: <40CD8055.7050209@visionsense.com> Message-ID: <1087829929.2619.72.camel@localhost.localdomain> On Mon, 2004-06-14 at 06:39, Nadav Horesh wrote: > For a simulation project I am working on I've subclasses ArrayType. I > was able to do much of my intentions until in one place when I tried to > make an array from a list of arrays I got an error message: > > . > . > . > File > "/usr/local/lib/python2.3/site-packages/numarray/numarraycore.py", line > 325, in array > return fromlist(sequence, type, shape) > File > "/usr/local/lib/python2.3/site-packages/numarray/numarraycore.py", line > 212, in fromlist > a = a.astype(type) > File > "/usr/local/lib/python2.3/site-packages/numarray/numarraycore.py", line > 630, in astype > retarr = self.__class__(buffer=None, shape=self._shape, type=type) > TypeError: __init__() got an unexpected keyword argument 'buffer' > > The analysis of the code showed that: > > 1. The NumArray class method definitions depends on the _PROTOTYPE flag > 2. The post-mortem debugging showed that when the error flagged, the > value of the variable _PROTOTYPE was 0 > > In a stand alone script there was no problem to do the list-> array > conversion: > > >>> import numarray as N > >>> import NumImage as NI # My module with the derived class > >>> a = N.arange(4) > >>> ia = NI.Cimage(N.arange(4)) # CImage is a derivative of NumImage > >>> a > array([0, 1, 2, 3]) > >>> ia > Cimage([0, 1, 2, 3]) > >>> N.array([a+i for i in range(3)]) > array([[0, 1, 2, 3], > [1, 2, 3, 4], > [2, 3, 4, 5]]) > >>> N.array([ia+i for i in range(3)]) # OK here, but failed as a part > of a complex script > Cimage([[0, 1, 2, 3], > [1, 2, 3, 4], > [2, 3, 4, 5]]) > > > My questions are: > > 1. Is this flag is in use? Yes. > If I set it to 0 will I be able to derive > a class from the "C code"? Exactly. _PROTOTYPE hides the original Python implementations of things that have since been moved to C; it is a compile time flag. I keep the prototype Python code around because it is much easier to debug and very useful when modifying or extending numarray. Regards, Todd From falted at pytables.org Mon Jun 21 11:40:06 2004 From: falted at pytables.org (Francesc Alted) Date: Mon Jun 21 11:40:06 2004 Subject: [Numpy-discussion] Unpickled numarray types unsable? Message-ID: <200406212039.18331.falted@pytables.org> Hi, I'm experiencing problems when using a numarray datatype after pickle/unpickle it: >>> import numarray >>> import pickle >>> c=pickle.dumps(numarray.Float64) >>> t=pickle.loads(c) >>> t Float64 >>> na=numarray.array(type=t, shape=(2,)) Traceback (most recent call last): File "", line 1, in ? File "/usr/local/lib/python2.3/site-packages/numarray/numarraycore.py", line 321, in array return NumArray(buffer=sequence, shape=shape, type=type) RuntimeError: _numarray_init: can't get typeno for type Perhaps this is a bug and the typeno attribute is missed when reconstructing the type? Cheers, -- Francesc Alted From jmiller at stsci.edu Mon Jun 21 13:06:04 2004 From: jmiller at stsci.edu (Todd Miller) Date: Mon Jun 21 13:06:04 2004 Subject: [Numpy-discussion] Unpickled numarray types unsable? In-Reply-To: <200406212039.18331.falted@pytables.org> References: <200406212039.18331.falted@pytables.org> Message-ID: <1087848295.2619.105.camel@localhost.localdomain> On Mon, 2004-06-21 at 14:39, Francesc Alted wrote: > Hi, > > I'm experiencing problems when using a numarray datatype after > pickle/unpickle it: > > >>> import numarray > >>> import pickle > >>> c=pickle.dumps(numarray.Float64) > >>> t=pickle.loads(c) > >>> t > Float64 > >>> na=numarray.array(type=t, shape=(2,)) > Traceback (most recent call last): > File "", line 1, in ? > File "/usr/local/lib/python2.3/site-packages/numarray/numarraycore.py", line 321, in array > return NumArray(buffer=sequence, shape=shape, type=type) > RuntimeError: _numarray_init: can't get typeno for type > > Perhaps this is a bug and the typeno attribute is missed when reconstructing > the type? It's a bug in the C-API / implementation of pickling for the type objects. The C-API keeps references to the type objects which are assumed to be unique: there is one Float64, just like there is one None. The pickling produces "functional equivalents" which lose their identities from the perspective of C. I'm not sure what the fix is yet. Regards, Todd From faheem at email.unc.edu Mon Jun 21 17:18:02 2004 From: faheem at email.unc.edu (Faheem Mitha) Date: Mon Jun 21 17:18:02 2004 Subject: [Numpy-discussion] numarray.concatenate for character arrays Message-ID: Dear People, Is the function numarray.concatenate supposed to work for character arrays? It doesn't for me. Do I need to write my own? Thanks in advance. Please cc me, I'm not subscribed. Faheem. In [17]: foo Out[17]: CharArray([['T', 'T'], ['C', 'A']]) In [18]: bar Out[18]: CharArray([['G', 'G'], ['G', 'G']]) In [23]: numarray.concatenate((foo,bar)) --------------------------------------------------------------------------- error Traceback (most recent call last) /home/faheem/wc/corrmodel/trunk/ /usr/lib/python2.3/site-packages/numarray/generic.py in concatenate(arrs, axis) 1018 arrs = map(_nc.asarray, arrs) 1019 if axis == 0: -> 1020 return _concat(arrs) 1021 else: 1022 return swapaxes(_concat([swapaxes(m,axis,0) for m in arrs]), axis, 0) /usr/lib/python2.3/site-packages/numarray/generic.py in _concat(arrs) 1000 convType = ufunc._maxPopType(arrs) 1001 except TypeError: -> 1002 dest = arrs[0]._clone(shape=destShape) 1003 else: 1004 dest = arrs[0].__class__(shape=destShape, type=convType) /usr/lib/python2.3/site-packages/numarray/generic.py in _clone(self, shape) 783 def _clone(self, shape): 784 c = self.copy() --> 785 c.resize(shape) 786 return c 787 /usr/lib/python2.3/site-packages/numarray/generic.py in resize(self, shape, *args) 854 self[offset:offset+olen] = self[0:olen] 855 offset += olen --> 856 self[offset:nlen] = self[0:nlen-offset] 857 else: # zero fill resized zero-length numarray 858 self[:] = 0 /usr/lib/python2.3/site-packages/numarray/strings.py in _copyFrom(self, arr) 217 me = self._byteView() 218 if self._itemsize <= arr._itemsize: --> 219 me[:] = it[..., :self._itemsize] 220 else: 221 me[...,:it._shape[-1]] = it error: copy1bytes: access beyond buffer. offset=8 buffersize=8 From Sebastien.deMentendeHorne at electrabel.com Tue Jun 22 04:59:32 2004 From: Sebastien.deMentendeHorne at electrabel.com (Sebastien.deMentendeHorne at electrabel.com) Date: Tue Jun 22 04:59:32 2004 Subject: [Numpy-discussion] lock in LinearAlgebra when embedding Message-ID: <035965348644D511A38C00508BF7EAEB145CAEE0@seacex03.eib.electrabel.be> Hi, There is a nasty behaviour in the LinearAlgebra package under windows NT. When I run the following script from the command-line (python2.3 on windows Enthought edition but I had the same problem with the original python2.3 + Numeric 23.3), it returns without problem. ##################### from LinearAlgebra import eigenvalues from Numeric import array a = array([[3,4],[1,6]]) print eigenvalues(a) ##################### Output [ 2. 7.] However, when I evaluate the same script embedded in C, it hangs while consuming 100% of the CPU: /* C Code */ #include int main(int argc, char *argv[]) { Py_Initialize(); PyRun_SimpleString("from LinearAlgebra import eigenvalues\nfrom Numeric import array\na = array([[3,4],[1,6]])\nprint eigenvalues(a)\n"); Py_Finalize(); return 0; } /* end of C code */ I compile the code with Mingw gcc embed.c -Ic:\python23\include -Lc:\python23 -lpython23 and gcc -v gives Reading specs from C:/Python23/Enthought/MingW/bin/../lib/gcc-lib/mingw32/3.2.3/specs Configured with: ../gcc/configure --with-gcc --with-gnu-ld --with-gnu-as --host=mingw32 --target=mingw32 --prefix=/mingw --enable-threads --disable-nls --enable-languages=c++,f77,objc --disable-win32-registry --disable-shared --enable-sjlj- exceptions Thread model: win32 gcc version 3.2.3 (mingw special 20030504-1) I must recompile lapack_lite from source with Mingw in order to get the correct behaviour. Any hint on a cleaner solution (not recompile lapack_lite whenever I install a new version of Numeric). Seb From mdehoon at ims.u-tokyo.ac.jp Tue Jun 22 05:22:17 2004 From: mdehoon at ims.u-tokyo.ac.jp (Michiel Jan Laurens de Hoon) Date: Tue Jun 22 05:22:17 2004 Subject: [Numpy-discussion] lock in LinearAlgebra when embedding In-Reply-To: <035965348644D511A38C00508BF7EAEB145CAEE0@seacex03.eib.electrabel.be> References: <035965348644D511A38C00508BF7EAEB145CAEE0@seacex03.eib.electrabel.be> Message-ID: <40D8240E.3000009@ims.u-tokyo.ac.jp> This may be related to patch 732520. The problem is that lapack_lite is not compiled correctly by setup.py; one part of it should be compiled without optimization, but is compiled with. On many platforms, this doesn't cause any problems, but on Cygwin (and therefore probably also on MinGW) the eigenvalues function hangs as a result. You can get more information by looking up the patch on sourceforge. This issue has come up a couple of times. Maybe we should fix the Numeric/numarray source code with this patch or something similar? --Michiel, U Tokyo. Sebastien.deMentendeHorne at electrabel.com wrote: > Hi, > > There is a nasty behaviour in the LinearAlgebra package under windows NT. > > When I run the following script from the command-line (python2.3 on windows > Enthought edition but I had the same problem with the original python2.3 + > Numeric 23.3), it returns without problem. > ##################### > from LinearAlgebra import eigenvalues > from Numeric import array > a = array([[3,4],[1,6]]) > print eigenvalues(a) > ##################### > Output > [ 2. 7.] > > However, when I evaluate the same script embedded in C, it hangs while > consuming 100% of the CPU: > /* C Code */ > #include > > int > main(int argc, char *argv[]) > { > Py_Initialize(); > PyRun_SimpleString("from LinearAlgebra import eigenvalues\nfrom Numeric > import array\na = array([[3,4],[1,6]])\nprint eigenvalues(a)\n"); > > Py_Finalize(); > return 0; > } > /* end of C code */ > > I compile the code with Mingw > gcc embed.c -Ic:\python23\include -Lc:\python23 -lpython23 > and gcc -v gives > > Reading specs from > C:/Python23/Enthought/MingW/bin/../lib/gcc-lib/mingw32/3.2.3/specs > Configured with: ../gcc/configure --with-gcc --with-gnu-ld --with-gnu-as > --host=mingw32 --target=mingw32 --prefix=/mingw > --enable-threads --disable-nls --enable-languages=c++,f77,objc > --disable-win32-registry --disable-shared --enable-sjlj- > exceptions > Thread model: win32 > gcc version 3.2.3 (mingw special 20030504-1) > > I must recompile lapack_lite from source with Mingw in order to get the > correct behaviour. > > Any hint on a cleaner solution (not recompile lapack_lite whenever I install > a new version of Numeric). > > Seb > > > ------------------------------------------------------- > This SF.Net email sponsored by Black Hat Briefings & Training. > Attend Black Hat Briefings & Training, Las Vegas July 24-29 - > digital self defense, top technical experts, no vendor pitches, > unmatched networking opportunities. Visit www.blackhat.com > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > -- Michiel de Hoon, Assistant Professor University of Tokyo, Institute of Medical Science Human Genome Center 4-6-1 Shirokane-dai, Minato-ku Tokyo 108-8639 Japan http://bonsai.ims.u-tokyo.ac.jp/~mdehoon From Sebastien.deMentendeHorne at electrabel.com Tue Jun 22 05:44:04 2004 From: Sebastien.deMentendeHorne at electrabel.com (Sebastien.deMentendeHorne at electrabel.com) Date: Tue Jun 22 05:44:04 2004 Subject: [Numpy-discussion] lock in LinearAlgebra when embedding Message-ID: <035965348644D511A38C00508BF7EAEB145CAEE1@seacex03.eib.electrabel.be> It looks like it is a similar problem. However, it is not yet resolved as the patch is from may/2003 and I have this problem in the latest Numeric version. Moreover, my problem lies in the embedded/not embedded difference when running the same script. In fact, I notice it when using Python embedded in Excel via VBA. Testing the same script from the command-line is OK while from Excel it hangs. And the python/numeric I use is the "native windows" version (i.e. compiled with VC++ and not cygwin/mingw). So, Numeric embedded works if lapack_lite is compile with mingw32 but NOT with VC++. Looks like this lapack interface is delicate... Seb -----Original Message----- From: Michiel Jan Laurens de Hoon [mailto:mdehoon at ims.u-tokyo.ac.jp] Sent: mardi 22 juin 2004 14:21 To: Sebastien.deMentendeHorne at electrabel.com Cc: numpy-discussion at lists.sourceforge.net Subject: Re: [Numpy-discussion] lock in LinearAlgebra when embedding This may be related to patch 732520. The problem is that lapack_lite is not compiled correctly by setup.py; one part of it should be compiled without optimization, but is compiled with. On many platforms, this doesn't cause any problems, but on Cygwin (and therefore probably also on MinGW) the eigenvalues function hangs as a result. You can get more information by looking up the patch on sourceforge. This issue has come up a couple of times. Maybe we should fix the Numeric/numarray source code with this patch or something similar? --Michiel, U Tokyo. Sebastien.deMentendeHorne at electrabel.com wrote: > Hi, > > There is a nasty behaviour in the LinearAlgebra package under windows NT. > > When I run the following script from the command-line (python2.3 on windows > Enthought edition but I had the same problem with the original python2.3 + > Numeric 23.3), it returns without problem. > ##################### > from LinearAlgebra import eigenvalues > from Numeric import array > a = array([[3,4],[1,6]]) > print eigenvalues(a) > ##################### > Output > [ 2. 7.] > > However, when I evaluate the same script embedded in C, it hangs while > consuming 100% of the CPU: > /* C Code */ > #include > > int > main(int argc, char *argv[]) > { > Py_Initialize(); > PyRun_SimpleString("from LinearAlgebra import eigenvalues\nfrom Numeric > import array\na = array([[3,4],[1,6]])\nprint eigenvalues(a)\n"); > > Py_Finalize(); > return 0; > } > /* end of C code */ > > I compile the code with Mingw > gcc embed.c -Ic:\python23\include -Lc:\python23 -lpython23 > and gcc -v gives > > Reading specs from > C:/Python23/Enthought/MingW/bin/../lib/gcc-lib/mingw32/3.2.3/specs > Configured with: ../gcc/configure --with-gcc --with-gnu-ld --with-gnu-as > --host=mingw32 --target=mingw32 --prefix=/mingw > --enable-threads --disable-nls --enable-languages=c++,f77,objc > --disable-win32-registry --disable-shared --enable-sjlj- > exceptions > Thread model: win32 > gcc version 3.2.3 (mingw special 20030504-1) > > I must recompile lapack_lite from source with Mingw in order to get the > correct behaviour. > > Any hint on a cleaner solution (not recompile lapack_lite whenever I install > a new version of Numeric). > > Seb > > > ------------------------------------------------------- > This SF.Net email sponsored by Black Hat Briefings & Training. > Attend Black Hat Briefings & Training, Las Vegas July 24-29 - > digital self defense, top technical experts, no vendor pitches, > unmatched networking opportunities. Visit www.blackhat.com > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > -- Michiel de Hoon, Assistant Professor University of Tokyo, Institute of Medical Science Human Genome Center 4-6-1 Shirokane-dai, Minato-ku Tokyo 108-8639 Japan http://bonsai.ims.u-tokyo.ac.jp/~mdehoon From jmiller at stsci.edu Tue Jun 22 06:05:11 2004 From: jmiller at stsci.edu (Todd Miller) Date: Tue Jun 22 06:05:11 2004 Subject: [Numpy-discussion] numarray.concatenate for character arrays In-Reply-To: References: Message-ID: <1087909484.14732.5.camel@halloween.stsci.edu> Hi Faheem, This is a bug. It'll be fixed in numarray-1.0 which should hopefully be released this week. Regards, Todd On Mon, 2004-06-21 at 20:17, Faheem Mitha wrote: > Dear People, > > Is the function numarray.concatenate supposed to work for character > arrays? It doesn't for me. Do I need to write my own? Thanks in > advance. Please cc me, I'm not subscribed. > > Faheem. > > In [17]: foo > Out[17]: > CharArray([['T', 'T'], > ['C', 'A']]) > > In [18]: bar > Out[18]: > CharArray([['G', 'G'], > ['G', 'G']]) > > In [23]: numarray.concatenate((foo,bar)) > --------------------------------------------------------------------------- > error Traceback (most recent call > last) > > /home/faheem/wc/corrmodel/trunk/ > > /usr/lib/python2.3/site-packages/numarray/generic.py in > concatenate(arrs, axis) > 1018 arrs = map(_nc.asarray, arrs) > 1019 if axis == 0: > -> 1020 return _concat(arrs) > 1021 else: > 1022 return swapaxes(_concat([swapaxes(m,axis,0) for m in > arrs]), axis, 0) > > /usr/lib/python2.3/site-packages/numarray/generic.py in _concat(arrs) > 1000 convType = ufunc._maxPopType(arrs) > 1001 except TypeError: > -> 1002 dest = arrs[0]._clone(shape=destShape) > 1003 else: > 1004 dest = arrs[0].__class__(shape=destShape, > type=convType) > > /usr/lib/python2.3/site-packages/numarray/generic.py in _clone(self, > shape) > 783 def _clone(self, shape): > 784 c = self.copy() > --> 785 c.resize(shape) > 786 return c > 787 > > /usr/lib/python2.3/site-packages/numarray/generic.py in resize(self, > shape, *args) > 854 self[offset:offset+olen] = self[0:olen] > 855 offset += olen > --> 856 self[offset:nlen] = self[0:nlen-offset] > 857 else: # zero fill resized zero-length numarray > 858 self[:] = 0 > > /usr/lib/python2.3/site-packages/numarray/strings.py in > _copyFrom(self, arr) > 217 me = self._byteView() > 218 if self._itemsize <= arr._itemsize: > --> 219 me[:] = it[..., :self._itemsize] > 220 else: > 221 me[...,:it._shape[-1]] = it > > error: copy1bytes: access beyond buffer. offset=8 buffersize=8 > > > > > > ------------------------------------------------------- > This SF.Net email sponsored by Black Hat Briefings & Training. > Attend Black Hat Briefings & Training, Las Vegas July 24-29 - > digital self defense, top technical experts, no vendor pitches, > unmatched networking opportunities. Visit www.blackhat.com > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- Todd Miller From sanner at scripps.edu Tue Jun 22 09:09:01 2004 From: sanner at scripps.edu (Michel Sanner) Date: Tue Jun 22 09:09:01 2004 Subject: [Numpy-discussion] convolve2d Message-ID: <40D8559D.8090107@scripps.edu> Hello, I started using numarray for doing 2D convolutions on images. I noticed that import numarray.examples.convolve.high_level as convolve convolve.Convolve2d(kernel, in, out) only works on square images. For images that are not square I get lots of noise in the background. Also I was wondering is using the high_level API is most efficient? Currently my image is a Numeric array (grabbed from the OpenGL frame buffer) which I convert to a numarray to do the convolution and back to a Numeric array. In the future I hope to completely replace Numeric by numarray. Thanks for any input -Michel -- ----------------------------------------------------------------------- o / Michel F. Sanner Ph.D. The Scripps Research Institute o Associate Professor Department of Molecular Biology \ 10550 North Torrey Pines Road o Tel. (858) 784-2341 La Jolla, CA 92037 / Fax. (858) 784-2860 o sanner at scripps.edu http://www.scripps.edu/~sanner ----------------------------------------------------------------------- From rowen at u.washington.edu Tue Jun 22 09:57:06 2004 From: rowen at u.washington.edu (Russell E Owen) Date: Tue Jun 22 09:57:06 2004 Subject: [Numpy-discussion] median filtering question -- masked arrays Message-ID: Does anyone have a hint for applying a median filter to a masked 2D array in numarray? I've been using the kludge of extracting a normal array with the masked data replaced by the overall median of the array and median filtering that (with the appropriate nd_image method). But this is a crude approximation and I'd like to do better -- preferably without coding my own median filter in C. -- Russell From jmiller at stsci.edu Tue Jun 22 11:09:55 2004 From: jmiller at stsci.edu (Todd Miller) Date: Tue Jun 22 11:09:55 2004 Subject: [Numpy-discussion] convolve2d In-Reply-To: <40D8559D.8090107@scripps.edu> References: <40D8559D.8090107@scripps.edu> Message-ID: <1087927670.14732.273.camel@halloween.stsci.edu> On Tue, 2004-06-22 at 11:51, Michel Sanner wrote: > Hello, > > I started using numarray for doing 2D convolutions on images. I noticed > that > > import numarray.examples.convolve.high_level as convolve > convolve.Convolve2d(kernel, in, out) > > only works on square images. For images that are not square I get lots > of noise in the background. I looked at the code and couldn't find any row/column typos. In theory, Convolve2d works for non-square arrays. What do kernel.info() and data.info() say? > Also I was wondering is using the high_level API is most efficient? For CPU, my guess is yes. The point access macros and functions are definitely slower than the high level API in any scenario where you're not running out of memory. The 1D API improves space usage but is less efficient in time. The Numeric compatible API is layered over the high level API. > Currently my image is a Numeric > array (grabbed from the OpenGL frame buffer) which I convert to a > numarray to do the convolution and back to a Numeric array. It's also possible something is happening during this conversion. It'd be good to round-trip the Numeric array and make sure the end product looks like the original. > In the future I hope to completely replace Numeric by numarray. Great! Regards, Todd From jmiller at stsci.edu Tue Jun 22 11:18:07 2004 From: jmiller at stsci.edu (Todd Miller) Date: Tue Jun 22 11:18:07 2004 Subject: [Numpy-discussion] convolve2d In-Reply-To: <40D8559D.8090107@scripps.edu> References: <40D8559D.8090107@scripps.edu> Message-ID: <1087928153.14732.286.camel@halloween.stsci.edu> On Tue, 2004-06-22 at 11:51, Michel Sanner wrote: > Hello, > > I started using numarray for doing 2D convolutions on images. I noticed > that > > import numarray.examples.convolve.high_level as convolve > convolve.Convolve2d(kernel, in, out) > Here's another (quick?) test which might help localize the problem. Do: convolve.Convolve2d(kernel, in, out, fft=1) The underlying Convolve2d implementations are very different depending on the use of FFT. Regards, Todd From jmiller at stsci.edu Tue Jun 22 13:30:12 2004 From: jmiller at stsci.edu (Todd Miller) Date: Tue Jun 22 13:30:12 2004 Subject: [Numpy-discussion] Java version for numarray? In-Reply-To: <200406071311.02485.falted@pytables.org> References: <200406071311.02485.falted@pytables.org> Message-ID: <1087936180.14732.349.camel@halloween.stsci.edu> On Mon, 2004-06-07 at 07:11, Francesc Alted wrote: > Hi, > > I'm using numarray objects basically like data containers for fast I/O > purposes in the context of pytables. I'm thinking now to port part of my > code to Java, and I'd like to use Jython to easy the transition. I've seen > that a there is a port of Numeric to Jython > (http://jnumerical.sourceforge.net/), and I thought that I could use these > Numeric objects as containers. Unfortunately, there are a couple of objects > that pytables relies on, namely RecArray and CharArray, that are not > supported by Numeric. > > My questions are: > > - I think that both RecArray and CharArray modules are written in pure > python, so the porting to Jython should be relatively easy, provided I'm > successful making them to use Numeric objects instead of NumArray objects. > What do you think? Sounds hard. I doubt there is enough commonality with the numarray to make bolting on the pure-Python RecArray and CharArray implementations reasonable. RecArray and CharArray are very much dependent on generic.NDArray; I doubt they're easy to port to Numeric. > Would that be feasible? I don't think so. > - Is there any effort going on in order to have an implementation of > numarray ready for use in Jython? No. Sorry about the slow response, Todd From faheem at email.unc.edu Wed Jun 23 10:58:21 2004 From: faheem at email.unc.edu (Faheem Mitha) Date: Wed Jun 23 10:58:21 2004 Subject: [Numpy-discussion] printing character array object Message-ID: Hi, it's me again. This time I am having problems with printing out a character array object (attached in cPickle format). I don't understand what the error message means. If you load the >>> import cPickle >>> cPickle.dump(foo,open('foo.save','w')) >>> foo [tons of error messages terminating in] libnumarray.error: copy1bytes: access beyond buffer. offset=6 buffersize=6 Looks like the problem is with numarray. The character array is 100x20 and I can't see any intrinsic problem with printing it out. >>> foo.getshape() Out[73]: (100, 20) Thanks for any clarification. Faheem. -------------- next part -------------- A non-text attachment was scrubbed... Name: foo.save Type: application/octet-stream Size: 2313 bytes Desc: URL: From haase at msg.ucsf.edu Wed Jun 23 15:07:04 2004 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Wed Jun 23 15:07:04 2004 Subject: [Numpy-discussion] numarray.sum should raise exception for wrong axis argument - and slicing ... Message-ID: <200406231506.24453.haase@msg.ucsf.edu> Hi, please take a look at this: >>> na.sum( na.zeros((2,6)) ) [0 0 0 0 0 0] >>> na.sum( na.zeros((2,6)) , 0) [0 0 0 0 0 0] >>> na.sum( na.zeros((2,6)) , 1) [0 0] >>> na.sum( na.zeros((2,6)) , 2) [0 0] >>> na.sum( na.zeros((2,6)) , 3) [0 0] >>> na.sum( na.zeros((2,6)) , 4) [0 0] >>> na.sum( na.zeros((2,6)) , -1) [0 0] >>> na.sum( na.zeros((2,6)) , -2) [0 0 0 0 0 0] >>> na.sum( na.zeros((2,6)) , -3) [0 0] >>> na.sum( na.zeros((2,6)) , -4) [0 0] >>> I think here should be a ValueError exception thrown rather than defaulting to the '-1'-axis. Comments ? Also this applies to (all?) other functions that have an 'axis' argument. And further I just found that putting "too many slicings" to an array also gets silently ignored: >>> b.shape (7, 128, 128, 128) >>> b[2,2,2,2,3] Traceback (most recent call last): File "", line 1, in ? IndexError: too many indices. BUT: >>> b[2:3 , 2:3 , 2:3 , 2:3 , 2:3 , 2:3] [[[[ 0.]]]] I find this very confusing !! Is there any good reason not to have the "IndexError" exception in all cases ? Thanks, Sebastian Haase From buu43vysbr at shentel.net Wed Jun 23 23:23:01 2004 From: buu43vysbr at shentel.net (Georgene Michiko) Date: Wed Jun 23 23:23:01 2004 Subject: [Numpy-discussion] THE CHEAPEST EVER SOFTWARES U SEE & WE SHIIP WORLDWIDE makes somebody Message-ID: making fellow stay fixed advantage neck secret pocket money believe important gotten learn nickel handwriting house -------------- next part -------------- An HTML attachment was scrubbed... URL: From curzio.basso at unibas.ch Thu Jun 24 03:15:00 2004 From: curzio.basso at unibas.ch (Curzio Basso) Date: Thu Jun 24 03:15:00 2004 Subject: [Numpy-discussion] type of matrixmultiply result Message-ID: <40DAA976.2010702@unibas.ch> Hi. I noticed that when multiplying two matrices of type Float32, the result is Float64: ----------------------------------------- In [103]: a=NA.ones((2,2), NA.Float32) In [104]: b=NA.ones((2,2), NA.Float32) In [105]: c=NA.matrixmultiply(a,b) In [106]: c.type() Out[106]: Float64 ----------------------------------------- Since the matrix I'm going to multiply in practice are quite big, I'd like to do the operation in Float32. Otherwise this is what I get: Traceback (most recent call last): File "/home/basso/work/python/port/apps/pca-heads.py", line 141, in ? pc = NA.array(NA.matrixmultiply(cent, c), NA.Float32) File "/home/basso/usr//lib/python/numarray/numarraycore.py", line 1150, in dot return ufunc.innerproduct(array1, _gen.swapaxes(array2, -1, -2)) File "/home/basso/usr//lib/python/numarray/ufunc.py", line 2047, in innerproduct r = a.__class__(shape=adots+bdots, type=rtype) MemoryError Any suggestion (apart from doing the operation one column at a time)? thanks From verveer at embl-heidelberg.de Thu Jun 24 05:34:07 2004 From: verveer at embl-heidelberg.de (Peter Verveer) Date: Thu Jun 24 05:34:07 2004 Subject: [Numpy-discussion] =?ISO-8859-1?Q?Re:_median_filtering_question_--_masked_arrays_?= =?ISO-8859-1?Q?=A0?= In-Reply-To: References: Message-ID: > Does anyone have a hint for applying a median filter to a masked 2D > array in numarray? > > I"ve been using the kludge of extracting a normal array with the > masked data replaced by the overall median of the array and median > filtering that (with the appropriate nd_image method). But this is a > crude approximation and I"d like to do better -- preferably without > coding my own median filter in C. Masks are not supported by the filters in nd_image. It could be done, but that would be a considerable effort for a rather exotic feature, so that won't happen soon. In the near future I will however add support for generic filters where you can provide your own filter function to be executed at each point. That may make it easy enough to roll your own function. Cheers, Peter From verveer at embl-heidelberg.de Thu Jun 24 05:39:00 2004 From: verveer at embl-heidelberg.de (Peter Verveer) Date: Thu Jun 24 05:39:00 2004 Subject: [Numpy-discussion] compress function Message-ID: <5900E940-C5DB-11D8-93CA-000A95C92C8E@embl-heidelberg.de> The documentation of the compress function states that the condition must be equal to the given axis of the array that is compressed. e.g.: >>> a = array([[1,2],[3,4]]) >>> print compress([1,0], a, axis = 1) [[1] [3]] However, this also works fine: >>> print compress([[1,0],[0,1]], a) [1, 4] which is great (I need that) but not documented. Is that behaviour intended? If so it maybe should be documented. Cheers, Peter From jmiller at stsci.edu Thu Jun 24 06:39:08 2004 From: jmiller at stsci.edu (Todd Miller) Date: Thu Jun 24 06:39:08 2004 Subject: [Numpy-discussion] type of matrixmultiply result In-Reply-To: <40DAA976.2010702@unibas.ch> References: <40DAA976.2010702@unibas.ch> Message-ID: <1088084311.26016.52.camel@halloween.stsci.edu> On Thu, 2004-06-24 at 06:14, Curzio Basso wrote: > Hi. > > I noticed that when multiplying two matrices of type Float32, the result > is Float64: > > ----------------------------------------- > In [103]: a=NA.ones((2,2), NA.Float32) > > In [104]: b=NA.ones((2,2), NA.Float32) > > In [105]: c=NA.matrixmultiply(a,b) > > In [106]: c.type() > Out[106]: Float64 > ----------------------------------------- > > Since the matrix I'm going to multiply in practice are quite big, I'd > like to do the operation in Float32. Otherwise this is what I get: > > Traceback (most recent call last): > File "/home/basso/work/python/port/apps/pca-heads.py", line 141, in ? > pc = NA.array(NA.matrixmultiply(cent, c), NA.Float32) > File "/home/basso/usr//lib/python/numarray/numarraycore.py", line > 1150, in dot return ufunc.innerproduct(array1, _gen.swapaxes(array2, > -1, -2)) > File "/home/basso/usr//lib/python/numarray/ufunc.py", line 2047, in > innerproduct > r = a.__class__(shape=adots+bdots, type=rtype) > MemoryError > > Any suggestion (apart from doing the operation one column at a time)? > I modified dot() and innerproduct() this morning to return Float32 and Complex32 for like inputs. This is in CVS now. numarray-1.0 is dragging out, but will nevertheless be released relatively soon. I'm curious about what your array dimensions are. When I implemented matrixmuliply for numarray, I was operating under the assumption that no one would be multiplying truly huge arrays because it's an O(N^3) algorithm. Regards, Todd > thanks > > > ------------------------------------------------------- > This SF.Net email sponsored by Black Hat Briefings & Training. > Attend Black Hat Briefings & Training, Las Vegas July 24-29 - > digital self defense, top technical experts, no vendor pitches, > unmatched networking opportunities. Visit www.blackhat.com > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- Todd Miller From rlw at stsci.edu Thu Jun 24 07:32:10 2004 From: rlw at stsci.edu (Rick White) Date: Thu Jun 24 07:32:10 2004 Subject: [Numpy-discussion] type of matrixmultiply result In-Reply-To: <1088084311.26016.52.camel@halloween.stsci.edu> Message-ID: On 24 Jun 2004, Todd Miller wrote: > On Thu, 2004-06-24 at 06:14, Curzio Basso wrote: > > > I noticed that when multiplying two matrices of type Float32, the result > > is Float64: > > I modified dot() and innerproduct() this morning to return Float32 and > Complex32 for like inputs. I wonder whether it would be worth providing an option to accumulate the sums using Float64 and to convert to Float32 before storing them in an array. I suspect that one reason this returned Float64 is that it is very easy to run into precision/roundoff problems in single-precision matrix multiplies. You could avoid that by using doubles for the sum while still returning the result as a single. Rick From perry at stsci.edu Thu Jun 24 08:09:11 2004 From: perry at stsci.edu (Perry Greenfield) Date: Thu Jun 24 08:09:11 2004 Subject: [Numpy-discussion] type of matrixmultiply result In-Reply-To: Message-ID: Rick White wrote: > On 24 Jun 2004, Todd Miller wrote: > > > On Thu, 2004-06-24 at 06:14, Curzio Basso wrote: > > > > > I noticed that when multiplying two matrices of type Float32, > the result > > > is Float64: > > > > I modified dot() and innerproduct() this morning to return Float32 and > > Complex32 for like inputs. > > I wonder whether it would be worth providing an option to accumulate > the sums using Float64 and to convert to Float32 before storing them in > an array. I suspect that one reason this returned Float64 is that it > is very easy to run into precision/roundoff problems in > single-precision matrix multiplies. You could avoid that by using > doubles for the sum while still returning the result as a single. > Rick > I definitely agree. I'm pretty certain the reason it was done with double precision floats is the sensitivity to roundoff issues with matrix operations. I think Rick is right though that only intermediate calculations need to be done in double precision and that doesn't require the whole output array to be kept that way. Perry From jmiller at stsci.edu Thu Jun 24 08:32:06 2004 From: jmiller at stsci.edu (Todd Miller) Date: Thu Jun 24 08:32:06 2004 Subject: [Numpy-discussion] type of matrixmultiply result In-Reply-To: References: Message-ID: <1088091080.26016.131.camel@halloween.stsci.edu> On Thu, 2004-06-24 at 11:08, Perry Greenfield wrote: > Rick White wrote: > > > On 24 Jun 2004, Todd Miller wrote: > > > > > On Thu, 2004-06-24 at 06:14, Curzio Basso wrote: > > > > > > > I noticed that when multiplying two matrices of type Float32, > > the result > > > > is Float64: > > > > > > I modified dot() and innerproduct() this morning to return Float32 and > > > Complex32 for like inputs. > > > > I wonder whether it would be worth providing an option to accumulate > > the sums using Float64 and to convert to Float32 before storing them in > > an array. I suspect that one reason this returned Float64 is that it > > is very easy to run into precision/roundoff problems in > > single-precision matrix multiplies. You could avoid that by using > > doubles for the sum while still returning the result as a single. > > Rick > > > I definitely agree. I'm pretty certain the reason it was done > with double precision floats is the sensitivity to roundoff > issues with matrix operations. I think Rick is right though that > only intermediate calculations need to be done in double precision > and that doesn't require the whole output array to be kept that way. > > Perry OK. I implemented intermediate sums using Float64 and Complex64 but single precision inputs will still result in single precision outputs. Todd From jmiller at stsci.edu Thu Jun 24 08:34:08 2004 From: jmiller at stsci.edu (Todd Miller) Date: Thu Jun 24 08:34:08 2004 Subject: [Numpy-discussion] type of matrixmultiply result In-Reply-To: References: Message-ID: <1088091221.26016.136.camel@halloween.stsci.edu> On Thu, 2004-06-24 at 10:30, Rick White wrote: > On 24 Jun 2004, Todd Miller wrote: > > > On Thu, 2004-06-24 at 06:14, Curzio Basso wrote: > > > > > I noticed that when multiplying two matrices of type Float32, the result > > > is Float64: > > > > I modified dot() and innerproduct() this morning to return Float32 and > > Complex32 for like inputs. > > I wonder whether it would be worth providing an option to accumulate > the sums using Float64 and to convert to Float32 before storing them in > an array. I suspect that one reason this returned Float64 is that it > is very easy to run into precision/roundoff problems in > single-precision matrix multiplies. You could avoid that by using > doubles for the sum while still returning the result as a single. > Rick OK. I implemented intermediate sums using Float64 and Complex64 but single precision inputs will still result in single precision outputs. Todd From tim.hochberg at cox.net Thu Jun 24 09:42:13 2004 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu Jun 24 09:42:13 2004 Subject: [Numpy-discussion] Compiling on windows? Message-ID: <40DB041B.1050201@cox.net> I've been using numarray 0.8 since 0.9 has some bugs that I can't easily work around. Since these have been fixed in CVS, I finally decided to try to download and compile a copy of numarray from CVS. Unfortunately, 'python setup.py build' bombs out almost imediately: creating build\temp.win32-2.3 creating build\temp.win32-2.3\Release creating build\temp.win32-2.3\Release\Src C:\Program Files\Microsoft Visual Studio\VC98\BIN\cl.exe /c /nologo /Ox /MD /W3 /GX /DNDEBUG -IInclude/numarray -IC:\python23\include -IC:\python23\PC /TcSrc\_convmodule.c /Fobuild\temp.win32-2.3\Release\Src\_convmodule.obj_convmodule.c Src\_convmodule.c(683) : warning C4244: '=' : conversion from 'int ' to 'float ', possible loss of data .... Src\_convmodule.c(2310) : error C2520: conversion from unsigned __int64 to double not implemented, use signed __int64 error: command '"C:\Program Files\Microsoft Visual Studio\VC98\BIN\cl.exe"' failed with exit status 2 This on Windows XP, Python 2.3.2, VC++ 6.0. Any hints as to what I could to make this work? I downloaded the source for version 0.9 and tried compiling it but got the same set of errors. Alternatively, is there going to be a 0.9.1 or 0.10 release soon? Thanks, -tim From jmiller at stsci.edu Thu Jun 24 09:52:49 2004 From: jmiller at stsci.edu (Todd Miller) Date: Thu Jun 24 09:52:49 2004 Subject: [Numpy-discussion] Compiling on windows? In-Reply-To: <40DB041B.1050201@cox.net> References: <40DB041B.1050201@cox.net> Message-ID: <1088095886.26016.143.camel@halloween.stsci.edu> On Thu, 2004-06-24 at 12:40, Tim Hochberg wrote: > I've been using numarray 0.8 since 0.9 has some bugs that I can't > easily work around. Since these have been fixed in CVS, I finally > decided to try to download and compile a copy of numarray from CVS. > Unfortunately, 'python setup.py build' bombs out almost imediately: > > creating build\temp.win32-2.3 > creating build\temp.win32-2.3\Release > creating build\temp.win32-2.3\Release\Src > C:\Program Files\Microsoft Visual Studio\VC98\BIN\cl.exe /c /nologo /Ox > /MD /W3 /GX /DNDEBUG -IInclude/numarray -IC:\python23\include > -IC:\python23\PC /TcSrc\_convmodule.c > /Fobuild\temp.win32-2.3\Release\Src\_convmodule.obj_convmodule.c > Src\_convmodule.c(683) : warning C4244: '=' : conversion from 'int ' to > 'float ', possible loss of data > > .... > > Src\_convmodule.c(2310) : error C2520: conversion from unsigned __int64 > to double not implemented, use signed __int64 > error: command '"C:\Program Files\Microsoft Visual > Studio\VC98\BIN\cl.exe"' failed with exit status 2 > > > This on Windows XP, Python 2.3.2, VC++ 6.0. Any hints as to what I could > to make this work? Try: python setup.py install --gencode Then try: python setup.py build ... if you must. :-) > I downloaded the source for version 0.9 and tried > compiling it but got the same set of errors. > > Alternatively, is there going to be a 0.9.1 or 0.10 release soon? numarray-1.0 should be out soon. Regards, Todd From haase at msg.ucsf.edu Thu Jun 24 10:07:10 2004 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Thu Jun 24 10:07:10 2004 Subject: [Numpy-discussion] comparing string array with None raises ValueError Message-ID: <200406241006.12875.haase@msg.ucsf.edu> Hi, I'm not sure if this is fixed already in CVS but here it goes: I'm working with record arrays, and trying to access a field with type '10a80' - that is, an array of 10 80 char 'strings' : >>> q.Mrc.hdr = q.Mrc.hdrArray[0].field >>> q.Mrc.hdr('title') != None Traceback (most recent call last): File "", line 1, in ? File "/jws30/haase/PrLin/numarray/strings.py", line 431, in __ne__ return self.StrCmp(other).__ne__(0) File "/jws30/haase/PrLin/numarray/strings.py", line 385, in StrCmp b = asarray(b0, kind=self.__class__) File "/jws30/haase/PrLin/numarray/strings.py", line 1024, in asarray return array(buffer, itemsize, shape, byteoffset, bytestride, kind) File "/jws30/haase/PrLin/numarray/strings.py", line 994, in array byteoffset=byteoffset, bytestride=bytestride) File "/jws30/haase/PrLin/numarray/strings.py", line 84, in __init__ raise ValueError("Must define both shape & itemsize if buffer is None") ValueError: Must define both shape & itemsize if buffer is None >>> >>> print q.Mrc.hdr('title') ['tit1e seb says hi' '' '' '' '' '' '' '' '' ''] >>> Thanks for numarray ;-) Sebastian Haase From jmiller at stsci.edu Thu Jun 24 10:32:03 2004 From: jmiller at stsci.edu (Todd Miller) Date: Thu Jun 24 10:32:03 2004 Subject: [Numpy-discussion] comparing string array with None raises ValueError In-Reply-To: <200406241006.12875.haase@msg.ucsf.edu> References: <200406241006.12875.haase@msg.ucsf.edu> Message-ID: <1088098275.26016.160.camel@halloween.stsci.edu> On Thu, 2004-06-24 at 13:06, Sebastian Haase wrote: > Hi, > I'm not sure if this is fixed already in CVS but here it goes: > I'm working with record arrays, and trying to access a field with > type '10a80' - that is, an array of 10 80 char 'strings' : > >>> q.Mrc.hdr = q.Mrc.hdrArray[0].field > >>> q.Mrc.hdr('title') != None Shouldn't this be: q.Mrc.hdr('title') != "" Regards, Todd From haase at msg.ucsf.edu Thu Jun 24 10:42:20 2004 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Thu Jun 24 10:42:20 2004 Subject: [Numpy-discussion] comparing string array with None raises ValueError In-Reply-To: <1088098275.26016.160.camel@halloween.stsci.edu> References: <200406241006.12875.haase@msg.ucsf.edu> <1088098275.26016.160.camel@halloween.stsci.edu> Message-ID: <200406241041.57810.haase@msg.ucsf.edu> On Thursday 24 June 2004 10:31 am, Todd Miller wrote: > On Thu, 2004-06-24 at 13:06, Sebastian Haase wrote: > > Hi, > > I'm not sure if this is fixed already in CVS but here it goes: > > I'm working with record arrays, and trying to access a field with > > > > type '10a80' - that is, an array of 10 80 char 'strings' : > > >>> q.Mrc.hdr = q.Mrc.hdrArray[0].field > > >>> q.Mrc.hdr('title') != None > > Shouldn't this be: > > q.Mrc.hdr('title') != "" No, I understand that this makes more sense, but I have some "display hook"-code that compares everything with None... In general it must be OK to compare anything with None, right ? (BTW, I get the same error with == and !=) Regards, Sebastian Haase From haase at msg.ucsf.edu Thu Jun 24 10:47:01 2004 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Thu Jun 24 10:47:01 2004 Subject: [Numpy-discussion] comparing string array with None raises ValueError In-Reply-To: <1088098275.26016.160.camel@halloween.stsci.edu> References: <200406241006.12875.haase@msg.ucsf.edu> <1088098275.26016.160.camel@halloween.stsci.edu> Message-ID: <200406241046.13941.haase@msg.ucsf.edu> On Thursday 24 June 2004 10:31 am, Todd Miller wrote: > On Thu, 2004-06-24 at 13:06, Sebastian Haase wrote: > > Hi, > > I'm not sure if this is fixed already in CVS but here it goes: > > I'm working with record arrays, and trying to access a field with > > > > type '10a80' - that is, an array of 10 80 char 'strings' : > > >>> q.Mrc.hdr = q.Mrc.hdrArray[0].field > > >>> q.Mrc.hdr('title') != None > > Shouldn't this be: > > q.Mrc.hdr('title') != "" (in my first reply, I forgot to point out that q.Mrc.hdr('title') is an ARRAY of strings ! ) No, I understand that this makes more sense, but I have some "display hook"-code that compares everything with None... In general it must be OK to compare anything with None, right ? (BTW, I get the same error with == and !=) Regards, Sebastian Haase From jmiller at stsci.edu Thu Jun 24 11:18:15 2004 From: jmiller at stsci.edu (Todd Miller) Date: Thu Jun 24 11:18:15 2004 Subject: [Numpy-discussion] ObjectArray paramter order poll Message-ID: <1088101018.26016.280.camel@halloween.stsci.edu> A while back Colin Williams suggested that we change the parameter order of numarray.objects.ObjectArray from: def __init__(self, shape=None, objects=None, rank=None) to: def __init__(self, objects=None, shape=None, rank=None) The new order emphasizes building object arrays from existing sequences, while the old order emphasizes building empty (full of None) object arrays of a specific shape. Since array() and fromlist() already exist to build from sequences, I thought it was fine to keep the order as is, plus I hate breaking s/w interfaces unless it was my idea. :-) Opinions please? Regards, Todd From jmiller at stsci.edu Thu Jun 24 11:47:02 2004 From: jmiller at stsci.edu (Todd Miller) Date: Thu Jun 24 11:47:02 2004 Subject: [Numpy-discussion] comparing string array with None raises ValueError In-Reply-To: <200406241046.13941.haase@msg.ucsf.edu> References: <200406241006.12875.haase@msg.ucsf.edu> <1088098275.26016.160.camel@halloween.stsci.edu> <200406241046.13941.haase@msg.ucsf.edu> Message-ID: <1088102785.26016.297.camel@halloween.stsci.edu> On Thu, 2004-06-24 at 13:46, Sebastian Haase wrote: > On Thursday 24 June 2004 10:31 am, Todd Miller wrote: > > On Thu, 2004-06-24 at 13:06, Sebastian Haase wrote: > > > Hi, > > > I'm not sure if this is fixed already in CVS but here it goes: > > > I'm working with record arrays, and trying to access a field with > > > > > > type '10a80' - that is, an array of 10 80 char 'strings' : > > > >>> q.Mrc.hdr = q.Mrc.hdrArray[0].field > > > >>> q.Mrc.hdr('title') != None > > > > Shouldn't this be: > > > > q.Mrc.hdr('title') != "" > > (in my first reply, I forgot to point out that q.Mrc.hdr('title') is an ARRAY > of strings ! ) > > No, I understand that this makes more sense, but I have some "display > hook"-code that compares everything with None... > In general it must be OK to compare anything with None, right ? > (BTW, I get the same error with == and !=) OK, I see your point. I talked it over with Perry and he made a reasonable case for allowing comparisons with None (or any object). Perry argued that since None is a common default parameter value, it might simplify code to not have to add logic to handle that case. If no one objects, I'll change numarray.strings so that comparison of a string array with any object not convertible to a string array results in an array of False values. Any objections? Regards, Todd From rlw at stsci.edu Thu Jun 24 11:54:05 2004 From: rlw at stsci.edu (Rick White) Date: Thu Jun 24 11:54:05 2004 Subject: [Numpy-discussion] comparing string array with None raises ValueError In-Reply-To: <1088102785.26016.297.camel@halloween.stsci.edu> Message-ID: On 24 Jun 2004, Todd Miller wrote: > OK, I see your point. I talked it over with Perry and he made a > reasonable case for allowing comparisons with None (or any object). > Perry argued that since None is a common default parameter value, it > might simplify code to not have to add logic to handle that case. > > If no one objects, I'll change numarray.strings so that comparison of a > string array with any object not convertible to a string array results > in an array of False values. > > Any objections? Hmm, before you do that you might look at this: >>> import numarray >>> b = numarray.zeros(10) >>> b==0 array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1], type=Bool) >>> b==None 0 So comparing numeric arrays to None returns a scalar false value since the variable b is not None. I figure that the behavior of comparisons to None for string arrays and numeric arrays ought to be consistent. I don't know which is preferred... Rick From haase at msg.ucsf.edu Thu Jun 24 12:14:31 2004 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Thu Jun 24 12:14:31 2004 Subject: [Numpy-discussion] comparing string array with None raises ValueError In-Reply-To: <1088102785.26016.297.camel@halloween.stsci.edu> References: <200406241006.12875.haase@msg.ucsf.edu> <200406241046.13941.haase@msg.ucsf.edu> <1088102785.26016.297.camel@halloween.stsci.edu> Message-ID: <200406241213.00537.haase@msg.ucsf.edu> On Thursday 24 June 2004 11:46 am, Todd Miller wrote: > On Thu, 2004-06-24 at 13:46, Sebastian Haase wrote: > > On Thursday 24 June 2004 10:31 am, Todd Miller wrote: > > > On Thu, 2004-06-24 at 13:06, Sebastian Haase wrote: > > > > Hi, > > > > I'm not sure if this is fixed already in CVS but here it goes: > > > > I'm working with record arrays, and trying to access a field with > > > > > > > > type '10a80' - that is, an array of 10 80 char 'strings' : > > > > >>> q.Mrc.hdr = q.Mrc.hdrArray[0].field > > > > >>> q.Mrc.hdr('title') != None > > > > > > Shouldn't this be: > > > > > > q.Mrc.hdr('title') != "" > > > > (in my first reply, I forgot to point out that q.Mrc.hdr('title') is an > > ARRAY of strings ! ) > > > > No, I understand that this makes more sense, but I have some "display > > hook"-code that compares everything with None... > > In general it must be OK to compare anything with None, right ? > > (BTW, I get the same error with == and !=) > > OK, I see your point. I talked it over with Perry and he made a > reasonable case for allowing comparisons with None (or any object). > Perry argued that since None is a common default parameter value, it > might simplify code to not have to add logic to handle that case. > > If no one objects, I'll change numarray.strings so that comparison of a > string array with any object not convertible to a string array results > in an array of False values. > > Any objections? Just my feeling is, that a single None would do it ... Regards, Sebastian From jmiller at stsci.edu Thu Jun 24 12:15:02 2004 From: jmiller at stsci.edu (Todd Miller) Date: Thu Jun 24 12:15:02 2004 Subject: [Numpy-discussion] comparing string array with None raises ValueError, more opinions please In-Reply-To: References: Message-ID: <1088104465.26016.310.camel@halloween.stsci.edu> On Thu, 2004-06-24 at 14:53, Rick White wrote: > On 24 Jun 2004, Todd Miller wrote: > > > OK, I see your point. I talked it over with Perry and he made a > > reasonable case for allowing comparisons with None (or any object). > > Perry argued that since None is a common default parameter value, it > > might simplify code to not have to add logic to handle that case. > > > > If no one objects, I'll change numarray.strings so that comparison of a > > string array with any object not convertible to a string array results > > in an array of False values. > > > > Any objections? > > Hmm, before you do that you might look at this: > > >>> import numarray > >>> b = numarray.zeros(10) > >>> b==0 > array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1], type=Bool) > >>> b==None > 0 > > So comparing numeric arrays to None returns a scalar false value since > the variable b is not None. Oh yeah... I forgot about that. Numeric comparisons have a special case for None and blow up for most other arbitrary objects. So there's also: >>> a= numarray.arange(100) >>> a == None 0 >>> class foo: .. pass .. >>> a == foo Traceback (most recent call last): .. TypeError: UFunc arguments must be numarray, scalars or numeric sequences > I figure that the behavior of comparisons to None for string arrays and > numeric arrays ought to be consistent. I don't know which is > preferred... So the two choices now on the table are: 1. Change string equality comparisons to return an array of false or true for objects which don't convert to strings. Change Numeric equality comparisons in a similar fashion. 2. Special case string equality comparisons for None, as is done with numerical comparisons. Raise a type error for objects (other than None) which don't convert to string arrays. I think I like 2. Other opinions on 1 & 2? Other choices? Regards, Todd From cookedm at physics.mcmaster.ca Thu Jun 24 12:27:07 2004 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Thu Jun 24 12:27:07 2004 Subject: [Numpy-discussion] comparing string array with None raises ValueError In-Reply-To: <200406241046.13941.haase@msg.ucsf.edu> References: <200406241006.12875.haase@msg.ucsf.edu> <1088098275.26016.160.camel@halloween.stsci.edu> <200406241046.13941.haase@msg.ucsf.edu> Message-ID: <200406241526.36796.cookedm@physics.mcmaster.ca> On June 24, 2004 01:46 pm, Sebastian Haase wrote: > In general it must be OK to compare anything with None, right ? > (BTW, I get the same error with == and !=) No! Not in general! I learnt this back when Numeric implemented rich comparisions; suddenly, lots of my code broke. You don't actually want "is this object _equal_ (or not equal) to None", you want "is this object None", as None is a singleton. A contrived example: >>> class A: ... def __eq__(self, other): return True ... >>> a = A() So, this is bad: >>> a == None True This is good: >>> a is None False In general, if you're testing if something is None, use 'is', not '=='. >>>q.Mrc.hdr('title') is not None -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From perry at stsci.edu Thu Jun 24 13:03:05 2004 From: perry at stsci.edu (Perry Greenfield) Date: Thu Jun 24 13:03:05 2004 Subject: [Numpy-discussion] comparing string array with None raises ValueError, more opinions please In-Reply-To: <1088104465.26016.310.camel@halloween.stsci.edu> Message-ID: Todd Miller wrote: > So the two choices now on the table are: > > 1. Change string equality comparisons to return an array of false or > true for objects which don't convert to strings. Change Numeric > equality comparisons in a similar fashion. > > 2. Special case string equality comparisons for None, as is done with > numerical comparisons. Raise a type error for objects (other than None) > which don't convert to string arrays. > > I think I like 2. Other opinions on 1 & 2? Other choices? > > Regards, > Todd > It would seem consistency with numerical arrays is important so unless someone presents a compelling argument for changing that behavior for arrays, the choice appears clear. Perry [wishing he had spent a little more time thinking about it and trying to remember why None is special cased for comparisons] From jmiller at stsci.edu Thu Jun 24 13:11:10 2004 From: jmiller at stsci.edu (Todd Miller) Date: Thu Jun 24 13:11:10 2004 Subject: [Numpy-discussion] comparing string array with None raises ValueError In-Reply-To: <200406241526.36796.cookedm@physics.mcmaster.ca> References: <200406241006.12875.haase@msg.ucsf.edu> <1088098275.26016.160.camel@halloween.stsci.edu> <200406241046.13941.haase@msg.ucsf.edu> <200406241526.36796.cookedm@physics.mcmaster.ca> Message-ID: <1088107839.26016.327.camel@halloween.stsci.edu> On Thu, 2004-06-24 at 15:26, David M. Cooke wrote: > On June 24, 2004 01:46 pm, Sebastian Haase wrote: > > > In general it must be OK to compare anything with None, right ? > > (BTW, I get the same error with == and !=) > > No! Not in general! Well, this is a good point. I think the current numerical behavior was a hack I stuck in for people who might not be aware of "is". It's looking like a mistake now. > I learnt this back when Numeric implemented rich > comparisions; suddenly, lots of my code broke. You don't actually want "is > this object _equal_ (or not equal) to None", you want "is this object None", > as None is a singleton. However, given the context of the original question, Sebastian's code doesn't read like *that* kind of None comparison: > > > > type '10a80' - that is, an array of 10 80 char 'strings' : > > > > >>> q.Mrc.hdr = q.Mrc.hdrArray[0].field > > > > >>> q.Mrc.hdr('title') != None q.Mrc.hdr('title') is pretty clearly a character array, ergo, it's not None. What did you want it to do Sebastian? Regards, Todd From barrett at stsci.edu Thu Jun 24 13:22:14 2004 From: barrett at stsci.edu (Paul Barrett) Date: Thu Jun 24 13:22:14 2004 Subject: [Numpy-discussion] ObjectArray paramter order poll In-Reply-To: <1088101018.26016.280.camel@halloween.stsci.edu> References: <1088101018.26016.280.camel@halloween.stsci.edu> Message-ID: <40DB37CB.1000408@stsci.edu> Todd Miller wrote: > A while back Colin Williams suggested that we change the parameter order > of numarray.objects.ObjectArray from: > > def __init__(self, shape=None, objects=None, rank=None) > > to: > > def __init__(self, objects=None, shape=None, rank=None) > > The new order emphasizes building object arrays from existing sequences, > while the old order emphasizes building empty (full of None) object > arrays of a specific shape. Since array() and fromlist() already exist > to build from sequences, I thought it was fine to keep the order as is, > plus I hate breaking s/w interfaces unless it was my idea. :-) > > Opinions please? +1 (for the second option) -- Paul Barrett, PhD Space Telescope Science Institute Phone: 410-338-4475 ESS/Science Software Branch FAX: 410-338-4767 Baltimore, MD 21218 From haase at msg.ucsf.edu Thu Jun 24 13:39:06 2004 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Thu Jun 24 13:39:06 2004 Subject: [Numpy-discussion] comparing string array with None raises ValueError In-Reply-To: <1088107839.26016.327.camel@halloween.stsci.edu> References: <200406241006.12875.haase@msg.ucsf.edu> <200406241526.36796.cookedm@physics.mcmaster.ca> <1088107839.26016.327.camel@halloween.stsci.edu> Message-ID: <200406241338.29060.haase@msg.ucsf.edu> On Thursday 24 June 2004 01:10 pm, Todd Miller wrote: > On Thu, 2004-06-24 at 15:26, David M. Cooke wrote: > > On June 24, 2004 01:46 pm, Sebastian Haase wrote: > > > In general it must be OK to compare anything with None, right ? > > > (BTW, I get the same error with == and !=) > > > > No! Not in general! > > Well, this is a good point. I think the current numerical behavior was > a hack I stuck in for people who might not be aware of "is". It's > looking like a mistake now. > > > I learnt this back when Numeric implemented rich > > comparisions; suddenly, lots of my code broke. You don't actually want > > "is this object _equal_ (or not equal) to None", you want "is this object > > None", as None is a singleton. > > However, given the context of the original question, Sebastian's code > > doesn't read like *that* kind of None comparison: > > > > > type '10a80' - that is, an array of 10 80 char 'strings' : > > > > > >>> q.Mrc.hdr = q.Mrc.hdrArray[0].field > > > > > >>> q.Mrc.hdr('title') != None > > q.Mrc.hdr('title') is pretty clearly a character array, ergo, it's not > None. What did you want it to do Sebastian? q.Mrc.hdr('title') is rather an array of 10 'character array' .. that's different, isn't it ? In any case, I actually already changed my code to the check of 'is None' - which is probably what I wanted anyway. So, the only argument left, is that the original error/exception message I got was kind of "ugly" ;-) And I think/thought if 'something' cannot be _converted_ to a string array it can't _be_ '==' a string array. That's just why I expected the simple result to be 'False'. But now I understand that if comparing an array to a scalar (maybe 'None') you always have to decide if this actually compares "the whole array" with the scalar, or if the comparison get "put through" and done on each element "separately" ... which it seems is already decided at least in the numerical array case. Regards, Sebastian From jmiller at stsci.edu Thu Jun 24 14:28:04 2004 From: jmiller at stsci.edu (Todd Miller) Date: Thu Jun 24 14:28:04 2004 Subject: [Numpy-discussion] comparing string array with None raises ValueError In-Reply-To: <200406241338.29060.haase@msg.ucsf.edu> References: <200406241006.12875.haase@msg.ucsf.edu> <200406241526.36796.cookedm@physics.mcmaster.ca> <1088107839.26016.327.camel@halloween.stsci.edu> <200406241338.29060.haase@msg.ucsf.edu> Message-ID: <1088112441.26016.339.camel@halloween.stsci.edu> On Thu, 2004-06-24 at 16:38, Sebastian Haase wrote: > On Thursday 24 June 2004 01:10 pm, Todd Miller wrote: > > On Thu, 2004-06-24 at 15:26, David M. Cooke wrote: > > > On June 24, 2004 01:46 pm, Sebastian Haase wrote: > > > > In general it must be OK to compare anything with None, right ? > > > > (BTW, I get the same error with == and !=) > > > > > > No! Not in general! > > > > Well, this is a good point. I think the current numerical behavior was > > a hack I stuck in for people who might not be aware of "is". It's > > looking like a mistake now. > > > > > I learnt this back when Numeric implemented rich > > > comparisions; suddenly, lots of my code broke. You don't actually want > > > "is this object _equal_ (or not equal) to None", you want "is this object > > > None", as None is a singleton. > > > > However, given the context of the original question, Sebastian's code > > > > doesn't read like *that* kind of None comparison: > > > > > > type '10a80' - that is, an array of 10 80 char 'strings' : > > > > > > >>> q.Mrc.hdr = q.Mrc.hdrArray[0].field > > > > > > >>> q.Mrc.hdr('title') != None > > > > q.Mrc.hdr('title') is pretty clearly a character array, ergo, it's not > > None. What did you want it to do Sebastian? > > q.Mrc.hdr('title') is rather an array of 10 'character array' I thought it was a single nrows x 10 CharArray with 80 character elements. > .. that's > different, isn't it ? It would be. What does q.Mrc.hdr('title').info() say? > In any case, I actually already changed my code to the check of 'is None' - > which is probably what I wanted anyway. > So, the only argument left, is that the original error/exception message I got > was kind of "ugly" ;-) And I think/thought if 'something' cannot be > _converted_ to a string array it can't _be_ '==' a string array. That's just > why I expected the simple result to be 'False'. > But now I understand that if comparing an array to a scalar (maybe 'None') you > always have to decide if this actually compares "the whole array" with the > scalar, or if the comparison get "put through" and done on each element > "separately" ... > which it seems is already decided at least in the numerical array case. I'm not really sure that the numerical array None hack should be replicated. Barring an edict from Perry or Rick or an outcry from the community, I think I'm going to just leave this alone. Regards, Todd From cookedm at physics.mcmaster.ca Thu Jun 24 22:45:16 2004 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Thu Jun 24 22:45:16 2004 Subject: [Numpy-discussion] comparing string array with None raises ValueError In-Reply-To: <1088107839.26016.327.camel@halloween.stsci.edu> References: <200406241006.12875.haase@msg.ucsf.edu> <200406241526.36796.cookedm@physics.mcmaster.ca> <1088107839.26016.327.camel@halloween.stsci.edu> Message-ID: <200406241636.01426.cookedm@physics.mcmaster.ca> On June 24, 2004 04:10 pm, Todd Miller wrote: > On Thu, 2004-06-24 at 15:26, David M. Cooke wrote: > > On June 24, 2004 01:46 pm, Sebastian Haase wrote: > > > In general it must be OK to compare anything with None, right ? > > > (BTW, I get the same error with == and !=) > > > > No! Not in general! > > Well, this is a good point. I think the current numerical behavior was > a hack I stuck in for people who might not be aware of "is". It's > looking like a mistake now. I've fixed up all uses of == None and != None in my copy of the numarray CVS, but the anonymous CVS on SourceForge is timing out :-(, so I haven't made a patch yet. Basically, I ran 'grep -R "[!=]= *None" *' to find all uses of == and != None, and replaced them with is/is not None. That would remove possible bugs. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From falted at pytables.org Fri Jun 25 02:49:13 2004 From: falted at pytables.org (Francesc Alted) Date: Fri Jun 25 02:49:13 2004 Subject: [Numpy-discussion] ObjectArray paramter order poll In-Reply-To: <1088101018.26016.280.camel@halloween.stsci.edu> References: <1088101018.26016.280.camel@halloween.stsci.edu> Message-ID: <200406251148.28153.falted@pytables.org> A Dijous 24 Juny 2004 20:16, Todd Miller va escriure: > A while back Colin Williams suggested that we change the parameter order > of numarray.objects.ObjectArray from: > > def __init__(self, shape=None, objects=None, rank=None) > > to: > > def __init__(self, objects=None, shape=None, rank=None) > > The new order emphasizes building object arrays from existing sequences, > while the old order emphasizes building empty (full of None) object > arrays of a specific shape. Since array() and fromlist() already exist > to build from sequences, I thought it was fine to keep the order as is, > plus I hate breaking s/w interfaces unless it was my idea. :-) > > Opinions please? +1 for the second option -- Francesc Alted From jmiller at stsci.edu Fri Jun 25 05:49:12 2004 From: jmiller at stsci.edu (Todd Miller) Date: Fri Jun 25 05:49:12 2004 Subject: [Numpy-discussion] numarray.sum should raise exception for wrong axis argument - and slicing ... In-Reply-To: <200406231506.24453.haase@msg.ucsf.edu> References: <200406231506.24453.haase@msg.ucsf.edu> Message-ID: <1088167715.32195.18.camel@halloween.stsci.edu> On Wed, 2004-06-23 at 18:06, Sebastian Haase wrote: > Hi, > please take a look at this: > >>> na.sum( na.zeros((2,6)) ) > [0 0 0 0 0 0] > >>> na.sum( na.zeros((2,6)) , 0) > [0 0 0 0 0 0] > >>> na.sum( na.zeros((2,6)) , 1) > [0 0] > >>> na.sum( na.zeros((2,6)) , 2) > [0 0] > >>> na.sum( na.zeros((2,6)) , 3) > [0 0] > >>> na.sum( na.zeros((2,6)) , 4) > [0 0] > >>> na.sum( na.zeros((2,6)) , -1) > [0 0] > >>> na.sum( na.zeros((2,6)) , -2) > [0 0 0 0 0 0] > >>> na.sum( na.zeros((2,6)) , -3) > [0 0] > >>> na.sum( na.zeros((2,6)) , -4) > [0 0] > >>> > > I think here should be a ValueError exception thrown rather than defaulting to > the '-1'-axis. Comments ? This is a bug. Hopefully I'll get it today. > Also this applies to (all?) other functions that have an 'axis' argument. > And further I just found that putting "too many slicings" to an array also > gets silently ignored: > >>> b.shape > (7, 128, 128, 128) > >>> b[2,2,2,2,3] > Traceback (most recent call last): > File "", line 1, in ? > IndexError: too many indices. > > BUT: > >>> b[2:3 , 2:3 , 2:3 , 2:3 , 2:3 , 2:3] > [[[[ 0.]]]] > > > I find this very confusing !! Is there any good reason not to have the > "IndexError" exception in all cases ? This is also a bug. Fixed in CVS now. Thanks for the report! Regards, Todd From haase at msg.ucsf.edu Fri Jun 25 08:24:07 2004 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Fri Jun 25 08:24:07 2004 Subject: [Numpy-discussion] numarray.sum should raise exception for wrong axis argument - and slicing ... In-Reply-To: <1088167715.32195.18.camel@halloween.stsci.edu> References: <200406231506.24453.haase@msg.ucsf.edu> <1088167715.32195.18.camel@halloween.stsci.edu> Message-ID: <200406250823.16156.haase@msg.ucsf.edu> On Friday 25 June 2004 05:48 am, Todd Miller wrote: > On Wed, 2004-06-23 at 18:06, Sebastian Haase wrote: > > Hi, > > > > please take a look at this: > > >>> na.sum( na.zeros((2,6)) ) > > > > [0 0 0 0 0 0] > > > > >>> na.sum( na.zeros((2,6)) , 0) > > > > [0 0 0 0 0 0] > > > > >>> na.sum( na.zeros((2,6)) , 1) > > > > [0 0] > > > > >>> na.sum( na.zeros((2,6)) , 2) > > > > [0 0] > > > > >>> na.sum( na.zeros((2,6)) , 3) > > > > [0 0] > > > > >>> na.sum( na.zeros((2,6)) , 4) > > > > [0 0] > > > > >>> na.sum( na.zeros((2,6)) , -1) > > > > [0 0] > > > > >>> na.sum( na.zeros((2,6)) , -2) > > > > [0 0 0 0 0 0] > > > > >>> na.sum( na.zeros((2,6)) , -3) > > > > [0 0] > > > > >>> na.sum( na.zeros((2,6)) , -4) > > > > [0 0] > > > > > > I think here should be a ValueError exception thrown rather than > > defaulting to the '-1'-axis. Comments ? > > This is a bug. Hopefully I'll get it today. > > > Also this applies to (all?) other functions that have an 'axis' argument. > > And further I just found that putting "too many slicings" to an array > > also > > > > gets silently ignored: > > >>> b.shape > > > > (7, 128, 128, 128) > > > > >>> b[2,2,2,2,3] > > > > Traceback (most recent call last): > > File "", line 1, in ? > > IndexError: too many indices. > > > > BUT: > > >>> b[2:3 , 2:3 , 2:3 , 2:3 , 2:3 , 2:3] > > > > [[[[ 0.]]]] > > > > > > I find this very confusing !! Is there any good reason not to have the > > "IndexError" exception in all cases ? > > This is also a bug. Fixed in CVS now. > > Thanks for the report! > > Regards, > Todd Thanks so much - I got worried there for a moment ;-) Regards, Sebastian From jmiller at stsci.edu Fri Jun 25 08:48:12 2004 From: jmiller at stsci.edu (Todd Miller) Date: Fri Jun 25 08:48:12 2004 Subject: [Numpy-discussion] compress function In-Reply-To: <5900E940-C5DB-11D8-93CA-000A95C92C8E@embl-heidelberg.de> References: <5900E940-C5DB-11D8-93CA-000A95C92C8E@embl-heidelberg.de> Message-ID: <1088178469.32195.142.camel@halloween.stsci.edu> On Thu, 2004-06-24 at 08:38, Peter Verveer wrote: > The documentation of the compress function states that the condition > must be equal to the given axis of the array that is compressed. e.g.: > > >>> a = array([[1,2],[3,4]]) > >>> print compress([1,0], a, axis = 1) > [[1] > [3]] > > However, this also works fine: > > >>> print compress([[1,0],[0,1]], a) > [1, 4] > > which is great (I need that) but not documented. Is that behaviour > intended? It is most likely accidental, but great is great. > If so it maybe should be documented. I filed a bug report but I'm not sure when I'll be *able* to do it. Regards, Todd From haase at msg.ucsf.edu Fri Jun 25 09:50:08 2004 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Fri Jun 25 09:50:08 2004 Subject: [Numpy-discussion] numarray speed - PySequence_GetItem Message-ID: <200406250949.44897.haase@msg.ucsf.edu> Hi, The long story is that I'm looking for a good/fast graph plotting programs; so I found WxPyPlot (http://www.cyberus.ca/~g_will/wxPython/wxpyplot.html) It uses wxPython and plots 25000 data points (with lines + square markers) in under one second - using Numeric that is. [the slow line in WxPyPlot is: dc.DrawLines(self.scaled) where self.scaled is an array of shape (25000,2) and type Float64 ] The short story is that numarray takes maybe 10 times as long as Numeric and I tracked the problem down into the wxPython SWIG typemap where he does this: wxPoint* wxPoint_LIST_helper(PyObject* source, int *count) { bool isFast = PyList_Check(source) || PyTuple_Check(source); for (x=0; x<*count; x++) { // Get an item: try fast way first. if (isFast) { o = PySequence_Fast_GET_ITEM(source, x); } else { o = PySequence_GetItem(source, x); if (o == NULL) { goto error1; } } I'm not 100% sure that this is where the problem lies - is there a chance (or a known issue) that numarray does PySequence_GetItem() slower than Numeric ? I just ran this again using the python profiler and I get this w/ numarray: ncalls tottime percall cumtime percall filename:lineno(function) 1 1.140 1.140 1.320 1.320 gdi.py:554(DrawLines) 1 1.250 1.250 1.520 1.520 gdi.py:792(_DrawRectangleList) 50230 0.450 0.000 0.450 0.000 numarraycore.py:501(__del__) and this with Numeric: 1 0.080 0.080 0.080 0.080 gdi.py:554(DrawLines) 1 0.090 0.090 0.090 0.090 gdi.py:792(_DrawRectangleList) Thanks, Sebastian Haase From jdhunter at ace.bsd.uchicago.edu Fri Jun 25 14:37:11 2004 From: jdhunter at ace.bsd.uchicago.edu (John Hunter) Date: Fri Jun 25 14:37:11 2004 Subject: [Numpy-discussion] numarray speed - PySequence_GetItem In-Reply-To: <200406250949.44897.haase@msg.ucsf.edu> (Sebastian Haase's message of "Fri, 25 Jun 2004 09:49:44 -0700") References: <200406250949.44897.haase@msg.ucsf.edu> Message-ID: >>>>> "Sebastian" == Sebastian Haase writes: Sebastian> Hi, The long story is that I'm looking for a good/fast Sebastian> graph plotting programs; so I found WxPyPlot Sebastian> (http://www.cyberus.ca/~g_will/wxPython/wxpyplot.html) Sebastian> It uses wxPython and plots 25000 data points (with Sebastian> lines + square markers) in under one second - using Sebastian> Numeric that is. Not an answer to your question .... matplotlib has full numarray support (no need to rely on sequence API). You need to set NUMERIX='numarray' in setup.py before building it *and* set numerix : numarray in the matplotlib rc file. If you don't do both of these things, your numarray performance will suffer, sometimes dramatically. With this test script from matplotlib.matlab import * N = 25000 x = rand(N) y = rand(N) scatter(x,y, marker='s') #savefig('test') show() You can do a scatter plot of squares, on my machine in under a second using numarray (wxagg or agg backend). Some fairly recent changes to matplotlib have moved this drawing into extension code, with an approx 10x performance boost from older versions. The latest version on the sf site (0.54.2) however, does have these changes. To plot markers with lines, you would need plot(x,y, marker='-s') instead of scatter. This is considerably slower (approx 3s on my system), mainly because I haven't ported the new fast drawing of marker code to the line class. This is an easy fix, however, and will be added in short order. JDH From haase at msg.ucsf.edu Fri Jun 25 15:34:01 2004 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Fri Jun 25 15:34:01 2004 Subject: [off topic] Re: [Numpy-discussion] numarray speed - PySequence_GetItem In-Reply-To: References: <200406250949.44897.haase@msg.ucsf.edu> Message-ID: <200406251533.27116.haase@msg.ucsf.edu> Hi John, I wanted to try matplotlib a few days ago, but first I had some trouble compiling it (my debian still uses gcc 2-95, which doesn't understand some 'std' namespace/template stuff) - and then it compiled, but segfaulted. Maybe I didn't get "set NUMERIX" stuff right - how do I know that it actually built _and_ uses the wx-backend ? BTW, from the profiling/timing I did you can tell that wxPyPlot actually plots 25000 data points in 0.1 secs - so it's _really_ fast ... So it would be nice to get to the ground of this ... Thanks for the comment, Sebastian On Friday 25 June 2004 02:12 pm, John Hunter wrote: > >>>>> "Sebastian" == Sebastian Haase writes: > > Sebastian> Hi, The long story is that I'm looking for a good/fast > Sebastian> graph plotting programs; so I found WxPyPlot > Sebastian> (http://www.cyberus.ca/~g_will/wxPython/wxpyplot.html) > Sebastian> It uses wxPython and plots 25000 data points (with > Sebastian> lines + square markers) in under one second - using > Sebastian> Numeric that is. > > Not an answer to your question .... > > matplotlib has full numarray support (no need to rely on sequence > API). You need to set NUMERIX='numarray' in setup.py before building > it *and* set numerix : numarray in the matplotlib rc file. If you > don't do both of these things, your numarray performance will suffer, > sometimes dramatically. > > With this test script > > from matplotlib.matlab import * > N = 25000 > x = rand(N) > y = rand(N) > scatter(x,y, marker='s') > #savefig('test') > show() > > You can do a scatter plot of squares, on my machine in under a second > using numarray (wxagg or agg backend). Some fairly recent changes to > matplotlib have moved this drawing into extension code, with an approx > 10x performance boost from older versions. The latest version on the > sf site (0.54.2) however, does have these changes. > > To plot markers with lines, you would need > > plot(x,y, marker='-s') > > instead of scatter. This is considerably slower (approx 3s on my > system), mainly because I haven't ported the new fast drawing of > marker code to the line class. This is an easy fix, however, and will > be added in short order. > > JDH > > > > ------------------------------------------------------- > This SF.Net email sponsored by Black Hat Briefings & Training. > Attend Black Hat Briefings & Training, Las Vegas July 24-29 - > digital self defense, top technical experts, no vendor pitches, > unmatched networking opportunities. Visit www.blackhat.com > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From jmiller at stsci.edu Sat Jun 26 05:23:03 2004 From: jmiller at stsci.edu (Todd Miller) Date: Sat Jun 26 05:23:03 2004 Subject: [Numpy-discussion] numarray.sum should raise exception for wrong axis argument - and slicing ... In-Reply-To: <200406250823.16156.haase@msg.ucsf.edu> References: <200406231506.24453.haase@msg.ucsf.edu> <1088167715.32195.18.camel@halloween.stsci.edu> <200406250823.16156.haase@msg.ucsf.edu> Message-ID: <1088252513.3744.2.camel@localhost.localdomain> On Fri, 2004-06-25 at 11:23, Sebastian Haase wrote: > > > > > Also this applies to (all?) other functions that have an 'axis' argument. > > > And further I just found that putting "too many slicings" to an array > > > also > > > > > > gets silently ignored: > > > >>> b.shape > > > > > > (7, 128, 128, 128) > > > > > > >>> b[2,2,2,2,3] > > > > > > Traceback (most recent call last): > > > File "", line 1, in ? > > > IndexError: too many indices. > > > > > > BUT: > > > >>> b[2:3 , 2:3 , 2:3 , 2:3 , 2:3 , 2:3] > > > > > > [[[[ 0.]]]] > > > > > > > > > I find this very confusing !! Is there any good reason not to have the > > > "IndexError" exception in all cases ? > > > > This is also a bug. Fixed in CVS now. I'm afraid I spoke too soon here... this will be fixed but is not yet. Thanks again for taking the time to bring this to our attention. Regards, Todd From Scott.Daniels at Acm.Org Sat Jun 26 19:11:01 2004 From: Scott.Daniels at Acm.Org (Scott David Daniels) Date: Sat Jun 26 19:11:01 2004 Subject: [Numpy-discussion] Re: numarray equivalent to sign() function In-Reply-To: References: Message-ID: Mike Zingale wrote: > Is there a replacement? def sign(x): return min(max(-1, math.frexp(x)[0]*2), 1) - Scott David Daniels Scott.Daniels at Acm.Org From martin.wiechert at gmx.de Mon Jun 28 10:23:02 2004 From: martin.wiechert at gmx.de (Martin Wiechert) Date: Mon Jun 28 10:23:02 2004 Subject: [Numpy-discussion] Re: [SciPy-user] IRIX problems In-Reply-To: References: <200406281455.50955.martin.wiechert@gmx.de> Message-ID: <200406281922.23886.martin.wiechert@gmx.de> Hi Pearu, thanks for your help. The solution is to replace libcomplib.sgimath with the newer libscs, which has the missing symbol. Our sysadmin pointed me to that. Martin. On Monday 28 June 2004 17:39, Pearu Peterson wrote: > On Mon, 28 Jun 2004, Martin Wiechert wrote: > > Hi all, > > > > I'm trying to install scipy on our dunno-how-old IRIX server. > > I've used a lib called complib.sgimath instead of atlas/LAPACK ... > > Now I get the following error: > > > > import lapack_lite > > ImportError: 563534:python2.3: rld: Fatal Error: unresolvable symbol > > in /user/wiechert/lib/python2.3/site-packages/Numeric/lapack_lite.so: > > zgelsd_ > > (This is a Numeric installation problem, not scipy.) > > > lapack_lite is linked against following libs: > > ldd lapack_lite.so > > libcomplib.sgimath.so => /usr/lib32/libcomplib.sgimath.so > > libblas.so => /usr/lib32/libblas.so > > libftn.so => /usr/lib32/libftn.so > > libm.so => /usr/lib32/libm.so > > libc.so.1 => /usr/lib32/libc.so.1 > > > > What can I do? > > Try to find out which libraries in your system contain zgelsd_ and use it > when linking lapack_lite. When that's unsuccessful then I suggest building > LAPACK libraries yourself, see the relevant section in > > http://www.scipy.org/documentation/buildatlas4scipy.txt > > HTH, > Pearu > > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.net > http://www.scipy.net/mailman/listinfo/scipy-user From Chris.Barker at noaa.gov Mon Jun 28 11:02:05 2004 From: Chris.Barker at noaa.gov (Chris Barker) Date: Mon Jun 28 11:02:05 2004 Subject: [off topic] Re: [Numpy-discussion] numarray speed - PySequence_GetItem In-Reply-To: <200406251533.27116.haase@msg.ucsf.edu> References: <200406250949.44897.haase@msg.ucsf.edu> <200406251533.27116.haase@msg.ucsf.edu> Message-ID: <40E05C8D.6050106@noaa.gov> Sebastian Haase wrote: > BTW, from the profiling/timing I did you can tell that wxPyPlot actually plots > 25000 data points in 0.1 secs - so it's _really_ fast ... Actually, it's probably not that fast, if you are timing on Linux/wxGTK/X-Windows. X is asyncronous, so what you are timing is how long it takes your program to tell X what to draw, but it may take longer than that to actually draw it. However, what you are timing is all the stuff that is effected by numarray/Numeric. I worked on part of the wxPython DC.DrawXXXList stuff, and I really wanted a Numeric native version, but Robin really didn't want an additional dependency. We discussed on this list a while back whether you could compile against Numeric, but let people run without it, and have it all work unless Someone actually used it. What makes that tricky is that the functions that test whether a PyObject is a Numeric array are in Numeric... but it could probably be done if you tried hard enough (maybe include just that function in wxPython...) The Same applies for numarray support. Anyway, as it stands, wxPython DC methods are faster with Lists or Tuples of values than Numeric or Numarray arrays. You might try converting to a list with numarray.tolist() before making the DC call. Another option is to write a few specialized DC functions that take numarray arrays to draw, but are not included in wxPython. I think you'd get it as fast as possible that way. I intend to do that some day. If you want to get it started, I'll help. You could probably get a particularly nice improvement in drawing a lot of rectangles, as looping through a N X 4 array of coords directly would be much, much faster that using the Sequence API on the whole thing, and on each item, and checking at every step what kind of object everything is. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From jmiller at stsci.edu Mon Jun 28 12:04:01 2004 From: jmiller at stsci.edu (Todd Miller) Date: Mon Jun 28 12:04:01 2004 Subject: [off topic] Re: [Numpy-discussion] numarray speed - PySequence_GetItem In-Reply-To: <40E05C8D.6050106@noaa.gov> References: <200406250949.44897.haase@msg.ucsf.edu> <200406251533.27116.haase@msg.ucsf.edu> <40E05C8D.6050106@noaa.gov> Message-ID: <1088449382.3744.138.camel@localhost.localdomain> On Mon, 2004-06-28 at 13:59, Chris Barker wrote: > Sebastian Haase wrote: > > BTW, from the profiling/timing I did you can tell that wxPyPlot actually plots > > 25000 data points in 0.1 secs - so it's _really_ fast ... > > Actually, it's probably not that fast, if you are timing on > Linux/wxGTK/X-Windows. X is asyncronous, so what you are timing is how > long it takes your program to tell X what to draw, but it may take > longer than that to actually draw it. However, what you are timing is > all the stuff that is effected by numarray/Numeric. > > I worked on part of the wxPython DC.DrawXXXList stuff, and I really > wanted a Numeric native version, but Robin really didn't want an > additional dependency. We discussed on this list a while back whether > you could compile against Numeric, but let people run without it, and > have it all work unless Someone actually used it. What makes that tricky > is that the functions that test whether a PyObject is a Numeric array > are in Numeric... but it could probably be done if you tried hard enough > (maybe include just that function in wxPython...) numarray-1.0 has two macros for dealing with this: PyArray_Present() and PyArray_isArray(obj). The former (safely) determines that numarray is installed, while the latter determines that numarray is installed and that obj is a NumArray. Both macros serve to guard sections of code which make more extensive use of the numarray C-API to keep them from segfaulting when numarray is not installed. I think this would be easy to do for Numeric as well. One problem is that compiling a "numarray improved" extension requires some portion of the numarray headers. I refactored the numarray includes so that a relatively simple set of 3 files can be used to support the Numeric compatible interface (for numarray). These could either be included in core Python (with a successful PEP) or included in interested packages. This approach adds a small source code burden somewhere, but eliminates the requirement for users to have numarray installed either to run or compile from source. I'll send out the draft PEP later today. Regards, Todd From jmiller at stsci.edu Mon Jun 28 12:42:02 2004 From: jmiller at stsci.edu (Todd Miller) Date: Mon Jun 28 12:42:02 2004 Subject: [Numpy-discussion] Numarray header PEP Message-ID: <1088451653.3744.200.camel@localhost.localdomain> Perry and I have written a draft PEP for the inclusion of some of the numarray headers within the Python distribution. The headers enable extension writers to provide better support for numarray without adding any additional dependencies (from a user's perspective) to their packages. For users who have numarray, an "improved extension" could provide better performance, and for those without numarray, the extension could work normally using the Sequence protocol. The PEP is attached. It is formatted using the docutils package which can be used to generate HTML or PDF. Comments and corrections would be appreciated. Regards, Todd -------------- next part -------------- PEP: XXX Title: numerical array headers Version: $Revision: 1.3 $ Last-Modified: $Date: 2002/08/30 04:11:20 $ Author: Todd Miller , Perry Greenfield Discussions-To: numpy-discussion at lists.sf.net Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 02-Jun-2004 Python-Version: 2.4 Post-History: 30-Aug-2002 Abstract ======== We propose the inclusion of three numarray header files within the CPython distribution to facilitate use of numarray array objects as an optional data data format for 3rd party modules. The PEP illustrates a simple technique by which a 3rd party extension may support numarrays as input or output values if numarray is installed, and yet the 3rd party extension does not require numarray to be installed to be built. Nothing needs to be changed in the setup.py or makefile for installing with or without numarray, and a subsequent installation of numarray will allow its use without rebuilding the 3rd party extension. Specification ============= This PEP applies only to the CPython platform and only to numarray. Analogous PEPs could be written for Jython and Python.NET and Numeric, but what is discussed here is a speed optimization that is tightly coupled to CPython and numarray. Three header files to support the numarray C-API should be included in the CPython distribution within a numarray subdirectory of the Python include directory: * numarray/arraybase.h * numarray/libnumeric.h * numarray/arrayobject.h The files are shown prefixed with "numarray" to leave the door open for doing similar PEPs with other packages, such as Numeric. If a plethora of such header contributions is anticipated, a further refinement would be to locate the headers under something like "third_party/numarray". In order to provide enhanced performance for array objects, an extension writer would start by including the numarray C-API in addition to any other Python headers: :: #include "numarray/arrayobject.h" Not shown in this PEP are the API calls which operate on numarrays. These are documented in the numarray manual. What is shown here are two calls which are guaranteed to be safe even when numarray is not installed: * PyArray_Present() * PyArray_isArray() In an extension function that wants to access the numarray API, a test needs to be performed to determine if the API functions are safely callable: :: PyObject * some_array_returning_function(PyObject *m, PyObject *args) { int param; PyObject *result; if (!PyArg_ParseTuple(args, "i", ¶m)) return NULL; if (PyArray_Present()) { result = numarray_returning_function(param); } else { result = list_returning_function(param); } return result; } Within **numarray_returning_function**, a subset of the numarray C-API (the Numeric compatible API) is available for use so it is possible to create and return numarrays. Within **list_returning_function**, only the standard Python C-API can be used because numarray is assumed to be unavailable in that particular Python installation. In an extension function that wants to accept numarrays as inputs and provide improved performance over the Python sequence protocol, an additional convenience function exists which diverts arrays to specialized code when numarray is present and the input is an array: :: PyObject * some_array_accepting_function(PyObject *m, PyObject *args) { PyObject *sequence, *result; if (!PyArg_ParseTuple(args, "O", &sequence)) return NULL; if (PyArray_isArray(sequence)) { result = numarray_input_function(sequence); } else { result = sequence_input_function(sequence); } return result; } During module initialization, a numarray enhanced extension must call **import_array()**, a macro which imports numarray and assigns a value to a static API pointer: PyArray_API. Since the API pointer starts with the value NULL and remains so if the numarray import fails, the API pointer serves as a flag that indicates that numarray was sucessfully imported whenever it is non-NULL. :: static void initfoo(void) { PyObject *m = Py_InitModule3( "foo", _foo_functions, _foo__doc__); if (m == NULL) return; import_array(); } **PyArray_Present()** indicates that numarray was successfully imported. It is defined in terms of the API function pointer as: :: #define PyArray_Present() (PyArray_API != NULL) **PyArray_isArray(s)** indicates that numarray was successfully imported and the given parameter is a numarray instance. It is defined as: :: #define PyArray_isArray(s) (PyArray_Present() && PyArray_Check(s)) Motivation ========== The use of numeric arrays as an interchange format is eminently sensible for many kinds of modules. For example, image, graphics, and audio modules all can accept or generate large amounts of numerical data that could easily use the numarray format. But since numarray is not part of the standard distribution, some authors of 3rd party extensions may be reluctant to add a dependency on a different 3rd party extension that isn't absolutely essential for its use fearing dissuading users who may be put off by extra installation requirements. Yet, not allowing easy interchange with numarray introduces annoyances that need not be present. Normally, in the absence of an explicit ability to generate or use numarray objects, one must write conversion utilities to convert from the data representation used to that for numarray. This typically involves excess copying of data (usually from internal to string to numarray). In cases where the 3rd party uses buffer objects, the data may not need copying at all. Either many users may have to develop their own conversion routines or numarray will have to include adapters for many other 3rd party packages. Since numarray is used by many projects, it makes more sense to put the conversion logic on the other side of the fence. There is a clear need for a mechanism that allows 3rd party software to use numarray objects if it is available without requiring numarray's presence to build and install properly. Rationale ========= One solution is to make numarray part of the standard distribution. That may be a good long-term solution, but at the moment, the numeric community is in transition period between the Numeric and numarray packages which may take years to complete. It is not likely that numarray will be considered for adoption until the transition is complete. Numarray is also a large package, and there is legitimate concern about its inclusion as regards the long-term commitment to support. We can solve that problem by making a few include files part of the Python Standard Distribution and demonstrating how extension writers can write code that uses numarray conditionally. The API submitted in this PEP is the subset of the numarray API which is most source compatible with Numeric. The headers consist of two handwritten files (arraybase.h and arrayobject.h) and one generated file (libnumeric.h). arraybase.h contains typedefs and enumerations which are important to both the API presented here and to the larger numarray specific API. arrayobject.h glues together arraybase and libnumeric and is needed for Numeric compatibility. libnumeric.h consists of macros generated from a template and a list of function prototypes. The macros themselves are somewhat intricate in order to provide the compile time checking effect of function prototypes. Further, the interface takes two forms: one form is used to compile numarray and defines static function prototypes. The other form is used to compile extensions which use the API and defines macros which execute function calls through pointers which are found in a table located using a single public API pointer. These macros also test the value of the API pointer in order to deliver a fatal error should a developer forget to initialize by calling import_array(). The interface chosen here is the subset of numarray most useful for porting existing Numeric code or creating new extensions which can be compiled for either numarray or Numeric. There are a number of other numarray API functions which are omitted here for the sake of simplicity. By choosing to support only the Numeric compatible subset of the numarray C-API, concerns about interface stability are minimized because the Numeric API is well established. However, it should be made clear that the numarray API subset proposed here is source compatible, not binary compatible, with Numeric. Resources ========= * numarray/arraybase.h (http://cvs.sourceforge.net/viewcvs.py/numpy/numarray/Include/numarray/arraybase.h) * numarray/libnumeric.h (http://cvs.sourceforge.net/viewcvs.py/numpy/numarray/Include/numarray/libnumeric.h) * numarray/arrayobject.h (http://cvs.sourceforge.net/viewcvs.py/numpy/numarray/Include/numarray/arrayobject.h) * numarray-1.0 manual PDF * numarray-1.0 source distribution * numarray website at STSCI (http://www.stsci.edu/resources/software_hardware/numarray) * example numarray enhanced extension References ========== . [1] PEP 1, PEP Purpose and Guidelines, Warsaw, Hylton (http://www.python.org/peps/pep-0001.html) . [2] PEP 9, Sample Plaintext PEP Template, Warsaw (http://www.python.org/peps/pep-0009.html) Copyright ========= This document has been placed in the public domain. . Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End: From haase at msg.ucsf.edu Mon Jun 28 14:23:50 2004 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Mon Jun 28 14:23:50 2004 Subject: [off topic] Re: [Numpy-discussion] numarray speed - PySequence_GetItem In-Reply-To: <1088449382.3744.138.camel@localhost.localdomain> References: <200406250949.44897.haase@msg.ucsf.edu> <40E05C8D.6050106@noaa.gov> <1088449382.3744.138.camel@localhost.localdomain> Message-ID: <200406281414.25400.haase@msg.ucsf.edu> On Monday 28 June 2004 12:03 pm, Todd Miller wrote: > On Mon, 2004-06-28 at 13:59, Chris Barker wrote: > > Sebastian Haase wrote: > > > BTW, from the profiling/timing I did you can tell that wxPyPlot > > > actually plots 25000 data points in 0.1 secs - so it's _really_ fast > > > ... > > > > Actually, it's probably not that fast, if you are timing on > > Linux/wxGTK/X-Windows. X is asyncronous, so what you are timing is how > > long it takes your program to tell X what to draw, but it may take > > longer than that to actually draw it. However, what you are timing is > > all the stuff that is effected by numarray/Numeric. > > > > I worked on part of the wxPython DC.DrawXXXList stuff, and I really > > wanted a Numeric native version, but Robin really didn't want an > > additional dependency. We discussed on this list a while back whether > > you could compile against Numeric, but let people run without it, and > > have it all work unless Someone actually used it. What makes that tricky > > is that the functions that test whether a PyObject is a Numeric array > > are in Numeric... but it could probably be done if you tried hard enough > > (maybe include just that function in wxPython...) > > numarray-1.0 has two macros for dealing with this: PyArray_Present() > and PyArray_isArray(obj). The former (safely) determines that numarray > is installed, while the latter determines that numarray is installed and > that obj is a NumArray. Both macros serve to guard sections of code > which make more extensive use of the numarray C-API to keep them from > segfaulting when numarray is not installed. I think this would be easy > to do for Numeric as well. > > One problem is that compiling a "numarray improved" extension requires > some portion of the numarray headers. I refactored the numarray > includes so that a relatively simple set of 3 files can be used to > support the Numeric compatible interface (for numarray). These could > either be included in core Python (with a successful PEP) or included in > interested packages. This approach adds a small source code burden > somewhere, but eliminates the requirement for users to have numarray > installed either to run or compile from source. > > I'll send out the draft PEP later today. > > Regards, > Todd My original question was just this: Does anyone know why numarray is maybe 10 times slower that Numeric with that particular code segment (PySequence_GetItem) ? - Sebastian From jmiller at stsci.edu Mon Jun 28 16:19:06 2004 From: jmiller at stsci.edu (Todd Miller) Date: Mon Jun 28 16:19:06 2004 Subject: [off topic] Re: [Numpy-discussion] numarray speed - PySequence_GetItem In-Reply-To: <200406281414.25400.haase@msg.ucsf.edu> References: <200406250949.44897.haase@msg.ucsf.edu> <40E05C8D.6050106@noaa.gov> <1088449382.3744.138.camel@localhost.localdomain> <200406281414.25400.haase@msg.ucsf.edu> Message-ID: <1088464671.3744.300.camel@localhost.localdomain> On Mon, 2004-06-28 at 17:14, Sebastian Haase wrote: > On Monday 28 June 2004 12:03 pm, Todd Miller wrote: > > On Mon, 2004-06-28 at 13:59, Chris Barker wrote: > > > Sebastian Haase wrote: > > > > BTW, from the profiling/timing I did you can tell that wxPyPlot > > > > actually plots 25000 data points in 0.1 secs - so it's _really_ fast > > > > ... > > > > > > Actually, it's probably not that fast, if you are timing on > > > Linux/wxGTK/X-Windows. X is asyncronous, so what you are timing is how > > > long it takes your program to tell X what to draw, but it may take > > > longer than that to actually draw it. However, what you are timing is > > > all the stuff that is effected by numarray/Numeric. > > > > > > I worked on part of the wxPython DC.DrawXXXList stuff, and I really > > > wanted a Numeric native version, but Robin really didn't want an > > > additional dependency. We discussed on this list a while back whether > > > you could compile against Numeric, but let people run without it, and > > > have it all work unless Someone actually used it. What makes that tricky > > > is that the functions that test whether a PyObject is a Numeric array > > > are in Numeric... but it could probably be done if you tried hard enough > > > (maybe include just that function in wxPython...) > > > > numarray-1.0 has two macros for dealing with this: PyArray_Present() > > and PyArray_isArray(obj). The former (safely) determines that numarray > > is installed, while the latter determines that numarray is installed and > > that obj is a NumArray. Both macros serve to guard sections of code > > which make more extensive use of the numarray C-API to keep them from > > segfaulting when numarray is not installed. I think this would be easy > > to do for Numeric as well. > > > > One problem is that compiling a "numarray improved" extension requires > > some portion of the numarray headers. I refactored the numarray > > includes so that a relatively simple set of 3 files can be used to > > support the Numeric compatible interface (for numarray). These could > > either be included in core Python (with a successful PEP) or included in > > interested packages. This approach adds a small source code burden > > somewhere, but eliminates the requirement for users to have numarray > > installed either to run or compile from source. > > > > I'll send out the draft PEP later today. > > > > Regards, > > Todd > > My original question was just this: Does anyone know why numarray is maybe 10 > times slower that Numeric with that particular code segment > (PySequence_GetItem) ? Well, the short answer is probably: no. Looking at the numarray sequence protocol benchmarks in Examples/bench.py, and looking at what wxPython is probably doing (fetching a 1x2 element array from an Nx2 and then fetching 2 numerical values from that)... I can't fully nail it down. My benchmarks show that numarray is 4x slower for fetching the two element array but only 1.1x slower for the two numbers; that makes me expect at most 4x slower. Noticing the 50k __del__ calls in your profile, I eliminated __del__ (breaking numarray) to see if that was the problem; the ratios changed to 2.5x slower and 0.9x slower (actually faster) respectively. The large number of "Check" routines preceding the numarray path (I count 7 looking at my copy of wxPython) has me a little concerned. I think those checks are more expensive for numarray because it is a new style class. I have a hard time imagining a 10x difference overall, but I think Python does have to traverse the numarray class hierarchy rather than do a type pointer comparison so they are more expensive. Is 10x a measured number or a gut feel? One last thought: because the sequence protocol is being used rather than raw array access, compiling matplotlib for numarray (or not) is not the issue. Regards, Todd From tim.hochberg at cox.net Tue Jun 29 08:15:51 2004 From: tim.hochberg at cox.net (Tim Hochberg) Date: Tue Jun 29 08:15:51 2004 Subject: [off topic] Re: [Numpy-discussion] numarray speed - PySequence_GetItem In-Reply-To: <1088518322.17789.104.camel@halloween.stsci.edu> References: <200406250949.44897.haase@msg.ucsf.edu> <40E05C8D.6050106@noaa.gov> <1088449382.3744.138.camel@localhost.localdomain> <200406281414.25400.haase@msg.ucsf.edu> <1088464671.3744.300.camel@localhost.localdomain> <40E0BB9C.5000000@cox.net> <1088518322.17789.104.camel@halloween.stsci.edu> Message-ID: <40E18678.1090800@cox.net> Todd Miller wrote: >On Mon, 2004-06-28 at 20:45, Tim Hochberg wrote: > > >>Todd Miller wrote: >> >> >> [SNIP] >>This reminds me, when profiling bits and pieces of my code I've often >>noticed that __del__ chews up a large chunk of time. Is there any >>prospect of this being knocked down at all, or is it inherent in the >>structure of numarray? >> >> > >__del__ is IMHO the elegant way to do numarray's shadowing of >"misbehaved arrays". misbehaved arrays are ones which don't meet the >requirements of a particular C-function, but generally that means >noncontiguous, byte-swapped, misaligned, or of the wrong type; it also >can mean some other sequence type like a list or tuple. I think using >the destructor is "necessary" for maintaining Numeric compatibility in C >because you can generally count on arrays being DECREF'd, but obviously >you couldn't count on some new API call being called. > > OK, that makes sense. In a general sense at least, I'll have to dig into the source to figure out the details. >__del__ used to be implemented in C as tp_dealloc, but I was running >into segfaults which I tracked down to the order in which a new style >class instance is torn down. The purpose of __del__ is to copy the >contents of a well behaved working array (the shadow) back onto the >original mis-behaved array. The problem was that, because of the >numarray class hierarchy, critical pieces of the shadow (the instance >dictionary) had already been torn down before the tp_dealloc was >called. The only way I could think of to fix it was to move the >destructor farther down in the class hierarchy, i.e. from >_numarray.tp_dealloc to NumArray.__del__ in Python. > > It seems that one could stash a reference to the instance dict somewhere (in PyArrayObject perhaps) to prevent the instance dict from being torn down when the rest of the subclass goes away. It would require another field in PyArrayObject , but that's big enough allready that no one would notice. I could easily be way off base here though. If I end up with too much spare time at some point, I may try to look into this. >If anyone can think of a way to get rid of __del__, I'm all for it. > > > [SNIP] >> >>I don't see any easy/obvious ways to speed up wxPySwigInstance_Check, >> >> > >Why no CheckExact, even if it's hand coded? Maybe the setup is tedious? > > I don't know although I suspect the setup being tedious might be part of it. I also believe that until recently, wxPython used old style classes and you couldn't use CheckExact. What I propose below is essentially a hand coded version of check exact for wxPoints. > > >>but I believe that wxPoints now obey the PySequence protocol, so I think >>that the whole wxPySwigInstance_Check branch could be removed. To get >>that into wxPython you'd probably have to convince Robin that it >>wouldn't hurt the speed of list of wxPoints unduly. >> >>Wait... If the above doesn't work, I think I do have a way that might >>work for speeding the check for a wxPoint. Before the loop starts, get a >>pointer to wx.core.Point (the class for wxPoints these days) and call it >>wxPoint_Type. Then just use for the check: >> o->ob_type == &wxPoint_Type >>Worth a try anyway. >> >>Unfortunately, I don't have any time to try any of this out right now. >> >>Chris, are you feeling bored? >> >>-tim >> >> > >What's the chance of adding direct support for numarray to wxPython? >Our PEP reduces the burden on a package to at worst adding 3 include >files for numarray plus the specialized package code. With those >files, the package can be compiled by users without numarray and also >run without numarray, but would receive a real boost for people willing >to install numarray since the sequence protocol could be bypassed. > > No idea, sorry. I haven't been keeping up with wxPython development lately. -tim From nadavh at visionsense.com Tue Jun 29 08:52:14 2004 From: nadavh at visionsense.com (Nadav Horesh) Date: Tue Jun 29 08:52:14 2004 Subject: [Numpy-discussion] Can not import image module Message-ID: <07C6A61102C94148B8104D42DE95F7E86DED4B@exchange2k.envision.co.il> >>> from numarray import image Traceback (most recent call last): File "", line 1, in -toplevel- from numarray import image File "/usr/local/lib/python2.3/site-packages/numarray/image/__init__.py", line 2, in -toplevel- from combine import * File "/usr/local/lib/python2.3/site-packages/numarray/image/combine.py", line 65, in -toplevel- _median = _combine.median AttributeError: 'module' object has no attribute 'median' numarray version: 1.0. I got this error also with version 0.9 Nadav. From curzio.basso at unibas.ch Tue Jun 29 09:05:39 2004 From: curzio.basso at unibas.ch (Curzio Basso) Date: Tue Jun 29 09:05:39 2004 Subject: [Numpy-discussion] generalized SVD Message-ID: <40E17833.7050400@unibas.ch> Hi. Has anyone a suggestion on where to find a GSVD implementation which I can use with numarray? I've seen GSVD is part of LAPACK, so maybe one option could be to use a LAPACK library implementing the required functions... thanks in advance. curzio. From jmiller at stsci.edu Tue Jun 29 09:05:49 2004 From: jmiller at stsci.edu (Todd Miller) Date: Tue Jun 29 09:05:49 2004 Subject: [off topic] Re: [Numpy-discussion] numarray speed - PySequence_GetItem In-Reply-To: <40E0BB9C.5000000@cox.net> References: <200406250949.44897.haase@msg.ucsf.edu> <40E05C8D.6050106@noaa.gov> <1088449382.3744.138.camel@localhost.localdomain> <200406281414.25400.haase@msg.ucsf.edu> <1088464671.3744.300.camel@localhost.localdomain> <40E0BB9C.5000000@cox.net> Message-ID: <1088518322.17789.104.camel@halloween.stsci.edu> On Mon, 2004-06-28 at 20:45, Tim Hochberg wrote: > Todd Miller wrote: > > >On Mon, 2004-06-28 at 17:14, Sebastian Haase wrote: > > > > > >> [SNIP] > >> > >>My original question was just this: Does anyone know why numarray is maybe 10 > >>times slower that Numeric with that particular code segment > >>(PySequence_GetItem) ? > >> > >> > > > >Well, the short answer is probably: no. > > > >Looking at the numarray sequence protocol benchmarks in > >Examples/bench.py, and looking at what wxPython is probably doing > >(fetching a 1x2 element array from an Nx2 and then fetching 2 numerical > >values from that)... I can't fully nail it down. My benchmarks show > >that numarray is 4x slower for fetching the two element array but only > >1.1x slower for the two numbers; that makes me expect at most 4x > >slower. > > > >Noticing the 50k __del__ calls in your profile, I eliminated __del__ > >(breaking numarray) to see if that was the problem; the ratios changed > >to 2.5x slower and 0.9x slower (actually faster) respectively. > > > > > This reminds me, when profiling bits and pieces of my code I've often > noticed that __del__ chews up a large chunk of time. Is there any > prospect of this being knocked down at all, or is it inherent in the > structure of numarray? __del__ is IMHO the elegant way to do numarray's shadowing of "misbehaved arrays". misbehaved arrays are ones which don't meet the requirements of a particular C-function, but generally that means noncontiguous, byte-swapped, misaligned, or of the wrong type; it also can mean some other sequence type like a list or tuple. I think using the destructor is "necessary" for maintaining Numeric compatibility in C because you can generally count on arrays being DECREF'd, but obviously you couldn't count on some new API call being called. __del__ used to be implemented in C as tp_dealloc, but I was running into segfaults which I tracked down to the order in which a new style class instance is torn down. The purpose of __del__ is to copy the contents of a well behaved working array (the shadow) back onto the original mis-behaved array. The problem was that, because of the numarray class hierarchy, critical pieces of the shadow (the instance dictionary) had already been torn down before the tp_dealloc was called. The only way I could think of to fix it was to move the destructor farther down in the class hierarchy, i.e. from _numarray.tp_dealloc to NumArray.__del__ in Python. If anyone can think of a way to get rid of __del__, I'm all for it. > >The large number of "Check" routines preceding the numarray path (I > >count 7 looking at my copy of wxPython) has me a little concerned. I > >think those checks are more expensive for numarray because it is a new > >style class. > > > If that's really a significant slowdown, the culprit's are likely > PyTuple_Check, PyList_Check and wxPySwigInstance_Check. > PySequence_Check appears to just be pointer compares and shouldn't > invoke any new style class machinery. PySequence_Length calls sq_length, > but appears also to not involve new class machinery. Of these, I think > PyTuple_Check and PyList_Check could be replaced with PyTuple_CheckExact > and PyList_CheckExact. This would slow down people using subclasses of > tuple/list, but speed everyone else up since the latter pair of > functions are just pointer compares. I think the former group is a very > small minority, possibly nonexistent, minority, so this would probably > be acceptable. > > I don't see any easy/obvious ways to speed up wxPySwigInstance_Check, Why no CheckExact, even if it's hand coded? Maybe the setup is tedious? > but I believe that wxPoints now obey the PySequence protocol, so I think > that the whole wxPySwigInstance_Check branch could be removed. To get > that into wxPython you'd probably have to convince Robin that it > wouldn't hurt the speed of list of wxPoints unduly. > > Wait... If the above doesn't work, I think I do have a way that might > work for speeding the check for a wxPoint. Before the loop starts, get a > pointer to wx.core.Point (the class for wxPoints these days) and call it > wxPoint_Type. Then just use for the check: > o->ob_type == &wxPoint_Type > Worth a try anyway. > > Unfortunately, I don't have any time to try any of this out right now. > > Chris, are you feeling bored? > > -tim What's the chance of adding direct support for numarray to wxPython? Our PEP reduces the burden on a package to at worst adding 3 include files for numarray plus the specialized package code. With those files, the package can be compiled by users without numarray and also run without numarray, but would receive a real boost for people willing to install numarray since the sequence protocol could be bypassed. Regards, Todd From jmiller at stsci.edu Tue Jun 29 09:12:48 2004 From: jmiller at stsci.edu (Todd Miller) Date: Tue Jun 29 09:12:48 2004 Subject: [off topic] Re: [Numpy-discussion] numarray speed - PySequence_GetItem In-Reply-To: <200406281638.03922.haase@msg.ucsf.edu> References: <200406250949.44897.haase@msg.ucsf.edu> <200406281414.25400.haase@msg.ucsf.edu> <1088464671.3744.300.camel@localhost.localdomain> <200406281638.03922.haase@msg.ucsf.edu> Message-ID: <1088520761.17789.184.camel@halloween.stsci.edu> On Mon, 2004-06-28 at 19:38, Sebastian Haase wrote: > > Is 10x a measured number or a gut feel? > > I put some time.clock() statements into the wxPyPlot code > I got this: (the times are differences: T_after-T_before) > one run with numarray: > <__main__.PolyLine instance at 0x868d414> time= 1.06 > <__main__.PolyMarker instance at 0x878e9c4> time= 1.37 > a second run with numarray: > <__main__.PolyLine instance at 0x875da1c> time= 0.85 > <__main__.PolyMarker instance at 0x86da034> time= 1.04 > first run with Numeric: > <__main__.PolyLine instance at 0x858babc> time= 0.07 > <__main__.PolyMarker instance at 0x858bc4c> time= 0.14 > a second one: > <__main__.PolyLine instance at 0x858cd7c> time= 0.08 > <__main__.PolyMarker instance at 0x8585d8c> time= 0.17 > This seems to be consistent with the profiling I did before: > I get this w/ numarray: > ncalls tottime percall cumtime percall filename:lineno(function) > 1 1.140 1.140 1.320 1.320 gdi.py:554(DrawLines) > 1 1.250 1.250 1.520 1.520 gdi.py:792(_DrawRectangleList) > 50230 0.450 0.000 0.450 0.000 numarraycore.py:501(__del__) > and this with Numeric: > 1 0.080 0.080 0.080 0.080 gdi.py:554(DrawLines) > 1 0.090 0.090 0.090 0.090 gdi.py:792(_DrawRectangleList) > > so this looks to me like a factor of 10x. Me too. Can you (or somebody) post the application code which does the drawlines? I can definitely instrument the bottleneck C-code, but I don't have time to ascend the wxPython learning curve. Todd From Chris.Barker at noaa.gov Tue Jun 29 09:36:03 2004 From: Chris.Barker at noaa.gov (Chris Barker) Date: Tue Jun 29 09:36:03 2004 Subject: [off topic] Re: [Numpy-discussion] numarray speed - PySequence_GetItem In-Reply-To: <40E18678.1090800@cox.net> References: <200406250949.44897.haase@msg.ucsf.edu> <40E05C8D.6050106@noaa.gov> <1088449382.3744.138.camel@localhost.localdomain> <200406281414.25400.haase@msg.ucsf.edu> <1088464671.3744.300.camel@localhost.localdomain> <40E0BB9C.5000000@cox.net> <1088518322.17789.104.camel@halloween.stsci.edu> <40E18678.1090800@cox.net> Message-ID: <40E199A7.3000301@noaa.gov> Tim Hochberg wrote: >>> but I believe that wxPoints now obey the PySequence protocol, so I >>> think that the whole wxPySwigInstance_Check branch could be removed. >>> To get that into wxPython you'd probably have to convince Robin that >>> it wouldn't hurt the speed of list of wxPoints unduly. >>> >>> Wait... If the above doesn't work, I think I do have a way that might >>> work for speeding the check for a wxPoint. Before the loop starts, >>> get a pointer to wx.core.Point (the class for wxPoints these days) >>> and call it wxPoint_Type. Then just use for the check: >>> o->ob_type == &wxPoint_Type >>> Worth a try anyway. >>> >>> Unfortunately, I don't have any time to try any of this out right now. >>> >>> Chris, are you feeling bored? Do you mean me? if so: A) I'm not bored. B) This is all kind of beyond me anyway, and C) I'm planning on implementing my own custom DC.DrawLotsOfStuff code, because I have some specialized needs that probably don't really belong in wxPython. My stuff will take Numeric arrays as input (This is for my FloatCanvas, if anyone cares). I'm still using Numeric, as numarray is a LOT slower when used in FloatCanvas, probably because I do a lot with small arrays, and maybe because of what we're talking about here as well. However, This may turn out to be important to me some day, so who knows? I'll keep this note around. >> What's the chance of adding direct support for numarray to wxPython? >> Our PEP reduces the burden on a package to at worst adding 3 include >> files for numarray plus the specialized package code. With those >> files, the package can be compiled by users without numarray and also >> run without numarray, but would receive a real boost for people willing >> to install numarray since the sequence protocol could be bypassed. If the PEP is accepted, and those include files are part of the standard Python distro, I suspect Robin would be quite happy to add direct support, at least if someone else writes the code. Whether he'd be open to including those files in the wxPython distribution itself, I don't know. Perhaps I'll drop him a line. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From haase at msg.ucsf.edu Tue Jun 29 09:44:46 2004 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue Jun 29 09:44:46 2004 Subject: [Numpy-discussion] bug ? in Records arrays in numarray Message-ID: <200406281700.37694.haase@msg.ucsf.edu> Hi, I have two record arrays. I was trying to assign one item from one recarray to the other: >>> omx.zext[0] = main.ring4ext[0,0] Traceback (most recent call last): File "", line 1, in ? File "X:/PrWin\numarray\records.py", line 744, in _setitem self.field(self._names[i])[row] = value.field(self._names[i]) File "C:\Python22\Lib\site-packages\numarray\numarraycore.py", line 619, in __tonumtype__ ValueError: Can't use non-rank-0 array as a scalar. >>> `omx.zext[0]` '' >>> `main.ring4ext[0,0]` '' >>> q = omx.zext[0] >>> w = main.ring4ext[0,0] >>> q (2097152107, -595656700, 91141, 1.0634608642000868e+037, -1.14841804241843e +018, 1.2771574333702815e-040) >>> w (array([428]), array([75]), array([124]), array([ 1.08846451e+09], type=Float32), array([ 99.25], type=Float32), array([ 1996.82995605], type=Float32)) >>> omx.zext._formats ['1Int32', '1Int32', '1Int32', '1Float32', '1Float32', '1Float32'] >>> main.ring4ext._formats ['1Int32', '1Int32', '1Int32', '1Float32', '1Float32', '1Float32'] I can't track down why one (q) contains scalars and the other (w) constains arrays (is array([428]) a 'rank-0 array'? ) ! This is how I generate them: main.ring4ext = rec.RecArray(main._extHdrRingBuffer, "i4,i4,i4,f4,f4,f4", shape=(ts,nc), names=("num","min","max","time","mean","darkVal"), aligned=1) omx.zext = rec.array(formats="i4,i4,i4,f4,f4,f4", names=("num","min","max","time","mean","darkVal"),shape=tuple(shapeZ),aligned=1 ) [[BTW: (shapeZ is a list, if I say: ...,shape=shapeZ I get NameError, "Illegal shape %s" % `shape` from "records.py" in line 462 -- usually type(shape) == types.ListType should be OK, right ?) ]] Any ideas? Thanks, Sebastian Haase From tim.hochberg at cox.net Tue Jun 29 10:12:38 2004 From: tim.hochberg at cox.net (Tim Hochberg) Date: Tue Jun 29 10:12:38 2004 Subject: [off topic] Re: [Numpy-discussion] numarray speed - PySequence_GetItem In-Reply-To: <1088464671.3744.300.camel@localhost.localdomain> References: <200406250949.44897.haase@msg.ucsf.edu> <40E05C8D.6050106@noaa.gov> <1088449382.3744.138.camel@localhost.localdomain> <200406281414.25400.haase@msg.ucsf.edu> <1088464671.3744.300.camel@localhost.localdomain> Message-ID: <40E0BB9C.5000000@cox.net> Todd Miller wrote: >On Mon, 2004-06-28 at 17:14, Sebastian Haase wrote: > > >> [SNIP] >> >>My original question was just this: Does anyone know why numarray is maybe 10 >>times slower that Numeric with that particular code segment >>(PySequence_GetItem) ? >> >> > >Well, the short answer is probably: no. > >Looking at the numarray sequence protocol benchmarks in >Examples/bench.py, and looking at what wxPython is probably doing >(fetching a 1x2 element array from an Nx2 and then fetching 2 numerical >values from that)... I can't fully nail it down. My benchmarks show >that numarray is 4x slower for fetching the two element array but only >1.1x slower for the two numbers; that makes me expect at most 4x >slower. > >Noticing the 50k __del__ calls in your profile, I eliminated __del__ >(breaking numarray) to see if that was the problem; the ratios changed >to 2.5x slower and 0.9x slower (actually faster) respectively. > > This reminds me, when profiling bits and pieces of my code I've often noticed that __del__ chews up a large chunk of time. Is there any prospect of this being knocked down at all, or is it inherent in the structure of numarray? >The large number of "Check" routines preceding the numarray path (I >count 7 looking at my copy of wxPython) has me a little concerned. I >think those checks are more expensive for numarray because it is a new >style class. > If that's really a significant slowdown, the culprit's are likely PyTuple_Check, PyList_Check and wxPySwigInstance_Check. PySequence_Check appears to just be pointer compares and shouldn't invoke any new style class machinery. PySequence_Length calls sq_length, but appears also to not involve new class machinery. Of these, I think PyTuple_Check and PyList_Check could be replaced with PyTuple_CheckExact and PyList_CheckExact. This would slow down people using subclasses of tuple/list, but speed everyone else up since the latter pair of functions are just pointer compares. I think the former group is a very small minority, possibly nonexistent, minority, so this would probably be acceptable. I don't see any easy/obvious ways to speed up wxPySwigInstance_Check, but I believe that wxPoints now obey the PySequence protocol, so I think that the whole wxPySwigInstance_Check branch could be removed. To get that into wxPython you'd probably have to convince Robin that it wouldn't hurt the speed of list of wxPoints unduly. Wait... If the above doesn't work, I think I do have a way that might work for speeding the check for a wxPoint. Before the loop starts, get a pointer to wx.core.Point (the class for wxPoints these days) and call it wxPoint_Type. Then just use for the check: o->ob_type == &wxPoint_Type Worth a try anyway. Unfortunately, I don't have any time to try any of this out right now. Chris, are you feeling bored? -tim >I have a hard time imagining a 10x difference overall, >but I think Python does have to traverse the numarray class hierarchy >rather than do a type pointer comparison so they are more expensive. > > From jmiller at stsci.edu Tue Jun 29 10:38:02 2004 From: jmiller at stsci.edu (Todd Miller) Date: Tue Jun 29 10:38:02 2004 Subject: [Numpy-discussion] Can not import image module In-Reply-To: <07C6A61102C94148B8104D42DE95F7E86DED4B@exchange2k.envision.co.il> References: <07C6A61102C94148B8104D42DE95F7E86DED4B@exchange2k.envision.co.il> Message-ID: <1088530609.17789.270.camel@halloween.stsci.edu> On Tue, 2004-06-29 at 07:14, Nadav Horesh wrote: > >>> from numarray import image I tried this and it worked fine for me under RH9.0 and Python-2.3.4 and also under Windows 2000 pro. I suggest deleting your entire numarray directory from site-packages and then re-installing. Also, what's your platform and shell? Regards, Todd > Traceback (most recent call last): > File "", line 1, in -toplevel- > from numarray import image > File "/usr/local/lib/python2.3/site-packages/numarray/image/__init__.py", line 2, in -toplevel- > from combine import * > File "/usr/local/lib/python2.3/site-packages/numarray/image/combine.py", line 65, in -toplevel- > _median = _combine.median > AttributeError: 'module' object has no attribute 'median' > > > > numarray version: 1.0. I got this error also with version 0.9 > > Nadav. > > > > > ------------------------------------------------------- > This SF.Net email sponsored by Black Hat Briefings & Training. > Attend Black Hat Briefings & Training, Las Vegas July 24-29 - > digital self defense, top technical experts, no vendor pitches, > unmatched networking opportunities. Visit www.blackhat.com > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- Todd Miller Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21218 (410) 338-4576 From gerard.vermeulen at grenoble.cnrs.fr Tue Jun 29 10:46:02 2004 From: gerard.vermeulen at grenoble.cnrs.fr (Gerard Vermeulen) Date: Tue Jun 29 10:46:02 2004 Subject: [Numpy-discussion] Numarray header PEP In-Reply-To: <1088451653.3744.200.camel@localhost.localdomain> References: <1088451653.3744.200.camel@localhost.localdomain> Message-ID: <20040629194456.44a1fa7f.gerard.vermeulen@grenoble.cnrs.fr> > > The PEP is attached. It is formatted using the docutils package which > can be used to generate HTML or PDF. Comments and corrections would be > appreciated. > PyQwt is a Python extension which can be conditionally compiled against Numeric and/or numarray (both, one of them or none). Your PEP excludes importing of Numeric and numarray in the same C-extension. All you need to do is to hide the macros PyArray_Present(), PyArray_isArray() and import_array() into a few functions with numarray specific names, so that the following becomes possible: #include /* defines the functions (no macros) int Numeric_Present(); int Numeric_isArray(); void import_numeric(); to hide the Numeric C-API stuff in a small Numeric/meta.c file. */ #include /* defines the functions (no macros) int numarray_Present(); int numarray_isArray(); void import_numarray(); to hide the numarray C-API stuff in a small numarray/meta.c file. */ PyObject * some_array_returning_function(PyObject *m, PyObject *args) { int param; PyObject *result; if (!PyArg_ParseTuple(args, "i", ¶m)) return NULL; if (Numeric_Present()) { result = numeric_returning_function(param); } else if (Numarray_Present()) { result = numarray_returning_function(param); } else { result = list_returning_function(param); } return result; } PyObject * some_array_accepting_function(PyObject *m, PyObject *args) { PyObject *sequence, *result; if (!PyArg_ParseTuple(args, "O", &sequence)) return NULL; if (Numeric_isArray(sequence)) { result = numeric_input_function(sequence); } else if (Numarray_isArray(sequence)) { result = numarray_input_function(sequence); } else { result = sequence_input_function(sequence); } return result; } /* the code for the numeric_whatever_functions and for the numarray_whatever_functions should be source files. */ static void initfoo(void) { PyObject *m = Py_InitModule3( "foo", _foo_functions, _foo__doc__); if (m == NULL) return; import_numeric(); import_numarray(); } Regards -- Gerard From nadavh at visionsense.com Tue Jun 29 11:32:02 2004 From: nadavh at visionsense.com (Nadav Horesh) Date: Tue Jun 29 11:32:02 2004 Subject: [Numpy-discussion] Can not import image module Message-ID: <07C6A61102C94148B8104D42DE95F7E86DED4F@exchange2k.envision.co.il> Same (RH9.0 + Python 2.3.4). It is OK on my home PC (Gentoo 1.4 +Python 2.3.4). I'll try your advice when I'll back at work. Thanks, Nadav. -----Original Message----- From: Todd Miller [mailto:jmiller at stsci.edu] Sent: Tue 29-Jun-04 20:36 To: Nadav Horesh Cc: numpy-discussion Subject: Re: [Numpy-discussion] Can not import image module On Tue, 2004-06-29 at 07:14, Nadav Horesh wrote: > >>> from numarray import image I tried this and it worked fine for me under RH9.0 and Python-2.3.4 and also under Windows 2000 pro. I suggest deleting your entire numarray directory from site-packages and then re-installing. Also, what's your platform and shell? Regards, Todd > Traceback (most recent call last): > File "", line 1, in -toplevel- > from numarray import image > File "/usr/local/lib/python2.3/site-packages/numarray/image/__init__.py", line 2, in -toplevel- > from combine import * > File "/usr/local/lib/python2.3/site-packages/numarray/image/combine.py", line 65, in -toplevel- > _median = _combine.median > AttributeError: 'module' object has no attribute 'median' > > > > numarray version: 1.0. I got this error also with version 0.9 > > Nadav. > > > > > ------------------------------------------------------- > This SF.Net email sponsored by Black Hat Briefings & Training. > Attend Black Hat Briefings & Training, Las Vegas July 24-29 - > digital self defense, top technical experts, no vendor pitches, > unmatched networking opportunities. Visit www.blackhat.com > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- Todd Miller Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21218 (410) 338-4576 From jmiller at stsci.edu Tue Jun 29 12:10:20 2004 From: jmiller at stsci.edu (Todd Miller) Date: Tue Jun 29 12:10:20 2004 Subject: [Numpy-discussion] Numarray header PEP In-Reply-To: <20040629194456.44a1fa7f.gerard.vermeulen@grenoble.cnrs.fr> References: <1088451653.3744.200.camel@localhost.localdomain> <20040629194456.44a1fa7f.gerard.vermeulen@grenoble.cnrs.fr> Message-ID: <1088536183.17789.346.camel@halloween.stsci.edu> On Tue, 2004-06-29 at 13:44, Gerard Vermeulen wrote: > > > > The PEP is attached. It is formatted using the docutils package which > > can be used to generate HTML or PDF. Comments and corrections would be > > appreciated. > > > PyQwt is a Python extension which can be conditionally compiled against > Numeric and/or numarray (both, one of them or none). Well that's cool! I'll have to keep the PyQwt guys in mind as potential first users. > Your PEP excludes importing of Numeric and numarray in the same C-extension. This is true but I don't understand your solution so I'll blather on below. > All you need to do is to hide the macros PyArray_Present(), PyArray_isArray() > and import_array() into a few functions with numarray specific names, so > that the following becomes possible: > > #include > /* defines the functions (no macros) > int Numeric_Present(); > int Numeric_isArray(); > void import_numeric(); > to hide the Numeric C-API stuff in a small Numeric/meta.c file. > */ > #include > /* defines the functions (no macros) > int numarray_Present(); > int numarray_isArray(); > void import_numarray(); > to hide the numarray C-API stuff in a small numarray/meta.c file. > */ > I may have come up with the wrong scheme for the Present() and isArray(). With my scheme, they *have* to be macros because the API functions are unsafe: when numarray or Numeric is not present, the API function table pointers are NULL so calling through them results in either a fatal error or a segfault. There is an additional problem that the "same functions" need to be called through different API pointers depending on whether numarray or Numeric is being supported. Redefinition of typedefs and enumerations (or perhaps conditional compilation short-circuited re-definitions) may also present a problem with compiling (or worse, running). I certainly like the idea of supporting both in the same extension module, but don't see how to get there, other than with separate compilation units. With seperate .c files, I'm not aware of a problem other than lots of global symbols. I haven't demoed that yet so I am interested if someone has made it work. Thanks very much for commenting on the PEP. Regards, Todd > PyObject * > some_array_returning_function(PyObject *m, PyObject *args) > { > int param; > PyObject *result; > > if (!PyArg_ParseTuple(args, "i", ¶m)) > return NULL; > > if (Numeric_Present()) { > result = numeric_returning_function(param); > } else if (Numarray_Present()) { > result = numarray_returning_function(param); > } else { > result = list_returning_function(param); > } > return result; > } > > PyObject * > some_array_accepting_function(PyObject *m, PyObject *args) > { > PyObject *sequence, *result; > > if (!PyArg_ParseTuple(args, "O", &sequence)) > return NULL; > > if (Numeric_isArray(sequence)) { > result = numeric_input_function(sequence); > } else if (Numarray_isArray(sequence)) { > result = numarray_input_function(sequence); > } else { > result = sequence_input_function(sequence); > } > return result; > } > > /* > the code for the numeric_whatever_functions and for the > numarray_whatever_functions should be source files. > */ > > static void > initfoo(void) > { > PyObject *m = Py_InitModule3( > "foo", > _foo_functions, > _foo__doc__); > if (m == NULL) return; > import_numeric(); > import_numarray(); > } > > Regards -- Gerard > > > > > > > ------------------------------------------------------- > This SF.Net email sponsored by Black Hat Briefings & Training. > Attend Black Hat Briefings & Training, Las Vegas July 24-29 - > digital self defense, top technical experts, no vendor pitches, > unmatched networking opportunities. Visit www.blackhat.com > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- Todd Miller Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21218 (410) 338-4576 From tim.hochberg at cox.net Tue Jun 29 12:22:11 2004 From: tim.hochberg at cox.net (Tim Hochberg) Date: Tue Jun 29 12:22:11 2004 Subject: [off topic] Re: [Numpy-discussion] numarray speed - PySequence_GetItem In-Reply-To: <1088520761.17789.184.camel@halloween.stsci.edu> References: <200406250949.44897.haase@msg.ucsf.edu> <200406281414.25400.haase@msg.ucsf.edu> <1088464671.3744.300.camel@localhost.localdomain> <200406281638.03922.haase@msg.ucsf.edu> <1088520761.17789.184.camel@halloween.stsci.edu> Message-ID: <40E1C12E.4060102@cox.net> Todd Miller wrote: >On Mon, 2004-06-28 at 19:38, Sebastian Haase wrote: > > > >>>Is 10x a measured number or a gut feel? >>> >>> >>[SNIP] >> >>so this looks to me like a factor of 10x. >> >> > >Me too. Can you (or somebody) post the application code which does the >drawlines? I can definitely instrument the bottleneck C-code, but I >don't have time to ascend the wxPython learning curve. > > Let me second that. With a little nudge from Chris Barker, I managed to get wxPython to compile here and I have some changes that may speed up drawlines, but it'd probably be best if we were all using the same benchmark. -tim From tim.hochberg at cox.net Tue Jun 29 13:21:07 2004 From: tim.hochberg at cox.net (Tim Hochberg) Date: Tue Jun 29 13:21:07 2004 Subject: [off topic] Re: [Numpy-discussion] numarray speed - PySequence_GetItem In-Reply-To: <1088520761.17789.184.camel@halloween.stsci.edu> References: <200406250949.44897.haase@msg.ucsf.edu> <200406281414.25400.haase@msg.ucsf.edu> <1088464671.3744.300.camel@localhost.localdomain> <200406281638.03922.haase@msg.ucsf.edu> <1088520761.17789.184.camel@halloween.stsci.edu> Message-ID: <40E1CEE6.3050108@cox.net> I'd bet a case of beer (or cash equivalent) that one of the main bottlenecks is the path PySequence_GetItem->_ndarray_item->_universalIndexing->_simpleIndexing->_simpleIndexingCore. The path through _universalIndexing in particular, if I deciphered it correctly, looks very slow. I don't think it needs to be that way though, _universalIndexing could probably be sped up, but more promising I think _ndarray_item could be made to call _simpleIndexingCore without all that much work. It appears that this would save the creation of several intermediate objects and it also looks like a couple of calls back to python! I'm not familiar with this code though, so I could easily be missing something that makes calling _simpleIndexingCore harder than it looks. -tim From Chris.Barker at noaa.gov Tue Jun 29 14:08:06 2004 From: Chris.Barker at noaa.gov (Chris Barker) Date: Tue Jun 29 14:08:06 2004 Subject: [off topic] Re: [Numpy-discussion] numarray speed - PySequence_GetItem In-Reply-To: <40E1C12E.4060102@cox.net> References: <200406250949.44897.haase@msg.ucsf.edu> <200406281414.25400.haase@msg.ucsf.edu> <1088464671.3744.300.camel@localhost.localdomain> <200406281638.03922.haase@msg.ucsf.edu> <1088520761.17789.184.camel@halloween.stsci.edu> <40E1C12E.4060102@cox.net> Message-ID: <40E1D998.8010204@noaa.gov> Tim Hochberg wrote: > Todd Miller wrote: >> Me too. Can you (or somebody) post the application code which does the >> drawlines? I can definitely instrument the bottleneck C-code, but I >> don't have time to ascend the wxPython learning curve. > Let me second that. With a little nudge from Chris Barker, I managed to > get wxPython to compile here and I have some changes that may speed up > drawlines, but it'd probably be best if we were all using the same > benchmark. Well, if I can't code, at least I can nudge! Perhaps I can also help get Todd what he's asking for, except that I'm not sure what you mean by "application code". Do you mean a small wxPython application that uses DC.DrawLines (and friends)? If so, yes, I can do that. What version of wxPython should I target? CVS head? 2.5.1? (The latest "released" version) there are some slight incompatibilities between versions, particularly with DC calls, unfortunately. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From gerard.vermeulen at grenoble.cnrs.fr Tue Jun 29 14:33:03 2004 From: gerard.vermeulen at grenoble.cnrs.fr (gerard.vermeulen at grenoble.cnrs.fr) Date: Tue Jun 29 14:33:03 2004 Subject: [Numpy-discussion] Numarray header PEP In-Reply-To: <1088536183.17789.346.camel@halloween.stsci.edu> References: <1088451653.3744.200.camel@localhost.localdomain> <20040629194456.44a1fa7f.gerard.vermeulen@grenoble.cnrs.fr> <1088536183.17789.346.camel@halloween.stsci.edu> Message-ID: <20040629211800.M55753@grenoble.cnrs.fr> On 29 Jun 2004 15:09:43 -0400, Todd Miller wrote > On Tue, 2004-06-29 at 13:44, Gerard Vermeulen wrote: > > > > > > The PEP is attached. It is formatted using the docutils package which > > > can be used to generate HTML or PDF. Comments and corrections would be > > > appreciated. > > > > > PyQwt is a Python extension which can be conditionally compiled against > > Numeric and/or numarray (both, one of them or none). > > Well that's cool! I'll have to keep the PyQwt guys in mind as potential > first users. > > > Your PEP excludes importing of Numeric and numarray in the same C-extension. > > This is true but I don't understand your solution so I'll blather on > below. > > > All you need to do is to hide the macros PyArray_Present(), PyArray_isArray() > > and import_array() into a few functions with numarray specific names, so > > that the following becomes possible: > > > > #include > > /* defines the functions (no macros) > > int Numeric_Present(); > > int Numeric_isArray(); > > void import_numeric(); > > to hide the Numeric C-API stuff in a small Numeric/meta.c file. > > */ > > #include > > /* defines the functions (no macros) > > int numarray_Present(); > > int numarray_isArray(); > > void import_numarray(); > > to hide the numarray C-API stuff in a small numarray/meta.c file. > > */ > > > > I may have come up with the wrong scheme for the Present() and > isArray(). With my scheme, they *have* to be macros because the API > functions are unsafe: when numarray or Numeric is not present, the API > function table pointers are NULL so calling through them results in > either a fatal error or a segfault. > The macros can be hidden from the module file scope by wrapping them in a function (see attached demo) > > There is an additional problem that the "same functions" need to be > called through different API pointers depending on whether numarray > or Numeric is being supported. Redefinition of typedefs and enumerations > > (or perhaps conditional compilation short-circuited re-definitions) may > also present a problem with compiling (or worse, running). > Tested and works. > > I certainly like the idea of supporting both in the same extension > module, but don't see how to get there, other than with separate > compilation units. With seperate .c files, I'm not aware of a problem > other than lots of global symbols. I haven't demoed that yet so I am > interested if someone has made it work. > Yes, you cannot mix the numarray API and Numeric API in the same .c file, but nevertheless you can hide the macros in small functions so that the macros don't pollute. Here you have a litte extension module 'pep' which tries to import Numeric and numarray. It has a method 'whatsThere' that prints the names of the Numerical/numarray extension modules it has imported. It has also a method 'whatsThis' that prints if an object is a Numeric array, a numarray array or something else: Python 2.3.3 (#2, Feb 17 2004, 11:45:40) [GCC 3.3.2 (Mandrake Linux 10.0 3.3.2-6mdk)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import pep >>> pep.whatsThere() Numeric! Numarray! >>> import Numeric >>> a = Numeric.arange(10) >>> pep.whatsThis(a) Numeric array >>> import numarray >>> a = numarray.arange(10) >>> pep.whatsThis(a) Numarray array >>> a = 10 >>> pep.whatsThis(a) Something else >>> The MANIFEST reads: pepmodule.c setup.py Numeric/meta.c Numeric/meta.h numarray/meta.c numarray/meta.h My initial idea was to add the meta.* files to the Numeric/numarray include files, but I did not try yet to solve the problem of the PyArray_API PY_ARRAY_UNIQUE_SYMBOL defines in the same way (I am sure it can be done. Regards -- Gerard -------------- next part -------------- A non-text attachment was scrubbed... Name: pep-0.0.1.tar.gz Type: application/gzip Size: 1421 bytes Desc: not available URL: From jmiller at stsci.edu Tue Jun 29 14:35:04 2004 From: jmiller at stsci.edu (Todd Miller) Date: Tue Jun 29 14:35:04 2004 Subject: [off topic] Re: [Numpy-discussion] numarray speed - PySequence_GetItem In-Reply-To: <40E1D998.8010204@noaa.gov> References: <200406250949.44897.haase@msg.ucsf.edu> <200406281414.25400.haase@msg.ucsf.edu> <1088464671.3744.300.camel@localhost.localdomain> <200406281638.03922.haase@msg.ucsf.edu> <1088520761.17789.184.camel@halloween.stsci.edu> <40E1C12E.4060102@cox.net> <40E1D998.8010204@noaa.gov> Message-ID: <1088544847.17789.393.camel@halloween.stsci.edu> On Tue, 2004-06-29 at 17:05, Chris Barker wrote: > Tim Hochberg wrote: > > Todd Miller wrote: > >> Me too. Can you (or somebody) post the application code which does the > >> drawlines? I can definitely instrument the bottleneck C-code, but I > >> don't have time to ascend the wxPython learning curve. > > > Let me second that. With a little nudge from Chris Barker, I managed to > > get wxPython to compile here and I have some changes that may speed up > > drawlines, but it'd probably be best if we were all using the same > > benchmark. > > Well, if I can't code, at least I can nudge! > > Perhaps I can also help get Todd what he's asking for, except that I'm > not sure what you mean by "application code". Do you mean a small > wxPython application that uses DC.DrawLines (and friends)? Yes. What I most want to do is a 50000 point drawlines, similar to what you profiled. Friends are fine too. > If so, yes, I > can do that. What version of wxPython should I target? > CVS head? > 2.5.1? (The latest "released" version) I'd prefer 2.5.1 unless Tim says otherwise. > there are some slight incompatibilities between versions, particularly > with DC calls, unfortunately. I'm hoping this won't affect the profile. Regards, Todd From jmiller at stsci.edu Tue Jun 29 14:43:10 2004 From: jmiller at stsci.edu (Todd Miller) Date: Tue Jun 29 14:43:10 2004 Subject: [Numpy-discussion] Numarray header PEP In-Reply-To: <20040629211800.M55753@grenoble.cnrs.fr> References: <1088451653.3744.200.camel@localhost.localdomain> <20040629194456.44a1fa7f.gerard.vermeulen@grenoble.cnrs.fr> <1088536183.17789.346.camel@halloween.stsci.edu> <20040629211800.M55753@grenoble.cnrs.fr> Message-ID: <1088545351.17789.406.camel@halloween.stsci.edu> Awesome! Thanks Gerard. This certainly sounds like the way to do it. I'll take a closer look tomorrow. Regards, Todd On Tue, 2004-06-29 at 17:32, gerard.vermeulen at grenoble.cnrs.fr wrote: > On 29 Jun 2004 15:09:43 -0400, Todd Miller wrote > > On Tue, 2004-06-29 at 13:44, Gerard Vermeulen wrote: > > > > > > > > The PEP is attached. It is formatted using the docutils package which > > > > can be used to generate HTML or PDF. Comments and corrections would be > > > > appreciated. > > > > > > > PyQwt is a Python extension which can be conditionally compiled against > > > Numeric and/or numarray (both, one of them or none). > > > > Well that's cool! I'll have to keep the PyQwt guys in mind as potential > > first users. > > > > > Your PEP excludes importing of Numeric and numarray in the same C-extension. > > > > This is true but I don't understand your solution so I'll blather on > > below. > > > > > All you need to do is to hide the macros PyArray_Present(), PyArray_isArray() > > > and import_array() into a few functions with numarray specific names, so > > > that the following becomes possible: > > > > > > #include > > > /* defines the functions (no macros) > > > int Numeric_Present(); > > > int Numeric_isArray(); > > > void import_numeric(); > > > to hide the Numeric C-API stuff in a small Numeric/meta.c file. > > > */ > > > #include > > > /* defines the functions (no macros) > > > int numarray_Present(); > > > int numarray_isArray(); > > > void import_numarray(); > > > to hide the numarray C-API stuff in a small numarray/meta.c file. > > > */ > > > > > > > I may have come up with the wrong scheme for the Present() and > > isArray(). With my scheme, they *have* to be macros because the API > > functions are unsafe: when numarray or Numeric is not present, the API > > function table pointers are NULL so calling through them results in > > either a fatal error or a segfault. > > > The macros can be hidden from the module file scope by wrapping them > in a function (see attached demo) > > > > There is an additional problem that the "same functions" need to be > > called through different API pointers depending on whether numarray > > or Numeric is being supported. Redefinition of typedefs and enumerations > > > > (or perhaps conditional compilation short-circuited re-definitions) may > > also present a problem with compiling (or worse, running). > > > Tested and works. > > > > I certainly like the idea of supporting both in the same extension > > module, but don't see how to get there, other than with separate > > compilation units. With seperate .c files, I'm not aware of a problem > > other than lots of global symbols. I haven't demoed that yet so I am > > interested if someone has made it work. > > > Yes, you cannot mix the numarray API and Numeric API in the same .c > file, but nevertheless you can hide the macros in small functions so > that the macros don't pollute. > > Here you have a litte extension module 'pep' which tries to import Numeric > and numarray. It has a method 'whatsThere' that prints the names of > the Numerical/numarray extension modules it has imported. > > It has also a method 'whatsThis' that prints if an object is a Numeric > array, a numarray array or something else: > > Python 2.3.3 (#2, Feb 17 2004, 11:45:40) > [GCC 3.3.2 (Mandrake Linux 10.0 3.3.2-6mdk)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> import pep > >>> pep.whatsThere() > Numeric! > Numarray! > >>> import Numeric > >>> a = Numeric.arange(10) > >>> pep.whatsThis(a) > Numeric array > >>> import numarray > >>> a = numarray.arange(10) > >>> pep.whatsThis(a) > Numarray array > >>> a = 10 > >>> pep.whatsThis(a) > Something else > >>> > > The MANIFEST reads: > pepmodule.c > setup.py > Numeric/meta.c > Numeric/meta.h > numarray/meta.c > numarray/meta.h > > My initial idea was to add the meta.* files to the Numeric/numarray include > files, but I did not try yet to solve the problem of the > PyArray_API PY_ARRAY_UNIQUE_SYMBOL defines in the same way (I am sure > it can be done. > > Regards -- Gerard -- From haase at msg.ucsf.edu Tue Jun 29 14:53:00 2004 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue Jun 29 14:53:00 2004 Subject: [Numpy-discussion] bug ? in Records arrays in numarray In-Reply-To: <200406281700.37694.haase@msg.ucsf.edu> References: <200406281700.37694.haase@msg.ucsf.edu> Message-ID: <200406291452.24673.haase@msg.ucsf.edu> OK, I'm still trying to get a handle on these record arrays - because I think they are pretty cool, if I could get them to work... Following the code from yesterday (see that posting below) I discovered this: main.ring4ext[0][0] is not the same as main.ring4ext[0,0] is this intended ?? >>> main.ring4ext[0][0] (2308, 76, 272, 1088481152.0, 104.18000030517578, 1994.949951171875) >>> main.ring4ext[0,0] (array([2308, 2309]), array([76, 76]), array([272, 269]), array([ 1.08848115e +09, 1.08848115e+09], type=Float32), array([ 104.18000031, 104.45999908], type=Float32), array([ 1994.94995117, 1994.95996094], type=Float32)) >>> main.ring4ext.shape # yesterday I had this different !!! (20,1) (20, 2) Any comments are appreciated, Thanks Sebastian On Monday 28 June 2004 05:00 pm, Sebastian Haase wrote: > Hi, > I have two record arrays. I was trying to assign one item from one recarray > to > > the other: > >>> omx.zext[0] = main.ring4ext[0,0] > > Traceback (most recent call last): > File "", line 1, in ? > File "X:/PrWin\numarray\records.py", line 744, in _setitem > self.field(self._names[i])[row] = value.field(self._names[i]) > File "C:\Python22\Lib\site-packages\numarray\numarraycore.py", line 619, > in __tonumtype__ > ValueError: Can't use non-rank-0 array as a scalar. > > >>> `omx.zext[0]` > > '' > > >>> `main.ring4ext[0,0]` > > '' > > >>> q = omx.zext[0] > >>> w = main.ring4ext[0,0] > >>> q > > (2097152107, -595656700, 91141, 1.0634608642000868e+037, -1.14841804241843e > +018, 1.2771574333702815e-040) > > >>> w > > (array([428]), array([75]), array([124]), array([ 1.08846451e+09], > type=Float32), array([ 99.25], type=Float32), array([ 1996.82995605], > type=Float32)) > > >>> omx.zext._formats > > ['1Int32', '1Int32', '1Int32', '1Float32', '1Float32', '1Float32'] > > >>> main.ring4ext._formats > > ['1Int32', '1Int32', '1Int32', '1Float32', '1Float32', '1Float32'] > > > I can't track down why one (q) contains scalars and the other (w) constains > arrays (is array([428]) a 'rank-0 array'? ) ! > This is how I generate them: > main.ring4ext = rec.RecArray(main._extHdrRingBuffer, "i4,i4,i4,f4,f4,f4", > shape=(ts,nc), names=("num","min","max","time","mean","darkVal"), > aligned=1) omx.zext = rec.array(formats="i4,i4,i4,f4,f4,f4", > > names=("num","min","max","time","mean","darkVal"),shape=tuple(shapeZ),align >ed=1 ) > > [[BTW: (shapeZ is a list, if I say: ...,shape=shapeZ I get > NameError, "Illegal shape %s" % `shape` from "records.py" in line 462 > -- usually type(shape) == types.ListType should be OK, right ?) > ]] > > > Any ideas? > > Thanks, > Sebastian Haase > > > > ------------------------------------------------------- > This SF.Net email sponsored by Black Hat Briefings & Training. > Attend Black Hat Briefings & Training, Las Vegas July 24-29 - > digital self defense, top technical experts, no vendor pitches, > unmatched networking opportunities. Visit www.blackhat.com > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From jmiller at stsci.edu Tue Jun 29 15:03:02 2004 From: jmiller at stsci.edu (Todd Miller) Date: Tue Jun 29 15:03:02 2004 Subject: [off topic] Re: [Numpy-discussion] numarray speed - PySequence_GetItem In-Reply-To: <40E1CEE6.3050108@cox.net> References: <200406250949.44897.haase@msg.ucsf.edu> <200406281414.25400.haase@msg.ucsf.edu> <1088464671.3744.300.camel@localhost.localdomain> <200406281638.03922.haase@msg.ucsf.edu> <1088520761.17789.184.camel@halloween.stsci.edu> <40E1CEE6.3050108@cox.net> Message-ID: <1088546550.17789.428.camel@halloween.stsci.edu> On Tue, 2004-06-29 at 16:19, Tim Hochberg wrote: > I'd bet a case of beer (or cash equivalent) that one of the main > bottlenecks is the path > PySequence_GetItem->_ndarray_item->_universalIndexing->_simpleIndexing->_simpleIndexingCore. I won't take the bet but if this works out, you get the beer. If it doesn't, well, I don't drink anymore anyway. > The path through _universalIndexing in particular, if I deciphered it > correctly, looks very slow. I don't think it needs to be that way > though, _universalIndexing could probably be sped up, but more promising > I think _ndarray_item could be made to call _simpleIndexingCore without > all that much work. It appears that this would save the creation of > several intermediate objects and it also looks like a couple of calls > back to python! I'm not familiar with this code though, so I could > easily be missing something that makes calling _simpleIndexingCore > harder than it looks. This looks very promising. I'll take a look tomorrow. Regards, Todd From Chris.Barker at noaa.gov Tue Jun 29 16:28:05 2004 From: Chris.Barker at noaa.gov (Chris Barker) Date: Tue Jun 29 16:28:05 2004 Subject: [off topic] Re: [Numpy-discussion] numarray speed - PySequence_GetItem In-Reply-To: <1088544847.17789.393.camel@halloween.stsci.edu> References: <200406250949.44897.haase@msg.ucsf.edu> <200406281414.25400.haase@msg.ucsf.edu> <1088464671.3744.300.camel@localhost.localdomain> <200406281638.03922.haase@msg.ucsf.edu> <1088520761.17789.184.camel@halloween.stsci.edu> <40E1C12E.4060102@cox.net> <40E1D998.8010204@noaa.gov> <1088544847.17789.393.camel@halloween.stsci.edu> Message-ID: <40E1FA6F.9090407@noaa.gov> Todd Miller wrote: > Yes. What I most want to do is a 50000 point drawlines, similar to what > you profiled. Friends are fine too. Actually, it was someone else that did the profiling, but here is a sample, about as simple as I could make it. It draws an N point line and M points. At the moment, it is using Numeric for the points, and numarray for the lines. Numeric is MUCH faster (which is the whole point of this discussion). Otherwise, it takes about the same amount of time to draw the lines as the points. Another note: if use the tolist() method in the numarray first, it's much faster also: dc.DrawLines(LinesPoints.tolist()) Obviously, the tolist method is much faster than wxPython's sequence methods, as would be expected. I'm going to send a note to Robin Dunn about this as well, and see what he thinks. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: DrawLinesTest.py URL: From jmiller at stsci.edu Tue Jun 29 16:51:09 2004 From: jmiller at stsci.edu (Todd Miller) Date: Tue Jun 29 16:51:09 2004 Subject: [Numpy-discussion] bug ? in Records arrays in numarray In-Reply-To: <200406291452.24673.haase@msg.ucsf.edu> References: <200406281700.37694.haase@msg.ucsf.edu> <200406291452.24673.haase@msg.ucsf.edu> Message-ID: <1088552996.3750.4.camel@localhost.localdomain> On Tue, 2004-06-29 at 17:52, Sebastian Haase wrote: > OK, > I'm still trying to get a handle on these record arrays - because I think they > are pretty cool, if I could get them to work... > Following the code from yesterday (see that posting below) I discovered this: > main.ring4ext[0][0] is not the same as main.ring4ext[0,0] > is this intended ?? > > >>> main.ring4ext[0][0] > (2308, 76, 272, 1088481152.0, 104.18000030517578, 1994.949951171875) > >>> main.ring4ext[0,0] > (array([2308, 2309]), array([76, 76]), array([272, 269]), array([ 1.08848115e > +09, 1.08848115e+09], type=Float32), array([ 104.18000031, 104.45999908], > type=Float32), array([ 1994.94995117, 1994.95996094], type=Float32)) > >>> main.ring4ext.shape # yesterday I had this different !!! (20,1) > (20, 2) > > Any comments are appreciated, I talked to JC Hsu, the numarray.records author, and he explained that we're probably looking at a limitation of numarray.records: it doesn't yet handle multi-dimensional arrays of records. JC indicated he had replied to Sebastian, but for the benefit of everyone else, that's the deal. Regards, Todd > Thanks > Sebastian > > > > > On Monday 28 June 2004 05:00 pm, Sebastian Haase wrote: > > Hi, > > I have two record arrays. I was trying to assign one item from one recarray > > to > > > > the other: > > >>> omx.zext[0] = main.ring4ext[0,0] > > > > Traceback (most recent call last): > > File "", line 1, in ? > > File "X:/PrWin\numarray\records.py", line 744, in _setitem > > self.field(self._names[i])[row] = value.field(self._names[i]) > > File "C:\Python22\Lib\site-packages\numarray\numarraycore.py", line 619, > > in __tonumtype__ > > ValueError: Can't use non-rank-0 array as a scalar. > > > > >>> `omx.zext[0]` > > > > '' > > > > >>> `main.ring4ext[0,0]` > > > > '' > > > > >>> q = omx.zext[0] > > >>> w = main.ring4ext[0,0] > > >>> q > > > > (2097152107, -595656700, 91141, 1.0634608642000868e+037, -1.14841804241843e > > +018, 1.2771574333702815e-040) > > > > >>> w > > > > (array([428]), array([75]), array([124]), array([ 1.08846451e+09], > > type=Float32), array([ 99.25], type=Float32), array([ 1996.82995605], > > type=Float32)) > > > > >>> omx.zext._formats > > > > ['1Int32', '1Int32', '1Int32', '1Float32', '1Float32', '1Float32'] > > > > >>> main.ring4ext._formats > > > > ['1Int32', '1Int32', '1Int32', '1Float32', '1Float32', '1Float32'] > > > > > > I can't track down why one (q) contains scalars and the other (w) constains > > arrays (is array([428]) a 'rank-0 array'? ) ! > > This is how I generate them: > > main.ring4ext = rec.RecArray(main._extHdrRingBuffer, "i4,i4,i4,f4,f4,f4", > > shape=(ts,nc), names=("num","min","max","time","mean","darkVal"), > > aligned=1) omx.zext = rec.array(formats="i4,i4,i4,f4,f4,f4", > > > > names=("num","min","max","time","mean","darkVal"),shape=tuple(shapeZ),align > >ed=1 ) > > > > [[BTW: (shapeZ is a list, if I say: ...,shape=shapeZ I get > > NameError, "Illegal shape %s" % `shape` from "records.py" in line 462 > > -- usually type(shape) == types.ListType should be OK, right ?) > > ]] > > > > > > Any ideas? > > > > Thanks, > > Sebastian Haase > > > > > > > > ------------------------------------------------------- > > This SF.Net email sponsored by Black Hat Briefings & Training. > > Attend Black Hat Briefings & Training, Las Vegas July 24-29 - > > digital self defense, top technical experts, no vendor pitches, > > unmatched networking opportunities. Visit www.blackhat.com > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > ------------------------------------------------------- > This SF.Net email sponsored by Black Hat Briefings & Training. > Attend Black Hat Briefings & Training, Las Vegas July 24-29 - > digital self defense, top technical experts, no vendor pitches, > unmatched networking opportunities. Visit www.blackhat.com > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- From haase at msg.ucsf.edu Tue Jun 29 17:01:07 2004 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue Jun 29 17:01:07 2004 Subject: [Numpy-discussion] bug ? in Records arrays in numarray In-Reply-To: <1088552996.3750.4.camel@localhost.localdomain> References: <200406281700.37694.haase@msg.ucsf.edu> <200406291452.24673.haase@msg.ucsf.edu> <1088552996.3750.4.camel@localhost.localdomain> Message-ID: <200406291700.20613.haase@msg.ucsf.edu> On Tuesday 29 June 2004 04:49 pm, Todd Miller wrote: > On Tue, 2004-06-29 at 17:52, Sebastian Haase wrote: > > OK, > > I'm still trying to get a handle on these record arrays - because I think > > they are pretty cool, if I could get them to work... > > Following the code from yesterday (see that posting below) I discovered > > this: main.ring4ext[0][0] is not the same as main.ring4ext[0,0] > > is this intended ?? > > > > >>> main.ring4ext[0][0] > > > > (2308, 76, 272, 1088481152.0, 104.18000030517578, 1994.949951171875) > > > > >>> main.ring4ext[0,0] > > > > (array([2308, 2309]), array([76, 76]), array([272, 269]), array([ > > 1.08848115e +09, 1.08848115e+09], type=Float32), array([ 104.18000031, > > 104.45999908], type=Float32), array([ 1994.94995117, 1994.95996094], > > type=Float32)) > > > > >>> main.ring4ext.shape # yesterday I had this different !!! (20,1) > > > > (20, 2) > > > > Any comments are appreciated, > > I talked to JC Hsu, the numarray.records author, and he explained that > we're probably looking at a limitation of numarray.records: it doesn't > yet handle multi-dimensional arrays of records. JC indicated he had > replied to Sebastian, but for the benefit of everyone else, that's the > deal. > > Regards, > Todd > Thanks, but it actually seems to work well now, as long as I use myRecArray[i][j] instead of myRecArray[i,j]. So, it looks like there is not missing much. I'll keep your "official statement" in mind when I explore this further... Thanks again, Sebastian Haase From haase at msg.ucsf.edu Tue Jun 29 17:08:01 2004 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue Jun 29 17:08:01 2004 Subject: [Numpy-discussion] bug in numarray.maximum.reduce ? Message-ID: <200406291705.55454.haase@msg.ucsf.edu> Hi, Is this a bug?: >>> # (import numarray as na ; 'd' is a 3 dimensional array) >>> d.type() Float32 >>> d[80, 136, 122] 80.3997039795 >>> na.maximum.reduce(d[:,136, 122]) 85.8426361084 >>> na.maximum.reduce(d) [136, 122] 37.3658103943 >>> >>> >>> na.maximum.reduce(d,0)[136, 122] 37.3658103943 >>> na.maximum.reduce(d,1)[136, 122] Traceback (most recent call last): File "", line 1, in ? IndexError: Index out of range I was using na.maximum.reduce(d) to get a "pixelwise" maximum along Z (axis 0). But as seen above it does not get it right. I then tried to reproduce this with some simple arrays, but here it works just fine: >>> a = na.arange(4*4*4) >>> a.shape=(4,4,4) >>> na.maximum.reduce(a) [[48 49 50 51] [52 53 54 55] [56 57 58 59] [60 61 62 63]] >>> a = na.arange(4*4*4).astype(na.Float32) >>> a.shape=(4,4,4) >>> na.maximum.reduce(a) [[ 48. 49. 50. 51.] [ 52. 53. 54. 55.] [ 56. 57. 58. 59.] [ 60. 61. 62. 63.]] >>> Any hint ? Regards, Sebastian Haase From bsder at mail.allcaps.org Tue Jun 29 17:15:42 2004 From: bsder at mail.allcaps.org (Andrew P. Lentvorski, Jr.) Date: Tue Jun 29 17:15:42 2004 Subject: [off topic] Re: [Numpy-discussion] numarray speed - PySequence_GetItem In-Reply-To: <40E1FA6F.9090407@noaa.gov> References: <200406250949.44897.haase@msg.ucsf.edu> <200406281414.25400.haase@msg.ucsf.edu> <1088464671.3744.300.camel@localhost.localdomain> <200406281638.03922.haase@msg.ucsf.edu> <1088520761.17789.184.camel@halloween.stsci.edu> <40E1C12E.4060102@cox.net> <40E1D998.8010204@noaa.gov> <1088544847.17789.393.camel@halloween.stsci.edu> <40E1FA6F.9090407@noaa.gov> Message-ID: <4C65F30B-CA2A-11D8-B6A8-000A95C874EE@mail.allcaps.org> On Jun 29, 2004, at 4:25 PM, Chris Barker wrote: > Todd Miller wrote: >> Yes. What I most want to do is a 50000 point drawlines, similar to >> what >> you profiled. Friends are fine too. > > Actually, it was someone else that did the profiling, but here is a > sample, about as simple as I could make it. > > It draws an N point line and M points. At the moment, it is using > Numeric for the points, and numarray for the lines. Numeric is MUCH > faster (which is the whole point of this discussion). Otherwise, it > takes about the same amount of time to draw the lines as the points. > > Another note: if use the tolist() method in the numarray first, it's > much faster also: > > dc.DrawLines(LinesPoints.tolist()) > If you are going to move lots of lines and points, I would recommend pushing this stuff through PyOpenGL with array objects and vertex objects. Letting OpenGL handle the transformations, clipping, movement and iteration in hardware stomps all over even the best written C code. Most UI toolkits have some form of OpenGL widget. For lots of the code I have written, even Mesa (the open-souce software OpenGL renderer) was fast enough, and not having to write all of the display transformation code by hand was a huge win even when the speed was somewhat lagging. -a From rowen at u.washington.edu Wed Jun 30 08:57:06 2004 From: rowen at u.washington.edu (Russell E Owen) Date: Wed Jun 30 08:57:06 2004 Subject: [Numpy-discussion] bug ? in Records arrays in numarray In-Reply-To: <1088552996.3750.4.camel@localhost.localdomain> References: <200406281700.37694.haase@msg.ucsf.edu> <200406291452.24673.haase@msg.ucsf.edu> <1088552996.3750.4.camel@localhost.localdomain> Message-ID: At 7:49 PM -0400 2004-06-29, Todd Miller wrote: >On Tue, 2004-06-29 at 17:52, Sebastian Haase wrote: >> OK, >> I'm still trying to get a handle on these record arrays - because >>I think they >> are pretty cool, if I could get them to work... >> Following the code from yesterday (see that posting below) I >>discovered this: >> main.ring4ext[0][0] is not the same as main.ring4ext[0,0] >> is this intended ?? >> >> >>> main.ring4ext[0][0] >> (2308, 76, 272, 1088481152.0, 104.18000030517578, 1994.949951171875) >> >>> main.ring4ext[0,0] >> (array([2308, 2309]), array([76, 76]), array([272, 269]), array([ >>1.08848115e >> +09, 1.08848115e+09], type=Float32), array([ 104.18000031, 104.45999908], >> type=Float32), array([ 1994.94995117, 1994.95996094], type=Float32)) >> >>> main.ring4ext.shape # yesterday I had this different !!! (20,1) >> (20, 2) >> >> Any comments are appreciated, > >I talked to JC Hsu, the numarray.records author, and he explained that >we're probably looking at a limitation of numarray.records: it doesn't >yet handle multi-dimensional arrays of records. JC indicated he had >replied to Sebastian, but for the benefit of everyone else, that's the >deal. I agree. I have gotten numarray.records to handle multi-dimensional arrays, but it's a terrible pain to create them, str(arry) fails and setting elements of records arrays is painful. I hope at some point they get a major redesign, as they don't actually seem to have been designed to fit in with numarray. The resulting code was so ugly that I gave up and used multiple identically shaped arrays instead. -- Russell From perry at stsci.edu Wed Jun 30 09:08:16 2004 From: perry at stsci.edu (Perry Greenfield) Date: Wed Jun 30 09:08:16 2004 Subject: [Numpy-discussion] bug ? in Records arrays in numarray In-Reply-To: References: <200406281700.37694.haase@msg.ucsf.edu> <200406291452.24673.haase@msg.ucsf.edu> <1088552996.3750.4.camel@localhost.localdomain> Message-ID: <925A9DEB-CAAF-11D8-80D7-000393989D66@stsci.edu> On Jun 30, 2004, at 11:55 AM, Russell E Owen wrote: > I agree. I have gotten numarray.records to handle multi-dimensional > arrays, but it's a terrible pain to create them, str(arry) fails and > setting elements of records arrays is painful. I hope at some point > they get a major redesign, as they don't actually seem to have been > designed to fit in with numarray. The resulting code was so ugly that > I gave up and used multiple identically shaped arrays instead. > I think we will be taking a look at that soon. I agree that they could be generalized to work better with numarray. Hopefully we will be soliciting comments in the next few weeks about the best way to do that. Perry From tim.hochberg at cox.net Wed Jun 30 12:58:17 2004 From: tim.hochberg at cox.net (Tim Hochberg) Date: Wed Jun 30 12:58:17 2004 Subject: [Numpy-discussion] Speeding up wxPython/numarray Message-ID: <40E31B31.7040105@cox.net> I spend some time seeing what I could do in the way of speeding up wxPoint_LIST_helper by tweaking the numarray code. My first suspect was _universalIndexing by way of _ndarray_item. However, due to some new-style machinations, _ndarray_item was never getting called. Instead, _ndarray_subscript was being called. So, I added a special case to _ndarray_subscript. This sped things up by 50% or so (I don't recall exactly). The code for that is at the end of this message; it's not gauranteed to be 100% correct; it's all experimental. After futzing around some more I figured out a way to trick python into using _ndarray_item. I added "type->tp_as_sequence->sq_item = _ndarray_item;" to _ndarray new. I then optimized _ndarray_item (code at end). This halved the execution time of my arbitrary benchmark. This trick may have horrible, unforseen consequences so use at your own risk. Finally I commented out the __del__ method numarraycore. This resulted in an additional speedup of 64% for a total speed up of 240%. Still not close to 10x, but a large improvement. However, this is obviously not viable for real use, but it's enough of a speedup that I'll try to see if there's anyway to move the shadow stuff back to tp_dealloc. In summary: Version Time Rel Speedup Abs Speedup Stock 0.398 ---- ---- _naarray_item mod 0.192 107% 107% del __del__ 0.117 64% 240% There were a couple of other things I tried that resulted in additional small speedups, but the tactics I used were too horrible to reproduce here. The main one of interest is that all of the calls to NA_updateDataPtr seem to burn some time. However, I don't have any idea what one could do about that. That's all for now. -tim static PyObject* _ndarray_subscript(PyArrayObject* self, PyObject* key) { PyObject *result; #ifdef TAH if (PyInt_CheckExact(key)) { long ikey = PyInt_AsLong(key); long offset; if (NA_getByteOffset(self, 1, &ikey, &offset) < 0) return NULL; if (!NA_updateDataPtr(self)) return NULL; return _simpleIndexingCore(self, offset, 1, Py_None); } #endif #if _PYTHON_CALLBACKS result = PyObject_CallMethod( (PyObject *) self, "_universalIndexing", "(OO)", key, Py_None); #else result = _universalIndexing(self, key, Py_None); #endif return result; } static PyObject * _ndarray_item(PyArrayObject *self, int i) { #ifdef TAH long offset; if (NA_getByteOffset(self, 1, &i, &offset) < 0) return NULL; if (!NA_updateDataPtr(self)) return NULL; return _simpleIndexingCore(self, offset, 1, Py_None); #else PyObject *result; PyObject *key = PyInt_FromLong(i); if (!key) return NULL; result = _universalIndexing(self, key, Py_None); Py_DECREF(key); return result; #endif } From jmiller at stsci.edu Wed Jun 30 14:48:09 2004 From: jmiller at stsci.edu (Todd Miller) Date: Wed Jun 30 14:48:09 2004 Subject: [Numpy-discussion] Speeding up wxPython/numarray In-Reply-To: <40E31B31.7040105@cox.net> References: <40E31B31.7040105@cox.net> Message-ID: <1088632048.7526.204.camel@halloween.stsci.edu> On Wed, 2004-06-30 at 15:57, Tim Hochberg wrote: > I spend some time seeing what I could do in the way of speeding up > wxPoint_LIST_helper by tweaking the numarray code. My first suspect was > _universalIndexing by way of _ndarray_item. However, due to some > new-style machinations, _ndarray_item was never getting called. Instead, > _ndarray_subscript was being called. So, I added a special case to > _ndarray_subscript. This sped things up by 50% or so (I don't recall > exactly). The code for that is at the end of this message; it's not > gauranteed to be 100% correct; it's all experimental. > > After futzing around some more I figured out a way to trick python into > using _ndarray_item. I added "type->tp_as_sequence->sq_item = > _ndarray_item;" to _ndarray new. I'm puzzled why you had to do this. You're using Python-2.3.x, right? There's conditionally compiled code which should be doing this statically. (At least I thought so.) > I then optimized _ndarray_item (code > at end). This halved the execution time of my arbitrary benchmark. This > trick may have horrible, unforseen consequences so use at your own risk. Right now the sq_item hack strikes me as somewhere between completely unnecessary and too scary for me! Maybe if python-dev blessed it. This optimization looks good to me. > Finally I commented out the __del__ method numarraycore. This resulted > in an additional speedup of 64% for a total speed up of 240%. Still not > close to 10x, but a large improvement. However, this is obviously not > viable for real use, but it's enough of a speedup that I'll try to see > if there's anyway to move the shadow stuff back to tp_dealloc. FYI, the issue with tp_dealloc may have to do with which mode Python is compiled in, --with-pydebug, or not. One approach which seems like it ought to work (just thought of this!) is to add an extra reference in C to the NumArray instance __dict__ (from NumArray.__init__ and stashed via a new attribute in the PyArrayObject struct) and then DECREF it as the last part of the tp_dealloc. > In summary: > > Version Time Rel Speedup Abs Speedup > Stock 0.398 ---- ---- > _naarray_item mod 0.192 107% 107% > del __del__ 0.117 64% 240% > > There were a couple of other things I tried that resulted in additional > small speedups, but the tactics I used were too horrible to reproduce > here. The main one of interest is that all of the calls to > NA_updateDataPtr seem to burn some time. However, I don't have any idea > what one could do about that. Francesc Alted had the same comment about NA_updateDataPtr a while ago. I tried to optimize it then but didn't get anywhere. NA_updateDataPtr() should be called at most once per extension function (more is unnecessary but not harmful) but needs to be called at least once as a consequence of the way the buffer protocol doesn't give locked pointers. > That's all for now. > > -tim Well, be picking out your beer. Todd > > > > static PyObject* > _ndarray_subscript(PyArrayObject* self, PyObject* key) > > { > PyObject *result; > #ifdef TAH > if (PyInt_CheckExact(key)) { > long ikey = PyInt_AsLong(key); > long offset; > if (NA_getByteOffset(self, 1, &ikey, &offset) < 0) > return NULL; > if (!NA_updateDataPtr(self)) > return NULL; > return _simpleIndexingCore(self, offset, 1, Py_None); > } > #endif > #if _PYTHON_CALLBACKS > result = PyObject_CallMethod( > (PyObject *) self, "_universalIndexing", "(OO)", key, Py_None); > #else > result = _universalIndexing(self, key, Py_None); > #endif > return result; > } > > > > static PyObject * > _ndarray_item(PyArrayObject *self, int i) > { > #ifdef TAH > long offset; > if (NA_getByteOffset(self, 1, &i, &offset) < 0) > return NULL; > if (!NA_updateDataPtr(self)) > return NULL; > return _simpleIndexingCore(self, offset, 1, Py_None); > #else > PyObject *result; > PyObject *key = PyInt_FromLong(i); > if (!key) return NULL; > result = _universalIndexing(self, key, Py_None); > Py_DECREF(key); > return result; > #endif > } > > > > > > ------------------------------------------------------- > This SF.Net email sponsored by Black Hat Briefings & Training. > Attend Black Hat Briefings & Training, Las Vegas July 24-29 - > digital self defense, top technical experts, no vendor pitches, > unmatched networking opportunities. Visit www.blackhat.com > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- From jmiller at stsci.edu Wed Jun 30 14:55:04 2004 From: jmiller at stsci.edu (Todd Miller) Date: Wed Jun 30 14:55:04 2004 Subject: [Numpy-discussion] Numarray header PEP In-Reply-To: <20040629211800.M55753@grenoble.cnrs.fr> References: <1088451653.3744.200.camel@localhost.localdomain> <20040629194456.44a1fa7f.gerard.vermeulen@grenoble.cnrs.fr> <1088536183.17789.346.camel@halloween.stsci.edu> <20040629211800.M55753@grenoble.cnrs.fr> Message-ID: <1088632459.7526.213.camel@halloween.stsci.edu> On Tue, 2004-06-29 at 17:32, gerard.vermeulen at grenoble.cnrs.fr wrote: > On 29 Jun 2004 15:09:43 -0400, Todd Miller wrote > > On Tue, 2004-06-29 at 13:44, Gerard Vermeulen wrote: > > > > > > > > The PEP is attached. It is formatted using the docutils package which > > > > can be used to generate HTML or PDF. Comments and corrections would be > > > > appreciated. > > > > > > > PyQwt is a Python extension which can be conditionally compiled against > > > Numeric and/or numarray (both, one of them or none). > > > > Well that's cool! I'll have to keep the PyQwt guys in mind as potential > > first users. > > > > > Your PEP excludes importing of Numeric and numarray in the same C-extension. > > > > This is true but I don't understand your solution so I'll blather on > > below. > > > > > All you need to do is to hide the macros PyArray_Present(), PyArray_isArray() > > > and import_array() into a few functions with numarray specific names, so > > > that the following becomes possible: > > > > > > #include > > > /* defines the functions (no macros) > > > int Numeric_Present(); > > > int Numeric_isArray(); > > > void import_numeric(); > > > to hide the Numeric C-API stuff in a small Numeric/meta.c file. > > > */ > > > #include > > > /* defines the functions (no macros) > > > int numarray_Present(); > > > int numarray_isArray(); > > > void import_numarray(); > > > to hide the numarray C-API stuff in a small numarray/meta.c file. > > > */ > > > > > > > I may have come up with the wrong scheme for the Present() and > > isArray(). With my scheme, they *have* to be macros because the API > > functions are unsafe: when numarray or Numeric is not present, the API > > function table pointers are NULL so calling through them results in > > either a fatal error or a segfault. > > > The macros can be hidden from the module file scope by wrapping them > in a function (see attached demo) Your demo is very clear... nice! > > There is an additional problem that the "same functions" need to be > > called through different API pointers depending on whether numarray > > or Numeric is being supported. Redefinition of typedefs and enumerations > > > > (or perhaps conditional compilation short-circuited re-definitions) may > > also present a problem with compiling (or worse, running). > > > Tested and works. > > > > I certainly like the idea of supporting both in the same extension > > module, but don't see how to get there, other than with separate > > compilation units. With seperate .c files, I'm not aware of a problem > > other than lots of global symbols. I haven't demoed that yet so I am > > interested if someone has made it work. > > > Yes, you cannot mix the numarray API and Numeric API in the same .c > file, but nevertheless you can hide the macros in small functions so > that the macros don't pollute. So... you use the "meta" code to provide package specific ordinary (not-macro-fied) functions to keep the different versions of the Present() and isArray() macros from conflicting. It would be nice to have a standard approach for using the same "extension enhancement code" for both numarray and Numeric. The PEP should really be expanded to provide an example of dual support for one complete and real function, guts and all, so people can see the process end-to-end; Something like a simple arrayprint. That process needs to be refined to remove as much tedium and duplication of effort as possible. The idea is to make it as close to providing one implementation to support both array packages as possible. I think it's important to illustrate how to partition the extension module into separate compilation units which correctly navigate the dual implementation mine field in the easiest possible way. It would also be nice to add some logic to the meta-functions so that which array package gets used is configurable. We did something like that for the matplotlib plotting software at the Python level with the "numerix" layer, an idea I think we copied from Chaco. The kind of dispatch I think might be good to support configurability looks like this: PyObject * whatsThis(PyObject *dummy, PyObject *args) { PyObject *result, *what = NULL; if (!PyArg_ParseTuple(args, "O", &what)) return 0; switch(PyArray_Which(what)) { USE_NUMERIC: result = Numeric_whatsThis(what); break; USE_NUMARRAY: result = Numarray_whatsThis(what); break; USE_SEQUENCE: result = Sequence_whatsThis(what); break; } Py_INCREF(Py_None); return Py_None; } In the above, I'm picturing a separate .c file for Numeric_whatsThis and for Numarray_whatsThis. It would be nice to streamline that to one .c and a process which somehow (simply) produces both functions. Or, ideally, the above would be done more like this: PyObject * whatsThis(PyObject *dummy, PyObject *args) { PyObject *result, *what = NULL; if (!PyArg_ParseTuple(args, "O", &what)) return 0; switch(Numerix_Which(what)) { USE_NUMERIX: result = Numerix_whatsThis(what); break; USE_SEQUENCE: result = Sequence_whatsThis(what); break; } Py_INCREF(Py_None); return Py_None; } Here, a common Numerix implementation supports both numarray and Numeric from a single simple .c. The extension module would do "#include numerix/arrayobject.h" and "import_numerix()" and otherwise just call PyArray_* functions. The current stumbling block is that numarray is not binary compatible with Numeric... so numerix in C falls apart. I haven't analyzed every symbol and struct to see if it is really feasible... but it seems like it is *almost* feasible, at least for typical usage. So, in a nutshell, I think the dual implementation support you demoed is important and we should work up an example and kick it around to make sure it's the best way we can think of doing it. Then we should add a section to the PEP describing dual support as well. Regards, Todd From tim.hochberg at cox.net Wed Jun 30 16:02:02 2004 From: tim.hochberg at cox.net (Tim Hochberg) Date: Wed Jun 30 16:02:02 2004 Subject: [Numpy-discussion] Speeding up wxPython/numarray In-Reply-To: <1088632048.7526.204.camel@halloween.stsci.edu> References: <40E31B31.7040105@cox.net> <1088632048.7526.204.camel@halloween.stsci.edu> Message-ID: <40E3462A.9080303@cox.net> Todd Miller wrote: >On Wed, 2004-06-30 at 15:57, Tim Hochberg wrote: > > >>[SNIP] >> >>After futzing around some more I figured out a way to trick python into >>using _ndarray_item. I added "type->tp_as_sequence->sq_item = >>_ndarray_item;" to _ndarray new. >> >> > >I'm puzzled why you had to do this. You're using Python-2.3.x, right? >There's conditionally compiled code which should be doing this >statically. (At least I thought so.) > > By this do you mean the "#if PY_VERSION_HEX >= 0x02030000 " that is wrapped around _ndarray_item? If so, I believe that it *is* getting compiled, it's just never getting called. What I think is happening is that the class NumArray inherits its sq_item from PyClassObject. In particular, I think it picks up instance_item from Objects/classobject.c. This appears to be fairly expensive and, I think, ends up calling tp_as_mapping->mp_subscript. Thus, _ndarray's sq_item slot never gets called. All of this is pretty iffy since I don't know this stuff very well and I didn't trace it all the way through. However, it explains what I've seen thus far. This is why I ended up using the horrible hack. I'm resetting NumArray's sq_item to point to _ndarray_item instead of instance_item. I believe that access at the python level goes through mp_subscrip, so it shouldn't be affected, and only objects at the C level should notice and they should just get the faster sq_item. You, will notice that there are an awful lot of I thinks in the above paragraphs though... >>I then optimized _ndarray_item (code >>at end). This halved the execution time of my arbitrary benchmark. This >>trick may have horrible, unforseen consequences so use at your own risk. >> >> > >Right now the sq_item hack strikes me as somewhere between completely >unnecessary and too scary for me! Maybe if python-dev blessed it. > > Yes, very scary. And it occurs to me that it will break subclasses of NumArray if they override __getitem__. When these subclasses are accessed from C they will see nd_array's sq_item instead of the overridden getitem. However, I think I also know how to fix it. But it does point out that it is very dangerous and there are probably dark corners of which I'm unaware. Asking on Python-List or PyDev would probably be a good idea. The nonscary, but painful, fix would to rewrite NumArray in C. >This optimization looks good to me. > > Unfortunately, I don't think the optimization to sq_item will affect much since NumArray appears to override it with >>Finally I commented out the __del__ method numarraycore. This resulted >>in an additional speedup of 64% for a total speed up of 240%. Still not >>close to 10x, but a large improvement. However, this is obviously not >>viable for real use, but it's enough of a speedup that I'll try to see >>if there's anyway to move the shadow stuff back to tp_dealloc. >> >> > >FYI, the issue with tp_dealloc may have to do with which mode Python is >compiled in, --with-pydebug, or not. One approach which seems like it >ought to work (just thought of this!) is to add an extra reference in C >to the NumArray instance __dict__ (from NumArray.__init__ and stashed >via a new attribute in the PyArrayObject struct) and then DECREF it as >the last part of the tp_dealloc. > > That sounds promising. [SNIP] > >Well, be picking out your beer. > > I was only about half right, so I'm not sure I qualify... -tim From gerard.vermeulen at grenoble.cnrs.fr Wed Jun 30 23:34:01 2004 From: gerard.vermeulen at grenoble.cnrs.fr (gerard.vermeulen at grenoble.cnrs.fr) Date: Wed Jun 30 23:34:01 2004 Subject: [Numpy-discussion] Numarray header PEP In-Reply-To: <1088632459.7526.213.camel@halloween.stsci.edu> References: <1088451653.3744.200.camel@localhost.localdomain> <20040629194456.44a1fa7f.gerard.vermeulen@grenoble.cnrs.fr> <1088536183.17789.346.camel@halloween.stsci.edu> <20040629211800.M55753@grenoble.cnrs.fr> <1088632459.7526.213.camel@halloween.stsci.edu> Message-ID: <20040701053355.M99698@grenoble.cnrs.fr> On 30 Jun 2004 17:54:19 -0400, Todd Miller wrote > > So... you use the "meta" code to provide package specific ordinary > (not-macro-fied) functions to keep the different versions of the > Present() and isArray() macros from conflicting. > > It would be nice to have a standard approach for using the same > "extension enhancement code" for both numarray and Numeric. The PEP > should really be expanded to provide an example of dual support for one > complete and real function, guts and all, so people can see the process > end-to-end; Something like a simple arrayprint. That process needs > to be refined to remove as much tedium and duplication of effort as > possible. The idea is to make it as close to providing one > implementation to support both array packages as possible. I think it's > important to illustrate how to partition the extension module into > separate compilation units which correctly navigate the dual > implementation mine field in the easiest possible way. > > It would also be nice to add some logic to the meta-functions so that > which array package gets used is configurable. We did something like > that for the matplotlib plotting software at the Python level with > the "numerix" layer, an idea I think we copied from Chaco. The kind > of dispatch I think might be good to support configurability looks like > this: > > PyObject * > whatsThis(PyObject *dummy, PyObject *args) > { > PyObject *result, *what = NULL; > if (!PyArg_ParseTuple(args, "O", &what)) > return 0; > switch(PyArray_Which(what)) { > USE_NUMERIC: > result = Numeric_whatsThis(what); break; > USE_NUMARRAY: > result = Numarray_whatsThis(what); break; > USE_SEQUENCE: > result = Sequence_whatsThis(what); break; > } > Py_INCREF(Py_None); > return Py_None; > } > > In the above, I'm picturing a separate .c file for Numeric_whatsThis > and for Numarray_whatsThis. It would be nice to streamline that to one > .c and a process which somehow (simply) produces both functions. > > Or, ideally, the above would be done more like this: > > PyObject * > whatsThis(PyObject *dummy, PyObject *args) > { > PyObject *result, *what = NULL; > if (!PyArg_ParseTuple(args, "O", &what)) > return 0; > switch(Numerix_Which(what)) { > USE_NUMERIX: > result = Numerix_whatsThis(what); break; > USE_SEQUENCE: > result = Sequence_whatsThis(what); break; > } > Py_INCREF(Py_None); > return Py_None; > } > > Here, a common Numerix implementation supports both numarray and Numeric > from a single simple .c. The extension module would do "#include > numerix/arrayobject.h" and "import_numerix()" and otherwise just call > PyArray_* functions. > > The current stumbling block is that numarray is not binary compatible > with Numeric... so numerix in C falls apart. I haven't analyzed > every symbol and struct to see if it is really feasible... but it > seems like it is *almost* feasible, at least for typical usage. > > So, in a nutshell, I think the dual implementation support you > demoed is important and we should work up an example and kick it > around to make sure it's the best way we can think of doing it. > Then we should add a section to the PEP describing dual support as well. > I would never apply numarray code to Numeric arrays and the inverse. It looks dangerous and I do not know if it is possible. The first thing coming to mind is that numarray and Numeric arrays refer to different type objects (this is what my pep module uses to differentiate them). So, even if numarray and Numeric are binary compatible, any 'alien' code referring the the 'Python-standard part' of the type objects may lead to surprises. A PEP proposing hacks will raise eyebrows at least. Secondly, most people use Numeric *or* numarray and not both. So, I prefer: Numeric In => Numeric Out or Numarray In => Numarray Out (NINO) Of course, Numeric or numarray output can be a user option if NINO does not apply. (explicit safe conversion between Numeric and numarray is possible if really needed). I'll try to flesh out the demo with real functions in the way you indicated (going as far as I consider safe). The problem of coding the Numeric (or numarray) functions in more than a single source file has also be addressed. It may take 2 weeks because I am off to a conference next week. Regards -- Gerard From nadavh at visionsense.com Tue Jun 1 06:32:05 2004 From: nadavh at visionsense.com (Nadav Horesh) Date: Tue Jun 1 06:32:05 2004 Subject: [Numpy-discussion] searchsorted Message-ID: <40BC84A5.2030402@visionsense.com> I am currently working on a simulation that makes a heavy use of searchsorted. But it does not precisely fit to what I need --- if a value v is between p and q searchsorted returns the index of q, while what I need is the index of p. Currently my solution is to turn to floating points numbers: ====================================== Python 2.3.4 (#1, May 31 2004, 09:13:03) [GCC 3.4.0] on linux2 Type "help", "copyright", "credits" or "license" for more information. from numarray import * bins = array((0,10,20,30)) val = array((10, 15)) searchsorted(bins, val) array([1, 2]) # I really would like to get array([1, 1]) # Here is the trick: fb = bins - 0.1 fb array([ -0.1, 9.9, 19.9, 29.9]) searchsorted(fb, val) - 1 array([1, 1]) # That's it! ============================================ My questions are: 1. Is there a more elegant solution? 2. I am thinking of letting "searchsorted" return a second boolean array which has the value True for every exact match: >>> searchsorted(bins, val) >>> [array([1, 2]), array([1, 0], type=Bool)] Any comments? Nadav. From perry at stsci.edu Tue Jun 1 09:17:00 2004 From: perry at stsci.edu (Perry Greenfield) Date: Tue Jun 1 09:17:00 2004 Subject: [Numpy-discussion] searchsorted In-Reply-To: <40BC84A5.2030402@visionsense.com> Message-ID: Nadav Horesh writes > I am currently working on a simulation that makes a heavy use of > searchsorted. But it does not precisely fit to what I need --- if a > value v is between p and q searchsorted returns the index of q, while > what I need is the index of p. > > Currently my solution is to turn to floating points numbers: > > ====================================== > > Python 2.3.4 (#1, May 31 2004, 09:13:03) > [GCC 3.4.0] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > from numarray import * > > bins = array((0,10,20,30)) > val = array((10, 15)) > searchsorted(bins, val) > array([1, 2]) # I really would like to get array([1, 1]) > > # Here is the trick: > > fb = bins - 0.1 > fb > array([ -0.1, 9.9, 19.9, 29.9]) > > searchsorted(fb, val) - 1 > array([1, 1]) # That's it! > This is only approximate, right? If val = array([9.95, 15]) you will get the wrong answer won't you? > ============================================ > > My questions are: > > 1. Is there a more elegant solution? > 2. I am thinking of letting "searchsorted" return a second boolean > array which has the value True for every exact match: > >>> searchsorted(bins, val) > >>> [array([1, 2]), array([1, 0], type=Bool)] > Any comments? > > Nadav. > To get the latter, you could so something like ind = searchsorted(bins, val) neq_mask = bins[ind]-val ind[neq_mask] -= 1. # well, you need to handle where ind = 0 and # is not equal as well Would that suffice? Perry From postmaster at framatome-anp.com Wed Jun 2 00:00:04 2004 From: postmaster at framatome-anp.com (System Administrator) Date: Wed Jun 2 00:00:04 2004 Subject: [Numpy-discussion] Undeliverable: Re: Your bill Message-ID: <72B401374280BA4897AF65F8853386C067884C@fpari01mxb.di.framatome.fr> Your message To: jacques.heliot at framatome-anp.com Subject: Re: Your bill Sent: Wed, 2 Jun 2004 08:58:40 +0200 did not reach the following recipient(s): jacques.heliot at framail.framatome-anp.com on Wed, 2 Jun 2004 08:58:45 +0200 The recipient name is not recognized The MTS-ID of the original message is: c=fr;a= ;p=fragroup;l=FPARI01MXB0406020658LWFQLW5H MSEXCH:IMS:FRAGROUP:FRAANP-FR-PARIS-PARIS:FPARI01MXB 0 (000C05A6) Unknown Recipient -------------- next part -------------- An embedded message was scrubbed... From: unknown sender Subject: no subject Date: no date Size: 38 URL: From numpy-discussion at lists.sourceforge.net Wed Jun 2 02:58:40 2004 From: numpy-discussion at lists.sourceforge.net (numpy-discussion at lists.sourceforge.net) Date: Wed, 2 Jun 2004 08:58:40 +0200 Subject: Your bill Message-ID: <200406020651.i526pUxl077148@mx.framatome-anp.com> ------------------ Virus Warning Message (on octopussy) Found virus WORM_NETSKY.D in file your_bill.pif The uncleanable file is deleted. --------------------------------------------------------- -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: ATT262686.txt URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: ATT262687.txt URL: From nadavh at visionsense.com Wed Jun 2 03:16:06 2004 From: nadavh at visionsense.com (Nadav Horesh) Date: Wed Jun 2 03:16:06 2004 Subject: [Numpy-discussion] searchsorted Message-ID: <07C6A61102C94148B8104D42DE95F7E86DED21@exchange2k.envision.co.il> -----Original Message----- From: Perry Greenfield [mailto:perry at stsci.edu] Sent: Tue 01-Jun-04 19:16 To: Nadav Horesh; numpy-discussion Cc: Subject: RE: [Numpy-discussion] searchsorted Nadav Horesh writes > I am currently working on a simulation that makes a heavy use of > searchsorted. But it does not precisely fit to what I need --- if a > value v is between p and q searchsorted returns the index of q, while > what I need is the index of p. > > Currently my solution is to turn to floating points numbers: > > ====================================== > > Python 2.3.4 (#1, May 31 2004, 09:13:03) > [GCC 3.4.0] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > from numarray import * > > bins = array((0,10,20,30)) > val = array((10, 15)) > searchsorted(bins, val) > array([1, 2]) # I really would like to get array([1, 1]) > > # Here is the trick: > > fb = bins - 0.1 > fb > array([ -0.1, 9.9, 19.9, 29.9]) > > searchsorted(fb, val) - 1 > array([1, 1]) # That's it! > This is only approximate, right? If val = array([9.95, 15]) you will get the wrong answer won't you? > ============================================ > > My questions are: > > 1. Is there a more elegant solution? > 2. I am thinking of letting "searchsorted" return a second boolean > array which has the value True for every exact match: > >>> searchsorted(bins, val) > >>> [array([1, 2]), array([1, 0], type=Bool)] > Any comments? > > Nadav. > To get the latter, you could so something like ind = searchsorted(bins, val) neq_mask = bins[ind]-val ind[neq_mask] -= 1. # well, you need to handle where ind = 0 and # is not equal as well Would that suffice? Perry ------------------------------------- Got the idea, the line should be really: ind = ind - (bins[ind] != val) You helped a lot. Thank you very much, Nadav. From karthik at james.hut.fi Wed Jun 2 07:28:03 2004 From: karthik at james.hut.fi (Karthikesh Raju) Date: Wed Jun 2 07:28:03 2004 Subject: [Numpy-discussion] Random Numbers Message-ID: Hi All, How close is the random number generation from numarray.random_array.normal(0,1,x) or numarray.linear_algebra.mlab.randn(x) to matlab's randn? i am having problems with an identical program written in matlab and python, with the results entirely different in both cases :( Warm regards karthik ----------------------------------------------------------------------- Karthikesh Raju, email: karthik at james.hut.fi Researcher, http://www.cis.hut.fi/karthik Helsinki University of Technology, Tel: +358-9-451 5389 Laboratory of Comp. & Info. Sc., Fax: +358-9-451 3277 Department of Computer Sc., P.O Box 5400, FIN 02015 HUT, Espoo, FINLAND ----------------------------------------------------------------------- From perry at stsci.edu Wed Jun 2 07:46:05 2004 From: perry at stsci.edu (Perry Greenfield) Date: Wed Jun 2 07:46:05 2004 Subject: [Numpy-discussion] Random Numbers In-Reply-To: Message-ID: Karthikesh Raju wrote: > How close is the random number generation from > numarray.random_array.normal(0,1,x) or > numarray.linear_algebra.mlab.randn(x) to matlab's randn? > > i am having problems with an identical program written in matlab and > python, with the results entirely different in both cases :( > Unfortunately, I'm not familiar with matlab and don't have it here to run. Have you tried comparing simple things between the two such as mean and standard deviation to see if you are getting the same result, followed by histograms if you see no differences there? Perry Greenfield From karthik at james.hut.fi Wed Jun 2 08:05:09 2004 From: karthik at james.hut.fi (Karthikesh Raju) Date: Wed Jun 2 08:05:09 2004 Subject: [Numpy-discussion] Random Numbers In-Reply-To: References: Message-ID: Hi Perry, Yes, i have been trying to compare random number generators. i plotted the histograms, they look quite "gaussian", i have also tried uniform numbers with "rand". May be there is some problem with one of my statements like: def AWGN(Tx_Symbols,No): len_Waveform = len(Tx_Waveform) std_No = sqrt(No/2.0) return Tx_Waveform + std_No*randn(len_Waveform) This function just takes in a bit stream in [1,-1] and adds gaussian noise. No defines the noise power. Now in the main function i am doing: Tx_Bit_Stream = random(n_Bits) > 0.5 Tx_Symbols = 2*Tx_Bit_Stream-1 Rx_Symbols = AWGN(Tx_Symbols, No) Rx_Bit_Stream = Rx_Symbols > 0 n_Errors = sum(not_equal(Tx_Bit_Stream,Rx_Bit_Stream)) Now i have noted that n_Errors is negative? How can that be so? n_Errors should be the number of places where Tx_Bit_Stream differs from Rx_Bit_Stream? How can the sum be negative? With warm regards karthik ----------------------------------------------------------------------- Karthikesh Raju, email: karthik at james.hut.fi Researcher, http://www.cis.hut.fi/karthik Helsinki University of Technology, Tel: +358-9-451 5389 Laboratory of Comp. & Info. Sc., Fax: +358-9-451 3277 Department of Computer Sc., P.O Box 5400, FIN 02015 HUT, Espoo, FINLAND ----------------------------------------------------------------------- On Wed, 2 Jun 2004, Perry Greenfield wrote: > Karthikesh Raju wrote: > > > How close is the random number generation from > > numarray.random_array.normal(0,1,x) or > > numarray.linear_algebra.mlab.randn(x) to matlab's randn? > > > > i am having problems with an identical program written in matlab and > > python, with the results entirely different in both cases :( > > > Unfortunately, I'm not familiar with matlab and don't have it here > to run. Have you tried comparing simple things between the two > such as mean and standard deviation to see if you are getting > the same result, followed by histograms if you see no differences > there? > > Perry Greenfield > > From southey at uiuc.edu Wed Jun 2 08:18:02 2004 From: southey at uiuc.edu (Bruce Southey) Date: Wed Jun 2 08:18:02 2004 Subject: [Numpy-discussion] Random Numbers Message-ID: <26daf8a1.4a3153d6.81ac900@expms6.cites.uiuc.edu> Hi, All three use different but well-known algorithms For Matlab 5 onwards, this references the randn (which is the standard normal): http://www.mathworks.com/company/newsletters/news_notes/clevescorner/spring01_cleve.html (You also note the link to Matlabs uniform generator that has excellent properties.) numarray.random_array.normal that uses ranlib snorm (snorm is the standard normal). numarray.linear_algebra.mlab.randn uses the Box-Muller method using random uniform numbers from ranlib. Your problems suggest that randn is not the cause. Without any code or what you want to do it hard to address your question except that you should ensure that your sampling does provide the normal distribution with your parameters. By that I mean draw many, many samples from one set of parameters and check the estimated mean and variance. Regards Bruce ---- Original message ---- >Date: Wed, 2 Jun 2004 17:27:02 +0300 >From: Karthikesh Raju >Subject: [Numpy-discussion] Random Numbers >To: numpy-discussion at lists.sourceforge.net > >Hi All, > >How close is the random number generation from >numarray.random_array.normal(0,1,x) or >numarray.linear_algebra.mlab.randn(x) to matlab's randn? > >i am having problems with an identical program written in matlab and >python, with the results entirely different in both cases :( > >Warm regards > >karthik > > >----------------------------------------------------------------------- >Karthikesh Raju, email: karthik at james.hut.fi >Researcher, http://www.cis.hut.fi/karthik >Helsinki University of Technology, Tel: +358-9-451 5389 >Laboratory of Comp. & Info. Sc., Fax: +358-9-451 3277 >Department of Computer Sc., >P.O Box 5400, FIN 02015 HUT, >Espoo, FINLAND >----------------------------------------------------------------------- > > >------------------------------------------------------- >This SF.Net email is sponsored by the new InstallShield X. >From Windows to Linux, servers to mobile, InstallShield X is the one >installation-authoring solution that does it all. Learn more and >evaluate today! http://www.installshield.com/Dev2Dev/0504 >_______________________________________________ >Numpy-discussion mailing list >Numpy-discussion at lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/numpy-discussion From karthik at james.hut.fi Wed Jun 2 08:29:03 2004 From: karthik at james.hut.fi (Karthikesh Raju) Date: Wed Jun 2 08:29:03 2004 Subject: [Numpy-discussion] Random Numbers In-Reply-To: <26daf8a1.4a3153d6.81ac900@expms6.cites.uiuc.edu> References: <26daf8a1.4a3153d6.81ac900@expms6.cites.uiuc.edu> Message-ID: Hi Bruce and All, Probably i jumped the wagon and thought the problem was with randn. Think i have isolated the problem: say: a = rand(10000) > 0.5 b = rand(10000) > 0.5 now to compare and find the number of differences or errors: sum(a!=b) or sum(not_equal(a,b)) How can this value be negative at times? With warm regards karthik ----------------------------------------------------------------------- Karthikesh Raju, email: karthik at james.hut.fi Researcher, http://www.cis.hut.fi/karthik Helsinki University of Technology, Tel: +358-9-451 5389 Laboratory of Comp. & Info. Sc., Fax: +358-9-451 3277 Department of Computer Sc., P.O Box 5400, FIN 02015 HUT, Espoo, FINLAND ----------------------------------------------------------------------- On Wed, 2 Jun 2004, Bruce Southey wrote: > Hi, > All three use different but well-known algorithms > > For Matlab 5 onwards, this references the randn (which is the standard normal): > http://www.mathworks.com/company/newsletters/news_notes/clevescorner/spring01_cleve.html > > (You also note the link to Matlabs uniform generator that has excellent > properties.) > > numarray.random_array.normal that uses ranlib snorm (snorm is the standard normal). > > numarray.linear_algebra.mlab.randn uses the Box-Muller method using random > uniform numbers from ranlib. > > Your problems suggest that randn is not the cause. Without any code or what you > want to do it hard to address your question except that you should ensure that > your sampling does provide the normal distribution with your parameters. By that > I mean draw many, many samples from one set of parameters and check the > estimated mean and variance. > > Regards > Bruce > > ---- Original message ---- > >Date: Wed, 2 Jun 2004 17:27:02 +0300 > >From: Karthikesh Raju > >Subject: [Numpy-discussion] Random Numbers > >To: numpy-discussion at lists.sourceforge.net > > > >Hi All, > > > >How close is the random number generation from > >numarray.random_array.normal(0,1,x) or > >numarray.linear_algebra.mlab.randn(x) to matlab's randn? > > > >i am having problems with an identical program written in matlab and > >python, with the results entirely different in both cases :( > > > >Warm regards > > > >karthik > > > > > >----------------------------------------------------------------------- > >Karthikesh Raju, email: karthik at james.hut.fi > >Researcher, http://www.cis.hut.fi/karthik > >Helsinki University of Technology, Tel: +358-9-451 5389 > >Laboratory of Comp. & Info. Sc., Fax: +358-9-451 3277 > >Department of Computer Sc., > >P.O Box 5400, FIN 02015 HUT, > >Espoo, FINLAND > >----------------------------------------------------------------------- > > > > > >------------------------------------------------------- > >This SF.Net email is sponsored by the new InstallShield X. > >From Windows to Linux, servers to mobile, InstallShield X is the one > >installation-authoring solution that does it all. Learn more and > >evaluate today! http://www.installshield.com/Dev2Dev/0504 > >_______________________________________________ > >Numpy-discussion mailing list > >Numpy-discussion at lists.sourceforge.net > >https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > From perry at stsci.edu Wed Jun 2 08:35:08 2004 From: perry at stsci.edu (Perry Greenfield) Date: Wed Jun 2 08:35:08 2004 Subject: [Numpy-discussion] Random Numbers In-Reply-To: Message-ID: Karthikesh Raju wrote: > > n_Errors = sum(not_equal(Tx_Bit_Stream,Rx_Bit_Stream)) > > Now i have noted that n_Errors is negative? How can that be so? > > n_Errors should be the number of places where Tx_Bit_Stream differs from > Rx_Bit_Stream? How can the sum be negative? > > I'm guessing you are seeing overflow problems with the boolean representation which is the result of logical comparisons (which uses only 8 bit values) try: sum(float(not_equal... to eliminate that problem. Perry From karthik at james.hut.fi Wed Jun 2 08:42:14 2004 From: karthik at james.hut.fi (Karthikesh Raju) Date: Wed Jun 2 08:42:14 2004 Subject: [Numpy-discussion] Random Numbers In-Reply-To: References: Message-ID: Hi Perry and All, Thankx a lot for your reply. i actually forgot to mention it, but a = rand(100000) > 0.5 b = rand(100000) > 0.5 sum(float(a,b)) results in a TypeError: Only rank-0 numarray can be cast to floats. With warm regards karthik ----------------------------------------------------------------------- Karthikesh Raju, email: karthik at james.hut.fi Researcher, http://www.cis.hut.fi/karthik Helsinki University of Technology, Tel: +358-9-451 5389 Laboratory of Comp. & Info. Sc., Fax: +358-9-451 3277 Department of Computer Sc., P.O Box 5400, FIN 02015 HUT, Espoo, FINLAND ----------------------------------------------------------------------- On Wed, 2 Jun 2004, Perry Greenfield wrote: > Karthikesh Raju wrote: > > > > n_Errors = sum(not_equal(Tx_Bit_Stream,Rx_Bit_Stream)) > > > > Now i have noted that n_Errors is negative? How can that be so? > > > > n_Errors should be the number of places where Tx_Bit_Stream differs from > > Rx_Bit_Stream? How can the sum be negative? > > > > > I'm guessing you are seeing overflow problems with the boolean > representation which is the result of logical comparisons > (which uses only 8 bit values) try: > > sum(float(not_equal... > > to eliminate that problem. > > Perry > > From perry at stsci.edu Wed Jun 2 08:43:02 2004 From: perry at stsci.edu (Perry Greenfield) Date: Wed Jun 2 08:43:02 2004 Subject: [Numpy-discussion] Random Numbers In-Reply-To: Message-ID: > I'm guessing you are seeing overflow problems with the boolean > representation which is the result of logical comparisons > (which uses only 8 bit values) try: > > sum(float(not_equal... > > to eliminate that problem. > > Perry > arg, I should have said: sum(not_equal(...).astype(Float)) From karthik at james.hut.fi Wed Jun 2 09:11:00 2004 From: karthik at james.hut.fi (Karthikesh Raju) Date: Wed Jun 2 09:11:00 2004 Subject: [Numpy-discussion] Random Numbers In-Reply-To: References: Message-ID: Thankx Perry, now i an getting somewhere near, but there are still some issues, matlab and python (exact algorithms) result in different values :(, looking at it though, Warm regards karthik ----------------------------------------------------------------------- Karthikesh Raju, email: karthik at james.hut.fi Researcher, http://www.cis.hut.fi/karthik Helsinki University of Technology, Tel: +358-9-451 5389 Laboratory of Comp. & Info. Sc., Fax: +358-9-451 3277 Department of Computer Sc., P.O Box 5400, FIN 02015 HUT, Espoo, FINLAND ----------------------------------------------------------------------- On Wed, 2 Jun 2004, Perry Greenfield wrote: > > I'm guessing you are seeing overflow problems with the boolean > > representation which is the result of logical comparisons > > (which uses only 8 bit values) try: > > > > sum(float(not_equal... > > > > to eliminate that problem. > > > > Perry > > > arg, I should have said: > > sum(not_equal(...).astype(Float)) > > From jdhunter at ace.bsd.uchicago.edu Wed Jun 2 09:28:03 2004 From: jdhunter at ace.bsd.uchicago.edu (John Hunter) Date: Wed Jun 2 09:28:03 2004 Subject: [Numpy-discussion] Random Numbers In-Reply-To: (Karthikesh Raju's message of "Wed, 2 Jun 2004 19:10:17 +0300") References: Message-ID: >>>>> "Karthikesh" == Karthikesh Raju writes: Karthikesh> Thankx Perry, now i an getting somewhere near, but Karthikesh> there are still some issues, matlab and python (exact Karthikesh> algorithms) result in different values :(, looking at Karthikesh> it though, One obvious difference, which you are likely aware of, is that rand(N) in matlab returns an NxN matrix whereas Numeric's MLab and numarray's linear_algebra.mlab version of rand returns a length N array. So if you naively use rand(N) in both cases, your sample sizes will be wildly different. Just a thought... JDH From karthik at james.hut.fi Wed Jun 2 09:30:03 2004 From: karthik at james.hut.fi (Karthikesh Raju) Date: Wed Jun 2 09:30:03 2004 Subject: [Numpy-discussion] Random Numbers In-Reply-To: References: Message-ID: Yes, that was something i knew, actually the real problem as Perry, they are a result of overflows due to bools .. With warm regards Karthik ----------------------------------------------------------------------- Karthikesh Raju, email: karthik at james.hut.fi Researcher, http://www.cis.hut.fi/karthik Helsinki University of Technology, Tel: +358-9-451 5389 Laboratory of Comp. & Info. Sc., Fax: +358-9-451 3277 Department of Computer Sc., P.O Box 5400, FIN 02015 HUT, Espoo, FINLAND ----------------------------------------------------------------------- On Wed, 2 Jun 2004, John Hunter wrote: > >>>>> "Karthikesh" == Karthikesh Raju writes: > > Karthikesh> Thankx Perry, now i an getting somewhere near, but > Karthikesh> there are still some issues, matlab and python (exact > Karthikesh> algorithms) result in different values :(, looking at > Karthikesh> it though, > > One obvious difference, which you are likely aware of, is that rand(N) > in matlab returns an NxN matrix whereas Numeric's MLab and numarray's > linear_algebra.mlab version of rand returns a length N array. So if > you naively use rand(N) in both cases, your sample sizes will be > wildly different. > > Just a thought... > > JDH > > From karthik at james.hut.fi Wed Jun 2 09:38:08 2004 From: karthik at james.hut.fi (Karthikesh Raju) Date: Wed Jun 2 09:38:08 2004 Subject: [Numpy-discussion] erfc Message-ID: Hi All, Thankx to everyone for my previous "Random Numbers" question. Ok, do we have the complementary error function in numarray? i mean erfc(z) = 1 - erf(z) = \frac{2}{\pi} \int_z^\infy e^{t^2}dt (just used LaTeX notation, ran out of ideas to represent it otherwise) Warm regards Karthik ----------------------------------------------------------------------- Karthikesh Raju, email: karthik at james.hut.fi Researcher, http://www.cis.hut.fi/karthik Helsinki University of Technology, Tel: +358-9-451 5389 Laboratory of Comp. & Info. Sc., Fax: +358-9-451 3277 Department of Computer Sc., P.O Box 5400, FIN 02015 HUT, Espoo, FINLAND ----------------------------------------------------------------------- From jdhunter at ace.bsd.uchicago.edu Wed Jun 2 09:47:06 2004 From: jdhunter at ace.bsd.uchicago.edu (John Hunter) Date: Wed Jun 2 09:47:06 2004 Subject: [Numpy-discussion] erfc In-Reply-To: (Karthikesh Raju's message of "Wed, 2 Jun 2004 19:37:17 +0300") References: Message-ID: >>>>> "GKarthikesh" == Karthikesh Raju writes: GKarthikesh> Hi All, Thankx to everyone for my previous "Random GKarthikesh> Numbers" question. Ok, do we have the complementary GKarthikesh> error function in numarray? As fas as I know, you'll need to use scipy for that In [3]: import scipy.stats In [4]: scipy.stats.erfc? Type: ufunc String Form: Namespace: Interactive Docstring: y=erfc(x) returns 1 - erf(x). or use pyrex or some other simple wrapper generator to wrap your math library's erfc function... JDH From a_wilson at mit.edu Wed Jun 2 11:49:01 2004 From: a_wilson at mit.edu (Andrew Wilson) Date: Wed Jun 2 11:49:01 2004 Subject: [Numpy-discussion] numarray.nd_image.label() help Message-ID: <40BDCCBE.9030000@mit.edu> Hello, I'm having some trouble with the numarray.nd_image.label() function as it is not connecting two objects (that are adjacent) on some images. I have tried using the struct: [[1,1,1] struct = [1,1,1] [1,1,1]] this does change how it is connecting various objects in my image (as compared to the default struct) but it still splits the objects eventhough they are touching. Any hints or clues? I could send the pictures, just don't want to post them to the entire list. Thanks for your help Andrew From rowen at u.washington.edu Wed Jun 2 12:43:03 2004 From: rowen at u.washington.edu (Russell E Owen) Date: Wed Jun 2 12:43:03 2004 Subject: [Numpy-discussion] numarray.nd_image.label() help In-Reply-To: <40BDCCBE.9030000@mit.edu> References: <40BDCCBE.9030000@mit.edu> Message-ID: At 2:49 PM +0200 2004-06-02, Andrew Wilson wrote: >Hello, > I'm having some trouble with the numarray.nd_image.label() >function as it is not connecting two objects (that are adjacent) on >some images. I have tried using the struct: > [[1,1,1] >struct = [1,1,1] > [1,1,1]] > >this does change how it is connecting various objects in my image >(as compared to the default struct) but it still splits the objects >eventhough they are touching. Any hints or clues? I could send the >pictures, just don't want to post them to the entire list. I use it and have not (yet) noticed this, so I hope it's something you're doing and not a bug. You could try generating a boolean version of your image, examine that to make sure the objects are still adjacent on that and use label on it. Here's my code: import numarray as num # smoothed data is a median-filtered version of my image # dataCut is a value above which pixels are considered data shapeArry = num.ones((3,3)) labels, numElts = num.nd_image.label(smoothedData>dataCut, shapeArry) If this doesn't do it (and nobody has anything better to offer), please put the image up somewhere that we can access it and show us your code. -- Russell From verveer at embl-heidelberg.de Wed Jun 2 13:38:05 2004 From: verveer at embl-heidelberg.de (Peter Verveer) Date: Wed Jun 2 13:38:05 2004 Subject: [Numpy-discussion] numarray.nd_image.label() help In-Reply-To: <40BDCCBE.9030000@mit.edu> References: <40BDCCBE.9030000@mit.edu> Message-ID: Hi Andrew, If you use the latest release or an older version you may have run into a bug that displayed that behaviour. I fixed that in the CVS version, so please try to use that if you are not already doing that. If you are using the CVS version and still get the problem, please show me the smallest possible array that displays the problem, and I will fix it. Cheers, Peter On Jun 2, 2004, at 2:49 PM, Andrew Wilson wrote: > Hello, > I'm having some trouble with the numarray.nd_image.label() function > as it is not connecting two objects (that are adjacent) on some > images. I have tried using the struct: > [[1,1,1] > struct = [1,1,1] > [1,1,1]] > > this does change how it is connecting various objects in my image (as > compared to the default struct) but it still splits the objects > eventhough they are touching. Any hints or clues? I could send the > pictures, just don't want to post them to the entire list. From Fernando.Perez at colorado.edu Wed Jun 2 17:53:08 2004 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Wed Jun 2 17:53:08 2004 Subject: [Numpy-discussion] Building rpm for 23.1 fails at the end Message-ID: <40BE765D.6020706@colorado.edu> Hi all, I just tried: root at planck[Numeric-23.1]# ./setup.py build bdist --formats=rpm and after a while I got this traceback: moving build/bdist.linux-i686/rpm/SRPMS/Numeric-23.1-1.src.rpm -> dist Traceback (most recent call last): File "./setup.py", line 182, in ? ext_modules = ext_modules File "/usr/src/build/394694-i386/install/usr/lib/python2.3/distutils/core.py", line 149, in setup File "/usr/src/build/394694-i386/install/usr/lib/python2.3/distutils/dist.py", line 907, in run_commands File "/usr/src/build/394694-i386/install/usr/lib/python2.3/distutils/dist.py", line 927, in run_command File "/usr/src/build/394694-i386/install/usr/lib/python2.3/distutils/command/bdist.py", line 146, in run File "/usr/lib/python2.3/cmd.py", line 333, in run_command File "/usr/src/build/394694-i386/install/usr/lib/python2.3/distutils/dist.py", line 927, in run_command File "/usr/src/build/394694-i386/install/usr/lib/python2.3/distutils/command/bdist_rpm.py", line 316, in run AssertionError: unexpected number of RPM files found: ['build/bdist.linux-i686/rpm/RPMS/i386/Numeric-23.1-1.i386.rpm', 'build/bdist.linux-i686/rpm/RPMS/i386/Numeric-debuginfo-23.1-1.i386.rpm'] As it turns out, the rpm DID get correctly built, but the -debuginfo- one is confusing setup.py. I simply copied the rpm by hand and moved on, but someone who knows more about distutils than me might want to look into this. Cheers, f From janne.sinkkonen at hut.fi Thu Jun 3 04:20:06 2004 From: janne.sinkkonen at hut.fi (Janne Sinkkonen) Date: Thu Jun 3 04:20:06 2004 Subject: [Numpy-discussion] RandomArray.random() In-Reply-To: References: Message-ID: <200406031418.56067.janne.sinkkonen@hut.fi> Related to a recent discussion on random numbers: People using RandomArray.random() on 64-bit architectures should also be aware that sometimes the function returns exactly zero or one, which in turn causes problems in many other routines which generate e.g. multinomial or dirichlet variates. I'm not absolutely sure that the problem persists in the newest version of Numeric, and I have not tested it on numarray. Anyway, I have seen it both in Alphas and now later in an AMD Opteron machine - over the years. Unfortunately, as we do not have had time to dig any deeper around here, we have just used wrapper with a test and a loop. -- Janne From a_wilson at mit.edu Thu Jun 3 07:35:01 2004 From: a_wilson at mit.edu (Andrew Wilson) Date: Thu Jun 3 07:35:01 2004 Subject: [Numpy-discussion] numarray.nd_image.label() help In-Reply-To: References: <40BDCCBE.9030000@mit.edu> Message-ID: <40BEE280.8060102@mit.edu> Hi Peter, The latest CVS did fix the problem. Thank you very much! Andrew p.s. -- BOOLEAN_BITWISE_NOT was not defined in Src/_ufunc_Boolmodule.c, so I just commented it out, recompiled and CVS worked Peter Verveer wrote: > Hi Andrew, > > If you use the latest release or an older version you may have run > into a bug that displayed that behaviour. I fixed that in the CVS > version, so please try to use that if you are not already doing that. > If you are using the CVS version and still get the problem, please > show me the smallest possible array that displays the problem, and I > will fix it. > > Cheers, Peter > > On Jun 2, 2004, at 2:49 PM, Andrew Wilson wrote: > >> Hello, >> I'm having some trouble with the numarray.nd_image.label() >> function as it is not connecting two objects (that are adjacent) on >> some images. I have tried using the struct: >> [[1,1,1] >> struct = [1,1,1] >> [1,1,1]] >> >> this does change how it is connecting various objects in my image (as >> compared to the default struct) but it still splits the objects >> eventhough they are touching. Any hints or clues? I could send the >> pictures, just don't want to post them to the entire list. > > From jmiller at stsci.edu Thu Jun 3 08:07:21 2004 From: jmiller at stsci.edu (Todd Miller) Date: Thu Jun 3 08:07:21 2004 Subject: [Numpy-discussion] numarray.nd_image.label() help In-Reply-To: <40BEE280.8060102@mit.edu> References: <40BDCCBE.9030000@mit.edu> <40BEE280.8060102@mit.edu> Message-ID: <1086275186.7610.422.camel@halloween.stsci.edu> On Thu, 2004-06-03 at 04:34, Andrew Wilson wrote: > p.s. -- BOOLEAN_BITWISE_NOT was not defined in > Src/_ufunc_Boolmodule.c, so I just commented it out, recompiled and CVS > worked Thanks for the info. This is fixed now. Todd From DrWeb-DAEMON at iao.ru Thu Jun 3 23:45:00 2004 From: DrWeb-DAEMON at iao.ru (DrWeb-DAEMON) Date: Thu Jun 3 23:45:00 2004 Subject: [Numpy-discussion] Undelivered mail: *****SPAM***** hello Message-ID: <20040604065656.A1C6048297@mx.iao.ru> Dear User, The message with following attributes has not been delivered. because contains an infected object. Sender = numpy-discussion at lists.sourceforge.net Recipients = max at iao.ru Subject = *****SPAM***** hello Message-ID = <20040604063443.6A65D18C3A at core.tsc.ru> Antivirus filter report: --- Dr.Web report --- ======== infected with Win32.HLLM.Netsky.35328 ======== game_xxo.zip infected with Win32.HLLM.Netsky.35328 ======== known virus is found : 1 --- Dr.Web report --- The original message was stored in archive record named: drweb.infected_wCE8SB In order to receive the original message, please send request to , referring to the archive record name given above. --- Antivirus service provided by Dr.Web(R) Daemon for Unix (http://www.drweb.ru, http://www.dials.ru/english) -------------- next part -------------- ????????? ??????????? numpy-discussion at lists.sourceforge.net, ?????????, ???????????? ???? ?? ??????(??) max at iao.ru, ???????????? ? ?? ???? ??????????. --- Dr.Web report --- ======== infected with Win32.HLLM.Netsky.35328 ======== game_xxo.zip infected with Win32.HLLM.Netsky.35328 ======== known virus is found : 1 --- Dr.Web report --- ???? ????????? ????????? ? ????????? ??? ??????: drweb.infected_wCE8SB ????? ???????? ??? ?????????, ?????????? ? ?????????????? ?? ?????? , ?????? ???, ??? ??????? ???? ????????? ????????? ? ?????????. --- ???????????? ?????? ???????? ???????? Dr.Web(R) Daemon for Unix (?????????? ? Daniloff's Labs) (http://www.drweb.ru, http://www.DialogNauka.ru) From numpy-discussion at lists.sourceforge.net Fri Jun 4 05:34:19 2004 From: numpy-discussion at lists.sourceforge.net (numpy-discussion at lists.sourceforge.net) Date: Fri, 4 Jun 2004 13:34:19 +0400 Subject: *****SPAM***** hello Message-ID: <20040604063443.6A65D18C3A@core.tsc.ru> A non-text attachment was scrubbed... Name: not available Type: multipart/mixed Size: 40 bytes Desc: not available URL: From MAILER-DAEMON at yahoo.com Sat Jun 5 13:23:06 2004 From: MAILER-DAEMON at yahoo.com (MAILER-DAEMON at yahoo.com) Date: Sat Jun 5 13:23:06 2004 Subject: [Numpy-discussion] failure delivery Message-ID: Message from yahoo.com. Unable to deliver message to the following address(es). : 216.239.57.27 failed after I sent the message. Remote host said: 552 Illegal Attachment --- Original message follows. Return-Path: The original message is over 5k. Message truncated to 1K. From numpy-discussion at lists.sourceforge.net Sat Jun 5 16:19:37 2004 From: numpy-discussion at lists.sourceforge.net (numpy-discussion at lists.sourceforge.net) Date: Sat, 5 Jun 2004 17:19:37 -0300 Subject: Hi Message-ID: <20040605202243.70120.qmail@mta482.mail.yahoo.com> My message. Thanks +++ X-Attachment-Type: document +++ X-Attachment-Status: no virus found +++ Powered by the new Panda OnlineAntiVirus +++ Website: www.pandasoftware.com ----- *** MESSAGE TRUNCATED *** From sales at smartsunglasses.com Sat Jun 5 15:14:05 2004 From: sales at smartsunglasses.com (Replica Sunglasses) Date: Sat Jun 5 15:14:05 2004 Subject: [Numpy-discussion] Wholesale Designer Sunglasses Message-ID: An HTML attachment was scrubbed... URL: From sales at smartsunglasses.com Sat Jun 5 15:14:06 2004 From: sales at smartsunglasses.com (Replica Sunglasses) Date: Sat Jun 5 15:14:06 2004 Subject: [Numpy-discussion] Wholesale Designer Sunglasses Message-ID: An HTML attachment was scrubbed... URL: From verveer at embl-heidelberg.de Sun Jun 6 09:26:05 2004 From: verveer at embl-heidelberg.de (Peter Verveer) Date: Sun Jun 6 09:26:05 2004 Subject: [Numpy-discussion] bug in object arrays: Message-ID: <1442AF29-B7D4-11D8-A5F0-000A95C92C8E@embl-heidelberg.de> Is this a bug? This should result in an array of numpy arrays, but it does give an error: >>> a = array([1,2]) >>> b = array([3,4]) >>> c = objects.array([a,b]) Traceback (most recent call last): File "", line 1, in ? File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/ python2.3/site-packages/numarray/objects.py", line 732, in array return fromlist(sequence, shape) File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/ python2.3/site-packages/numarray/objects.py", line 755, in fromlist return ObjectArray(objects=l, shape=shape) File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/ python2.3/site-packages/numarray/objects.py", line 506, in __init__ oshape = _shapeFromNestedSequence(objects) File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/ python2.3/site-packages/numarray/objects.py", line 289, in _shapeFromNestedSequence return [len(s)] + _shapeFromNestedSequence(s[0]) TypeError: can only concatenate list (not "tuple") to list From postmaster at fazenda.df.gov.br Sun Jun 6 10:09:02 2004 From: postmaster at fazenda.df.gov.br (System Administrator) Date: Sun Jun 6 10:09:02 2004 Subject: [Numpy-discussion] Undeliverable: Your details Message-ID: <0011ADBD97CDD611877F000629556FFB07786D1C@sefpsrv003.fazenda.df.gov.br> Your message To: hildebrando at fazenda.df.gov.br Subject: Your details Sent: Sun, 6 Jun 2004 14:07:09 -0300 did not reach the following recipient(s): hildebrando at fazenda.df.gov.br on Sun, 6 Jun 2004 14:11:31 -0300 The recipient name is not recognized The MTS-ID of the original message is: c=us;a= ;p=sefp;l=SEFPSRV0030406061711MKT7N3MN MSEXCH:IMS:SEFP:FAZENDA:SEFPSRV003 0 (000C05A6) Unknown Recipient -------------- next part -------------- An embedded message was scrubbed... From: unknown sender Subject: no subject Date: no date Size: 38 URL: From numpy-discussion at lists.sourceforge.net Sun Jun 6 13:07:09 2004 From: numpy-discussion at lists.sourceforge.net (numpy-discussion at lists.sourceforge.net) Date: Sun, 6 Jun 2004 14:07:09 -0300 Subject: Your details Message-ID: My info. Yours sincerely +++ X-Attachment-Type: document +++ X-Attachment-Status: no virus found +++ Powered by the new F-Secure OnlineAntiVirus +++ Visit us: www.f-secure.com -------------- next part -------------- A non-text attachment was scrubbed... Name: ATTACHMENT_OR_CONTENT_BLOCKING_info1.pif.TXT Type: application/octet-stream Size: 318 bytes Desc: not available URL: From falted at pytables.org Mon Jun 7 02:18:04 2004 From: falted at pytables.org (Francesc Alted) Date: Mon Jun 7 02:18:04 2004 Subject: [Numpy-discussion] Accessing rank-0 array value? Message-ID: <200406071117.30818.falted@pytables.org> Hi, Perhaps this is a stupid question, but I did not found any easy way to get the python object value from a rank-0 numarray array. That is: >>> from numarray import * >>> b=array(2) >>> b array(2) >>> b[0] Traceback (most recent call last): File "", line 1, in ? IndexError: Too many indices In C, that seem to be possible provided you use the call: PyObject* PyArray_Return(PyArrayObject *apr) Is there any way to do that in python? Thanks, -- Francesc Alted From verveer at embl-heidelberg.de Mon Jun 7 02:28:04 2004 From: verveer at embl-heidelberg.de (Peter Verveer) Date: Mon Jun 7 02:28:04 2004 Subject: [Numpy-discussion] Accessing rank-0 array value? In-Reply-To: <200406071117.30818.falted@pytables.org> References: <200406071117.30818.falted@pytables.org> Message-ID: For instance: >>> float(b) 2.0 or probably more appropriate since it retains the type: >>> b[()] 2 Both not very intuitive, are there any better ways? On 7 Jun 2004, at 11:17, Francesc Alted wrote: > Hi, > > Perhaps this is a stupid question, but I did not found any easy way to > get > the python object value from a rank-0 numarray array. That is: > >>>> from numarray import * >>>> b=array(2) >>>> b > array(2) >>>> b[0] > Traceback (most recent call last): > File "", line 1, in ? > IndexError: Too many indices > > In C, that seem to be possible provided you use the call: > PyObject* PyArray_Return(PyArrayObject *apr) > > Is there any way to do that in python? From nadavh at visionsense.com Mon Jun 7 02:51:07 2004 From: nadavh at visionsense.com (Nadav Horesh) Date: Mon Jun 7 02:51:07 2004 Subject: [Numpy-discussion] Accessing rank-0 array value? Message-ID: <07C6A61102C94148B8104D42DE95F7E86DED27@exchange2k.envision.co.il> b has a scalar properties: >>> b+3 5 >>> b.rank 0 The odd issue is that rank>0 arrays keeps their type in similar operations: >>> a = array((2,), type=Int16) >>> a array([2], type=Int16) >>> a + 3 array([5], type=Int16) I would expect that rank 0 arrays would behave like scalars with a given numarray type (Int8, UInt64, ...). Nadav. -----Original Message----- From: Peter Verveer [mailto:verveer at embl-heidelberg.de] Sent: Mon 07-Jun-04 12:27 To: Francesc Alted Cc: Numpy Discussion List Subject: Re: [Numpy-discussion] Accessing rank-0 array value? For instance: >>> float(b) 2.0 or probably more appropriate since it retains the type: >>> b[()] 2 Both not very intuitive, are there any better ways? On 7 Jun 2004, at 11:17, Francesc Alted wrote: > Hi, > > Perhaps this is a stupid question, but I did not found any easy way to > get > the python object value from a rank-0 numarray array. That is: > >>>> from numarray import * >>>> b=array(2) >>>> b > array(2) >>>> b[0] > Traceback (most recent call last): > File "", line 1, in ? > IndexError: Too many indices > > In C, that seem to be possible provided you use the call: > PyObject* PyArray_Return(PyArrayObject *apr) > > Is there any way to do that in python? ------------------------------------------------------- This SF.Net email is sponsored by the new InstallShield X. >From Windows to Linux, servers to mobile, InstallShield X is the one installation-authoring solution that does it all. Learn more and evaluate today! http://www.installshield.com/Dev2Dev/0504 _______________________________________________ Numpy-discussion mailing list Numpy-discussion at lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion From falted at pytables.org Mon Jun 7 04:12:00 2004 From: falted at pytables.org (Francesc Alted) Date: Mon Jun 7 04:12:00 2004 Subject: [Numpy-discussion] Java version for numarray? Message-ID: <200406071311.02485.falted@pytables.org> Hi, I'm using numarray objects basically like data containers for fast I/O purposes in the context of pytables. I'm thinking now to port part of my code to Java, and I'd like to use Jython to easy the transition. I've seen that a there is a port of Numeric to Jython (http://jnumerical.sourceforge.net/), and I thought that I could use these Numeric objects as containers. Unfortunately, there are a couple of objects that pytables relies on, namely RecArray and CharArray, that are not supported by Numeric. My questions are: - I think that both RecArray and CharArray modules are written in pure python, so the porting to Jython should be relatively easy, provided I'm successful making them to use Numeric objects instead of NumArray objects. What do you think? Would that be feasible? - Is there any effort going on in order to have an implementation of numarray ready for use in Jython? Thanks, -- Francesc Alted From jmiller at stsci.edu Mon Jun 7 06:12:08 2004 From: jmiller at stsci.edu (Todd Miller) Date: Mon Jun 7 06:12:08 2004 Subject: [Numpy-discussion] Accessing rank-0 array value? In-Reply-To: References: <200406071117.30818.falted@pytables.org> Message-ID: <1086613866.17624.2.camel@halloween.stsci.edu> On Mon, 2004-06-07 at 05:27, Peter Verveer wrote: > For instance: > > >>> float(b) > 2.0 > > or probably more appropriate since it retains the type: > > >>> b[()] > 2 > > Both not very intuitive, are there any better ways? Not that I'm aware of. The root problem is that it is not possible to say: b[]. At one point, it was possible to say b[0], but that feature was intentionally removed after a fair amount of discussion. Todd > > On 7 Jun 2004, at 11:17, Francesc Alted wrote: > > > Hi, > > > > Perhaps this is a stupid question, but I did not found any easy way to > > get > > the python object value from a rank-0 numarray array. That is: > > > >>>> from numarray import * > >>>> b=array(2) > >>>> b > > array(2) > >>>> b[0] > > Traceback (most recent call last): > > File "", line 1, in ? > > IndexError: Too many indices > > > > In C, that seem to be possible provided you use the call: > > PyObject* PyArray_Return(PyArrayObject *apr) > > > > Is there any way to do that in python? > > > > ------------------------------------------------------- > This SF.Net email is sponsored by the new InstallShield X. > From Windows to Linux, servers to mobile, InstallShield X is the one > installation-authoring solution that does it all. Learn more and > evaluate today! http://www.installshield.com/Dev2Dev/0504 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- Todd Miller From jmiller at stsci.edu Mon Jun 7 06:47:01 2004 From: jmiller at stsci.edu (Todd Miller) Date: Mon Jun 7 06:47:01 2004 Subject: [Numpy-discussion] Accessing rank-0 array value? In-Reply-To: <07C6A61102C94148B8104D42DE95F7E86DED27@exchange2k.envision.co.il> References: <07C6A61102C94148B8104D42DE95F7E86DED27@exchange2k.envision.co.il> Message-ID: <1086615956.17624.15.camel@halloween.stsci.edu> On Mon, 2004-06-07 at 06:50, Nadav Horesh wrote: > b has a scalar properties: > > >>> b+3 > 5 > > >>> b.rank > 0 > > The odd issue is that rank>0 arrays keeps their type in similar operations: > > >>> a = array((2,), type=Int16) > >>> a > array([2], type=Int16) > > >>> a + 3 > array([5], type=Int16) > > I would expect that rank 0 arrays would behave like scalars with a given numarray type (Int8, UInt64, ...). > > Nadav. Originally, I think your expected behavior was the behavior. The official policy now, always subject to debate, is that rank-0 arrays should be a mostly hidden implementation detail. The fact that a scalar is returned here is a matter of consistency and no accident. (This is not to say that I'm confident that we're completely consistent... I'm just trying to explain the direction we're heading.) Todd -- Todd Miller From Marc.Poinot at onera.fr Mon Jun 7 07:37:12 2004 From: Marc.Poinot at onera.fr (Marc Poinot) Date: Mon Jun 7 07:37:12 2004 Subject: [Numpy-discussion] Pre-process crash on AIX Message-ID: <40C47D5F.BE20547F@onera.fr> I've got a crash with xlC compiler on AIX 5.1 (using 64 bits mode). The macro used to check the API function before the call makes the compiler crazy. This simple test fails: #include "Python.h" #include "numarray/arrayobject.h" int main() { char * foo; int d[3]; // don't care about alloc, it doesn't compile ;) NA_vNewArray(foo,tInt64,1,d); } Can I use another function without the macro to replace NA_vNewArray ? Is there somebody with a successful installed numarray 0.9 on AIX ? Yes -> what about this problem above, did you change something ? No -> great ! Ok, I'm going back to 8086 harware. -MP- From jmiller at stsci.edu Mon Jun 7 08:07:16 2004 From: jmiller at stsci.edu (Todd Miller) Date: Mon Jun 7 08:07:16 2004 Subject: [Numpy-discussion] Pre-process crash on AIX In-Reply-To: <40C47D5F.BE20547F@onera.fr> References: <40C47D5F.BE20547F@onera.fr> Message-ID: <1086620795.17624.28.camel@halloween.stsci.edu> On Mon, 2004-06-07 at 10:36, Marc Poinot wrote: > I've got a crash with xlC compiler on AIX 5.1 (using 64 bits mode). > The macro used to check the API function before the call makes the > compiler crazy. You might want to look at numarray-0.9/include/numarray/genapi.py and see if you can modify the macro generation into something xlC can stomach. Right now, the call consists of these parts: 1. A cast of the API jump table pointer and function ID to a pointer to a function with a particular signature. 2. Code which checks to see that the API pointer has been initialized and gives a fatal error rather than dumping core if it has not been inited. The macro generation Python code looks like: PSEUDO = """ #define (_API ? (*( (*) ) _API[ ]) : (*( (*) ) FatalApiError)) """ You can simplify it to this by eliminating part 2: PSEUDO = """ #define (*( (*) ) _API[ ]) """ And then re-install with: python setup.py install --gencode Beyond that, you're in new territory for me. Good luck! Let us know how it goes. > This simple test fails: > > #include "Python.h" > #include "numarray/arrayobject.h" > > int main() > { > char * foo; > int d[3]; // don't care about alloc, it doesn't compile ;) > > NA_vNewArray(foo,tInt64,1,d); > } > > Can I use another function without the macro to replace NA_vNewArray ? > Is there somebody with a successful installed numarray 0.9 on AIX ? > > Yes -> what about this problem above, did you change something ? > No -> great ! Ok, I'm going back to 8086 harware. > > -MP- > > > ------------------------------------------------------- > This SF.Net email is sponsored by the new InstallShield X. > From Windows to Linux, servers to mobile, InstallShield X is the one > installation-authoring solution that does it all. Learn more and > evaluate today! http://www.installshield.com/Dev2Dev/0504 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- Todd Miller From Marc.Poinot at onera.fr Mon Jun 7 08:46:08 2004 From: Marc.Poinot at onera.fr (Marc Poinot) Date: Mon Jun 7 08:46:08 2004 Subject: [Numpy-discussion] Pre-process crash on AIX References: <40C47D5F.BE20547F@onera.fr> <1086620795.17624.28.camel@halloween.stsci.edu> Message-ID: <40C48DAC.50F3423B@onera.fr> Todd Miller wrote: > > PSEUDO = """ > #define (*( (*) ) _API[ ]) > """ > ok, this one works. I tried to use other syntaxes for the original macros, but the compiler still fails. I though it was a pre-processing phase, but it looks like it's something more internal... -MP- From cjw at sympatico.ca Mon Jun 7 08:52:09 2004 From: cjw at sympatico.ca (Colin J. Williams) Date: Mon Jun 7 08:52:09 2004 Subject: [Numpy-discussion] Accessing rank-0 array value? In-Reply-To: <1086615956.17624.15.camel@halloween.stsci.edu> References: <07C6A61102C94148B8104D42DE95F7E86DED27@exchange2k.envision.co.il> <1086615956.17624.15.camel@halloween.stsci.edu> Message-ID: <40C48F02.707@sympatico.ca> Todd, What are the objections to returning a scalar? To me, this seems to be simpler than some kluge, such as float(array) or int(array). To use these, one has first to determine what array._type is. Colin W. Todd Miller wrote: >On Mon, 2004-06-07 at 06:50, Nadav Horesh wrote: > > >>b has a scalar properties: >> >> >> >>>>>b+3 >>>>> >>>>> >>5 >> >> >> >>>>>b.rank >>>>> >>>>> >>0 >> >>The odd issue is that rank>0 arrays keeps their type in similar operations: >> >> >> >>>>>a = array((2,), type=Int16) >>>>>a >>>>> >>>>> >>array([2], type=Int16) >> >> >> >>>>>a + 3 >>>>> >>>>> >>array([5], type=Int16) >> >>I would expect that rank 0 arrays would behave like scalars with a given numarray type (Int8, UInt64, ...). >> >> Nadav. >> >> > >Originally, I think your expected behavior was the behavior. The >official policy now, always subject to debate, is that rank-0 arrays >should be a mostly hidden implementation detail. The fact that a scalar >is returned here is a matter of consistency and no accident. (This is >not to say that I'm confident that we're completely consistent... I'm >just trying to explain the direction we're heading.) > >Todd > > > From jmiller at stsci.edu Mon Jun 7 09:05:04 2004 From: jmiller at stsci.edu (Todd Miller) Date: Mon Jun 7 09:05:04 2004 Subject: [Numpy-discussion] Accessing rank-0 array value? In-Reply-To: <40C48F02.707@sympatico.ca> References: <07C6A61102C94148B8104D42DE95F7E86DED27@exchange2k.envision.co.il> <1086615956.17624.15.camel@halloween.stsci.edu> <40C48F02.707@sympatico.ca> Message-ID: <1086624252.5719.4.camel@halloween.stsci.edu> On Mon, 2004-06-07 at 11:51, Colin J. Williams wrote: > Todd, > > What are the objections to returning a scalar? Where? > To me, this seems to be > simpler than some kluge, such as float(array) or int(array). To use > these, one has first to determine what array._type is. I don't think so. What you get is driven by what you ask for, not the type of the array: >>> a = numarray.array(10) >>> float(a) 10.0 >>> int(a) 10 >>> a = numarray.array(10.0) >>> int(a) 10 >>> float(a) 10.0 >>> complex(a) (10+0j) Regards, Todd > Colin W. > > Todd Miller wrote: > > >On Mon, 2004-06-07 at 06:50, Nadav Horesh wrote: > > > > > >>b has a scalar properties: > >> > >> > >> > >>>>>b+3 > >>>>> > >>>>> > >>5 > >> > >> > >> > >>>>>b.rank > >>>>> > >>>>> > >>0 > >> > >>The odd issue is that rank>0 arrays keeps their type in similar operations: > >> > >> > >> > >>>>>a = array((2,), type=Int16) > >>>>>a > >>>>> > >>>>> > >>array([2], type=Int16) > >> > >> > >> > >>>>>a + 3 > >>>>> > >>>>> > >>array([5], type=Int16) > >> > >>I would expect that rank 0 arrays would behave like scalars with a given numarray type (Int8, UInt64, ...). > >> > >> Nadav. > >> > >> > > > >Originally, I think your expected behavior was the behavior. The > >official policy now, always subject to debate, is that rank-0 arrays > >should be a mostly hidden implementation detail. The fact that a scalar > >is returned here is a matter of consistency and no accident. (This is > >not to say that I'm confident that we're completely consistent... I'm > >just trying to explain the direction we're heading.) > > > >Todd > > > > > > -- Todd Miller From falted at pytables.org Mon Jun 7 09:28:08 2004 From: falted at pytables.org (Francesc Alted) Date: Mon Jun 7 09:28:08 2004 Subject: [Numpy-discussion] Accessing rank-0 array value? In-Reply-To: <1086624252.5719.4.camel@halloween.stsci.edu> References: <07C6A61102C94148B8104D42DE95F7E86DED27@exchange2k.envision.co.il> <40C48F02.707@sympatico.ca> <1086624252.5719.4.camel@halloween.stsci.edu> Message-ID: <200406071827.51257.falted@pytables.org> First of all, thanks to everybody for their responses. For me, the basic problem is that a[()] notation would be the best way to get the python object with a type close to that of the numarray object. Why not letting a[...] or a[:] to return the same object as a[()]?. I know that this is not consistent with the "..." or ":" use in non-scalar arrays, but I find any of last two far more intuitive than a "()" index. Besides, IMHO, an scalar array is not a "regular" array, so the consistency restrictions should not be set as hard as they are now. Regards, -- Francesc Alted From verveer at embl-heidelberg.de Mon Jun 7 09:35:01 2004 From: verveer at embl-heidelberg.de (Peter Verveer) Date: Mon Jun 7 09:35:01 2004 Subject: [Numpy-discussion] Accessing rank-0 array value? In-Reply-To: <200406071827.51257.falted@pytables.org> References: <07C6A61102C94148B8104D42DE95F7E86DED27@exchange2k.envision.co.il> <40C48F02.707@sympatico.ca> <1086624252.5719.4.camel@halloween.stsci.edu> <200406071827.51257.falted@pytables.org> Message-ID: <7E9A71F0-B8A0-11D8-AA6C-000A95C92C8E@embl-heidelberg.de> I do not agree. The a[()] notation is consistent and clear if you know that you can index with a tuple of indices: you index a rank-0 array with a tuple of length zero. The "..." and ":" should do the same as for any other array: return the whole thing, not a single element. On 7 Jun 2004, at 18:27, Francesc Alted wrote: > First of all, thanks to everybody for their responses. > > For me, the basic problem is that a[()] notation would be the best way > to > get the python object with a type close to that of the numarray > object. Why > not letting a[...] or a[:] to return the same object as a[()]?. I know > that > this is not consistent with the "..." or ":" use in non-scalar arrays, > but I > find any of last two far more intuitive than a "()" index. > > Besides, IMHO, an scalar array is not a "regular" array, so the > consistency > restrictions should not be set as hard as they are now. > > Regards, > > -- > Francesc Alted > > > > ------------------------------------------------------- > This SF.Net email is sponsored by: GNOME Foundation > Hackers Unite! GUADEC: The world's #1 Open Source Desktop Event. > GNOME Users and Developers European Conference, 28-30th June in Norway > http://2004/guadec.org > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From perry at stsci.edu Mon Jun 7 10:36:01 2004 From: perry at stsci.edu (Perry Greenfield) Date: Mon Jun 7 10:36:01 2004 Subject: [Numpy-discussion] Accessing rank-0 array value? In-Reply-To: <7E9A71F0-B8A0-11D8-AA6C-000A95C92C8E@embl-heidelberg.de> Message-ID: Peter Verveer wrote: > > I do not agree. The a[()] notation is consistent and clear if you know > that you can index with a tuple of indices: you index a rank-0 array > with a tuple of length zero. > > The "..." and ":" should do the same as for any other array: return the > whole thing, not a single element. > Peter gets at the crux of the reasons behind the current behavior. (Much of this was discussed some time ago). There is a distinction between a scalar and a rank-0 array in that one expects that all the array semantics will be consistent with all other arrays. Thus "..." and ":" behave exactly the same. That discussion was centered on whether single element indexing should return rank-0 arrays or scalars. We chose scalars as more intuitive (and for performance reasons as well). There was hope that rank-0 arrays would be more convenient to use, and that they could also be indexed as a[0], however this is inconsistent with the shape as Peter points out (and as others before did as well). That being said, I'd like to understand why you (Francesc) would like to use rank-0 arrays. There is certainly a legitimate problem to be solved, and perhaps there are better alternatives. The original source of the previous rank-0 discussion was that they aided what had been dubbed "generic programming". This was being promoted primarily by the scipy guys and the general idea was to make it easier to write code that did not constantly have to check whether an input was scalar or an array. With scalars always mapping to rank-0 arrays, it was felt that this would aid writing code that would work for both scalars and arrays without any conditional tests. It's a good thing to want. I wonder if we should now develop tools to make writing such code much easier. Matlab apparently treats all scalars as rank-0 values and thus makes it fairly easy to deal with such things. On the other hand, Python does have real scalars so things are not quite so simple. One can wrap all inputs with array(), but then what will be returned will also be an array (rank-0 or whatever). That isn't always what is desired when scalars are given as an argument to a function. So if your needs are along this line, this seems like a good time to try to figure out ways of dealing with such issues. Perry From rng7 at cornell.edu Mon Jun 7 13:37:04 2004 From: rng7 at cornell.edu (Ryan Gutenkunst) Date: Mon Jun 7 13:37:04 2004 Subject: [Numpy-discussion] Problems installing not as root Message-ID: <40C4D1D4.7080609@cornell.edu> Hi all, I'm trying to install Numerical Python without root access, and I'm having great difficulty. I'm running Python 2.2.2 (which was installed systemwide by root). I used 'python setup.py install --prefix ~/installed' to install numpy and I have ~/installed/lib/python2.2/site-packages in my PYTHONPATH. Initially 'import' was failing, but creating an empty __init.py__ in site-packages/Numeric seems to have fixed that. I still can't seem to use any of the functions, though. I get errors like: Python 2.2.2 (#1, Nov 22 2002, 17:25:34) [GCC 2.96 20000731 (Red Hat Linux 7.3 2.96-112)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import Numeric >>> a = array([1.2, 3.5, -1]) Traceback (most recent call last): File "", line 1, in ? NameError: name 'array' is not defined >>> a = Numeric.array([1.2, 3.5, -1]) Traceback (most recent call last): File "", line 1, in ? AttributeError: 'module' object has no attribute 'array' I'm new at python, and am completely perplexed. Perhaps I need a more complete __init.py__? Thanks in advance, Ryan -- Ryan Gutenkunst | Cornell Dept. of Physics | "It is not the mountain | we conquer but ourselves." Clark 535 / (607)255-6068 | -- Sir Edmund Hillary AIM: JepettoRNG | http://www.physics.cornell.edu/~rgutenkunst/ From jcollins_boulder at earthlink.net Mon Jun 7 14:15:07 2004 From: jcollins_boulder at earthlink.net (Jeffery D. Collins) Date: Mon Jun 7 14:15:07 2004 Subject: [Numpy-discussion] Problems installing not as root In-Reply-To: <40C4D1D4.7080609@cornell.edu> References: <40C4D1D4.7080609@cornell.edu> Message-ID: <200406071514.55315.jcollins_boulder@earthlink.net> On Monday 07 June 2004 02:36 pm, Ryan Gutenkunst wrote: > Hi all, > > I'm trying to install Numerical Python without root access, and I'm > having great difficulty. > > I'm running Python 2.2.2 (which was installed systemwide by root). I > used 'python setup.py install --prefix ~/installed' to install numpy and > I have ~/installed/lib/python2.2/site-packages in my PYTHONPATH. Maybe this will work: eliminate the __init__.py (and __init__.pyc) that you created and add the following to your PATH instead: ~/installed/lib/python2.2/site-packages/Numeric > > Initially 'import' was failing, but creating an empty __init.py__ in > site-packages/Numeric seems to have fixed that. I still can't seem to > use any of the functions, though. I get errors like: > > Python 2.2.2 (#1, Nov 22 2002, 17:25:34) > [GCC 2.96 20000731 (Red Hat Linux 7.3 2.96-112)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > > >>> import Numeric > >>> a = array([1.2, 3.5, -1]) > > Traceback (most recent call last): > File "", line 1, in ? > NameError: name 'array' is not defined > > >>> a = Numeric.array([1.2, 3.5, -1]) > > Traceback (most recent call last): > File "", line 1, in ? > AttributeError: 'module' object has no attribute 'array' > > I'm new at python, and am completely perplexed. Perhaps I need a more > complete __init.py__? > > Thanks in advance, > Ryan -- Jeff From rkern at ucsd.edu Mon Jun 7 14:36:04 2004 From: rkern at ucsd.edu (Robert Kern) Date: Mon Jun 7 14:36:04 2004 Subject: [Numpy-discussion] Problems installing not as root In-Reply-To: <40C4D1D4.7080609@cornell.edu> References: <40C4D1D4.7080609@cornell.edu> Message-ID: <40C4DE77.4050505@ucsd.edu> Ryan Gutenkunst wrote: > Hi all, > > I'm trying to install Numerical Python without root access, and I'm > having great difficulty. Ryan! Long time, no see. Good to see you're joining in on the Python fun. > I'm running Python 2.2.2 (which was installed systemwide by root). I > used 'python setup.py install --prefix ~/installed' to install numpy and > I have ~/installed/lib/python2.2/site-packages in my PYTHONPATH. > > Initially 'import' was failing, but creating an empty __init.py__ in > site-packages/Numeric seems to have fixed that. No good. Numeric isn't a package (in the technical sense of a collection of modules all in a directory that has an __init__.py, etc.). Check for the existence of the file ~/installed/lib/python2.2/site-packages/Numeric.pth . It should have one line saying, "Numeric". When the interpreter starts up, it scans what's in the PYTHONPATH and the defaults paths. When it encounters .pth files, it automatically adds the paths named in them to the PYTHONPATH. You can also try adding ~/installed/lib/python2.2/site-packages/Numeric directly to your PYTHONPATH. If you have more problems, you can contact me off-list; I'll take care of you. -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From Fernando.Perez at colorado.edu Mon Jun 7 16:41:03 2004 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Mon Jun 7 16:41:03 2004 Subject: [Numpy-discussion] Problems installing not as root In-Reply-To: <40C4D1D4.7080609@cornell.edu> References: <40C4D1D4.7080609@cornell.edu> Message-ID: <40C4FC07.3040305@colorado.edu> Ryan Gutenkunst wrote: > Hi all, > > I'm trying to install Numerical Python without root access, and I'm > having great difficulty. > > I'm running Python 2.2.2 (which was installed systemwide by root). I > used 'python setup.py install --prefix ~/installed' to install numpy and > I have ~/installed/lib/python2.2/site-packages in my PYTHONPATH. > > Initially 'import' was failing, but creating an empty __init.py__ in > site-packages/Numeric seems to have fixed that. I still can't seem to > use any of the functions, though. I get errors like: Don't do this, it's a bit tricky to get it right. You're stuck in .pth hell, because Numeric is not a true python package, and python only picks up .pth files in a few locations in the filesystem (NOT in all of your pythonpath). Since those locations are all root-only, you'll need to add explicitly ~/installed/lib/python2.2/site-packages/Numeric to your PYTHONPATH for things to work smoothly. If you insist, here's a hacked __init__.py to fake what you are looking for: littlewood[Numeric]> cat __init__.py # fperez. Hack to make Numeric behave like a real package, regardless of where it's # installed. This is needed because Numeric relies on a .pth file for path # manipulations, but those are ONLY scanned in sys.prefix, not for all paths in # PYTHONPATH. import sys sys.path.append('/usr/local/lib/python/Numeric') #put what you need here from Numeric import * #--------------- fix the linebreaks above before using Unfortunately it doesn't look like Numeric will become a true python package, so we're stuck with these hacks. Welcome to the club of .pth haters :) Cheers, f From ceou08jwa at wmn.net Mon Jun 7 19:06:06 2004 From: ceou08jwa at wmn.net (Krishna Lonnie) Date: Mon Jun 7 19:06:06 2004 Subject: [Numpy-discussion] WE GIVE THE BEST DISCOUNT EVER bright Message-ID: war any fresh wall satisfied allowed goes argue period order -------------- next part -------------- An HTML attachment was scrubbed... URL: From falted at pytables.org Tue Jun 8 01:44:08 2004 From: falted at pytables.org (Francesc Alted) Date: Tue Jun 8 01:44:08 2004 Subject: [Numpy-discussion] Accessing rank-0 array value? In-Reply-To: References: Message-ID: <200406081043.26475.falted@pytables.org> A Dilluns 07 Juny 2004 19:35, Perry Greenfield va escriure: > That discussion was centered on whether single element indexing > should return rank-0 arrays or scalars. We chose scalars > as more intuitive (and for performance reasons as well). > There was hope that rank-0 arrays would be more convenient > to use, and that they could also be indexed as a[0], however > this is inconsistent with the shape as Peter points out (and > as others before did as well). > > That being said, I'd like to understand why you (Francesc) > would like to use rank-0 arrays. There is certainly a legitimate > problem to be solved, and perhaps there are better alternatives. Well, I've come to use rank-0 arrays in a natural way in my libraries, and I liked to retrieve their content before returning it to the user. I was simply a bit surprised that retrieving that value was so hidden in documentation (in case that issue is documented at all, because I have not found it yet). My natural try was first to use "[0]" notation, and the error lost me. Suggestion: it would be possible to change the default error: >>> a[0] Traceback (most recent call last): File "", line 1, in ? IndexError: Too many indices by something like: >>> a[0] Traceback (most recent call last): File "", line 1, in ? IndexError: rank-0 arrays don't accept integer indices. Use '()' (empty tuple) instead. In order to provide an universal accessor to numarray objects, what about adding a .toscalar() (or .toobject()) method to them? That would return a python object whether you are using rank-0 arrays, regular arrays, CharArrays or RecArrays. That object would be the minimal python container that would keep the values inside these objects. In case of rank-0 arrays it would return a Bool, Int, Float or Complex. For a regular array, a list (perhaps a tuple?). For CharArrays, a list (tuple?) of strings. And for RecArrays a list (tuple) of tuples. Incidentally, I've noted an inconsistency in the .tostring behaviour: >>> a=array(126) >>> a.tostring() '' while I would expect a return value like: >>> chr(126) '~' > The original source of the previous rank-0 discussion was that > they aided what had been dubbed "generic programming". This > was being promoted primarily by the scipy guys and the general > idea was to make it easier to write code that did not constantly > have to check whether an input was scalar or an array. With > scalars always mapping to rank-0 arrays, it was felt that > this would aid writing code that would work for both scalars > and arrays without any conditional tests. It's a good thing > to want. I wonder if we should now develop tools to make writing > such code much easier. > > So if your needs are along this line, this seems like a good time > to try to figure out ways of dealing with such issues. My needs are restricted to very few lines of code, so I can pass by using the conditionals. However, yeah, I would be happy as well if I would be able to implement that kind of "generic programming". I find that to be elegant, although it may perfectly not worth the effort, I don't know. Well, for me the whole issue was getting rank-0 values as python objects. Sorry if my point of view is adding confusion to these issues. Cheers, -- Francesc Alted From tim.hochberg at cox.net Tue Jun 8 08:44:10 2004 From: tim.hochberg at cox.net (Tim Hochberg) Date: Tue Jun 8 08:44:10 2004 Subject: [Numpy-discussion] Accessing rank-0 array value? In-Reply-To: <200406081043.26475.falted@pytables.org> References: <200406081043.26475.falted@pytables.org> Message-ID: <40C5DC95.2050307@cox.net> [SNIP] >>>>a[0] >>>> >>>> >Traceback (most recent call last): > File "", line 1, in ? >IndexError: Too many indices > >by something like: > > > >>>>a[0] >>>> >>>> >Traceback (most recent call last): > File "", line 1, in ? >IndexError: rank-0 arrays don't accept integer indices. Use '()' (empty tuple) >instead. > This seems like a sensible idea. One might further restrict this and only raise it when a rank-0 array is indexed with zero, which is the only real case that causes confusion. In that case, I suggest the error message similar to: IndexError: Too many indices (Use A[()] to get scalar from rank-0 array A) >In order to provide an universal accessor to numarray objects, what about >adding a .toscalar() (or .toobject()) method to them? That would return a >python object whether you are using rank-0 arrays, regular arrays, >CharArrays or RecArrays. That object would be the minimal python container >that would keep the values inside these objects. In case of rank-0 arrays it >would return a Bool, Int, Float or Complex. For a regular array, a list >(perhaps a tuple?). For CharArrays, a list (tuple?) of strings. And for >RecArrays a list (tuple) of tuples. That's an interesting idea. `tolist` doesn't work on rank-0 arrays, sensibly enough. I would expect something called toscalar to only work on rank-0 arrays. I don't like that since I would like to see fewer special cases for rank-0 arrays not less. However, something called `toobject` (or `topyobject`, `tocoreobject`, etc) could return a core python object that was equivalent to the original array, which I believe is what you describe above. For example, after: obj = anArray.toobject() type = anArray.type() newArray = numarray.array(obj, type) `newArray` would always be equal to `anArray` in both value and type. The implementation of toobject could be as simple as: def toobject(self): if self.rank == 0: return self[()] else: return self.tolist() I'm not entirely convinced it's a good idea yet; I'd have to see some use cases, but it's an interesting idea in any event. >Incidentally, I've noted an inconsistency in the .tostring behaviour: > > > >>>>a=array(126) >>>>a.tostring() >>>> >>>> >'' > > Interesting. That's better than the memory error I get with Numarray-0.8. Or the indefinite hang I get with numarray 0.9. >while I would expect a return value like: > > > >>>>chr(126) >>>> >>>> >'~' > > I think it should actually be returning the same things as array([126]).tostring(). That is: '\x80\x00\x00\x00' [SNIP] Regards, -tim From bens at MIT.EDU Tue Jun 8 09:08:03 2004 From: bens at MIT.EDU (Benjamin M. Schwartz) Date: Tue Jun 8 09:08:03 2004 Subject: [Numpy-discussion] Accessing rank-0 array value? In-Reply-To: <200406071827.51257.falted@pytables.org> References: <07C6A61102C94148B8104D42DE95F7E86DED27@exchange2k.envision.co.il> <40C48F02.707@sympatico.ca> <1086624252.5719.4.camel@halloween.stsci.edu> <200406071827.51257.falted@pytables.org> Message-ID: <40C5ABBB.3050304@mit.edu> Francesc Alted wrote: >For me, the basic problem is that a[()] notation would be the best way to >get the python object with a type close to that of the numarray object. Why >not letting a[...] or a[:] to return the same object as a[()]?. I know that >this is not consistent with the "..." or ":" use in non-scalar arrays, but I >find any of last two far more intuitive than a "()" index. > > I don't agree with this idea as written; I think that a[:] should always return a. However, I think that there is another relevant point here: single-element arrays have no rank! In other words, from a purely mathematical point of view, it makes sense not to distinguish between the number 5, the vector with a single element 5, the 1x1 matrix 5, the 1x1x1 object-of-some-sort 5, etc. Accordingly, if a is a single-element array, it would be sensible for a[()]==a[0]==a[0,0]==a[0,0,0]==a[(0)]==a[(0, 0, 0, 0, 0)] Asking for the array's dimensions would then require a convention. -long-time-reader, first-time-poster Ben Schwartz From jmiller at stsci.edu Tue Jun 8 12:37:38 2004 From: jmiller at stsci.edu (Todd Miller) Date: Tue Jun 8 12:37:38 2004 Subject: [Numpy-discussion] bug in object arrays: In-Reply-To: <1442AF29-B7D4-11D8-A5F0-000A95C92C8E@embl-heidelberg.de> References: <1442AF29-B7D4-11D8-A5F0-000A95C92C8E@embl-heidelberg.de> Message-ID: <1086723359.8041.89.camel@halloween.stsci.edu> On Sun, 2004-06-06 at 12:10, Peter Verveer wrote: > Is this a bug? I think this is more of a bonanza: a bug and an ambiguity. > This should result in an array of numpy arrays, but it does give an > error: > > >>> a = array([1,2]) > >>> b = array([3,4]) > >>> c = objects.array([a,b]) > Traceback (most recent call last): > File "", line 1, in ? > File > "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/ > python2.3/site-packages/numarray/objects.py", line 732, in array > return fromlist(sequence, shape) > File > "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/ > python2.3/site-packages/numarray/objects.py", line 755, in fromlist > return ObjectArray(objects=l, shape=shape) > File > "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/ > python2.3/site-packages/numarray/objects.py", line 506, in __init__ > oshape = _shapeFromNestedSequence(objects) > File > "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/ > python2.3/site-packages/numarray/objects.py", line 289, in > _shapeFromNestedSequence > return [len(s)] + _shapeFromNestedSequence(s[0]) > TypeError: can only concatenate list (not "tuple") to list The bug is that your example didn't work as follows: >>> import numarray.objects as obj >>> obj.array([a,b]) ObjectArray([[1, 2], [3, 4]]) Returning a shape=(2,) object array is another obvious behavior but turns out not to be what Numeric does and hence not what numarray does either. Faced with sequence objects, Numeric and numarray both just keep recursing (logically anyway) until they hit something which isn't a sequence. Thus, they return not a 1D array of arrays, but a 2D array of numbers: >>> import Numeric Numeric.array([a,b], typecode='O') array([[1 , 2 ], [3 , 4 ]],'O') With some new code I added today, numarray of the future will support your preferred behavior like this, barring further discussion: >>> obj.array([a,b], rank=1) ObjectArray([array([1, 2]), array([3, 4])]) Here, the new "rank" parameter explicitly specifies the expected rank of the resulting array, defining the point at which nested sequences become "objects in their own right". Perry's inclination was that rank should default to 1 so that your expected behavior was the default. Since that's not backward compatible with Numeric (or numarray-0.9) I think maybe rank=None is better. In this case, rank=None is equivalent to rank=2. How does the rank parameter sound? Regards, Todd From verveer at embl-heidelberg.de Tue Jun 8 18:21:47 2004 From: verveer at embl-heidelberg.de (Peter Verveer) Date: Tue Jun 8 18:21:47 2004 Subject: [Numpy-discussion] bug in object arrays: In-Reply-To: <1086723359.8041.89.camel@halloween.stsci.edu> References: <1442AF29-B7D4-11D8-A5F0-000A95C92C8E@embl-heidelberg.de> <1086723359.8041.89.camel@halloween.stsci.edu> Message-ID: > The bug is that your example didn't work as follows: > >>>> import numarray.objects as obj >>>> obj.array([a,b]) > ObjectArray([[1, 2], > [3, 4]]) > > Returning a shape=(2,) object array is another obvious behavior but > turns out not to be what Numeric does and hence not what numarray does > either. Faced with sequence objects, Numeric and numarray both just > keep recursing (logically anyway) until they hit something which isn't > a > sequence. Thus, they return not a 1D array of arrays, but a 2D array > of > numbers: > >>>> import Numeric > Numeric.array([a,b], typecode='O') > array([[1 , 2 ], > [3 , 4 ]],'O') > > With some new code I added today, numarray of the future will support > your preferred behavior like this, barring further discussion: > >>>> obj.array([a,b], rank=1) > ObjectArray([array([1, 2]), array([3, 4])]) > > Here, the new "rank" parameter explicitly specifies the expected rank > of > the resulting array, defining the point at which nested sequences > become > "objects in their own right". Perry's inclination was that rank should > default to 1 so that your expected behavior was the default. Since > that's not backward compatible with Numeric (or numarray-0.9) I think > maybe rank=None is better. In this case, rank=None is equivalent to > rank=2. > > How does the rank parameter sound? > > Regards, > Todd I see the logic of the recursing but it was unexpected in this particular example. The rank parameter is a solution, but I don't think it is very elegant to be honest. rank=1 should certainly not be the default, because it would not work if the rank of the desired object array is not one... For example, I want to make object arrays of arbitrary rank containing numarrays, so I would need to specify the rank anyway. How about stopping the recursion as soon as an object is found that is not a sequence, or that is a sequence of but of a different type? In my example the outer sequence is a list and the objects are arrays, so as soon as a array is seen the recursion would stop, giving the desired behavior. Is this idea too complex? If not, could some optional argument switch to this behaviour? Or maybe some special value for rank (-1) could be used for that? The rank parameter would still be useful if you want to stop the recursion in a nested list, then you must give some kind of hint anyway. It would give the desired behavior in many cases, so I would actually prefer to see something like that as the default. Maybe my suggestion would not break things to much since nesting different types of sequence to generate a homogeneous object array may not be done often anyway (its not what you want usually, I guess). Cheers, Peter From nadavh at visionsense.com Wed Jun 9 03:28:01 2004 From: nadavh at visionsense.com (Nadav Horesh) Date: Wed Jun 9 03:28:01 2004 Subject: [Numpy-discussion] rank-0 arrays: Something useful Message-ID: <40C6E59B.2070403@visionsense.com> One can use rank 0 arrays instead of scalars to automatically promote type of scalar-array operations: >>> from numarray import * >>> a = arange(10, type=Int16) >>> a * 10 array([ 0, 10, 20, 30, 40, 50, 60, 70, 80, 90], type=Int16) # same type >>> a * array(10, type=Int32) array([ 0, 10, 20, 30, 40, 50, 60, 70, 80, 90]) # Type promoted to Int32 Is there any way to economically set the type of the operation result? something like: multiply(a,b, type=xxx) # Doesn't work instead of: (a*b).astype(xxx) Nadav. From southey at uiuc.edu Wed Jun 9 08:06:11 2004 From: southey at uiuc.edu (Bruce Southey) Date: Wed Jun 9 08:06:11 2004 Subject: [Numpy-discussion] RandomArray.random() Message-ID: Hi, I would be prepared to at least look at it with numarray if you are prepared to share the scripts or rather the relevant part of them. On the surface, it would appear that this is a 32 vs 64 bit problem in defining the type precision. Regards Bruce ---- Original message ---- >Date: Thu, 3 Jun 2004 14:18:56 +0300 >From: Janne Sinkkonen >Subject: [Numpy-discussion] RandomArray.random() >To: numpy-discussion at lists.sourceforge.net > >Related to a recent discussion on random numbers: > >People using RandomArray.random() on 64-bit architectures should also be aware >that sometimes the function returns exactly zero or one, which in turn causes >problems in many other routines which generate e.g. multinomial or dirichlet >variates. > >I'm not absolutely sure that the problem persists in the newest version of >Numeric, and I have not tested it on numarray. Anyway, I have seen it both in >Alphas and now later in an AMD Opteron machine - over the years. > >Unfortunately, as we do not have had time to dig any deeper around here, we >have just used wrapper with a test and a loop. > >-- >Janne > > > >------------------------------------------------------- >This SF.Net email is sponsored by the new InstallShield X. >From Windows to Linux, servers to mobile, InstallShield X is the one >installation-authoring solution that does it all. Learn more and >evaluate today! http://www.installshield.com/Dev2Dev/0504 >_______________________________________________ >Numpy-discussion mailing list >Numpy-discussion at lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/numpy-discussion From cjw at sympatico.ca Thu Jun 10 07:27:03 2004 From: cjw at sympatico.ca (Colin J. Williams) Date: Thu Jun 10 07:27:03 2004 Subject: [Numpy-discussion] ObjectArray Message-ID: <40C86F7D.20309@sympatico.ca> ObjectArray, in objects.py, seems to offer a neat packaging capability. The examples focus on sting data, but it appears capable of handling all simple types and some container types. What is the intent? Dictionaries seem OK, tuples are OK when the shape is not specified, but an instance of ObjectArray fails. Colin W. From Jason.Ruiter at altarum.org Fri Jun 11 06:40:04 2004 From: Jason.Ruiter at altarum.org (Jason Ruiter) Date: Fri Jun 11 06:40:04 2004 Subject: [Numpy-discussion] Numarray problem on Opteron system, Suse 9.1 Message-ID: <40C9B597.2000600@altarum.org> Greetings, I'm running Suse 9.1 on a dual opteron system with 16GB of RAM. I'm using Python 2.3.3, Numeric 23.1 and numarray 0.9. I'm trying to allocate large (>4GB) arrays. Under Numeric, I get: Python 2.3.3 (#1, Apr 6 2004, 09:45:08) [GCC 3.3.3 (SuSE Linux)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from Numeric import * >>> a=ones(2000000000,Int16); Traceback (most recent call last): File "", line 1, in ? File "/usr/lib64/python2.3/site-packages/Numeric/Numeric.py", line 578, in ones a=zeros(shape, typecode, savespace) MemoryError: can't allocate memory for array >>> Under numarray: Python 2.3.3 (#1, Apr 6 2004, 09:45:08) [GCC 3.3.3 (SuSE Linux)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from numarray import * >>> a=ones(2000000000,Int16) Traceback (most recent call last): File "", line 1, in ? File "/usr/lib64/python2.3/site-packages/numarray/numarraycore.py", line 1111, in ones retarr = _fillarray(_gen.product(shape), 1, 0, type) File "/usr/lib64/python2.3/site-packages/numarray/numarraycore.py", line 144, in _fillarray outarr = NumArray((size,), outtype) ValueError: new_memory: invalid region size: -294967296. >>> I've verified that python, Numeric, and numarray are built and linked against the 64bit libraries. I've also verified that it's possible to allocate a >16GB Array with the following program I found on one of the debian mailling lists. :q:q:q! #include #include int main() { size_t n; void *p; double gb; for(gb=20;gb>.3;gb-=.5) { n= 1024L * 1024L * 1024L * gb; p = malloc( n ); printf("%12lu %4.1lfGb %p\n",n,n/1024./1024./1024.,p); free(p); } return 0; } 21474836480 20.0Gb (nil) 20937965568 19.5Gb (nil) 20401094656 19.0Gb (nil) 19864223744 18.5Gb (nil) 19327352832 18.0Gb (nil) 18790481920 17.5Gb (nil) 18253611008 17.0Gb (nil) 17716740096 16.5Gb 0x3995a02010 17179869184 16.0Gb 0x3995a02010 16642998272 15.5Gb 0x3995a02010 16106127360 15.0Gb 0x3995a02010 15569256448 14.5Gb 0x3995a02010 15032385536 14.0Gb 0x3995a02010 14495514624 13.5Gb 0x3995a02010 13958643712 13.0Gb 0x3995a02010 13421772800 12.5Gb 0x3995a02010 12884901888 12.0Gb 0x3995a02010 12348030976 11.5Gb 0x3995a02010 11811160064 11.0Gb 0x3995a02010 11274289152 10.5Gb 0x3995a02010 10737418240 10.0Gb 0x3995a02010 10200547328 9.5Gb 0x3995a02010 9663676416 9.0Gb 0x3995a02010 9126805504 8.5Gb 0x3995a02010 8589934592 8.0Gb 0x3995a02010 8053063680 7.5Gb 0x3995a02010 7516192768 7.0Gb 0x3995a02010 6979321856 6.5Gb 0x3995a02010 6442450944 6.0Gb 0x3995a02010 5905580032 5.5Gb 0x3995a02010 5368709120 5.0Gb 0x3995a02010 4831838208 4.5Gb 0x3995a02010 4294967296 4.0Gb 0x3995a02010 3758096384 3.5Gb 0x3995a02010 3221225472 3.0Gb 0x3995a02010 2684354560 2.5Gb 0x3995a02010 2147483648 2.0Gb 0x3995a02010 1610612736 1.5Gb 0x3995a02010 1073741824 1.0Gb 0x3995a02010 536870912 0.5Gb 0x3995a02010 Can someone point me in the right direction? Thanks Jason -- Jason Ruiter Research Scientist Environment and Emerging Technologies Division Altarum Institute Ann Arbor, Michigan, USA (v)734.302.4724 (f)734.302.4991 (m)248.345.4598 Jason.Ruiter at Altarum.org From jmiller at stsci.edu Fri Jun 11 08:50:12 2004 From: jmiller at stsci.edu (Todd Miller) Date: Fri Jun 11 08:50:12 2004 Subject: [Numpy-discussion] Numarray problem on Opteron system, Suse 9.1 In-Reply-To: <40C9B597.2000600@altarum.org> References: <40C9B597.2000600@altarum.org> Message-ID: <1086968959.5585.61.camel@halloween.stsci.edu> On Fri, 2004-06-11 at 09:37, Jason Ruiter wrote: > Greetings, > Hi, I think you're getting into bleeding edge territory. For numarray, the particular problem you exposed should be solved for 1.0, but I still think the overall 64-bit prognosis is not good. There are API issues in Python itself which are going to make really large arrays a problem until they're solved; for instance, the sequence protocol returns the length of a sequence as an int (32-bits on most platforms). Regards, Todd > I'm running Suse 9.1 on a dual opteron system with 16GB of RAM. I'm > using Python 2.3.3, Numeric 23.1 and numarray 0.9. > > I'm trying to allocate large (>4GB) arrays. Under Numeric, I get: > > Python 2.3.3 (#1, Apr 6 2004, 09:45:08) > [GCC 3.3.3 (SuSE Linux)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from Numeric import * > >>> a=ones(2000000000,Int16); > Traceback (most recent call last): > File "", line 1, in ? > File "/usr/lib64/python2.3/site-packages/Numeric/Numeric.py", line > 578, in ones > a=zeros(shape, typecode, savespace) > MemoryError: can't allocate memory for array > >>> > > Under numarray: > > Python 2.3.3 (#1, Apr 6 2004, 09:45:08) > [GCC 3.3.3 (SuSE Linux)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from numarray import * > >>> a=ones(2000000000,Int16) > Traceback (most recent call last): > File "", line 1, in ? > File "/usr/lib64/python2.3/site-packages/numarray/numarraycore.py", > line 1111, in ones > retarr = _fillarray(_gen.product(shape), 1, 0, type) > File "/usr/lib64/python2.3/site-packages/numarray/numarraycore.py", > line 144, in _fillarray > outarr = NumArray((size,), outtype) > ValueError: new_memory: invalid region size: -294967296. > >>> > > > I've verified that python, Numeric, and numarray are built and linked > against the 64bit libraries. > > I've also verified that it's possible to allocate a >16GB Array with the > following program I found on one of the debian mailling lists. > :q:q:q! > #include > #include > int main() { size_t n; void *p; double gb; > for(gb=20;gb>.3;gb-=.5) { > n= 1024L * 1024L * 1024L * gb; > p = malloc( n ); > printf("%12lu %4.1lfGb %p\n",n,n/1024./1024./1024.,p); > free(p); } return 0; } > > 21474836480 20.0Gb (nil) > 20937965568 19.5Gb (nil) > 20401094656 19.0Gb (nil) > 19864223744 18.5Gb (nil) > 19327352832 18.0Gb (nil) > 18790481920 17.5Gb (nil) > 18253611008 17.0Gb (nil) > 17716740096 16.5Gb 0x3995a02010 > 17179869184 16.0Gb 0x3995a02010 > 16642998272 15.5Gb 0x3995a02010 > 16106127360 15.0Gb 0x3995a02010 > 15569256448 14.5Gb 0x3995a02010 > 15032385536 14.0Gb 0x3995a02010 > 14495514624 13.5Gb 0x3995a02010 > 13958643712 13.0Gb 0x3995a02010 > 13421772800 12.5Gb 0x3995a02010 > 12884901888 12.0Gb 0x3995a02010 > 12348030976 11.5Gb 0x3995a02010 > 11811160064 11.0Gb 0x3995a02010 > 11274289152 10.5Gb 0x3995a02010 > 10737418240 10.0Gb 0x3995a02010 > 10200547328 9.5Gb 0x3995a02010 > 9663676416 9.0Gb 0x3995a02010 > 9126805504 8.5Gb 0x3995a02010 > 8589934592 8.0Gb 0x3995a02010 > 8053063680 7.5Gb 0x3995a02010 > 7516192768 7.0Gb 0x3995a02010 > 6979321856 6.5Gb 0x3995a02010 > 6442450944 6.0Gb 0x3995a02010 > 5905580032 5.5Gb 0x3995a02010 > 5368709120 5.0Gb 0x3995a02010 > 4831838208 4.5Gb 0x3995a02010 > 4294967296 4.0Gb 0x3995a02010 > 3758096384 3.5Gb 0x3995a02010 > 3221225472 3.0Gb 0x3995a02010 > 2684354560 2.5Gb 0x3995a02010 > 2147483648 2.0Gb 0x3995a02010 > 1610612736 1.5Gb 0x3995a02010 > 1073741824 1.0Gb 0x3995a02010 > 536870912 0.5Gb 0x3995a02010 > > > Can someone point me in the right direction? > > Thanks > Jason -- Todd Miller From jmiller at stsci.edu Fri Jun 11 13:56:03 2004 From: jmiller at stsci.edu (Todd Miller) Date: Fri Jun 11 13:56:03 2004 Subject: [Numpy-discussion] ObjectArray In-Reply-To: <40C9C779.6090700@sympatico.ca> References: <40C86F7D.20309@sympatico.ca> <1086899001.4044.24.camel@halloween.stsci.edu> <40C9C779.6090700@sympatico.ca> Message-ID: <1086987202.5585.366.camel@halloween.stsci.edu> On Fri, 2004-06-11 at 10:53, Colin J. Williams wrote: > Todd, > > I'm trying out objects.py from the CVS as of June 11, with PROTOTYPE= > 1 > > Suggestions and comments based on that. > > 1. change the order in ObjectsArray so the __init__ becomes: > def __init__(self, objects=None, shape=None, > rank=None): > The thinking is that, in the absence of others, objects is the > argument most likely to be specified. I think this can go either way. For people that care, there are already array() and fromlist(). Subclasses can override the order. Am I missing something? > 2. add a check in __init__ to ensure that the shape is in fact > shape-like. Right now it raises a TypeError, and although it's ugly, it does at least trap the error and give a clue in the right direction. > 3. real doc strings would be helpful. Currently, the doc strings are > monopolized by the doctest stuff, > which was not the original intent of doc strings. True enough. Maybe for 1.1. > 4. I wonder about the need for both rank and shape. NumArray gets > along fine with just shape. This is a new feature, and is useful in the context of object arrays where a sequence object is an ambiguous thing: should the sequence be seen as containing elements of an object array, or itself an element of the object array. rank lets you specify the rank of the resulting object array explicitly. It is also computed implicitly if not specified based on the first object which doesn't match the type of the original sequence. This is new for 1.0. > 5. It is assumed that objects (in __init__) is a sequence, but no > check to make sure that it is. Actually, for 1.0 I don't even think it has to be a sequence anymore. It's possible to make a rank-0 object array. > 6. _byteorder is specified for NumArray, but not for ObjectArray, > this upsets a utility which > looks at array characteristics. _byteorder is a private attribute which doesn't make sense for ObjectArray. Add exception handling to the utility if it must be used with ObjectArrays. > 7. How is an ObjectArray better than a nested list of objects? It > seems to provide speedier access > to object pointers for larger n. It may offer the opportunity > to present data in a more organized > tabular manner. Dictionaries are better, in that they provide > access by means of an arbitrary key, > for the cost of hashing. > > NumArray appears to offer storage saving, in addition to the > above, but not here. > > Am I missing something? ObjectArrays can be indexed, manipulated, and printed like numarrays. ObjectArrays also work with object array ufuncs, so for instance, adding two ObjectArrays together implicitly applies the add builtin to each pair of elements. These ufuncs even support reductions and accumulations. This *is* a good question, but I still think ObjectArrays are useful in some contexts. Thanks for the feedback. Regards, Todd From nadavh at visionsense.com Mon Jun 14 03:41:01 2004 From: nadavh at visionsense.com (Nadav Horesh) Date: Mon Jun 14 03:41:01 2004 Subject: [Numpy-discussion] Subclassing NumArray: _PROTOTYPE flag issue Message-ID: <40CD8055.7050209@visionsense.com> For a simulation project I am working on I've subclasses ArrayType. I was able to do much of my intentions until in one place when I tried to make an array from a list of arrays I got an error message: . . . File "/usr/local/lib/python2.3/site-packages/numarray/numarraycore.py", line 325, in array return fromlist(sequence, type, shape) File "/usr/local/lib/python2.3/site-packages/numarray/numarraycore.py", line 212, in fromlist a = a.astype(type) File "/usr/local/lib/python2.3/site-packages/numarray/numarraycore.py", line 630, in astype retarr = self.__class__(buffer=None, shape=self._shape, type=type) TypeError: __init__() got an unexpected keyword argument 'buffer' The analysis of the code showed that: 1. The NumArray class method definitions depends on the _PROTOTYPE flag 2. The post-mortem debugging showed that when the error flagged, the value of the variable _PROTOTYPE was 0 In a stand alone script there was no problem to do the list-> array conversion: >>> import numarray as N >>> import NumImage as NI # My module with the derived class >>> a = N.arange(4) >>> ia = NI.Cimage(N.arange(4)) # CImage is a derivative of NumImage >>> a array([0, 1, 2, 3]) >>> ia Cimage([0, 1, 2, 3]) >>> N.array([a+i for i in range(3)]) array([[0, 1, 2, 3], [1, 2, 3, 4], [2, 3, 4, 5]]) >>> N.array([ia+i for i in range(3)]) # OK here, but failed as a part of a complex script Cimage([[0, 1, 2, 3], [1, 2, 3, 4], [2, 3, 4, 5]]) My questions are: 1. Is this flag is in use? If I set it to 0 will I be able to derive a class from the "C code"? 2. Any intelligent solution? Nadav. From cjw at sympatico.ca Mon Jun 14 09:46:46 2004 From: cjw at sympatico.ca (Colin J. Williams) Date: Mon Jun 14 09:46:46 2004 Subject: [Numpy-discussion] ObjectArray In-Reply-To: <1086987202.5585.366.camel@halloween.stsci.edu> References: <40C86F7D.20309@sympatico.ca> <1086899001.4044.24.camel@halloween.stsci.edu> <40C9C779.6090700@sympatico.ca> <1086987202.5585.366.camel@halloween.stsci.edu> Message-ID: <40CDD617.2010605@sympatico.ca> Todd Miller wrote: >On Fri, 2004-06-11 at 10:53, Colin J. Williams wrote: > > >>Todd, >> >>I'm trying out objects.py from the CVS as of June 11, with PROTOTYPE= >>1 >> >>Suggestions and comments based on that. >> >>1. change the order in ObjectsArray so the __init__ becomes: >> def __init__(self, objects=None, shape=None, >>rank=None): >> The thinking is that, in the absence of others, objects is the >>argument most likely to be specified. >> >> > >I think this can go either way. For people that care, there are >already array() and fromlist(). Subclasses can override the order. Am >I missing something? > The convenience of being able to call the ObjectArray constructor without using keywords. For this purpose, I suggest that it's best to order the parameters in the most likely frequency of usage. I've found that it's not good to have a subclass with a different signature. A generic call to a constructor is as likely to call the subclass as the class. >>2. add a check in __init__ to ensure that the shape is in fact >>shape-like. >> >> > >Right now it raises a TypeError, and although it's ugly, it does at >least trap the error and give a clue in the right direction. > The error traceback is rather deep and doesn't directly point to the source of the problem. >>3. real doc strings would be helpful. Currently, the doc strings are >>monopolized by the doctest stuff, >> which was not the original intent of doc strings. >> >> > >True enough. Maybe for 1.1. > > This is particularly important for objects which are not elsewhere documented, including private functions. >>4. I wonder about the need for both rank and shape. NumArray gets >>along fine with just shape. >> >> > >This is a new feature, and is useful in the context of object arrays >where a sequence object is an ambiguous thing: should the sequence be >seen as containing elements of an object array, or itself an element of >the object array. rank lets you specify the rank of the resulting >object array explicitly. It is also computed implicitly if not >specified based on the first object which doesn't match the type of the >original sequence. This is new for 1.0. > I hadn't spotted that rank gives one the opportunity to specify the dimensionality of objects, but shape provides another way of doing it. >>5. It is assumed that objects (in __init__) is a sequence, but no >>check to make sure that it is. >> >> > >Actually, for 1.0 I don't even think it has to be a sequence anymore. >It's possible to make a rank-0 object array. > > Yes, it is, this adds to the generality. >>6. _byteorder is specified for NumArray, but not for ObjectArray, >>this upsets a utility which >> looks at array characteristics. >> >> > >_byteorder is a private attribute which doesn't make sense for >ObjectArray. Add exception handling to the utility if it must be used >with ObjectArrays. > I'll do that. > > > >>7. How is an ObjectArray better than a nested list of objects? It >>seems to provide speedier access >> to object pointers for larger n. It may offer the opportunity >>to present data in a more organized >> tabular manner. Dictionaries are better, in that they provide >>access by means of an arbitrary key, >> for the cost of hashing. >> >> NumArray appears to offer storage saving, in addition to the >>above, but not here. >> >> Am I missing something? >> >> > >ObjectArrays can be indexed, manipulated, and printed like numarrays. >ObjectArrays also work with object array ufuncs, so for instance, adding >two ObjectArrays together implicitly applies the add builtin to each >pair of elements. These ufuncs even support reductions and >accumulations. This *is* a good question, but I still think >ObjectArrays are useful in some contexts. > The ability to use the ufuncs is potentially good, but presumably assumes a homogeneous ObjectArray. We can't mix strings and dictionaries. > >Thanks for the feedback. > >Regards, >Todd > > > Many thanks for your response. ObjectArray provides a flexible addition to numarray. Colin W > > > From perry at stsci.edu Mon Jun 14 11:28:04 2004 From: perry at stsci.edu (Perry Greenfield) Date: Mon Jun 14 11:28:04 2004 Subject: [Numpy-discussion] Subclassing NumArray: _PROTOTYPE flag issue In-Reply-To: <40CD8055.7050209@visionsense.com> Message-ID: Unfortunately Todd is away this week. I can try to help if you can illustrate what is being tried in the more complex script that you are referring to that is failing. Offhand, I didn't believe that _PROTOTYPE should ever be set to 0 unless it was done so explicitly for testing or debugging purposes (but perhaps I misremember). Thanks, Perry Nadav Horesh wrote: > For a simulation project I am working on I've subclasses ArrayType. I > was able to do much of my intentions until in one place when I tried to > make an array from a list of arrays I got an error message: > > . > . > . > File > "/usr/local/lib/python2.3/site-packages/numarray/numarraycore.py", line > 325, in array > return fromlist(sequence, type, shape) > File > "/usr/local/lib/python2.3/site-packages/numarray/numarraycore.py", line > 212, in fromlist > a = a.astype(type) > File > "/usr/local/lib/python2.3/site-packages/numarray/numarraycore.py", line > 630, in astype > retarr = self.__class__(buffer=None, shape=self._shape, type=type) > TypeError: __init__() got an unexpected keyword argument 'buffer' > > The analysis of the code showed that: > > 1. The NumArray class method definitions depends on the _PROTOTYPE flag > 2. The post-mortem debugging showed that when the error flagged, the > value of the variable _PROTOTYPE was 0 > > In a stand alone script there was no problem to do the list-> array > conversion: > > >>> import numarray as N > >>> import NumImage as NI # My module with the derived class > >>> a = N.arange(4) > >>> ia = NI.Cimage(N.arange(4)) # CImage is a derivative of NumImage > >>> a > array([0, 1, 2, 3]) > >>> ia > Cimage([0, 1, 2, 3]) > >>> N.array([a+i for i in range(3)]) > array([[0, 1, 2, 3], > [1, 2, 3, 4], > [2, 3, 4, 5]]) > >>> N.array([ia+i for i in range(3)]) # OK here, but failed as a part > of a complex script > Cimage([[0, 1, 2, 3], > [1, 2, 3, 4], > [2, 3, 4, 5]]) > > > My questions are: > > 1. Is this flag is in use? If I set it to 0 will I be able to derive > a class from the "C code"? > 2. Any intelligent solution? > > Nadav. > > > ------------------------------------------------------- > This SF.Net email is sponsored by the new InstallShield X. > From Windows to Linux, servers to mobile, InstallShield X is the > one installation-authoring solution that does it all. Learn more and > evaluate today! http://www.installshield.com/Dev2Dev/0504 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From haase at msg.ucsf.edu Mon Jun 14 16:03:02 2004 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Mon Jun 14 16:03:02 2004 Subject: [Numpy-discussion] numarray.sum should raise exception for wrong axis argument Message-ID: <200406141602.07650.haase@msg.ucsf.edu> Hi, please take a look at this: >>> na.sum( na.zeros((2,6)) ) [0 0 0 0 0 0] >>> na.sum( na.zeros((2,6)) , 0) [0 0 0 0 0 0] >>> na.sum( na.zeros((2,6)) , 1) [0 0] >>> na.sum( na.zeros((2,6)) , 2) [0 0] >>> na.sum( na.zeros((2,6)) , 3) [0 0] >>> na.sum( na.zeros((2,6)) , 4) [0 0] >>> na.sum( na.zeros((2,6)) , -1) [0 0] >>> na.sum( na.zeros((2,6)) , -2) [0 0 0 0 0 0] >>> na.sum( na.zeros((2,6)) , -3) [0 0] >>> na.sum( na.zeros((2,6)) , -4) [0 0] >>> I think here should be a ValueError exception thrown rather than defaulting to the '-1'-axis. Comments ? Thanks, Sebastian Haase From nadavh at visionsense.com Wed Jun 16 08:43:14 2004 From: nadavh at visionsense.com (Nadav Horesh) Date: Wed Jun 16 08:43:14 2004 Subject: [Numpy-discussion] Subclassing NumArray: _PROTOTYPE flag issue Message-ID: <07C6A61102C94148B8104D42DE95F7E86DED3D@exchange2k.envision.co.il> OK it is my mistake, I did not implement the buffer argument in the constructor of my derived class. Still I have some minor problems, but nothing to bother about for the time being. It is very compelling to subclass NumArray, I wish that it will be documented when version 1.0 will be released. Nadav -----Original Message----- From: Perry Greenfield [mailto:perry at stsci.edu] Sent: Mon 14-Jun-04 21:27 To: Nadav Horesh; numpy Cc: Subject: RE: [Numpy-discussion] Subclassing NumArray: _PROTOTYPE flag issue Unfortunately Todd is away this week. I can try to help if you can illustrate what is being tried in the more complex script that you are referring to that is failing. Offhand, I didn't believe that _PROTOTYPE should ever be set to 0 unless it was done so explicitly for testing or debugging purposes (but perhaps I misremember). Thanks, Perry Nadav Horesh wrote: > For a simulation project I am working on I've subclasses ArrayType. I > was able to do much of my intentions until in one place when I tried to > make an array from a list of arrays I got an error message: > > . > . > . > File > "/usr/local/lib/python2.3/site-packages/numarray/numarraycore.py", line > 325, in array > return fromlist(sequence, type, shape) > File > "/usr/local/lib/python2.3/site-packages/numarray/numarraycore.py", line > 212, in fromlist > a = a.astype(type) > File > "/usr/local/lib/python2.3/site-packages/numarray/numarraycore.py", line > 630, in astype > retarr = self.__class__(buffer=None, shape=self._shape, type=type) > TypeError: __init__() got an unexpected keyword argument 'buffer' > > The analysis of the code showed that: > > 1. The NumArray class method definitions depends on the _PROTOTYPE flag > 2. The post-mortem debugging showed that when the error flagged, the > value of the variable _PROTOTYPE was 0 > > In a stand alone script there was no problem to do the list-> array > conversion: > > >>> import numarray as N > >>> import NumImage as NI # My module with the derived class > >>> a = N.arange(4) > >>> ia = NI.Cimage(N.arange(4)) # CImage is a derivative of NumImage > >>> a > array([0, 1, 2, 3]) > >>> ia > Cimage([0, 1, 2, 3]) > >>> N.array([a+i for i in range(3)]) > array([[0, 1, 2, 3], > [1, 2, 3, 4], > [2, 3, 4, 5]]) > >>> N.array([ia+i for i in range(3)]) # OK here, but failed as a part > of a complex script > Cimage([[0, 1, 2, 3], > [1, 2, 3, 4], > [2, 3, 4, 5]]) > > > My questions are: > > 1. Is this flag is in use? If I set it to 0 will I be able to derive > a class from the "C code"? > 2. Any intelligent solution? > > Nadav. > > > ------------------------------------------------------- > This SF.Net email is sponsored by the new InstallShield X. > From Windows to Linux, servers to mobile, InstallShield X is the > one installation-authoring solution that does it all. Learn more and > evaluate today! http://www.installshield.com/Dev2Dev/0504 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From jbaddor at physics.mcgill.ca Thu Jun 17 11:47:03 2004 From: jbaddor at physics.mcgill.ca (Jean-Bernard Addor) Date: Thu Jun 17 11:47:03 2004 Subject: [Numpy-discussion] arrayfns.histogram does it exist in numarray? Message-ID: Hello ! I am analysing data in hdf files with Numeric. For different reasons, including better support of hdf files (huge datasets), I try to switch to numarray. I uses a lot the arrayfns.histogram function of Numeric, does something similar exist in nummarray? See you, Jean-Bernard From perry at stsci.edu Thu Jun 17 13:28:10 2004 From: perry at stsci.edu (Perry Greenfield) Date: Thu Jun 17 13:28:10 2004 Subject: [Numpy-discussion] arrayfns.histogram does it exist in numarray? In-Reply-To: Message-ID: Jean-Bernard Addor wrote: > I am analysing data in hdf files with Numeric. For different reasons, > including better support of hdf files (huge datasets), I try to switch to > numarray. I uses a lot the arrayfns.histogram function of Numeric, does > something similar exist in nummarray? > Not as such yet (we have our own more specialized histogram function for internal projects, but hadn't gotten around to generalizing it for general use). I think we'll take a quick look at seeing if the Numeric arrayfns extension can be recompiled to use numarray with no or few changes. Since Todd is away, this won't be done until next week. I also have a simple Python function to do this, albeit more slowly (but not hugely slower) if you need something quickly. Perry From Fernando.Perez at colorado.edu Thu Jun 17 16:28:00 2004 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Thu Jun 17 16:28:00 2004 Subject: [Numpy-discussion] Using dotblas Message-ID: <40D228F0.2050209@colorado.edu> Hi all, I've built NumPy 23.1 with dotblas enabled, and as per the readme I'm starting to pepper my code with things like: # External packages import Numeric as N # Use BLAS-optimized versions when available try: import dotblas N.dot = dotblas._dotblas.matrixproduct N.multiarray.innerproduct = dotblas._dotblas.innerproduct except ImportError: print '*** dotblas not available, using regular Numeric functions' I wonder if this is really necessary, and if so, why. Is there any reason _not_ to use the dotblas versions when available? If not, shouldn't Numeric itself pick them up internally so that it always exposes the fastest possible version? I can always send in a patch to do this ( a trivial 5-liner like the above code), but I may just be missing something, as it seems too obvious not to be already the default. Regards, f. From Fernando.Perez at colorado.edu Thu Jun 17 16:50:06 2004 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Thu Jun 17 16:50:06 2004 Subject: [Numpy-discussion] Using dotblas In-Reply-To: <40D22C18.4040804@ucsd.edu> References: <40D228F0.2050209@colorado.edu> <40D22C18.4040804@ucsd.edu> Message-ID: <40D22DF1.5020306@colorado.edu> Robert Kern wrote: > Check the last few lines of Numeric.py . Argh. Could've saved myself the embarrasment :) In my meager defense, the README.html file for doblas then needs to be updated, since that's what I followed: # quote Finally, if you don't know whether dotblas will be available on all the machines where you whish to run your code, you can simply use a try-except statement like this: import Numeric try: import dotblas Numeric.dot = dotblas.dot if we can't find the fast dotblas, just use the normal dot except ImportError: pass # /quote Thanks, though. I'll clean up my codes f From shawnemch at hotmail.com Sat Jun 19 21:56:01 2004 From: shawnemch at hotmail.com (OEMcd--Cheap--software--Inc.) Date: Sat Jun 19 21:56:01 2004 Subject: [Numpy-discussion] Review your account 555418212 Message-ID: <36149808074438614442897@hotmail.com> An HTML attachment was scrubbed... URL: From jmiller at stsci.edu Mon Jun 21 07:47:02 2004 From: jmiller at stsci.edu (Todd Miller) Date: Mon Jun 21 07:47:02 2004 Subject: [Numpy-discussion] arrayfns.histogram does it exist in numarray? In-Reply-To: References: Message-ID: <1087829108.2619.66.camel@localhost.localdomain> On Thu, 2004-06-17 at 16:27, Perry Greenfield wrote: > Jean-Bernard Addor wrote: > > I am analysing data in hdf files with Numeric. For different reasons, > > including better support of hdf files (huge datasets), I try to switch to > > numarray. I uses a lot the arrayfns.histogram function of Numeric, does > > something similar exist in nummarray? > > > Not as such yet (we have our own more specialized histogram > function for internal projects, but hadn't gotten around to > generalizing it for general use). I think we'll take a quick > look at seeing if the Numeric arrayfns extension can be recompiled > to use numarray with no or few changes. Since Todd is away, > this won't be done until next week. I also have a simple > Python function to do this, albeit more slowly (but not hugely > slower) if you need something quickly. I ported histogram() this morning to numarray.numeric. It's in CVS now and will be part of numarray-1.0 in the next week or so. Regards, Todd From jmiller at stsci.edu Mon Jun 21 08:00:05 2004 From: jmiller at stsci.edu (Todd Miller) Date: Mon Jun 21 08:00:05 2004 Subject: [Numpy-discussion] Subclassing NumArray: _PROTOTYPE flag issue In-Reply-To: <40CD8055.7050209@visionsense.com> References: <40CD8055.7050209@visionsense.com> Message-ID: <1087829929.2619.72.camel@localhost.localdomain> On Mon, 2004-06-14 at 06:39, Nadav Horesh wrote: > For a simulation project I am working on I've subclasses ArrayType. I > was able to do much of my intentions until in one place when I tried to > make an array from a list of arrays I got an error message: > > . > . > . > File > "/usr/local/lib/python2.3/site-packages/numarray/numarraycore.py", line > 325, in array > return fromlist(sequence, type, shape) > File > "/usr/local/lib/python2.3/site-packages/numarray/numarraycore.py", line > 212, in fromlist > a = a.astype(type) > File > "/usr/local/lib/python2.3/site-packages/numarray/numarraycore.py", line > 630, in astype > retarr = self.__class__(buffer=None, shape=self._shape, type=type) > TypeError: __init__() got an unexpected keyword argument 'buffer' > > The analysis of the code showed that: > > 1. The NumArray class method definitions depends on the _PROTOTYPE flag > 2. The post-mortem debugging showed that when the error flagged, the > value of the variable _PROTOTYPE was 0 > > In a stand alone script there was no problem to do the list-> array > conversion: > > >>> import numarray as N > >>> import NumImage as NI # My module with the derived class > >>> a = N.arange(4) > >>> ia = NI.Cimage(N.arange(4)) # CImage is a derivative of NumImage > >>> a > array([0, 1, 2, 3]) > >>> ia > Cimage([0, 1, 2, 3]) > >>> N.array([a+i for i in range(3)]) > array([[0, 1, 2, 3], > [1, 2, 3, 4], > [2, 3, 4, 5]]) > >>> N.array([ia+i for i in range(3)]) # OK here, but failed as a part > of a complex script > Cimage([[0, 1, 2, 3], > [1, 2, 3, 4], > [2, 3, 4, 5]]) > > > My questions are: > > 1. Is this flag is in use? Yes. > If I set it to 0 will I be able to derive > a class from the "C code"? Exactly. _PROTOTYPE hides the original Python implementations of things that have since been moved to C; it is a compile time flag. I keep the prototype Python code around because it is much easier to debug and very useful when modifying or extending numarray. Regards, Todd From falted at pytables.org Mon Jun 21 11:40:06 2004 From: falted at pytables.org (Francesc Alted) Date: Mon Jun 21 11:40:06 2004 Subject: [Numpy-discussion] Unpickled numarray types unsable? Message-ID: <200406212039.18331.falted@pytables.org> Hi, I'm experiencing problems when using a numarray datatype after pickle/unpickle it: >>> import numarray >>> import pickle >>> c=pickle.dumps(numarray.Float64) >>> t=pickle.loads(c) >>> t Float64 >>> na=numarray.array(type=t, shape=(2,)) Traceback (most recent call last): File "", line 1, in ? File "/usr/local/lib/python2.3/site-packages/numarray/numarraycore.py", line 321, in array return NumArray(buffer=sequence, shape=shape, type=type) RuntimeError: _numarray_init: can't get typeno for type Perhaps this is a bug and the typeno attribute is missed when reconstructing the type? Cheers, -- Francesc Alted From jmiller at stsci.edu Mon Jun 21 13:06:04 2004 From: jmiller at stsci.edu (Todd Miller) Date: Mon Jun 21 13:06:04 2004 Subject: [Numpy-discussion] Unpickled numarray types unsable? In-Reply-To: <200406212039.18331.falted@pytables.org> References: <200406212039.18331.falted@pytables.org> Message-ID: <1087848295.2619.105.camel@localhost.localdomain> On Mon, 2004-06-21 at 14:39, Francesc Alted wrote: > Hi, > > I'm experiencing problems when using a numarray datatype after > pickle/unpickle it: > > >>> import numarray > >>> import pickle > >>> c=pickle.dumps(numarray.Float64) > >>> t=pickle.loads(c) > >>> t > Float64 > >>> na=numarray.array(type=t, shape=(2,)) > Traceback (most recent call last): > File "", line 1, in ? > File "/usr/local/lib/python2.3/site-packages/numarray/numarraycore.py", line 321, in array > return NumArray(buffer=sequence, shape=shape, type=type) > RuntimeError: _numarray_init: can't get typeno for type > > Perhaps this is a bug and the typeno attribute is missed when reconstructing > the type? It's a bug in the C-API / implementation of pickling for the type objects. The C-API keeps references to the type objects which are assumed to be unique: there is one Float64, just like there is one None. The pickling produces "functional equivalents" which lose their identities from the perspective of C. I'm not sure what the fix is yet. Regards, Todd From faheem at email.unc.edu Mon Jun 21 17:18:02 2004 From: faheem at email.unc.edu (Faheem Mitha) Date: Mon Jun 21 17:18:02 2004 Subject: [Numpy-discussion] numarray.concatenate for character arrays Message-ID: Dear People, Is the function numarray.concatenate supposed to work for character arrays? It doesn't for me. Do I need to write my own? Thanks in advance. Please cc me, I'm not subscribed. Faheem. In [17]: foo Out[17]: CharArray([['T', 'T'], ['C', 'A']]) In [18]: bar Out[18]: CharArray([['G', 'G'], ['G', 'G']]) In [23]: numarray.concatenate((foo,bar)) --------------------------------------------------------------------------- error Traceback (most recent call last) /home/faheem/wc/corrmodel/trunk/ /usr/lib/python2.3/site-packages/numarray/generic.py in concatenate(arrs, axis) 1018 arrs = map(_nc.asarray, arrs) 1019 if axis == 0: -> 1020 return _concat(arrs) 1021 else: 1022 return swapaxes(_concat([swapaxes(m,axis,0) for m in arrs]), axis, 0) /usr/lib/python2.3/site-packages/numarray/generic.py in _concat(arrs) 1000 convType = ufunc._maxPopType(arrs) 1001 except TypeError: -> 1002 dest = arrs[0]._clone(shape=destShape) 1003 else: 1004 dest = arrs[0].__class__(shape=destShape, type=convType) /usr/lib/python2.3/site-packages/numarray/generic.py in _clone(self, shape) 783 def _clone(self, shape): 784 c = self.copy() --> 785 c.resize(shape) 786 return c 787 /usr/lib/python2.3/site-packages/numarray/generic.py in resize(self, shape, *args) 854 self[offset:offset+olen] = self[0:olen] 855 offset += olen --> 856 self[offset:nlen] = self[0:nlen-offset] 857 else: # zero fill resized zero-length numarray 858 self[:] = 0 /usr/lib/python2.3/site-packages/numarray/strings.py in _copyFrom(self, arr) 217 me = self._byteView() 218 if self._itemsize <= arr._itemsize: --> 219 me[:] = it[..., :self._itemsize] 220 else: 221 me[...,:it._shape[-1]] = it error: copy1bytes: access beyond buffer. offset=8 buffersize=8 From Sebastien.deMentendeHorne at electrabel.com Tue Jun 22 04:59:32 2004 From: Sebastien.deMentendeHorne at electrabel.com (Sebastien.deMentendeHorne at electrabel.com) Date: Tue Jun 22 04:59:32 2004 Subject: [Numpy-discussion] lock in LinearAlgebra when embedding Message-ID: <035965348644D511A38C00508BF7EAEB145CAEE0@seacex03.eib.electrabel.be> Hi, There is a nasty behaviour in the LinearAlgebra package under windows NT. When I run the following script from the command-line (python2.3 on windows Enthought edition but I had the same problem with the original python2.3 + Numeric 23.3), it returns without problem. ##################### from LinearAlgebra import eigenvalues from Numeric import array a = array([[3,4],[1,6]]) print eigenvalues(a) ##################### Output [ 2. 7.] However, when I evaluate the same script embedded in C, it hangs while consuming 100% of the CPU: /* C Code */ #include int main(int argc, char *argv[]) { Py_Initialize(); PyRun_SimpleString("from LinearAlgebra import eigenvalues\nfrom Numeric import array\na = array([[3,4],[1,6]])\nprint eigenvalues(a)\n"); Py_Finalize(); return 0; } /* end of C code */ I compile the code with Mingw gcc embed.c -Ic:\python23\include -Lc:\python23 -lpython23 and gcc -v gives Reading specs from C:/Python23/Enthought/MingW/bin/../lib/gcc-lib/mingw32/3.2.3/specs Configured with: ../gcc/configure --with-gcc --with-gnu-ld --with-gnu-as --host=mingw32 --target=mingw32 --prefix=/mingw --enable-threads --disable-nls --enable-languages=c++,f77,objc --disable-win32-registry --disable-shared --enable-sjlj- exceptions Thread model: win32 gcc version 3.2.3 (mingw special 20030504-1) I must recompile lapack_lite from source with Mingw in order to get the correct behaviour. Any hint on a cleaner solution (not recompile lapack_lite whenever I install a new version of Numeric). Seb From mdehoon at ims.u-tokyo.ac.jp Tue Jun 22 05:22:17 2004 From: mdehoon at ims.u-tokyo.ac.jp (Michiel Jan Laurens de Hoon) Date: Tue Jun 22 05:22:17 2004 Subject: [Numpy-discussion] lock in LinearAlgebra when embedding In-Reply-To: <035965348644D511A38C00508BF7EAEB145CAEE0@seacex03.eib.electrabel.be> References: <035965348644D511A38C00508BF7EAEB145CAEE0@seacex03.eib.electrabel.be> Message-ID: <40D8240E.3000009@ims.u-tokyo.ac.jp> This may be related to patch 732520. The problem is that lapack_lite is not compiled correctly by setup.py; one part of it should be compiled without optimization, but is compiled with. On many platforms, this doesn't cause any problems, but on Cygwin (and therefore probably also on MinGW) the eigenvalues function hangs as a result. You can get more information by looking up the patch on sourceforge. This issue has come up a couple of times. Maybe we should fix the Numeric/numarray source code with this patch or something similar? --Michiel, U Tokyo. Sebastien.deMentendeHorne at electrabel.com wrote: > Hi, > > There is a nasty behaviour in the LinearAlgebra package under windows NT. > > When I run the following script from the command-line (python2.3 on windows > Enthought edition but I had the same problem with the original python2.3 + > Numeric 23.3), it returns without problem. > ##################### > from LinearAlgebra import eigenvalues > from Numeric import array > a = array([[3,4],[1,6]]) > print eigenvalues(a) > ##################### > Output > [ 2. 7.] > > However, when I evaluate the same script embedded in C, it hangs while > consuming 100% of the CPU: > /* C Code */ > #include > > int > main(int argc, char *argv[]) > { > Py_Initialize(); > PyRun_SimpleString("from LinearAlgebra import eigenvalues\nfrom Numeric > import array\na = array([[3,4],[1,6]])\nprint eigenvalues(a)\n"); > > Py_Finalize(); > return 0; > } > /* end of C code */ > > I compile the code with Mingw > gcc embed.c -Ic:\python23\include -Lc:\python23 -lpython23 > and gcc -v gives > > Reading specs from > C:/Python23/Enthought/MingW/bin/../lib/gcc-lib/mingw32/3.2.3/specs > Configured with: ../gcc/configure --with-gcc --with-gnu-ld --with-gnu-as > --host=mingw32 --target=mingw32 --prefix=/mingw > --enable-threads --disable-nls --enable-languages=c++,f77,objc > --disable-win32-registry --disable-shared --enable-sjlj- > exceptions > Thread model: win32 > gcc version 3.2.3 (mingw special 20030504-1) > > I must recompile lapack_lite from source with Mingw in order to get the > correct behaviour. > > Any hint on a cleaner solution (not recompile lapack_lite whenever I install > a new version of Numeric). > > Seb > > > ------------------------------------------------------- > This SF.Net email sponsored by Black Hat Briefings & Training. > Attend Black Hat Briefings & Training, Las Vegas July 24-29 - > digital self defense, top technical experts, no vendor pitches, > unmatched networking opportunities. Visit www.blackhat.com > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > -- Michiel de Hoon, Assistant Professor University of Tokyo, Institute of Medical Science Human Genome Center 4-6-1 Shirokane-dai, Minato-ku Tokyo 108-8639 Japan http://bonsai.ims.u-tokyo.ac.jp/~mdehoon From Sebastien.deMentendeHorne at electrabel.com Tue Jun 22 05:44:04 2004 From: Sebastien.deMentendeHorne at electrabel.com (Sebastien.deMentendeHorne at electrabel.com) Date: Tue Jun 22 05:44:04 2004 Subject: [Numpy-discussion] lock in LinearAlgebra when embedding Message-ID: <035965348644D511A38C00508BF7EAEB145CAEE1@seacex03.eib.electrabel.be> It looks like it is a similar problem. However, it is not yet resolved as the patch is from may/2003 and I have this problem in the latest Numeric version. Moreover, my problem lies in the embedded/not embedded difference when running the same script. In fact, I notice it when using Python embedded in Excel via VBA. Testing the same script from the command-line is OK while from Excel it hangs. And the python/numeric I use is the "native windows" version (i.e. compiled with VC++ and not cygwin/mingw). So, Numeric embedded works if lapack_lite is compile with mingw32 but NOT with VC++. Looks like this lapack interface is delicate... Seb -----Original Message----- From: Michiel Jan Laurens de Hoon [mailto:mdehoon at ims.u-tokyo.ac.jp] Sent: mardi 22 juin 2004 14:21 To: Sebastien.deMentendeHorne at electrabel.com Cc: numpy-discussion at lists.sourceforge.net Subject: Re: [Numpy-discussion] lock in LinearAlgebra when embedding This may be related to patch 732520. The problem is that lapack_lite is not compiled correctly by setup.py; one part of it should be compiled without optimization, but is compiled with. On many platforms, this doesn't cause any problems, but on Cygwin (and therefore probably also on MinGW) the eigenvalues function hangs as a result. You can get more information by looking up the patch on sourceforge. This issue has come up a couple of times. Maybe we should fix the Numeric/numarray source code with this patch or something similar? --Michiel, U Tokyo. Sebastien.deMentendeHorne at electrabel.com wrote: > Hi, > > There is a nasty behaviour in the LinearAlgebra package under windows NT. > > When I run the following script from the command-line (python2.3 on windows > Enthought edition but I had the same problem with the original python2.3 + > Numeric 23.3), it returns without problem. > ##################### > from LinearAlgebra import eigenvalues > from Numeric import array > a = array([[3,4],[1,6]]) > print eigenvalues(a) > ##################### > Output > [ 2. 7.] > > However, when I evaluate the same script embedded in C, it hangs while > consuming 100% of the CPU: > /* C Code */ > #include > > int > main(int argc, char *argv[]) > { > Py_Initialize(); > PyRun_SimpleString("from LinearAlgebra import eigenvalues\nfrom Numeric > import array\na = array([[3,4],[1,6]])\nprint eigenvalues(a)\n"); > > Py_Finalize(); > return 0; > } > /* end of C code */ > > I compile the code with Mingw > gcc embed.c -Ic:\python23\include -Lc:\python23 -lpython23 > and gcc -v gives > > Reading specs from > C:/Python23/Enthought/MingW/bin/../lib/gcc-lib/mingw32/3.2.3/specs > Configured with: ../gcc/configure --with-gcc --with-gnu-ld --with-gnu-as > --host=mingw32 --target=mingw32 --prefix=/mingw > --enable-threads --disable-nls --enable-languages=c++,f77,objc > --disable-win32-registry --disable-shared --enable-sjlj- > exceptions > Thread model: win32 > gcc version 3.2.3 (mingw special 20030504-1) > > I must recompile lapack_lite from source with Mingw in order to get the > correct behaviour. > > Any hint on a cleaner solution (not recompile lapack_lite whenever I install > a new version of Numeric). > > Seb > > > ------------------------------------------------------- > This SF.Net email sponsored by Black Hat Briefings & Training. > Attend Black Hat Briefings & Training, Las Vegas July 24-29 - > digital self defense, top technical experts, no vendor pitches, > unmatched networking opportunities. Visit www.blackhat.com > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > -- Michiel de Hoon, Assistant Professor University of Tokyo, Institute of Medical Science Human Genome Center 4-6-1 Shirokane-dai, Minato-ku Tokyo 108-8639 Japan http://bonsai.ims.u-tokyo.ac.jp/~mdehoon From jmiller at stsci.edu Tue Jun 22 06:05:11 2004 From: jmiller at stsci.edu (Todd Miller) Date: Tue Jun 22 06:05:11 2004 Subject: [Numpy-discussion] numarray.concatenate for character arrays In-Reply-To: References: Message-ID: <1087909484.14732.5.camel@halloween.stsci.edu> Hi Faheem, This is a bug. It'll be fixed in numarray-1.0 which should hopefully be released this week. Regards, Todd On Mon, 2004-06-21 at 20:17, Faheem Mitha wrote: > Dear People, > > Is the function numarray.concatenate supposed to work for character > arrays? It doesn't for me. Do I need to write my own? Thanks in > advance. Please cc me, I'm not subscribed. > > Faheem. > > In [17]: foo > Out[17]: > CharArray([['T', 'T'], > ['C', 'A']]) > > In [18]: bar > Out[18]: > CharArray([['G', 'G'], > ['G', 'G']]) > > In [23]: numarray.concatenate((foo,bar)) > --------------------------------------------------------------------------- > error Traceback (most recent call > last) > > /home/faheem/wc/corrmodel/trunk/ > > /usr/lib/python2.3/site-packages/numarray/generic.py in > concatenate(arrs, axis) > 1018 arrs = map(_nc.asarray, arrs) > 1019 if axis == 0: > -> 1020 return _concat(arrs) > 1021 else: > 1022 return swapaxes(_concat([swapaxes(m,axis,0) for m in > arrs]), axis, 0) > > /usr/lib/python2.3/site-packages/numarray/generic.py in _concat(arrs) > 1000 convType = ufunc._maxPopType(arrs) > 1001 except TypeError: > -> 1002 dest = arrs[0]._clone(shape=destShape) > 1003 else: > 1004 dest = arrs[0].__class__(shape=destShape, > type=convType) > > /usr/lib/python2.3/site-packages/numarray/generic.py in _clone(self, > shape) > 783 def _clone(self, shape): > 784 c = self.copy() > --> 785 c.resize(shape) > 786 return c > 787 > > /usr/lib/python2.3/site-packages/numarray/generic.py in resize(self, > shape, *args) > 854 self[offset:offset+olen] = self[0:olen] > 855 offset += olen > --> 856 self[offset:nlen] = self[0:nlen-offset] > 857 else: # zero fill resized zero-length numarray > 858 self[:] = 0 > > /usr/lib/python2.3/site-packages/numarray/strings.py in > _copyFrom(self, arr) > 217 me = self._byteView() > 218 if self._itemsize <= arr._itemsize: > --> 219 me[:] = it[..., :self._itemsize] > 220 else: > 221 me[...,:it._shape[-1]] = it > > error: copy1bytes: access beyond buffer. offset=8 buffersize=8 > > > > > > ------------------------------------------------------- > This SF.Net email sponsored by Black Hat Briefings & Training. > Attend Black Hat Briefings & Training, Las Vegas July 24-29 - > digital self defense, top technical experts, no vendor pitches, > unmatched networking opportunities. Visit www.blackhat.com > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- Todd Miller From sanner at scripps.edu Tue Jun 22 09:09:01 2004 From: sanner at scripps.edu (Michel Sanner) Date: Tue Jun 22 09:09:01 2004 Subject: [Numpy-discussion] convolve2d Message-ID: <40D8559D.8090107@scripps.edu> Hello, I started using numarray for doing 2D convolutions on images. I noticed that import numarray.examples.convolve.high_level as convolve convolve.Convolve2d(kernel, in, out) only works on square images. For images that are not square I get lots of noise in the background. Also I was wondering is using the high_level API is most efficient? Currently my image is a Numeric array (grabbed from the OpenGL frame buffer) which I convert to a numarray to do the convolution and back to a Numeric array. In the future I hope to completely replace Numeric by numarray. Thanks for any input -Michel -- ----------------------------------------------------------------------- o / Michel F. Sanner Ph.D. The Scripps Research Institute o Associate Professor Department of Molecular Biology \ 10550 North Torrey Pines Road o Tel. (858) 784-2341 La Jolla, CA 92037 / Fax. (858) 784-2860 o sanner at scripps.edu http://www.scripps.edu/~sanner ----------------------------------------------------------------------- From rowen at u.washington.edu Tue Jun 22 09:57:06 2004 From: rowen at u.washington.edu (Russell E Owen) Date: Tue Jun 22 09:57:06 2004 Subject: [Numpy-discussion] median filtering question -- masked arrays Message-ID: Does anyone have a hint for applying a median filter to a masked 2D array in numarray? I've been using the kludge of extracting a normal array with the masked data replaced by the overall median of the array and median filtering that (with the appropriate nd_image method). But this is a crude approximation and I'd like to do better -- preferably without coding my own median filter in C. -- Russell From jmiller at stsci.edu Tue Jun 22 11:09:55 2004 From: jmiller at stsci.edu (Todd Miller) Date: Tue Jun 22 11:09:55 2004 Subject: [Numpy-discussion] convolve2d In-Reply-To: <40D8559D.8090107@scripps.edu> References: <40D8559D.8090107@scripps.edu> Message-ID: <1087927670.14732.273.camel@halloween.stsci.edu> On Tue, 2004-06-22 at 11:51, Michel Sanner wrote: > Hello, > > I started using numarray for doing 2D convolutions on images. I noticed > that > > import numarray.examples.convolve.high_level as convolve > convolve.Convolve2d(kernel, in, out) > > only works on square images. For images that are not square I get lots > of noise in the background. I looked at the code and couldn't find any row/column typos. In theory, Convolve2d works for non-square arrays. What do kernel.info() and data.info() say? > Also I was wondering is using the high_level API is most efficient? For CPU, my guess is yes. The point access macros and functions are definitely slower than the high level API in any scenario where you're not running out of memory. The 1D API improves space usage but is less efficient in time. The Numeric compatible API is layered over the high level API. > Currently my image is a Numeric > array (grabbed from the OpenGL frame buffer) which I convert to a > numarray to do the convolution and back to a Numeric array. It's also possible something is happening during this conversion. It'd be good to round-trip the Numeric array and make sure the end product looks like the original. > In the future I hope to completely replace Numeric by numarray. Great! Regards, Todd From jmiller at stsci.edu Tue Jun 22 11:18:07 2004 From: jmiller at stsci.edu (Todd Miller) Date: Tue Jun 22 11:18:07 2004 Subject: [Numpy-discussion] convolve2d In-Reply-To: <40D8559D.8090107@scripps.edu> References: <40D8559D.8090107@scripps.edu> Message-ID: <1087928153.14732.286.camel@halloween.stsci.edu> On Tue, 2004-06-22 at 11:51, Michel Sanner wrote: > Hello, > > I started using numarray for doing 2D convolutions on images. I noticed > that > > import numarray.examples.convolve.high_level as convolve > convolve.Convolve2d(kernel, in, out) > Here's another (quick?) test which might help localize the problem. Do: convolve.Convolve2d(kernel, in, out, fft=1) The underlying Convolve2d implementations are very different depending on the use of FFT. Regards, Todd From jmiller at stsci.edu Tue Jun 22 13:30:12 2004 From: jmiller at stsci.edu (Todd Miller) Date: Tue Jun 22 13:30:12 2004 Subject: [Numpy-discussion] Java version for numarray? In-Reply-To: <200406071311.02485.falted@pytables.org> References: <200406071311.02485.falted@pytables.org> Message-ID: <1087936180.14732.349.camel@halloween.stsci.edu> On Mon, 2004-06-07 at 07:11, Francesc Alted wrote: > Hi, > > I'm using numarray objects basically like data containers for fast I/O > purposes in the context of pytables. I'm thinking now to port part of my > code to Java, and I'd like to use Jython to easy the transition. I've seen > that a there is a port of Numeric to Jython > (http://jnumerical.sourceforge.net/), and I thought that I could use these > Numeric objects as containers. Unfortunately, there are a couple of objects > that pytables relies on, namely RecArray and CharArray, that are not > supported by Numeric. > > My questions are: > > - I think that both RecArray and CharArray modules are written in pure > python, so the porting to Jython should be relatively easy, provided I'm > successful making them to use Numeric objects instead of NumArray objects. > What do you think? Sounds hard. I doubt there is enough commonality with the numarray to make bolting on the pure-Python RecArray and CharArray implementations reasonable. RecArray and CharArray are very much dependent on generic.NDArray; I doubt they're easy to port to Numeric. > Would that be feasible? I don't think so. > - Is there any effort going on in order to have an implementation of > numarray ready for use in Jython? No. Sorry about the slow response, Todd From faheem at email.unc.edu Wed Jun 23 10:58:21 2004 From: faheem at email.unc.edu (Faheem Mitha) Date: Wed Jun 23 10:58:21 2004 Subject: [Numpy-discussion] printing character array object Message-ID: Hi, it's me again. This time I am having problems with printing out a character array object (attached in cPickle format). I don't understand what the error message means. If you load the >>> import cPickle >>> cPickle.dump(foo,open('foo.save','w')) >>> foo [tons of error messages terminating in] libnumarray.error: copy1bytes: access beyond buffer. offset=6 buffersize=6 Looks like the problem is with numarray. The character array is 100x20 and I can't see any intrinsic problem with printing it out. >>> foo.getshape() Out[73]: (100, 20) Thanks for any clarification. Faheem. -------------- next part -------------- A non-text attachment was scrubbed... Name: foo.save Type: application/octet-stream Size: 2313 bytes Desc: URL: From haase at msg.ucsf.edu Wed Jun 23 15:07:04 2004 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Wed Jun 23 15:07:04 2004 Subject: [Numpy-discussion] numarray.sum should raise exception for wrong axis argument - and slicing ... Message-ID: <200406231506.24453.haase@msg.ucsf.edu> Hi, please take a look at this: >>> na.sum( na.zeros((2,6)) ) [0 0 0 0 0 0] >>> na.sum( na.zeros((2,6)) , 0) [0 0 0 0 0 0] >>> na.sum( na.zeros((2,6)) , 1) [0 0] >>> na.sum( na.zeros((2,6)) , 2) [0 0] >>> na.sum( na.zeros((2,6)) , 3) [0 0] >>> na.sum( na.zeros((2,6)) , 4) [0 0] >>> na.sum( na.zeros((2,6)) , -1) [0 0] >>> na.sum( na.zeros((2,6)) , -2) [0 0 0 0 0 0] >>> na.sum( na.zeros((2,6)) , -3) [0 0] >>> na.sum( na.zeros((2,6)) , -4) [0 0] >>> I think here should be a ValueError exception thrown rather than defaulting to the '-1'-axis. Comments ? Also this applies to (all?) other functions that have an 'axis' argument. And further I just found that putting "too many slicings" to an array also gets silently ignored: >>> b.shape (7, 128, 128, 128) >>> b[2,2,2,2,3] Traceback (most recent call last): File "", line 1, in ? IndexError: too many indices. BUT: >>> b[2:3 , 2:3 , 2:3 , 2:3 , 2:3 , 2:3] [[[[ 0.]]]] I find this very confusing !! Is there any good reason not to have the "IndexError" exception in all cases ? Thanks, Sebastian Haase From buu43vysbr at shentel.net Wed Jun 23 23:23:01 2004 From: buu43vysbr at shentel.net (Georgene Michiko) Date: Wed Jun 23 23:23:01 2004 Subject: [Numpy-discussion] THE CHEAPEST EVER SOFTWARES U SEE & WE SHIIP WORLDWIDE makes somebody Message-ID: making fellow stay fixed advantage neck secret pocket money believe important gotten learn nickel handwriting house -------------- next part -------------- An HTML attachment was scrubbed... URL: From curzio.basso at unibas.ch Thu Jun 24 03:15:00 2004 From: curzio.basso at unibas.ch (Curzio Basso) Date: Thu Jun 24 03:15:00 2004 Subject: [Numpy-discussion] type of matrixmultiply result Message-ID: <40DAA976.2010702@unibas.ch> Hi. I noticed that when multiplying two matrices of type Float32, the result is Float64: ----------------------------------------- In [103]: a=NA.ones((2,2), NA.Float32) In [104]: b=NA.ones((2,2), NA.Float32) In [105]: c=NA.matrixmultiply(a,b) In [106]: c.type() Out[106]: Float64 ----------------------------------------- Since the matrix I'm going to multiply in practice are quite big, I'd like to do the operation in Float32. Otherwise this is what I get: Traceback (most recent call last): File "/home/basso/work/python/port/apps/pca-heads.py", line 141, in ? pc = NA.array(NA.matrixmultiply(cent, c), NA.Float32) File "/home/basso/usr//lib/python/numarray/numarraycore.py", line 1150, in dot return ufunc.innerproduct(array1, _gen.swapaxes(array2, -1, -2)) File "/home/basso/usr//lib/python/numarray/ufunc.py", line 2047, in innerproduct r = a.__class__(shape=adots+bdots, type=rtype) MemoryError Any suggestion (apart from doing the operation one column at a time)? thanks From verveer at embl-heidelberg.de Thu Jun 24 05:34:07 2004 From: verveer at embl-heidelberg.de (Peter Verveer) Date: Thu Jun 24 05:34:07 2004 Subject: [Numpy-discussion] =?ISO-8859-1?Q?Re:_median_filtering_question_--_masked_arrays_?= =?ISO-8859-1?Q?=A0?= In-Reply-To: References: Message-ID: > Does anyone have a hint for applying a median filter to a masked 2D > array in numarray? > > I"ve been using the kludge of extracting a normal array with the > masked data replaced by the overall median of the array and median > filtering that (with the appropriate nd_image method). But this is a > crude approximation and I"d like to do better -- preferably without > coding my own median filter in C. Masks are not supported by the filters in nd_image. It could be done, but that would be a considerable effort for a rather exotic feature, so that won't happen soon. In the near future I will however add support for generic filters where you can provide your own filter function to be executed at each point. That may make it easy enough to roll your own function. Cheers, Peter From verveer at embl-heidelberg.de Thu Jun 24 05:39:00 2004 From: verveer at embl-heidelberg.de (Peter Verveer) Date: Thu Jun 24 05:39:00 2004 Subject: [Numpy-discussion] compress function Message-ID: <5900E940-C5DB-11D8-93CA-000A95C92C8E@embl-heidelberg.de> The documentation of the compress function states that the condition must be equal to the given axis of the array that is compressed. e.g.: >>> a = array([[1,2],[3,4]]) >>> print compress([1,0], a, axis = 1) [[1] [3]] However, this also works fine: >>> print compress([[1,0],[0,1]], a) [1, 4] which is great (I need that) but not documented. Is that behaviour intended? If so it maybe should be documented. Cheers, Peter From jmiller at stsci.edu Thu Jun 24 06:39:08 2004 From: jmiller at stsci.edu (Todd Miller) Date: Thu Jun 24 06:39:08 2004 Subject: [Numpy-discussion] type of matrixmultiply result In-Reply-To: <40DAA976.2010702@unibas.ch> References: <40DAA976.2010702@unibas.ch> Message-ID: <1088084311.26016.52.camel@halloween.stsci.edu> On Thu, 2004-06-24 at 06:14, Curzio Basso wrote: > Hi. > > I noticed that when multiplying two matrices of type Float32, the result > is Float64: > > ----------------------------------------- > In [103]: a=NA.ones((2,2), NA.Float32) > > In [104]: b=NA.ones((2,2), NA.Float32) > > In [105]: c=NA.matrixmultiply(a,b) > > In [106]: c.type() > Out[106]: Float64 > ----------------------------------------- > > Since the matrix I'm going to multiply in practice are quite big, I'd > like to do the operation in Float32. Otherwise this is what I get: > > Traceback (most recent call last): > File "/home/basso/work/python/port/apps/pca-heads.py", line 141, in ? > pc = NA.array(NA.matrixmultiply(cent, c), NA.Float32) > File "/home/basso/usr//lib/python/numarray/numarraycore.py", line > 1150, in dot return ufunc.innerproduct(array1, _gen.swapaxes(array2, > -1, -2)) > File "/home/basso/usr//lib/python/numarray/ufunc.py", line 2047, in > innerproduct > r = a.__class__(shape=adots+bdots, type=rtype) > MemoryError > > Any suggestion (apart from doing the operation one column at a time)? > I modified dot() and innerproduct() this morning to return Float32 and Complex32 for like inputs. This is in CVS now. numarray-1.0 is dragging out, but will nevertheless be released relatively soon. I'm curious about what your array dimensions are. When I implemented matrixmuliply for numarray, I was operating under the assumption that no one would be multiplying truly huge arrays because it's an O(N^3) algorithm. Regards, Todd > thanks > > > ------------------------------------------------------- > This SF.Net email sponsored by Black Hat Briefings & Training. > Attend Black Hat Briefings & Training, Las Vegas July 24-29 - > digital self defense, top technical experts, no vendor pitches, > unmatched networking opportunities. Visit www.blackhat.com > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- Todd Miller From rlw at stsci.edu Thu Jun 24 07:32:10 2004 From: rlw at stsci.edu (Rick White) Date: Thu Jun 24 07:32:10 2004 Subject: [Numpy-discussion] type of matrixmultiply result In-Reply-To: <1088084311.26016.52.camel@halloween.stsci.edu> Message-ID: On 24 Jun 2004, Todd Miller wrote: > On Thu, 2004-06-24 at 06:14, Curzio Basso wrote: > > > I noticed that when multiplying two matrices of type Float32, the result > > is Float64: > > I modified dot() and innerproduct() this morning to return Float32 and > Complex32 for like inputs. I wonder whether it would be worth providing an option to accumulate the sums using Float64 and to convert to Float32 before storing them in an array. I suspect that one reason this returned Float64 is that it is very easy to run into precision/roundoff problems in single-precision matrix multiplies. You could avoid that by using doubles for the sum while still returning the result as a single. Rick From perry at stsci.edu Thu Jun 24 08:09:11 2004 From: perry at stsci.edu (Perry Greenfield) Date: Thu Jun 24 08:09:11 2004 Subject: [Numpy-discussion] type of matrixmultiply result In-Reply-To: Message-ID: Rick White wrote: > On 24 Jun 2004, Todd Miller wrote: > > > On Thu, 2004-06-24 at 06:14, Curzio Basso wrote: > > > > > I noticed that when multiplying two matrices of type Float32, > the result > > > is Float64: > > > > I modified dot() and innerproduct() this morning to return Float32 and > > Complex32 for like inputs. > > I wonder whether it would be worth providing an option to accumulate > the sums using Float64 and to convert to Float32 before storing them in > an array. I suspect that one reason this returned Float64 is that it > is very easy to run into precision/roundoff problems in > single-precision matrix multiplies. You could avoid that by using > doubles for the sum while still returning the result as a single. > Rick > I definitely agree. I'm pretty certain the reason it was done with double precision floats is the sensitivity to roundoff issues with matrix operations. I think Rick is right though that only intermediate calculations need to be done in double precision and that doesn't require the whole output array to be kept that way. Perry From jmiller at stsci.edu Thu Jun 24 08:32:06 2004 From: jmiller at stsci.edu (Todd Miller) Date: Thu Jun 24 08:32:06 2004 Subject: [Numpy-discussion] type of matrixmultiply result In-Reply-To: References: Message-ID: <1088091080.26016.131.camel@halloween.stsci.edu> On Thu, 2004-06-24 at 11:08, Perry Greenfield wrote: > Rick White wrote: > > > On 24 Jun 2004, Todd Miller wrote: > > > > > On Thu, 2004-06-24 at 06:14, Curzio Basso wrote: > > > > > > > I noticed that when multiplying two matrices of type Float32, > > the result > > > > is Float64: > > > > > > I modified dot() and innerproduct() this morning to return Float32 and > > > Complex32 for like inputs. > > > > I wonder whether it would be worth providing an option to accumulate > > the sums using Float64 and to convert to Float32 before storing them in > > an array. I suspect that one reason this returned Float64 is that it > > is very easy to run into precision/roundoff problems in > > single-precision matrix multiplies. You could avoid that by using > > doubles for the sum while still returning the result as a single. > > Rick > > > I definitely agree. I'm pretty certain the reason it was done > with double precision floats is the sensitivity to roundoff > issues with matrix operations. I think Rick is right though that > only intermediate calculations need to be done in double precision > and that doesn't require the whole output array to be kept that way. > > Perry OK. I implemented intermediate sums using Float64 and Complex64 but single precision inputs will still result in single precision outputs. Todd From jmiller at stsci.edu Thu Jun 24 08:34:08 2004 From: jmiller at stsci.edu (Todd Miller) Date: Thu Jun 24 08:34:08 2004 Subject: [Numpy-discussion] type of matrixmultiply result In-Reply-To: References: Message-ID: <1088091221.26016.136.camel@halloween.stsci.edu> On Thu, 2004-06-24 at 10:30, Rick White wrote: > On 24 Jun 2004, Todd Miller wrote: > > > On Thu, 2004-06-24 at 06:14, Curzio Basso wrote: > > > > > I noticed that when multiplying two matrices of type Float32, the result > > > is Float64: > > > > I modified dot() and innerproduct() this morning to return Float32 and > > Complex32 for like inputs. > > I wonder whether it would be worth providing an option to accumulate > the sums using Float64 and to convert to Float32 before storing them in > an array. I suspect that one reason this returned Float64 is that it > is very easy to run into precision/roundoff problems in > single-precision matrix multiplies. You could avoid that by using > doubles for the sum while still returning the result as a single. > Rick OK. I implemented intermediate sums using Float64 and Complex64 but single precision inputs will still result in single precision outputs. Todd From tim.hochberg at cox.net Thu Jun 24 09:42:13 2004 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu Jun 24 09:42:13 2004 Subject: [Numpy-discussion] Compiling on windows? Message-ID: <40DB041B.1050201@cox.net> I've been using numarray 0.8 since 0.9 has some bugs that I can't easily work around. Since these have been fixed in CVS, I finally decided to try to download and compile a copy of numarray from CVS. Unfortunately, 'python setup.py build' bombs out almost imediately: creating build\temp.win32-2.3 creating build\temp.win32-2.3\Release creating build\temp.win32-2.3\Release\Src C:\Program Files\Microsoft Visual Studio\VC98\BIN\cl.exe /c /nologo /Ox /MD /W3 /GX /DNDEBUG -IInclude/numarray -IC:\python23\include -IC:\python23\PC /TcSrc\_convmodule.c /Fobuild\temp.win32-2.3\Release\Src\_convmodule.obj_convmodule.c Src\_convmodule.c(683) : warning C4244: '=' : conversion from 'int ' to 'float ', possible loss of data .... Src\_convmodule.c(2310) : error C2520: conversion from unsigned __int64 to double not implemented, use signed __int64 error: command '"C:\Program Files\Microsoft Visual Studio\VC98\BIN\cl.exe"' failed with exit status 2 This on Windows XP, Python 2.3.2, VC++ 6.0. Any hints as to what I could to make this work? I downloaded the source for version 0.9 and tried compiling it but got the same set of errors. Alternatively, is there going to be a 0.9.1 or 0.10 release soon? Thanks, -tim From jmiller at stsci.edu Thu Jun 24 09:52:49 2004 From: jmiller at stsci.edu (Todd Miller) Date: Thu Jun 24 09:52:49 2004 Subject: [Numpy-discussion] Compiling on windows? In-Reply-To: <40DB041B.1050201@cox.net> References: <40DB041B.1050201@cox.net> Message-ID: <1088095886.26016.143.camel@halloween.stsci.edu> On Thu, 2004-06-24 at 12:40, Tim Hochberg wrote: > I've been using numarray 0.8 since 0.9 has some bugs that I can't > easily work around. Since these have been fixed in CVS, I finally > decided to try to download and compile a copy of numarray from CVS. > Unfortunately, 'python setup.py build' bombs out almost imediately: > > creating build\temp.win32-2.3 > creating build\temp.win32-2.3\Release > creating build\temp.win32-2.3\Release\Src > C:\Program Files\Microsoft Visual Studio\VC98\BIN\cl.exe /c /nologo /Ox > /MD /W3 /GX /DNDEBUG -IInclude/numarray -IC:\python23\include > -IC:\python23\PC /TcSrc\_convmodule.c > /Fobuild\temp.win32-2.3\Release\Src\_convmodule.obj_convmodule.c > Src\_convmodule.c(683) : warning C4244: '=' : conversion from 'int ' to > 'float ', possible loss of data > > .... > > Src\_convmodule.c(2310) : error C2520: conversion from unsigned __int64 > to double not implemented, use signed __int64 > error: command '"C:\Program Files\Microsoft Visual > Studio\VC98\BIN\cl.exe"' failed with exit status 2 > > > This on Windows XP, Python 2.3.2, VC++ 6.0. Any hints as to what I could > to make this work? Try: python setup.py install --gencode Then try: python setup.py build ... if you must. :-) > I downloaded the source for version 0.9 and tried > compiling it but got the same set of errors. > > Alternatively, is there going to be a 0.9.1 or 0.10 release soon? numarray-1.0 should be out soon. Regards, Todd From haase at msg.ucsf.edu Thu Jun 24 10:07:10 2004 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Thu Jun 24 10:07:10 2004 Subject: [Numpy-discussion] comparing string array with None raises ValueError Message-ID: <200406241006.12875.haase@msg.ucsf.edu> Hi, I'm not sure if this is fixed already in CVS but here it goes: I'm working with record arrays, and trying to access a field with type '10a80' - that is, an array of 10 80 char 'strings' : >>> q.Mrc.hdr = q.Mrc.hdrArray[0].field >>> q.Mrc.hdr('title') != None Traceback (most recent call last): File "", line 1, in ? File "/jws30/haase/PrLin/numarray/strings.py", line 431, in __ne__ return self.StrCmp(other).__ne__(0) File "/jws30/haase/PrLin/numarray/strings.py", line 385, in StrCmp b = asarray(b0, kind=self.__class__) File "/jws30/haase/PrLin/numarray/strings.py", line 1024, in asarray return array(buffer, itemsize, shape, byteoffset, bytestride, kind) File "/jws30/haase/PrLin/numarray/strings.py", line 994, in array byteoffset=byteoffset, bytestride=bytestride) File "/jws30/haase/PrLin/numarray/strings.py", line 84, in __init__ raise ValueError("Must define both shape & itemsize if buffer is None") ValueError: Must define both shape & itemsize if buffer is None >>> >>> print q.Mrc.hdr('title') ['tit1e seb says hi' '' '' '' '' '' '' '' '' ''] >>> Thanks for numarray ;-) Sebastian Haase From jmiller at stsci.edu Thu Jun 24 10:32:03 2004 From: jmiller at stsci.edu (Todd Miller) Date: Thu Jun 24 10:32:03 2004 Subject: [Numpy-discussion] comparing string array with None raises ValueError In-Reply-To: <200406241006.12875.haase@msg.ucsf.edu> References: <200406241006.12875.haase@msg.ucsf.edu> Message-ID: <1088098275.26016.160.camel@halloween.stsci.edu> On Thu, 2004-06-24 at 13:06, Sebastian Haase wrote: > Hi, > I'm not sure if this is fixed already in CVS but here it goes: > I'm working with record arrays, and trying to access a field with > type '10a80' - that is, an array of 10 80 char 'strings' : > >>> q.Mrc.hdr = q.Mrc.hdrArray[0].field > >>> q.Mrc.hdr('title') != None Shouldn't this be: q.Mrc.hdr('title') != "" Regards, Todd From haase at msg.ucsf.edu Thu Jun 24 10:42:20 2004 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Thu Jun 24 10:42:20 2004 Subject: [Numpy-discussion] comparing string array with None raises ValueError In-Reply-To: <1088098275.26016.160.camel@halloween.stsci.edu> References: <200406241006.12875.haase@msg.ucsf.edu> <1088098275.26016.160.camel@halloween.stsci.edu> Message-ID: <200406241041.57810.haase@msg.ucsf.edu> On Thursday 24 June 2004 10:31 am, Todd Miller wrote: > On Thu, 2004-06-24 at 13:06, Sebastian Haase wrote: > > Hi, > > I'm not sure if this is fixed already in CVS but here it goes: > > I'm working with record arrays, and trying to access a field with > > > > type '10a80' - that is, an array of 10 80 char 'strings' : > > >>> q.Mrc.hdr = q.Mrc.hdrArray[0].field > > >>> q.Mrc.hdr('title') != None > > Shouldn't this be: > > q.Mrc.hdr('title') != "" No, I understand that this makes more sense, but I have some "display hook"-code that compares everything with None... In general it must be OK to compare anything with None, right ? (BTW, I get the same error with == and !=) Regards, Sebastian Haase From haase at msg.ucsf.edu Thu Jun 24 10:47:01 2004 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Thu Jun 24 10:47:01 2004 Subject: [Numpy-discussion] comparing string array with None raises ValueError In-Reply-To: <1088098275.26016.160.camel@halloween.stsci.edu> References: <200406241006.12875.haase@msg.ucsf.edu> <1088098275.26016.160.camel@halloween.stsci.edu> Message-ID: <200406241046.13941.haase@msg.ucsf.edu> On Thursday 24 June 2004 10:31 am, Todd Miller wrote: > On Thu, 2004-06-24 at 13:06, Sebastian Haase wrote: > > Hi, > > I'm not sure if this is fixed already in CVS but here it goes: > > I'm working with record arrays, and trying to access a field with > > > > type '10a80' - that is, an array of 10 80 char 'strings' : > > >>> q.Mrc.hdr = q.Mrc.hdrArray[0].field > > >>> q.Mrc.hdr('title') != None > > Shouldn't this be: > > q.Mrc.hdr('title') != "" (in my first reply, I forgot to point out that q.Mrc.hdr('title') is an ARRAY of strings ! ) No, I understand that this makes more sense, but I have some "display hook"-code that compares everything with None... In general it must be OK to compare anything with None, right ? (BTW, I get the same error with == and !=) Regards, Sebastian Haase From jmiller at stsci.edu Thu Jun 24 11:18:15 2004 From: jmiller at stsci.edu (Todd Miller) Date: Thu Jun 24 11:18:15 2004 Subject: [Numpy-discussion] ObjectArray paramter order poll Message-ID: <1088101018.26016.280.camel@halloween.stsci.edu> A while back Colin Williams suggested that we change the parameter order of numarray.objects.ObjectArray from: def __init__(self, shape=None, objects=None, rank=None) to: def __init__(self, objects=None, shape=None, rank=None) The new order emphasizes building object arrays from existing sequences, while the old order emphasizes building empty (full of None) object arrays of a specific shape. Since array() and fromlist() already exist to build from sequences, I thought it was fine to keep the order as is, plus I hate breaking s/w interfaces unless it was my idea. :-) Opinions please? Regards, Todd From jmiller at stsci.edu Thu Jun 24 11:47:02 2004 From: jmiller at stsci.edu (Todd Miller) Date: Thu Jun 24 11:47:02 2004 Subject: [Numpy-discussion] comparing string array with None raises ValueError In-Reply-To: <200406241046.13941.haase@msg.ucsf.edu> References: <200406241006.12875.haase@msg.ucsf.edu> <1088098275.26016.160.camel@halloween.stsci.edu> <200406241046.13941.haase@msg.ucsf.edu> Message-ID: <1088102785.26016.297.camel@halloween.stsci.edu> On Thu, 2004-06-24 at 13:46, Sebastian Haase wrote: > On Thursday 24 June 2004 10:31 am, Todd Miller wrote: > > On Thu, 2004-06-24 at 13:06, Sebastian Haase wrote: > > > Hi, > > > I'm not sure if this is fixed already in CVS but here it goes: > > > I'm working with record arrays, and trying to access a field with > > > > > > type '10a80' - that is, an array of 10 80 char 'strings' : > > > >>> q.Mrc.hdr = q.Mrc.hdrArray[0].field > > > >>> q.Mrc.hdr('title') != None > > > > Shouldn't this be: > > > > q.Mrc.hdr('title') != "" > > (in my first reply, I forgot to point out that q.Mrc.hdr('title') is an ARRAY > of strings ! ) > > No, I understand that this makes more sense, but I have some "display > hook"-code that compares everything with None... > In general it must be OK to compare anything with None, right ? > (BTW, I get the same error with == and !=) OK, I see your point. I talked it over with Perry and he made a reasonable case for allowing comparisons with None (or any object). Perry argued that since None is a common default parameter value, it might simplify code to not have to add logic to handle that case. If no one objects, I'll change numarray.strings so that comparison of a string array with any object not convertible to a string array results in an array of False values. Any objections? Regards, Todd From rlw at stsci.edu Thu Jun 24 11:54:05 2004 From: rlw at stsci.edu (Rick White) Date: Thu Jun 24 11:54:05 2004 Subject: [Numpy-discussion] comparing string array with None raises ValueError In-Reply-To: <1088102785.26016.297.camel@halloween.stsci.edu> Message-ID: On 24 Jun 2004, Todd Miller wrote: > OK, I see your point. I talked it over with Perry and he made a > reasonable case for allowing comparisons with None (or any object). > Perry argued that since None is a common default parameter value, it > might simplify code to not have to add logic to handle that case. > > If no one objects, I'll change numarray.strings so that comparison of a > string array with any object not convertible to a string array results > in an array of False values. > > Any objections? Hmm, before you do that you might look at this: >>> import numarray >>> b = numarray.zeros(10) >>> b==0 array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1], type=Bool) >>> b==None 0 So comparing numeric arrays to None returns a scalar false value since the variable b is not None. I figure that the behavior of comparisons to None for string arrays and numeric arrays ought to be consistent. I don't know which is preferred... Rick From haase at msg.ucsf.edu Thu Jun 24 12:14:31 2004 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Thu Jun 24 12:14:31 2004 Subject: [Numpy-discussion] comparing string array with None raises ValueError In-Reply-To: <1088102785.26016.297.camel@halloween.stsci.edu> References: <200406241006.12875.haase@msg.ucsf.edu> <200406241046.13941.haase@msg.ucsf.edu> <1088102785.26016.297.camel@halloween.stsci.edu> Message-ID: <200406241213.00537.haase@msg.ucsf.edu> On Thursday 24 June 2004 11:46 am, Todd Miller wrote: > On Thu, 2004-06-24 at 13:46, Sebastian Haase wrote: > > On Thursday 24 June 2004 10:31 am, Todd Miller wrote: > > > On Thu, 2004-06-24 at 13:06, Sebastian Haase wrote: > > > > Hi, > > > > I'm not sure if this is fixed already in CVS but here it goes: > > > > I'm working with record arrays, and trying to access a field with > > > > > > > > type '10a80' - that is, an array of 10 80 char 'strings' : > > > > >>> q.Mrc.hdr = q.Mrc.hdrArray[0].field > > > > >>> q.Mrc.hdr('title') != None > > > > > > Shouldn't this be: > > > > > > q.Mrc.hdr('title') != "" > > > > (in my first reply, I forgot to point out that q.Mrc.hdr('title') is an > > ARRAY of strings ! ) > > > > No, I understand that this makes more sense, but I have some "display > > hook"-code that compares everything with None... > > In general it must be OK to compare anything with None, right ? > > (BTW, I get the same error with == and !=) > > OK, I see your point. I talked it over with Perry and he made a > reasonable case for allowing comparisons with None (or any object). > Perry argued that since None is a common default parameter value, it > might simplify code to not have to add logic to handle that case. > > If no one objects, I'll change numarray.strings so that comparison of a > string array with any object not convertible to a string array results > in an array of False values. > > Any objections? Just my feeling is, that a single None would do it ... Regards, Sebastian From jmiller at stsci.edu Thu Jun 24 12:15:02 2004 From: jmiller at stsci.edu (Todd Miller) Date: Thu Jun 24 12:15:02 2004 Subject: [Numpy-discussion] comparing string array with None raises ValueError, more opinions please In-Reply-To: References: Message-ID: <1088104465.26016.310.camel@halloween.stsci.edu> On Thu, 2004-06-24 at 14:53, Rick White wrote: > On 24 Jun 2004, Todd Miller wrote: > > > OK, I see your point. I talked it over with Perry and he made a > > reasonable case for allowing comparisons with None (or any object). > > Perry argued that since None is a common default parameter value, it > > might simplify code to not have to add logic to handle that case. > > > > If no one objects, I'll change numarray.strings so that comparison of a > > string array with any object not convertible to a string array results > > in an array of False values. > > > > Any objections? > > Hmm, before you do that you might look at this: > > >>> import numarray > >>> b = numarray.zeros(10) > >>> b==0 > array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1], type=Bool) > >>> b==None > 0 > > So comparing numeric arrays to None returns a scalar false value since > the variable b is not None. Oh yeah... I forgot about that. Numeric comparisons have a special case for None and blow up for most other arbitrary objects. So there's also: >>> a= numarray.arange(100) >>> a == None 0 >>> class foo: .. pass .. >>> a == foo Traceback (most recent call last): .. TypeError: UFunc arguments must be numarray, scalars or numeric sequences > I figure that the behavior of comparisons to None for string arrays and > numeric arrays ought to be consistent. I don't know which is > preferred... So the two choices now on the table are: 1. Change string equality comparisons to return an array of false or true for objects which don't convert to strings. Change Numeric equality comparisons in a similar fashion. 2. Special case string equality comparisons for None, as is done with numerical comparisons. Raise a type error for objects (other than None) which don't convert to string arrays. I think I like 2. Other opinions on 1 & 2? Other choices? Regards, Todd From cookedm at physics.mcmaster.ca Thu Jun 24 12:27:07 2004 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Thu Jun 24 12:27:07 2004 Subject: [Numpy-discussion] comparing string array with None raises ValueError In-Reply-To: <200406241046.13941.haase@msg.ucsf.edu> References: <200406241006.12875.haase@msg.ucsf.edu> <1088098275.26016.160.camel@halloween.stsci.edu> <200406241046.13941.haase@msg.ucsf.edu> Message-ID: <200406241526.36796.cookedm@physics.mcmaster.ca> On June 24, 2004 01:46 pm, Sebastian Haase wrote: > In general it must be OK to compare anything with None, right ? > (BTW, I get the same error with == and !=) No! Not in general! I learnt this back when Numeric implemented rich comparisions; suddenly, lots of my code broke. You don't actually want "is this object _equal_ (or not equal) to None", you want "is this object None", as None is a singleton. A contrived example: >>> class A: ... def __eq__(self, other): return True ... >>> a = A() So, this is bad: >>> a == None True This is good: >>> a is None False In general, if you're testing if something is None, use 'is', not '=='. >>>q.Mrc.hdr('title') is not None -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From perry at stsci.edu Thu Jun 24 13:03:05 2004 From: perry at stsci.edu (Perry Greenfield) Date: Thu Jun 24 13:03:05 2004 Subject: [Numpy-discussion] comparing string array with None raises ValueError, more opinions please In-Reply-To: <1088104465.26016.310.camel@halloween.stsci.edu> Message-ID: Todd Miller wrote: > So the two choices now on the table are: > > 1. Change string equality comparisons to return an array of false or > true for objects which don't convert to strings. Change Numeric > equality comparisons in a similar fashion. > > 2. Special case string equality comparisons for None, as is done with > numerical comparisons. Raise a type error for objects (other than None) > which don't convert to string arrays. > > I think I like 2. Other opinions on 1 & 2? Other choices? > > Regards, > Todd > It would seem consistency with numerical arrays is important so unless someone presents a compelling argument for changing that behavior for arrays, the choice appears clear. Perry [wishing he had spent a little more time thinking about it and trying to remember why None is special cased for comparisons] From jmiller at stsci.edu Thu Jun 24 13:11:10 2004 From: jmiller at stsci.edu (Todd Miller) Date: Thu Jun 24 13:11:10 2004 Subject: [Numpy-discussion] comparing string array with None raises ValueError In-Reply-To: <200406241526.36796.cookedm@physics.mcmaster.ca> References: <200406241006.12875.haase@msg.ucsf.edu> <1088098275.26016.160.camel@halloween.stsci.edu> <200406241046.13941.haase@msg.ucsf.edu> <200406241526.36796.cookedm@physics.mcmaster.ca> Message-ID: <1088107839.26016.327.camel@halloween.stsci.edu> On Thu, 2004-06-24 at 15:26, David M. Cooke wrote: > On June 24, 2004 01:46 pm, Sebastian Haase wrote: > > > In general it must be OK to compare anything with None, right ? > > (BTW, I get the same error with == and !=) > > No! Not in general! Well, this is a good point. I think the current numerical behavior was a hack I stuck in for people who might not be aware of "is". It's looking like a mistake now. > I learnt this back when Numeric implemented rich > comparisions; suddenly, lots of my code broke. You don't actually want "is > this object _equal_ (or not equal) to None", you want "is this object None", > as None is a singleton. However, given the context of the original question, Sebastian's code doesn't read like *that* kind of None comparison: > > > > type '10a80' - that is, an array of 10 80 char 'strings' : > > > > >>> q.Mrc.hdr = q.Mrc.hdrArray[0].field > > > > >>> q.Mrc.hdr('title') != None q.Mrc.hdr('title') is pretty clearly a character array, ergo, it's not None. What did you want it to do Sebastian? Regards, Todd From barrett at stsci.edu Thu Jun 24 13:22:14 2004 From: barrett at stsci.edu (Paul Barrett) Date: Thu Jun 24 13:22:14 2004 Subject: [Numpy-discussion] ObjectArray paramter order poll In-Reply-To: <1088101018.26016.280.camel@halloween.stsci.edu> References: <1088101018.26016.280.camel@halloween.stsci.edu> Message-ID: <40DB37CB.1000408@stsci.edu> Todd Miller wrote: > A while back Colin Williams suggested that we change the parameter order > of numarray.objects.ObjectArray from: > > def __init__(self, shape=None, objects=None, rank=None) > > to: > > def __init__(self, objects=None, shape=None, rank=None) > > The new order emphasizes building object arrays from existing sequences, > while the old order emphasizes building empty (full of None) object > arrays of a specific shape. Since array() and fromlist() already exist > to build from sequences, I thought it was fine to keep the order as is, > plus I hate breaking s/w interfaces unless it was my idea. :-) > > Opinions please? +1 (for the second option) -- Paul Barrett, PhD Space Telescope Science Institute Phone: 410-338-4475 ESS/Science Software Branch FAX: 410-338-4767 Baltimore, MD 21218 From haase at msg.ucsf.edu Thu Jun 24 13:39:06 2004 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Thu Jun 24 13:39:06 2004 Subject: [Numpy-discussion] comparing string array with None raises ValueError In-Reply-To: <1088107839.26016.327.camel@halloween.stsci.edu> References: <200406241006.12875.haase@msg.ucsf.edu> <200406241526.36796.cookedm@physics.mcmaster.ca> <1088107839.26016.327.camel@halloween.stsci.edu> Message-ID: <200406241338.29060.haase@msg.ucsf.edu> On Thursday 24 June 2004 01:10 pm, Todd Miller wrote: > On Thu, 2004-06-24 at 15:26, David M. Cooke wrote: > > On June 24, 2004 01:46 pm, Sebastian Haase wrote: > > > In general it must be OK to compare anything with None, right ? > > > (BTW, I get the same error with == and !=) > > > > No! Not in general! > > Well, this is a good point. I think the current numerical behavior was > a hack I stuck in for people who might not be aware of "is". It's > looking like a mistake now. > > > I learnt this back when Numeric implemented rich > > comparisions; suddenly, lots of my code broke. You don't actually want > > "is this object _equal_ (or not equal) to None", you want "is this object > > None", as None is a singleton. > > However, given the context of the original question, Sebastian's code > > doesn't read like *that* kind of None comparison: > > > > > type '10a80' - that is, an array of 10 80 char 'strings' : > > > > > >>> q.Mrc.hdr = q.Mrc.hdrArray[0].field > > > > > >>> q.Mrc.hdr('title') != None > > q.Mrc.hdr('title') is pretty clearly a character array, ergo, it's not > None. What did you want it to do Sebastian? q.Mrc.hdr('title') is rather an array of 10 'character array' .. that's different, isn't it ? In any case, I actually already changed my code to the check of 'is None' - which is probably what I wanted anyway. So, the only argument left, is that the original error/exception message I got was kind of "ugly" ;-) And I think/thought if 'something' cannot be _converted_ to a string array it can't _be_ '==' a string array. That's just why I expected the simple result to be 'False'. But now I understand that if comparing an array to a scalar (maybe 'None') you always have to decide if this actually compares "the whole array" with the scalar, or if the comparison get "put through" and done on each element "separately" ... which it seems is already decided at least in the numerical array case. Regards, Sebastian From jmiller at stsci.edu Thu Jun 24 14:28:04 2004 From: jmiller at stsci.edu (Todd Miller) Date: Thu Jun 24 14:28:04 2004 Subject: [Numpy-discussion] comparing string array with None raises ValueError In-Reply-To: <200406241338.29060.haase@msg.ucsf.edu> References: <200406241006.12875.haase@msg.ucsf.edu> <200406241526.36796.cookedm@physics.mcmaster.ca> <1088107839.26016.327.camel@halloween.stsci.edu> <200406241338.29060.haase@msg.ucsf.edu> Message-ID: <1088112441.26016.339.camel@halloween.stsci.edu> On Thu, 2004-06-24 at 16:38, Sebastian Haase wrote: > On Thursday 24 June 2004 01:10 pm, Todd Miller wrote: > > On Thu, 2004-06-24 at 15:26, David M. Cooke wrote: > > > On June 24, 2004 01:46 pm, Sebastian Haase wrote: > > > > In general it must be OK to compare anything with None, right ? > > > > (BTW, I get the same error with == and !=) > > > > > > No! Not in general! > > > > Well, this is a good point. I think the current numerical behavior was > > a hack I stuck in for people who might not be aware of "is". It's > > looking like a mistake now. > > > > > I learnt this back when Numeric implemented rich > > > comparisions; suddenly, lots of my code broke. You don't actually want > > > "is this object _equal_ (or not equal) to None", you want "is this object > > > None", as None is a singleton. > > > > However, given the context of the original question, Sebastian's code > > > > doesn't read like *that* kind of None comparison: > > > > > > type '10a80' - that is, an array of 10 80 char 'strings' : > > > > > > >>> q.Mrc.hdr = q.Mrc.hdrArray[0].field > > > > > > >>> q.Mrc.hdr('title') != None > > > > q.Mrc.hdr('title') is pretty clearly a character array, ergo, it's not > > None. What did you want it to do Sebastian? > > q.Mrc.hdr('title') is rather an array of 10 'character array' I thought it was a single nrows x 10 CharArray with 80 character elements. > .. that's > different, isn't it ? It would be. What does q.Mrc.hdr('title').info() say? > In any case, I actually already changed my code to the check of 'is None' - > which is probably what I wanted anyway. > So, the only argument left, is that the original error/exception message I got > was kind of "ugly" ;-) And I think/thought if 'something' cannot be > _converted_ to a string array it can't _be_ '==' a string array. That's just > why I expected the simple result to be 'False'. > But now I understand that if comparing an array to a scalar (maybe 'None') you > always have to decide if this actually compares "the whole array" with the > scalar, or if the comparison get "put through" and done on each element > "separately" ... > which it seems is already decided at least in the numerical array case. I'm not really sure that the numerical array None hack should be replicated. Barring an edict from Perry or Rick or an outcry from the community, I think I'm going to just leave this alone. Regards, Todd From cookedm at physics.mcmaster.ca Thu Jun 24 22:45:16 2004 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Thu Jun 24 22:45:16 2004 Subject: [Numpy-discussion] comparing string array with None raises ValueError In-Reply-To: <1088107839.26016.327.camel@halloween.stsci.edu> References: <200406241006.12875.haase@msg.ucsf.edu> <200406241526.36796.cookedm@physics.mcmaster.ca> <1088107839.26016.327.camel@halloween.stsci.edu> Message-ID: <200406241636.01426.cookedm@physics.mcmaster.ca> On June 24, 2004 04:10 pm, Todd Miller wrote: > On Thu, 2004-06-24 at 15:26, David M. Cooke wrote: > > On June 24, 2004 01:46 pm, Sebastian Haase wrote: > > > In general it must be OK to compare anything with None, right ? > > > (BTW, I get the same error with == and !=) > > > > No! Not in general! > > Well, this is a good point. I think the current numerical behavior was > a hack I stuck in for people who might not be aware of "is". It's > looking like a mistake now. I've fixed up all uses of == None and != None in my copy of the numarray CVS, but the anonymous CVS on SourceForge is timing out :-(, so I haven't made a patch yet. Basically, I ran 'grep -R "[!=]= *None" *' to find all uses of == and != None, and replaced them with is/is not None. That would remove possible bugs. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From falted at pytables.org Fri Jun 25 02:49:13 2004 From: falted at pytables.org (Francesc Alted) Date: Fri Jun 25 02:49:13 2004 Subject: [Numpy-discussion] ObjectArray paramter order poll In-Reply-To: <1088101018.26016.280.camel@halloween.stsci.edu> References: <1088101018.26016.280.camel@halloween.stsci.edu> Message-ID: <200406251148.28153.falted@pytables.org> A Dijous 24 Juny 2004 20:16, Todd Miller va escriure: > A while back Colin Williams suggested that we change the parameter order > of numarray.objects.ObjectArray from: > > def __init__(self, shape=None, objects=None, rank=None) > > to: > > def __init__(self, objects=None, shape=None, rank=None) > > The new order emphasizes building object arrays from existing sequences, > while the old order emphasizes building empty (full of None) object > arrays of a specific shape. Since array() and fromlist() already exist > to build from sequences, I thought it was fine to keep the order as is, > plus I hate breaking s/w interfaces unless it was my idea. :-) > > Opinions please? +1 for the second option -- Francesc Alted From jmiller at stsci.edu Fri Jun 25 05:49:12 2004 From: jmiller at stsci.edu (Todd Miller) Date: Fri Jun 25 05:49:12 2004 Subject: [Numpy-discussion] numarray.sum should raise exception for wrong axis argument - and slicing ... In-Reply-To: <200406231506.24453.haase@msg.ucsf.edu> References: <200406231506.24453.haase@msg.ucsf.edu> Message-ID: <1088167715.32195.18.camel@halloween.stsci.edu> On Wed, 2004-06-23 at 18:06, Sebastian Haase wrote: > Hi, > please take a look at this: > >>> na.sum( na.zeros((2,6)) ) > [0 0 0 0 0 0] > >>> na.sum( na.zeros((2,6)) , 0) > [0 0 0 0 0 0] > >>> na.sum( na.zeros((2,6)) , 1) > [0 0] > >>> na.sum( na.zeros((2,6)) , 2) > [0 0] > >>> na.sum( na.zeros((2,6)) , 3) > [0 0] > >>> na.sum( na.zeros((2,6)) , 4) > [0 0] > >>> na.sum( na.zeros((2,6)) , -1) > [0 0] > >>> na.sum( na.zeros((2,6)) , -2) > [0 0 0 0 0 0] > >>> na.sum( na.zeros((2,6)) , -3) > [0 0] > >>> na.sum( na.zeros((2,6)) , -4) > [0 0] > >>> > > I think here should be a ValueError exception thrown rather than defaulting to > the '-1'-axis. Comments ? This is a bug. Hopefully I'll get it today. > Also this applies to (all?) other functions that have an 'axis' argument. > And further I just found that putting "too many slicings" to an array also > gets silently ignored: > >>> b.shape > (7, 128, 128, 128) > >>> b[2,2,2,2,3] > Traceback (most recent call last): > File "", line 1, in ? > IndexError: too many indices. > > BUT: > >>> b[2:3 , 2:3 , 2:3 , 2:3 , 2:3 , 2:3] > [[[[ 0.]]]] > > > I find this very confusing !! Is there any good reason not to have the > "IndexError" exception in all cases ? This is also a bug. Fixed in CVS now. Thanks for the report! Regards, Todd From haase at msg.ucsf.edu Fri Jun 25 08:24:07 2004 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Fri Jun 25 08:24:07 2004 Subject: [Numpy-discussion] numarray.sum should raise exception for wrong axis argument - and slicing ... In-Reply-To: <1088167715.32195.18.camel@halloween.stsci.edu> References: <200406231506.24453.haase@msg.ucsf.edu> <1088167715.32195.18.camel@halloween.stsci.edu> Message-ID: <200406250823.16156.haase@msg.ucsf.edu> On Friday 25 June 2004 05:48 am, Todd Miller wrote: > On Wed, 2004-06-23 at 18:06, Sebastian Haase wrote: > > Hi, > > > > please take a look at this: > > >>> na.sum( na.zeros((2,6)) ) > > > > [0 0 0 0 0 0] > > > > >>> na.sum( na.zeros((2,6)) , 0) > > > > [0 0 0 0 0 0] > > > > >>> na.sum( na.zeros((2,6)) , 1) > > > > [0 0] > > > > >>> na.sum( na.zeros((2,6)) , 2) > > > > [0 0] > > > > >>> na.sum( na.zeros((2,6)) , 3) > > > > [0 0] > > > > >>> na.sum( na.zeros((2,6)) , 4) > > > > [0 0] > > > > >>> na.sum( na.zeros((2,6)) , -1) > > > > [0 0] > > > > >>> na.sum( na.zeros((2,6)) , -2) > > > > [0 0 0 0 0 0] > > > > >>> na.sum( na.zeros((2,6)) , -3) > > > > [0 0] > > > > >>> na.sum( na.zeros((2,6)) , -4) > > > > [0 0] > > > > > > I think here should be a ValueError exception thrown rather than > > defaulting to the '-1'-axis. Comments ? > > This is a bug. Hopefully I'll get it today. > > > Also this applies to (all?) other functions that have an 'axis' argument. > > And further I just found that putting "too many slicings" to an array > > also > > > > gets silently ignored: > > >>> b.shape > > > > (7, 128, 128, 128) > > > > >>> b[2,2,2,2,3] > > > > Traceback (most recent call last): > > File "", line 1, in ? > > IndexError: too many indices. > > > > BUT: > > >>> b[2:3 , 2:3 , 2:3 , 2:3 , 2:3 , 2:3] > > > > [[[[ 0.]]]] > > > > > > I find this very confusing !! Is there any good reason not to have the > > "IndexError" exception in all cases ? > > This is also a bug. Fixed in CVS now. > > Thanks for the report! > > Regards, > Todd Thanks so much - I got worried there for a moment ;-) Regards, Sebastian From jmiller at stsci.edu Fri Jun 25 08:48:12 2004 From: jmiller at stsci.edu (Todd Miller) Date: Fri Jun 25 08:48:12 2004 Subject: [Numpy-discussion] compress function In-Reply-To: <5900E940-C5DB-11D8-93CA-000A95C92C8E@embl-heidelberg.de> References: <5900E940-C5DB-11D8-93CA-000A95C92C8E@embl-heidelberg.de> Message-ID: <1088178469.32195.142.camel@halloween.stsci.edu> On Thu, 2004-06-24 at 08:38, Peter Verveer wrote: > The documentation of the compress function states that the condition > must be equal to the given axis of the array that is compressed. e.g.: > > >>> a = array([[1,2],[3,4]]) > >>> print compress([1,0], a, axis = 1) > [[1] > [3]] > > However, this also works fine: > > >>> print compress([[1,0],[0,1]], a) > [1, 4] > > which is great (I need that) but not documented. Is that behaviour > intended? It is most likely accidental, but great is great. > If so it maybe should be documented. I filed a bug report but I'm not sure when I'll be *able* to do it. Regards, Todd From haase at msg.ucsf.edu Fri Jun 25 09:50:08 2004 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Fri Jun 25 09:50:08 2004 Subject: [Numpy-discussion] numarray speed - PySequence_GetItem Message-ID: <200406250949.44897.haase@msg.ucsf.edu> Hi, The long story is that I'm looking for a good/fast graph plotting programs; so I found WxPyPlot (http://www.cyberus.ca/~g_will/wxPython/wxpyplot.html) It uses wxPython and plots 25000 data points (with lines + square markers) in under one second - using Numeric that is. [the slow line in WxPyPlot is: dc.DrawLines(self.scaled) where self.scaled is an array of shape (25000,2) and type Float64 ] The short story is that numarray takes maybe 10 times as long as Numeric and I tracked the problem down into the wxPython SWIG typemap where he does this: wxPoint* wxPoint_LIST_helper(PyObject* source, int *count) { bool isFast = PyList_Check(source) || PyTuple_Check(source); for (x=0; x<*count; x++) { // Get an item: try fast way first. if (isFast) { o = PySequence_Fast_GET_ITEM(source, x); } else { o = PySequence_GetItem(source, x); if (o == NULL) { goto error1; } } I'm not 100% sure that this is where the problem lies - is there a chance (or a known issue) that numarray does PySequence_GetItem() slower than Numeric ? I just ran this again using the python profiler and I get this w/ numarray: ncalls tottime percall cumtime percall filename:lineno(function) 1 1.140 1.140 1.320 1.320 gdi.py:554(DrawLines) 1 1.250 1.250 1.520 1.520 gdi.py:792(_DrawRectangleList) 50230 0.450 0.000 0.450 0.000 numarraycore.py:501(__del__) and this with Numeric: 1 0.080 0.080 0.080 0.080 gdi.py:554(DrawLines) 1 0.090 0.090 0.090 0.090 gdi.py:792(_DrawRectangleList) Thanks, Sebastian Haase From jdhunter at ace.bsd.uchicago.edu Fri Jun 25 14:37:11 2004 From: jdhunter at ace.bsd.uchicago.edu (John Hunter) Date: Fri Jun 25 14:37:11 2004 Subject: [Numpy-discussion] numarray speed - PySequence_GetItem In-Reply-To: <200406250949.44897.haase@msg.ucsf.edu> (Sebastian Haase's message of "Fri, 25 Jun 2004 09:49:44 -0700") References: <200406250949.44897.haase@msg.ucsf.edu> Message-ID: >>>>> "Sebastian" == Sebastian Haase writes: Sebastian> Hi, The long story is that I'm looking for a good/fast Sebastian> graph plotting programs; so I found WxPyPlot Sebastian> (http://www.cyberus.ca/~g_will/wxPython/wxpyplot.html) Sebastian> It uses wxPython and plots 25000 data points (with Sebastian> lines + square markers) in under one second - using Sebastian> Numeric that is. Not an answer to your question .... matplotlib has full numarray support (no need to rely on sequence API). You need to set NUMERIX='numarray' in setup.py before building it *and* set numerix : numarray in the matplotlib rc file. If you don't do both of these things, your numarray performance will suffer, sometimes dramatically. With this test script from matplotlib.matlab import * N = 25000 x = rand(N) y = rand(N) scatter(x,y, marker='s') #savefig('test') show() You can do a scatter plot of squares, on my machine in under a second using numarray (wxagg or agg backend). Some fairly recent changes to matplotlib have moved this drawing into extension code, with an approx 10x performance boost from older versions. The latest version on the sf site (0.54.2) however, does have these changes. To plot markers with lines, you would need plot(x,y, marker='-s') instead of scatter. This is considerably slower (approx 3s on my system), mainly because I haven't ported the new fast drawing of marker code to the line class. This is an easy fix, however, and will be added in short order. JDH From haase at msg.ucsf.edu Fri Jun 25 15:34:01 2004 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Fri Jun 25 15:34:01 2004 Subject: [off topic] Re: [Numpy-discussion] numarray speed - PySequence_GetItem In-Reply-To: References: <200406250949.44897.haase@msg.ucsf.edu> Message-ID: <200406251533.27116.haase@msg.ucsf.edu> Hi John, I wanted to try matplotlib a few days ago, but first I had some trouble compiling it (my debian still uses gcc 2-95, which doesn't understand some 'std' namespace/template stuff) - and then it compiled, but segfaulted. Maybe I didn't get "set NUMERIX" stuff right - how do I know that it actually built _and_ uses the wx-backend ? BTW, from the profiling/timing I did you can tell that wxPyPlot actually plots 25000 data points in 0.1 secs - so it's _really_ fast ... So it would be nice to get to the ground of this ... Thanks for the comment, Sebastian On Friday 25 June 2004 02:12 pm, John Hunter wrote: > >>>>> "Sebastian" == Sebastian Haase writes: > > Sebastian> Hi, The long story is that I'm looking for a good/fast > Sebastian> graph plotting programs; so I found WxPyPlot > Sebastian> (http://www.cyberus.ca/~g_will/wxPython/wxpyplot.html) > Sebastian> It uses wxPython and plots 25000 data points (with > Sebastian> lines + square markers) in under one second - using > Sebastian> Numeric that is. > > Not an answer to your question .... > > matplotlib has full numarray support (no need to rely on sequence > API). You need to set NUMERIX='numarray' in setup.py before building > it *and* set numerix : numarray in the matplotlib rc file. If you > don't do both of these things, your numarray performance will suffer, > sometimes dramatically. > > With this test script > > from matplotlib.matlab import * > N = 25000 > x = rand(N) > y = rand(N) > scatter(x,y, marker='s') > #savefig('test') > show() > > You can do a scatter plot of squares, on my machine in under a second > using numarray (wxagg or agg backend). Some fairly recent changes to > matplotlib have moved this drawing into extension code, with an approx > 10x performance boost from older versions. The latest version on the > sf site (0.54.2) however, does have these changes. > > To plot markers with lines, you would need > > plot(x,y, marker='-s') > > instead of scatter. This is considerably slower (approx 3s on my > system), mainly because I haven't ported the new fast drawing of > marker code to the line class. This is an easy fix, however, and will > be added in short order. > > JDH > > > > ------------------------------------------------------- > This SF.Net email sponsored by Black Hat Briefings & Training. > Attend Black Hat Briefings & Training, Las Vegas July 24-29 - > digital self defense, top technical experts, no vendor pitches, > unmatched networking opportunities. Visit www.blackhat.com > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From jmiller at stsci.edu Sat Jun 26 05:23:03 2004 From: jmiller at stsci.edu (Todd Miller) Date: Sat Jun 26 05:23:03 2004 Subject: [Numpy-discussion] numarray.sum should raise exception for wrong axis argument - and slicing ... In-Reply-To: <200406250823.16156.haase@msg.ucsf.edu> References: <200406231506.24453.haase@msg.ucsf.edu> <1088167715.32195.18.camel@halloween.stsci.edu> <200406250823.16156.haase@msg.ucsf.edu> Message-ID: <1088252513.3744.2.camel@localhost.localdomain> On Fri, 2004-06-25 at 11:23, Sebastian Haase wrote: > > > > > Also this applies to (all?) other functions that have an 'axis' argument. > > > And further I just found that putting "too many slicings" to an array > > > also > > > > > > gets silently ignored: > > > >>> b.shape > > > > > > (7, 128, 128, 128) > > > > > > >>> b[2,2,2,2,3] > > > > > > Traceback (most recent call last): > > > File "", line 1, in ? > > > IndexError: too many indices. > > > > > > BUT: > > > >>> b[2:3 , 2:3 , 2:3 , 2:3 , 2:3 , 2:3] > > > > > > [[[[ 0.]]]] > > > > > > > > > I find this very confusing !! Is there any good reason not to have the > > > "IndexError" exception in all cases ? > > > > This is also a bug. Fixed in CVS now. I'm afraid I spoke too soon here... this will be fixed but is not yet. Thanks again for taking the time to bring this to our attention. Regards, Todd From Scott.Daniels at Acm.Org Sat Jun 26 19:11:01 2004 From: Scott.Daniels at Acm.Org (Scott David Daniels) Date: Sat Jun 26 19:11:01 2004 Subject: [Numpy-discussion] Re: numarray equivalent to sign() function In-Reply-To: References: Message-ID: Mike Zingale wrote: > Is there a replacement? def sign(x): return min(max(-1, math.frexp(x)[0]*2), 1) - Scott David Daniels Scott.Daniels at Acm.Org From martin.wiechert at gmx.de Mon Jun 28 10:23:02 2004 From: martin.wiechert at gmx.de (Martin Wiechert) Date: Mon Jun 28 10:23:02 2004 Subject: [Numpy-discussion] Re: [SciPy-user] IRIX problems In-Reply-To: References: <200406281455.50955.martin.wiechert@gmx.de> Message-ID: <200406281922.23886.martin.wiechert@gmx.de> Hi Pearu, thanks for your help. The solution is to replace libcomplib.sgimath with the newer libscs, which has the missing symbol. Our sysadmin pointed me to that. Martin. On Monday 28 June 2004 17:39, Pearu Peterson wrote: > On Mon, 28 Jun 2004, Martin Wiechert wrote: > > Hi all, > > > > I'm trying to install scipy on our dunno-how-old IRIX server. > > I've used a lib called complib.sgimath instead of atlas/LAPACK ... > > Now I get the following error: > > > > import lapack_lite > > ImportError: 563534:python2.3: rld: Fatal Error: unresolvable symbol > > in /user/wiechert/lib/python2.3/site-packages/Numeric/lapack_lite.so: > > zgelsd_ > > (This is a Numeric installation problem, not scipy.) > > > lapack_lite is linked against following libs: > > ldd lapack_lite.so > > libcomplib.sgimath.so => /usr/lib32/libcomplib.sgimath.so > > libblas.so => /usr/lib32/libblas.so > > libftn.so => /usr/lib32/libftn.so > > libm.so => /usr/lib32/libm.so > > libc.so.1 => /usr/lib32/libc.so.1 > > > > What can I do? > > Try to find out which libraries in your system contain zgelsd_ and use it > when linking lapack_lite. When that's unsuccessful then I suggest building > LAPACK libraries yourself, see the relevant section in > > http://www.scipy.org/documentation/buildatlas4scipy.txt > > HTH, > Pearu > > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.net > http://www.scipy.net/mailman/listinfo/scipy-user From Chris.Barker at noaa.gov Mon Jun 28 11:02:05 2004 From: Chris.Barker at noaa.gov (Chris Barker) Date: Mon Jun 28 11:02:05 2004 Subject: [off topic] Re: [Numpy-discussion] numarray speed - PySequence_GetItem In-Reply-To: <200406251533.27116.haase@msg.ucsf.edu> References: <200406250949.44897.haase@msg.ucsf.edu> <200406251533.27116.haase@msg.ucsf.edu> Message-ID: <40E05C8D.6050106@noaa.gov> Sebastian Haase wrote: > BTW, from the profiling/timing I did you can tell that wxPyPlot actually plots > 25000 data points in 0.1 secs - so it's _really_ fast ... Actually, it's probably not that fast, if you are timing on Linux/wxGTK/X-Windows. X is asyncronous, so what you are timing is how long it takes your program to tell X what to draw, but it may take longer than that to actually draw it. However, what you are timing is all the stuff that is effected by numarray/Numeric. I worked on part of the wxPython DC.DrawXXXList stuff, and I really wanted a Numeric native version, but Robin really didn't want an additional dependency. We discussed on this list a while back whether you could compile against Numeric, but let people run without it, and have it all work unless Someone actually used it. What makes that tricky is that the functions that test whether a PyObject is a Numeric array are in Numeric... but it could probably be done if you tried hard enough (maybe include just that function in wxPython...) The Same applies for numarray support. Anyway, as it stands, wxPython DC methods are faster with Lists or Tuples of values than Numeric or Numarray arrays. You might try converting to a list with numarray.tolist() before making the DC call. Another option is to write a few specialized DC functions that take numarray arrays to draw, but are not included in wxPython. I think you'd get it as fast as possible that way. I intend to do that some day. If you want to get it started, I'll help. You could probably get a particularly nice improvement in drawing a lot of rectangles, as looping through a N X 4 array of coords directly would be much, much faster that using the Sequence API on the whole thing, and on each item, and checking at every step what kind of object everything is. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From jmiller at stsci.edu Mon Jun 28 12:04:01 2004 From: jmiller at stsci.edu (Todd Miller) Date: Mon Jun 28 12:04:01 2004 Subject: [off topic] Re: [Numpy-discussion] numarray speed - PySequence_GetItem In-Reply-To: <40E05C8D.6050106@noaa.gov> References: <200406250949.44897.haase@msg.ucsf.edu> <200406251533.27116.haase@msg.ucsf.edu> <40E05C8D.6050106@noaa.gov> Message-ID: <1088449382.3744.138.camel@localhost.localdomain> On Mon, 2004-06-28 at 13:59, Chris Barker wrote: > Sebastian Haase wrote: > > BTW, from the profiling/timing I did you can tell that wxPyPlot actually plots > > 25000 data points in 0.1 secs - so it's _really_ fast ... > > Actually, it's probably not that fast, if you are timing on > Linux/wxGTK/X-Windows. X is asyncronous, so what you are timing is how > long it takes your program to tell X what to draw, but it may take > longer than that to actually draw it. However, what you are timing is > all the stuff that is effected by numarray/Numeric. > > I worked on part of the wxPython DC.DrawXXXList stuff, and I really > wanted a Numeric native version, but Robin really didn't want an > additional dependency. We discussed on this list a while back whether > you could compile against Numeric, but let people run without it, and > have it all work unless Someone actually used it. What makes that tricky > is that the functions that test whether a PyObject is a Numeric array > are in Numeric... but it could probably be done if you tried hard enough > (maybe include just that function in wxPython...) numarray-1.0 has two macros for dealing with this: PyArray_Present() and PyArray_isArray(obj). The former (safely) determines that numarray is installed, while the latter determines that numarray is installed and that obj is a NumArray. Both macros serve to guard sections of code which make more extensive use of the numarray C-API to keep them from segfaulting when numarray is not installed. I think this would be easy to do for Numeric as well. One problem is that compiling a "numarray improved" extension requires some portion of the numarray headers. I refactored the numarray includes so that a relatively simple set of 3 files can be used to support the Numeric compatible interface (for numarray). These could either be included in core Python (with a successful PEP) or included in interested packages. This approach adds a small source code burden somewhere, but eliminates the requirement for users to have numarray installed either to run or compile from source. I'll send out the draft PEP later today. Regards, Todd From jmiller at stsci.edu Mon Jun 28 12:42:02 2004 From: jmiller at stsci.edu (Todd Miller) Date: Mon Jun 28 12:42:02 2004 Subject: [Numpy-discussion] Numarray header PEP Message-ID: <1088451653.3744.200.camel@localhost.localdomain> Perry and I have written a draft PEP for the inclusion of some of the numarray headers within the Python distribution. The headers enable extension writers to provide better support for numarray without adding any additional dependencies (from a user's perspective) to their packages. For users who have numarray, an "improved extension" could provide better performance, and for those without numarray, the extension could work normally using the Sequence protocol. The PEP is attached. It is formatted using the docutils package which can be used to generate HTML or PDF. Comments and corrections would be appreciated. Regards, Todd -------------- next part -------------- PEP: XXX Title: numerical array headers Version: $Revision: 1.3 $ Last-Modified: $Date: 2002/08/30 04:11:20 $ Author: Todd Miller , Perry Greenfield Discussions-To: numpy-discussion at lists.sf.net Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 02-Jun-2004 Python-Version: 2.4 Post-History: 30-Aug-2002 Abstract ======== We propose the inclusion of three numarray header files within the CPython distribution to facilitate use of numarray array objects as an optional data data format for 3rd party modules. The PEP illustrates a simple technique by which a 3rd party extension may support numarrays as input or output values if numarray is installed, and yet the 3rd party extension does not require numarray to be installed to be built. Nothing needs to be changed in the setup.py or makefile for installing with or without numarray, and a subsequent installation of numarray will allow its use without rebuilding the 3rd party extension. Specification ============= This PEP applies only to the CPython platform and only to numarray. Analogous PEPs could be written for Jython and Python.NET and Numeric, but what is discussed here is a speed optimization that is tightly coupled to CPython and numarray. Three header files to support the numarray C-API should be included in the CPython distribution within a numarray subdirectory of the Python include directory: * numarray/arraybase.h * numarray/libnumeric.h * numarray/arrayobject.h The files are shown prefixed with "numarray" to leave the door open for doing similar PEPs with other packages, such as Numeric. If a plethora of such header contributions is anticipated, a further refinement would be to locate the headers under something like "third_party/numarray". In order to provide enhanced performance for array objects, an extension writer would start by including the numarray C-API in addition to any other Python headers: :: #include "numarray/arrayobject.h" Not shown in this PEP are the API calls which operate on numarrays. These are documented in the numarray manual. What is shown here are two calls which are guaranteed to be safe even when numarray is not installed: * PyArray_Present() * PyArray_isArray() In an extension function that wants to access the numarray API, a test needs to be performed to determine if the API functions are safely callable: :: PyObject * some_array_returning_function(PyObject *m, PyObject *args) { int param; PyObject *result; if (!PyArg_ParseTuple(args, "i", ¶m)) return NULL; if (PyArray_Present()) { result = numarray_returning_function(param); } else { result = list_returning_function(param); } return result; } Within **numarray_returning_function**, a subset of the numarray C-API (the Numeric compatible API) is available for use so it is possible to create and return numarrays. Within **list_returning_function**, only the standard Python C-API can be used because numarray is assumed to be unavailable in that particular Python installation. In an extension function that wants to accept numarrays as inputs and provide improved performance over the Python sequence protocol, an additional convenience function exists which diverts arrays to specialized code when numarray is present and the input is an array: :: PyObject * some_array_accepting_function(PyObject *m, PyObject *args) { PyObject *sequence, *result; if (!PyArg_ParseTuple(args, "O", &sequence)) return NULL; if (PyArray_isArray(sequence)) { result = numarray_input_function(sequence); } else { result = sequence_input_function(sequence); } return result; } During module initialization, a numarray enhanced extension must call **import_array()**, a macro which imports numarray and assigns a value to a static API pointer: PyArray_API. Since the API pointer starts with the value NULL and remains so if the numarray import fails, the API pointer serves as a flag that indicates that numarray was sucessfully imported whenever it is non-NULL. :: static void initfoo(void) { PyObject *m = Py_InitModule3( "foo", _foo_functions, _foo__doc__); if (m == NULL) return; import_array(); } **PyArray_Present()** indicates that numarray was successfully imported. It is defined in terms of the API function pointer as: :: #define PyArray_Present() (PyArray_API != NULL) **PyArray_isArray(s)** indicates that numarray was successfully imported and the given parameter is a numarray instance. It is defined as: :: #define PyArray_isArray(s) (PyArray_Present() && PyArray_Check(s)) Motivation ========== The use of numeric arrays as an interchange format is eminently sensible for many kinds of modules. For example, image, graphics, and audio modules all can accept or generate large amounts of numerical data that could easily use the numarray format. But since numarray is not part of the standard distribution, some authors of 3rd party extensions may be reluctant to add a dependency on a different 3rd party extension that isn't absolutely essential for its use fearing dissuading users who may be put off by extra installation requirements. Yet, not allowing easy interchange with numarray introduces annoyances that need not be present. Normally, in the absence of an explicit ability to generate or use numarray objects, one must write conversion utilities to convert from the data representation used to that for numarray. This typically involves excess copying of data (usually from internal to string to numarray). In cases where the 3rd party uses buffer objects, the data may not need copying at all. Either many users may have to develop their own conversion routines or numarray will have to include adapters for many other 3rd party packages. Since numarray is used by many projects, it makes more sense to put the conversion logic on the other side of the fence. There is a clear need for a mechanism that allows 3rd party software to use numarray objects if it is available without requiring numarray's presence to build and install properly. Rationale ========= One solution is to make numarray part of the standard distribution. That may be a good long-term solution, but at the moment, the numeric community is in transition period between the Numeric and numarray packages which may take years to complete. It is not likely that numarray will be considered for adoption until the transition is complete. Numarray is also a large package, and there is legitimate concern about its inclusion as regards the long-term commitment to support. We can solve that problem by making a few include files part of the Python Standard Distribution and demonstrating how extension writers can write code that uses numarray conditionally. The API submitted in this PEP is the subset of the numarray API which is most source compatible with Numeric. The headers consist of two handwritten files (arraybase.h and arrayobject.h) and one generated file (libnumeric.h). arraybase.h contains typedefs and enumerations which are important to both the API presented here and to the larger numarray specific API. arrayobject.h glues together arraybase and libnumeric and is needed for Numeric compatibility. libnumeric.h consists of macros generated from a template and a list of function prototypes. The macros themselves are somewhat intricate in order to provide the compile time checking effect of function prototypes. Further, the interface takes two forms: one form is used to compile numarray and defines static function prototypes. The other form is used to compile extensions which use the API and defines macros which execute function calls through pointers which are found in a table located using a single public API pointer. These macros also test the value of the API pointer in order to deliver a fatal error should a developer forget to initialize by calling import_array(). The interface chosen here is the subset of numarray most useful for porting existing Numeric code or creating new extensions which can be compiled for either numarray or Numeric. There are a number of other numarray API functions which are omitted here for the sake of simplicity. By choosing to support only the Numeric compatible subset of the numarray C-API, concerns about interface stability are minimized because the Numeric API is well established. However, it should be made clear that the numarray API subset proposed here is source compatible, not binary compatible, with Numeric. Resources ========= * numarray/arraybase.h (http://cvs.sourceforge.net/viewcvs.py/numpy/numarray/Include/numarray/arraybase.h) * numarray/libnumeric.h (http://cvs.sourceforge.net/viewcvs.py/numpy/numarray/Include/numarray/libnumeric.h) * numarray/arrayobject.h (http://cvs.sourceforge.net/viewcvs.py/numpy/numarray/Include/numarray/arrayobject.h) * numarray-1.0 manual PDF * numarray-1.0 source distribution * numarray website at STSCI (http://www.stsci.edu/resources/software_hardware/numarray) * example numarray enhanced extension References ========== . [1] PEP 1, PEP Purpose and Guidelines, Warsaw, Hylton (http://www.python.org/peps/pep-0001.html) . [2] PEP 9, Sample Plaintext PEP Template, Warsaw (http://www.python.org/peps/pep-0009.html) Copyright ========= This document has been placed in the public domain. . Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End: From haase at msg.ucsf.edu Mon Jun 28 14:23:50 2004 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Mon Jun 28 14:23:50 2004 Subject: [off topic] Re: [Numpy-discussion] numarray speed - PySequence_GetItem In-Reply-To: <1088449382.3744.138.camel@localhost.localdomain> References: <200406250949.44897.haase@msg.ucsf.edu> <40E05C8D.6050106@noaa.gov> <1088449382.3744.138.camel@localhost.localdomain> Message-ID: <200406281414.25400.haase@msg.ucsf.edu> On Monday 28 June 2004 12:03 pm, Todd Miller wrote: > On Mon, 2004-06-28 at 13:59, Chris Barker wrote: > > Sebastian Haase wrote: > > > BTW, from the profiling/timing I did you can tell that wxPyPlot > > > actually plots 25000 data points in 0.1 secs - so it's _really_ fast > > > ... > > > > Actually, it's probably not that fast, if you are timing on > > Linux/wxGTK/X-Windows. X is asyncronous, so what you are timing is how > > long it takes your program to tell X what to draw, but it may take > > longer than that to actually draw it. However, what you are timing is > > all the stuff that is effected by numarray/Numeric. > > > > I worked on part of the wxPython DC.DrawXXXList stuff, and I really > > wanted a Numeric native version, but Robin really didn't want an > > additional dependency. We discussed on this list a while back whether > > you could compile against Numeric, but let people run without it, and > > have it all work unless Someone actually used it. What makes that tricky > > is that the functions that test whether a PyObject is a Numeric array > > are in Numeric... but it could probably be done if you tried hard enough > > (maybe include just that function in wxPython...) > > numarray-1.0 has two macros for dealing with this: PyArray_Present() > and PyArray_isArray(obj). The former (safely) determines that numarray > is installed, while the latter determines that numarray is installed and > that obj is a NumArray. Both macros serve to guard sections of code > which make more extensive use of the numarray C-API to keep them from > segfaulting when numarray is not installed. I think this would be easy > to do for Numeric as well. > > One problem is that compiling a "numarray improved" extension requires > some portion of the numarray headers. I refactored the numarray > includes so that a relatively simple set of 3 files can be used to > support the Numeric compatible interface (for numarray). These could > either be included in core Python (with a successful PEP) or included in > interested packages. This approach adds a small source code burden > somewhere, but eliminates the requirement for users to have numarray > installed either to run or compile from source. > > I'll send out the draft PEP later today. > > Regards, > Todd My original question was just this: Does anyone know why numarray is maybe 10 times slower that Numeric with that particular code segment (PySequence_GetItem) ? - Sebastian From jmiller at stsci.edu Mon Jun 28 16:19:06 2004 From: jmiller at stsci.edu (Todd Miller) Date: Mon Jun 28 16:19:06 2004 Subject: [off topic] Re: [Numpy-discussion] numarray speed - PySequence_GetItem In-Reply-To: <200406281414.25400.haase@msg.ucsf.edu> References: <200406250949.44897.haase@msg.ucsf.edu> <40E05C8D.6050106@noaa.gov> <1088449382.3744.138.camel@localhost.localdomain> <200406281414.25400.haase@msg.ucsf.edu> Message-ID: <1088464671.3744.300.camel@localhost.localdomain> On Mon, 2004-06-28 at 17:14, Sebastian Haase wrote: > On Monday 28 June 2004 12:03 pm, Todd Miller wrote: > > On Mon, 2004-06-28 at 13:59, Chris Barker wrote: > > > Sebastian Haase wrote: > > > > BTW, from the profiling/timing I did you can tell that wxPyPlot > > > > actually plots 25000 data points in 0.1 secs - so it's _really_ fast > > > > ... > > > > > > Actually, it's probably not that fast, if you are timing on > > > Linux/wxGTK/X-Windows. X is asyncronous, so what you are timing is how > > > long it takes your program to tell X what to draw, but it may take > > > longer than that to actually draw it. However, what you are timing is > > > all the stuff that is effected by numarray/Numeric. > > > > > > I worked on part of the wxPython DC.DrawXXXList stuff, and I really > > > wanted a Numeric native version, but Robin really didn't want an > > > additional dependency. We discussed on this list a while back whether > > > you could compile against Numeric, but let people run without it, and > > > have it all work unless Someone actually used it. What makes that tricky > > > is that the functions that test whether a PyObject is a Numeric array > > > are in Numeric... but it could probably be done if you tried hard enough > > > (maybe include just that function in wxPython...) > > > > numarray-1.0 has two macros for dealing with this: PyArray_Present() > > and PyArray_isArray(obj). The former (safely) determines that numarray > > is installed, while the latter determines that numarray is installed and > > that obj is a NumArray. Both macros serve to guard sections of code > > which make more extensive use of the numarray C-API to keep them from > > segfaulting when numarray is not installed. I think this would be easy > > to do for Numeric as well. > > > > One problem is that compiling a "numarray improved" extension requires > > some portion of the numarray headers. I refactored the numarray > > includes so that a relatively simple set of 3 files can be used to > > support the Numeric compatible interface (for numarray). These could > > either be included in core Python (with a successful PEP) or included in > > interested packages. This approach adds a small source code burden > > somewhere, but eliminates the requirement for users to have numarray > > installed either to run or compile from source. > > > > I'll send out the draft PEP later today. > > > > Regards, > > Todd > > My original question was just this: Does anyone know why numarray is maybe 10 > times slower that Numeric with that particular code segment > (PySequence_GetItem) ? Well, the short answer is probably: no. Looking at the numarray sequence protocol benchmarks in Examples/bench.py, and looking at what wxPython is probably doing (fetching a 1x2 element array from an Nx2 and then fetching 2 numerical values from that)... I can't fully nail it down. My benchmarks show that numarray is 4x slower for fetching the two element array but only 1.1x slower for the two numbers; that makes me expect at most 4x slower. Noticing the 50k __del__ calls in your profile, I eliminated __del__ (breaking numarray) to see if that was the problem; the ratios changed to 2.5x slower and 0.9x slower (actually faster) respectively. The large number of "Check" routines preceding the numarray path (I count 7 looking at my copy of wxPython) has me a little concerned. I think those checks are more expensive for numarray because it is a new style class. I have a hard time imagining a 10x difference overall, but I think Python does have to traverse the numarray class hierarchy rather than do a type pointer comparison so they are more expensive. Is 10x a measured number or a gut feel? One last thought: because the sequence protocol is being used rather than raw array access, compiling matplotlib for numarray (or not) is not the issue. Regards, Todd From tim.hochberg at cox.net Tue Jun 29 08:15:51 2004 From: tim.hochberg at cox.net (Tim Hochberg) Date: Tue Jun 29 08:15:51 2004 Subject: [off topic] Re: [Numpy-discussion] numarray speed - PySequence_GetItem In-Reply-To: <1088518322.17789.104.camel@halloween.stsci.edu> References: <200406250949.44897.haase@msg.ucsf.edu> <40E05C8D.6050106@noaa.gov> <1088449382.3744.138.camel@localhost.localdomain> <200406281414.25400.haase@msg.ucsf.edu> <1088464671.3744.300.camel@localhost.localdomain> <40E0BB9C.5000000@cox.net> <1088518322.17789.104.camel@halloween.stsci.edu> Message-ID: <40E18678.1090800@cox.net> Todd Miller wrote: >On Mon, 2004-06-28 at 20:45, Tim Hochberg wrote: > > >>Todd Miller wrote: >> >> >> [SNIP] >>This reminds me, when profiling bits and pieces of my code I've often >>noticed that __del__ chews up a large chunk of time. Is there any >>prospect of this being knocked down at all, or is it inherent in the >>structure of numarray? >> >> > >__del__ is IMHO the elegant way to do numarray's shadowing of >"misbehaved arrays". misbehaved arrays are ones which don't meet the >requirements of a particular C-function, but generally that means >noncontiguous, byte-swapped, misaligned, or of the wrong type; it also >can mean some other sequence type like a list or tuple. I think using >the destructor is "necessary" for maintaining Numeric compatibility in C >because you can generally count on arrays being DECREF'd, but obviously >you couldn't count on some new API call being called. > > OK, that makes sense. In a general sense at least, I'll have to dig into the source to figure out the details. >__del__ used to be implemented in C as tp_dealloc, but I was running >into segfaults which I tracked down to the order in which a new style >class instance is torn down. The purpose of __del__ is to copy the >contents of a well behaved working array (the shadow) back onto the >original mis-behaved array. The problem was that, because of the >numarray class hierarchy, critical pieces of the shadow (the instance >dictionary) had already been torn down before the tp_dealloc was >called. The only way I could think of to fix it was to move the >destructor farther down in the class hierarchy, i.e. from >_numarray.tp_dealloc to NumArray.__del__ in Python. > > It seems that one could stash a reference to the instance dict somewhere (in PyArrayObject perhaps) to prevent the instance dict from being torn down when the rest of the subclass goes away. It would require another field in PyArrayObject , but that's big enough allready that no one would notice. I could easily be way off base here though. If I end up with too much spare time at some point, I may try to look into this. >If anyone can think of a way to get rid of __del__, I'm all for it. > > > [SNIP] >> >>I don't see any easy/obvious ways to speed up wxPySwigInstance_Check, >> >> > >Why no CheckExact, even if it's hand coded? Maybe the setup is tedious? > > I don't know although I suspect the setup being tedious might be part of it. I also believe that until recently, wxPython used old style classes and you couldn't use CheckExact. What I propose below is essentially a hand coded version of check exact for wxPoints. > > >>but I believe that wxPoints now obey the PySequence protocol, so I think >>that the whole wxPySwigInstance_Check branch could be removed. To get >>that into wxPython you'd probably have to convince Robin that it >>wouldn't hurt the speed of list of wxPoints unduly. >> >>Wait... If the above doesn't work, I think I do have a way that might >>work for speeding the check for a wxPoint. Before the loop starts, get a >>pointer to wx.core.Point (the class for wxPoints these days) and call it >>wxPoint_Type. Then just use for the check: >> o->ob_type == &wxPoint_Type >>Worth a try anyway. >> >>Unfortunately, I don't have any time to try any of this out right now. >> >>Chris, are you feeling bored? >> >>-tim >> >> > >What's the chance of adding direct support for numarray to wxPython? >Our PEP reduces the burden on a package to at worst adding 3 include >files for numarray plus the specialized package code. With those >files, the package can be compiled by users without numarray and also >run without numarray, but would receive a real boost for people willing >to install numarray since the sequence protocol could be bypassed. > > No idea, sorry. I haven't been keeping up with wxPython development lately. -tim From nadavh at visionsense.com Tue Jun 29 08:52:14 2004 From: nadavh at visionsense.com (Nadav Horesh) Date: Tue Jun 29 08:52:14 2004 Subject: [Numpy-discussion] Can not import image module Message-ID: <07C6A61102C94148B8104D42DE95F7E86DED4B@exchange2k.envision.co.il> >>> from numarray import image Traceback (most recent call last): File "", line 1, in -toplevel- from numarray import image File "/usr/local/lib/python2.3/site-packages/numarray/image/__init__.py", line 2, in -toplevel- from combine import * File "/usr/local/lib/python2.3/site-packages/numarray/image/combine.py", line 65, in -toplevel- _median = _combine.median AttributeError: 'module' object has no attribute 'median' numarray version: 1.0. I got this error also with version 0.9 Nadav. From curzio.basso at unibas.ch Tue Jun 29 09:05:39 2004 From: curzio.basso at unibas.ch (Curzio Basso) Date: Tue Jun 29 09:05:39 2004 Subject: [Numpy-discussion] generalized SVD Message-ID: <40E17833.7050400@unibas.ch> Hi. Has anyone a suggestion on where to find a GSVD implementation which I can use with numarray? I've seen GSVD is part of LAPACK, so maybe one option could be to use a LAPACK library implementing the required functions... thanks in advance. curzio. From jmiller at stsci.edu Tue Jun 29 09:05:49 2004 From: jmiller at stsci.edu (Todd Miller) Date: Tue Jun 29 09:05:49 2004 Subject: [off topic] Re: [Numpy-discussion] numarray speed - PySequence_GetItem In-Reply-To: <40E0BB9C.5000000@cox.net> References: <200406250949.44897.haase@msg.ucsf.edu> <40E05C8D.6050106@noaa.gov> <1088449382.3744.138.camel@localhost.localdomain> <200406281414.25400.haase@msg.ucsf.edu> <1088464671.3744.300.camel@localhost.localdomain> <40E0BB9C.5000000@cox.net> Message-ID: <1088518322.17789.104.camel@halloween.stsci.edu> On Mon, 2004-06-28 at 20:45, Tim Hochberg wrote: > Todd Miller wrote: > > >On Mon, 2004-06-28 at 17:14, Sebastian Haase wrote: > > > > > >> [SNIP] > >> > >>My original question was just this: Does anyone know why numarray is maybe 10 > >>times slower that Numeric with that particular code segment > >>(PySequence_GetItem) ? > >> > >> > > > >Well, the short answer is probably: no. > > > >Looking at the numarray sequence protocol benchmarks in > >Examples/bench.py, and looking at what wxPython is probably doing > >(fetching a 1x2 element array from an Nx2 and then fetching 2 numerical > >values from that)... I can't fully nail it down. My benchmarks show > >that numarray is 4x slower for fetching the two element array but only > >1.1x slower for the two numbers; that makes me expect at most 4x > >slower. > > > >Noticing the 50k __del__ calls in your profile, I eliminated __del__ > >(breaking numarray) to see if that was the problem; the ratios changed > >to 2.5x slower and 0.9x slower (actually faster) respectively. > > > > > This reminds me, when profiling bits and pieces of my code I've often > noticed that __del__ chews up a large chunk of time. Is there any > prospect of this being knocked down at all, or is it inherent in the > structure of numarray? __del__ is IMHO the elegant way to do numarray's shadowing of "misbehaved arrays". misbehaved arrays are ones which don't meet the requirements of a particular C-function, but generally that means noncontiguous, byte-swapped, misaligned, or of the wrong type; it also can mean some other sequence type like a list or tuple. I think using the destructor is "necessary" for maintaining Numeric compatibility in C because you can generally count on arrays being DECREF'd, but obviously you couldn't count on some new API call being called. __del__ used to be implemented in C as tp_dealloc, but I was running into segfaults which I tracked down to the order in which a new style class instance is torn down. The purpose of __del__ is to copy the contents of a well behaved working array (the shadow) back onto the original mis-behaved array. The problem was that, because of the numarray class hierarchy, critical pieces of the shadow (the instance dictionary) had already been torn down before the tp_dealloc was called. The only way I could think of to fix it was to move the destructor farther down in the class hierarchy, i.e. from _numarray.tp_dealloc to NumArray.__del__ in Python. If anyone can think of a way to get rid of __del__, I'm all for it. > >The large number of "Check" routines preceding the numarray path (I > >count 7 looking at my copy of wxPython) has me a little concerned. I > >think those checks are more expensive for numarray because it is a new > >style class. > > > If that's really a significant slowdown, the culprit's are likely > PyTuple_Check, PyList_Check and wxPySwigInstance_Check. > PySequence_Check appears to just be pointer compares and shouldn't > invoke any new style class machinery. PySequence_Length calls sq_length, > but appears also to not involve new class machinery. Of these, I think > PyTuple_Check and PyList_Check could be replaced with PyTuple_CheckExact > and PyList_CheckExact. This would slow down people using subclasses of > tuple/list, but speed everyone else up since the latter pair of > functions are just pointer compares. I think the former group is a very > small minority, possibly nonexistent, minority, so this would probably > be acceptable. > > I don't see any easy/obvious ways to speed up wxPySwigInstance_Check, Why no CheckExact, even if it's hand coded? Maybe the setup is tedious? > but I believe that wxPoints now obey the PySequence protocol, so I think > that the whole wxPySwigInstance_Check branch could be removed. To get > that into wxPython you'd probably have to convince Robin that it > wouldn't hurt the speed of list of wxPoints unduly. > > Wait... If the above doesn't work, I think I do have a way that might > work for speeding the check for a wxPoint. Before the loop starts, get a > pointer to wx.core.Point (the class for wxPoints these days) and call it > wxPoint_Type. Then just use for the check: > o->ob_type == &wxPoint_Type > Worth a try anyway. > > Unfortunately, I don't have any time to try any of this out right now. > > Chris, are you feeling bored? > > -tim What's the chance of adding direct support for numarray to wxPython? Our PEP reduces the burden on a package to at worst adding 3 include files for numarray plus the specialized package code. With those files, the package can be compiled by users without numarray and also run without numarray, but would receive a real boost for people willing to install numarray since the sequence protocol could be bypassed. Regards, Todd From jmiller at stsci.edu Tue Jun 29 09:12:48 2004 From: jmiller at stsci.edu (Todd Miller) Date: Tue Jun 29 09:12:48 2004 Subject: [off topic] Re: [Numpy-discussion] numarray speed - PySequence_GetItem In-Reply-To: <200406281638.03922.haase@msg.ucsf.edu> References: <200406250949.44897.haase@msg.ucsf.edu> <200406281414.25400.haase@msg.ucsf.edu> <1088464671.3744.300.camel@localhost.localdomain> <200406281638.03922.haase@msg.ucsf.edu> Message-ID: <1088520761.17789.184.camel@halloween.stsci.edu> On Mon, 2004-06-28 at 19:38, Sebastian Haase wrote: > > Is 10x a measured number or a gut feel? > > I put some time.clock() statements into the wxPyPlot code > I got this: (the times are differences: T_after-T_before) > one run with numarray: > <__main__.PolyLine instance at 0x868d414> time= 1.06 > <__main__.PolyMarker instance at 0x878e9c4> time= 1.37 > a second run with numarray: > <__main__.PolyLine instance at 0x875da1c> time= 0.85 > <__main__.PolyMarker instance at 0x86da034> time= 1.04 > first run with Numeric: > <__main__.PolyLine instance at 0x858babc> time= 0.07 > <__main__.PolyMarker instance at 0x858bc4c> time= 0.14 > a second one: > <__main__.PolyLine instance at 0x858cd7c> time= 0.08 > <__main__.PolyMarker instance at 0x8585d8c> time= 0.17 > This seems to be consistent with the profiling I did before: > I get this w/ numarray: > ncalls tottime percall cumtime percall filename:lineno(function) > 1 1.140 1.140 1.320 1.320 gdi.py:554(DrawLines) > 1 1.250 1.250 1.520 1.520 gdi.py:792(_DrawRectangleList) > 50230 0.450 0.000 0.450 0.000 numarraycore.py:501(__del__) > and this with Numeric: > 1 0.080 0.080 0.080 0.080 gdi.py:554(DrawLines) > 1 0.090 0.090 0.090 0.090 gdi.py:792(_DrawRectangleList) > > so this looks to me like a factor of 10x. Me too. Can you (or somebody) post the application code which does the drawlines? I can definitely instrument the bottleneck C-code, but I don't have time to ascend the wxPython learning curve. Todd From Chris.Barker at noaa.gov Tue Jun 29 09:36:03 2004 From: Chris.Barker at noaa.gov (Chris Barker) Date: Tue Jun 29 09:36:03 2004 Subject: [off topic] Re: [Numpy-discussion] numarray speed - PySequence_GetItem In-Reply-To: <40E18678.1090800@cox.net> References: <200406250949.44897.haase@msg.ucsf.edu> <40E05C8D.6050106@noaa.gov> <1088449382.3744.138.camel@localhost.localdomain> <200406281414.25400.haase@msg.ucsf.edu> <1088464671.3744.300.camel@localhost.localdomain> <40E0BB9C.5000000@cox.net> <1088518322.17789.104.camel@halloween.stsci.edu> <40E18678.1090800@cox.net> Message-ID: <40E199A7.3000301@noaa.gov> Tim Hochberg wrote: >>> but I believe that wxPoints now obey the PySequence protocol, so I >>> think that the whole wxPySwigInstance_Check branch could be removed. >>> To get that into wxPython you'd probably have to convince Robin that >>> it wouldn't hurt the speed of list of wxPoints unduly. >>> >>> Wait... If the above doesn't work, I think I do have a way that might >>> work for speeding the check for a wxPoint. Before the loop starts, >>> get a pointer to wx.core.Point (the class for wxPoints these days) >>> and call it wxPoint_Type. Then just use for the check: >>> o->ob_type == &wxPoint_Type >>> Worth a try anyway. >>> >>> Unfortunately, I don't have any time to try any of this out right now. >>> >>> Chris, are you feeling bored? Do you mean me? if so: A) I'm not bored. B) This is all kind of beyond me anyway, and C) I'm planning on implementing my own custom DC.DrawLotsOfStuff code, because I have some specialized needs that probably don't really belong in wxPython. My stuff will take Numeric arrays as input (This is for my FloatCanvas, if anyone cares). I'm still using Numeric, as numarray is a LOT slower when used in FloatCanvas, probably because I do a lot with small arrays, and maybe because of what we're talking about here as well. However, This may turn out to be important to me some day, so who knows? I'll keep this note around. >> What's the chance of adding direct support for numarray to wxPython? >> Our PEP reduces the burden on a package to at worst adding 3 include >> files for numarray plus the specialized package code. With those >> files, the package can be compiled by users without numarray and also >> run without numarray, but would receive a real boost for people willing >> to install numarray since the sequence protocol could be bypassed. If the PEP is accepted, and those include files are part of the standard Python distro, I suspect Robin would be quite happy to add direct support, at least if someone else writes the code. Whether he'd be open to including those files in the wxPython distribution itself, I don't know. Perhaps I'll drop him a line. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From haase at msg.ucsf.edu Tue Jun 29 09:44:46 2004 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue Jun 29 09:44:46 2004 Subject: [Numpy-discussion] bug ? in Records arrays in numarray Message-ID: <200406281700.37694.haase@msg.ucsf.edu> Hi, I have two record arrays. I was trying to assign one item from one recarray to the other: >>> omx.zext[0] = main.ring4ext[0,0] Traceback (most recent call last): File "", line 1, in ? File "X:/PrWin\numarray\records.py", line 744, in _setitem self.field(self._names[i])[row] = value.field(self._names[i]) File "C:\Python22\Lib\site-packages\numarray\numarraycore.py", line 619, in __tonumtype__ ValueError: Can't use non-rank-0 array as a scalar. >>> `omx.zext[0]` '' >>> `main.ring4ext[0,0]` '' >>> q = omx.zext[0] >>> w = main.ring4ext[0,0] >>> q (2097152107, -595656700, 91141, 1.0634608642000868e+037, -1.14841804241843e +018, 1.2771574333702815e-040) >>> w (array([428]), array([75]), array([124]), array([ 1.08846451e+09], type=Float32), array([ 99.25], type=Float32), array([ 1996.82995605], type=Float32)) >>> omx.zext._formats ['1Int32', '1Int32', '1Int32', '1Float32', '1Float32', '1Float32'] >>> main.ring4ext._formats ['1Int32', '1Int32', '1Int32', '1Float32', '1Float32', '1Float32'] I can't track down why one (q) contains scalars and the other (w) constains arrays (is array([428]) a 'rank-0 array'? ) ! This is how I generate them: main.ring4ext = rec.RecArray(main._extHdrRingBuffer, "i4,i4,i4,f4,f4,f4", shape=(ts,nc), names=("num","min","max","time","mean","darkVal"), aligned=1) omx.zext = rec.array(formats="i4,i4,i4,f4,f4,f4", names=("num","min","max","time","mean","darkVal"),shape=tuple(shapeZ),aligned=1 ) [[BTW: (shapeZ is a list, if I say: ...,shape=shapeZ I get NameError, "Illegal shape %s" % `shape` from "records.py" in line 462 -- usually type(shape) == types.ListType should be OK, right ?) ]] Any ideas? Thanks, Sebastian Haase From tim.hochberg at cox.net Tue Jun 29 10:12:38 2004 From: tim.hochberg at cox.net (Tim Hochberg) Date: Tue Jun 29 10:12:38 2004 Subject: [off topic] Re: [Numpy-discussion] numarray speed - PySequence_GetItem In-Reply-To: <1088464671.3744.300.camel@localhost.localdomain> References: <200406250949.44897.haase@msg.ucsf.edu> <40E05C8D.6050106@noaa.gov> <1088449382.3744.138.camel@localhost.localdomain> <200406281414.25400.haase@msg.ucsf.edu> <1088464671.3744.300.camel@localhost.localdomain> Message-ID: <40E0BB9C.5000000@cox.net> Todd Miller wrote: >On Mon, 2004-06-28 at 17:14, Sebastian Haase wrote: > > >> [SNIP] >> >>My original question was just this: Does anyone know why numarray is maybe 10 >>times slower that Numeric with that particular code segment >>(PySequence_GetItem) ? >> >> > >Well, the short answer is probably: no. > >Looking at the numarray sequence protocol benchmarks in >Examples/bench.py, and looking at what wxPython is probably doing >(fetching a 1x2 element array from an Nx2 and then fetching 2 numerical >values from that)... I can't fully nail it down. My benchmarks show >that numarray is 4x slower for fetching the two element array but only >1.1x slower for the two numbers; that makes me expect at most 4x >slower. > >Noticing the 50k __del__ calls in your profile, I eliminated __del__ >(breaking numarray) to see if that was the problem; the ratios changed >to 2.5x slower and 0.9x slower (actually faster) respectively. > > This reminds me, when profiling bits and pieces of my code I've often noticed that __del__ chews up a large chunk of time. Is there any prospect of this being knocked down at all, or is it inherent in the structure of numarray? >The large number of "Check" routines preceding the numarray path (I >count 7 looking at my copy of wxPython) has me a little concerned. I >think those checks are more expensive for numarray because it is a new >style class. > If that's really a significant slowdown, the culprit's are likely PyTuple_Check, PyList_Check and wxPySwigInstance_Check. PySequence_Check appears to just be pointer compares and shouldn't invoke any new style class machinery. PySequence_Length calls sq_length, but appears also to not involve new class machinery. Of these, I think PyTuple_Check and PyList_Check could be replaced with PyTuple_CheckExact and PyList_CheckExact. This would slow down people using subclasses of tuple/list, but speed everyone else up since the latter pair of functions are just pointer compares. I think the former group is a very small minority, possibly nonexistent, minority, so this would probably be acceptable. I don't see any easy/obvious ways to speed up wxPySwigInstance_Check, but I believe that wxPoints now obey the PySequence protocol, so I think that the whole wxPySwigInstance_Check branch could be removed. To get that into wxPython you'd probably have to convince Robin that it wouldn't hurt the speed of list of wxPoints unduly. Wait... If the above doesn't work, I think I do have a way that might work for speeding the check for a wxPoint. Before the loop starts, get a pointer to wx.core.Point (the class for wxPoints these days) and call it wxPoint_Type. Then just use for the check: o->ob_type == &wxPoint_Type Worth a try anyway. Unfortunately, I don't have any time to try any of this out right now. Chris, are you feeling bored? -tim >I have a hard time imagining a 10x difference overall, >but I think Python does have to traverse the numarray class hierarchy >rather than do a type pointer comparison so they are more expensive. > > From jmiller at stsci.edu Tue Jun 29 10:38:02 2004 From: jmiller at stsci.edu (Todd Miller) Date: Tue Jun 29 10:38:02 2004 Subject: [Numpy-discussion] Can not import image module In-Reply-To: <07C6A61102C94148B8104D42DE95F7E86DED4B@exchange2k.envision.co.il> References: <07C6A61102C94148B8104D42DE95F7E86DED4B@exchange2k.envision.co.il> Message-ID: <1088530609.17789.270.camel@halloween.stsci.edu> On Tue, 2004-06-29 at 07:14, Nadav Horesh wrote: > >>> from numarray import image I tried this and it worked fine for me under RH9.0 and Python-2.3.4 and also under Windows 2000 pro. I suggest deleting your entire numarray directory from site-packages and then re-installing. Also, what's your platform and shell? Regards, Todd > Traceback (most recent call last): > File "", line 1, in -toplevel- > from numarray import image > File "/usr/local/lib/python2.3/site-packages/numarray/image/__init__.py", line 2, in -toplevel- > from combine import * > File "/usr/local/lib/python2.3/site-packages/numarray/image/combine.py", line 65, in -toplevel- > _median = _combine.median > AttributeError: 'module' object has no attribute 'median' > > > > numarray version: 1.0. I got this error also with version 0.9 > > Nadav. > > > > > ------------------------------------------------------- > This SF.Net email sponsored by Black Hat Briefings & Training. > Attend Black Hat Briefings & Training, Las Vegas July 24-29 - > digital self defense, top technical experts, no vendor pitches, > unmatched networking opportunities. Visit www.blackhat.com > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- Todd Miller Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21218 (410) 338-4576 From gerard.vermeulen at grenoble.cnrs.fr Tue Jun 29 10:46:02 2004 From: gerard.vermeulen at grenoble.cnrs.fr (Gerard Vermeulen) Date: Tue Jun 29 10:46:02 2004 Subject: [Numpy-discussion] Numarray header PEP In-Reply-To: <1088451653.3744.200.camel@localhost.localdomain> References: <1088451653.3744.200.camel@localhost.localdomain> Message-ID: <20040629194456.44a1fa7f.gerard.vermeulen@grenoble.cnrs.fr> > > The PEP is attached. It is formatted using the docutils package which > can be used to generate HTML or PDF. Comments and corrections would be > appreciated. > PyQwt is a Python extension which can be conditionally compiled against Numeric and/or numarray (both, one of them or none). Your PEP excludes importing of Numeric and numarray in the same C-extension. All you need to do is to hide the macros PyArray_Present(), PyArray_isArray() and import_array() into a few functions with numarray specific names, so that the following becomes possible: #include /* defines the functions (no macros) int Numeric_Present(); int Numeric_isArray(); void import_numeric(); to hide the Numeric C-API stuff in a small Numeric/meta.c file. */ #include /* defines the functions (no macros) int numarray_Present(); int numarray_isArray(); void import_numarray(); to hide the numarray C-API stuff in a small numarray/meta.c file. */ PyObject * some_array_returning_function(PyObject *m, PyObject *args) { int param; PyObject *result; if (!PyArg_ParseTuple(args, "i", ¶m)) return NULL; if (Numeric_Present()) { result = numeric_returning_function(param); } else if (Numarray_Present()) { result = numarray_returning_function(param); } else { result = list_returning_function(param); } return result; } PyObject * some_array_accepting_function(PyObject *m, PyObject *args) { PyObject *sequence, *result; if (!PyArg_ParseTuple(args, "O", &sequence)) return NULL; if (Numeric_isArray(sequence)) { result = numeric_input_function(sequence); } else if (Numarray_isArray(sequence)) { result = numarray_input_function(sequence); } else { result = sequence_input_function(sequence); } return result; } /* the code for the numeric_whatever_functions and for the numarray_whatever_functions should be source files. */ static void initfoo(void) { PyObject *m = Py_InitModule3( "foo", _foo_functions, _foo__doc__); if (m == NULL) return; import_numeric(); import_numarray(); } Regards -- Gerard From nadavh at visionsense.com Tue Jun 29 11:32:02 2004 From: nadavh at visionsense.com (Nadav Horesh) Date: Tue Jun 29 11:32:02 2004 Subject: [Numpy-discussion] Can not import image module Message-ID: <07C6A61102C94148B8104D42DE95F7E86DED4F@exchange2k.envision.co.il> Same (RH9.0 + Python 2.3.4). It is OK on my home PC (Gentoo 1.4 +Python 2.3.4). I'll try your advice when I'll back at work. Thanks, Nadav. -----Original Message----- From: Todd Miller [mailto:jmiller at stsci.edu] Sent: Tue 29-Jun-04 20:36 To: Nadav Horesh Cc: numpy-discussion Subject: Re: [Numpy-discussion] Can not import image module On Tue, 2004-06-29 at 07:14, Nadav Horesh wrote: > >>> from numarray import image I tried this and it worked fine for me under RH9.0 and Python-2.3.4 and also under Windows 2000 pro. I suggest deleting your entire numarray directory from site-packages and then re-installing. Also, what's your platform and shell? Regards, Todd > Traceback (most recent call last): > File "", line 1, in -toplevel- > from numarray import image > File "/usr/local/lib/python2.3/site-packages/numarray/image/__init__.py", line 2, in -toplevel- > from combine import * > File "/usr/local/lib/python2.3/site-packages/numarray/image/combine.py", line 65, in -toplevel- > _median = _combine.median > AttributeError: 'module' object has no attribute 'median' > > > > numarray version: 1.0. I got this error also with version 0.9 > > Nadav. > > > > > ------------------------------------------------------- > This SF.Net email sponsored by Black Hat Briefings & Training. > Attend Black Hat Briefings & Training, Las Vegas July 24-29 - > digital self defense, top technical experts, no vendor pitches, > unmatched networking opportunities. Visit www.blackhat.com > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- Todd Miller Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21218 (410) 338-4576 From jmiller at stsci.edu Tue Jun 29 12:10:20 2004 From: jmiller at stsci.edu (Todd Miller) Date: Tue Jun 29 12:10:20 2004 Subject: [Numpy-discussion] Numarray header PEP In-Reply-To: <20040629194456.44a1fa7f.gerard.vermeulen@grenoble.cnrs.fr> References: <1088451653.3744.200.camel@localhost.localdomain> <20040629194456.44a1fa7f.gerard.vermeulen@grenoble.cnrs.fr> Message-ID: <1088536183.17789.346.camel@halloween.stsci.edu> On Tue, 2004-06-29 at 13:44, Gerard Vermeulen wrote: > > > > The PEP is attached. It is formatted using the docutils package which > > can be used to generate HTML or PDF. Comments and corrections would be > > appreciated. > > > PyQwt is a Python extension which can be conditionally compiled against > Numeric and/or numarray (both, one of them or none). Well that's cool! I'll have to keep the PyQwt guys in mind as potential first users. > Your PEP excludes importing of Numeric and numarray in the same C-extension. This is true but I don't understand your solution so I'll blather on below. > All you need to do is to hide the macros PyArray_Present(), PyArray_isArray() > and import_array() into a few functions with numarray specific names, so > that the following becomes possible: > > #include > /* defines the functions (no macros) > int Numeric_Present(); > int Numeric_isArray(); > void import_numeric(); > to hide the Numeric C-API stuff in a small Numeric/meta.c file. > */ > #include > /* defines the functions (no macros) > int numarray_Present(); > int numarray_isArray(); > void import_numarray(); > to hide the numarray C-API stuff in a small numarray/meta.c file. > */ > I may have come up with the wrong scheme for the Present() and isArray(). With my scheme, they *have* to be macros because the API functions are unsafe: when numarray or Numeric is not present, the API function table pointers are NULL so calling through them results in either a fatal error or a segfault. There is an additional problem that the "same functions" need to be called through different API pointers depending on whether numarray or Numeric is being supported. Redefinition of typedefs and enumerations (or perhaps conditional compilation short-circuited re-definitions) may also present a problem with compiling (or worse, running). I certainly like the idea of supporting both in the same extension module, but don't see how to get there, other than with separate compilation units. With seperate .c files, I'm not aware of a problem other than lots of global symbols. I haven't demoed that yet so I am interested if someone has made it work. Thanks very much for commenting on the PEP. Regards, Todd > PyObject * > some_array_returning_function(PyObject *m, PyObject *args) > { > int param; > PyObject *result; > > if (!PyArg_ParseTuple(args, "i", ¶m)) > return NULL; > > if (Numeric_Present()) { > result = numeric_returning_function(param); > } else if (Numarray_Present()) { > result = numarray_returning_function(param); > } else { > result = list_returning_function(param); > } > return result; > } > > PyObject * > some_array_accepting_function(PyObject *m, PyObject *args) > { > PyObject *sequence, *result; > > if (!PyArg_ParseTuple(args, "O", &sequence)) > return NULL; > > if (Numeric_isArray(sequence)) { > result = numeric_input_function(sequence); > } else if (Numarray_isArray(sequence)) { > result = numarray_input_function(sequence); > } else { > result = sequence_input_function(sequence); > } > return result; > } > > /* > the code for the numeric_whatever_functions and for the > numarray_whatever_functions should be source files. > */ > > static void > initfoo(void) > { > PyObject *m = Py_InitModule3( > "foo", > _foo_functions, > _foo__doc__); > if (m == NULL) return; > import_numeric(); > import_numarray(); > } > > Regards -- Gerard > > > > > > > ------------------------------------------------------- > This SF.Net email sponsored by Black Hat Briefings & Training. > Attend Black Hat Briefings & Training, Las Vegas July 24-29 - > digital self defense, top technical experts, no vendor pitches, > unmatched networking opportunities. Visit www.blackhat.com > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- Todd Miller Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21218 (410) 338-4576 From tim.hochberg at cox.net Tue Jun 29 12:22:11 2004 From: tim.hochberg at cox.net (Tim Hochberg) Date: Tue Jun 29 12:22:11 2004 Subject: [off topic] Re: [Numpy-discussion] numarray speed - PySequence_GetItem In-Reply-To: <1088520761.17789.184.camel@halloween.stsci.edu> References: <200406250949.44897.haase@msg.ucsf.edu> <200406281414.25400.haase@msg.ucsf.edu> <1088464671.3744.300.camel@localhost.localdomain> <200406281638.03922.haase@msg.ucsf.edu> <1088520761.17789.184.camel@halloween.stsci.edu> Message-ID: <40E1C12E.4060102@cox.net> Todd Miller wrote: >On Mon, 2004-06-28 at 19:38, Sebastian Haase wrote: > > > >>>Is 10x a measured number or a gut feel? >>> >>> >>[SNIP] >> >>so this looks to me like a factor of 10x. >> >> > >Me too. Can you (or somebody) post the application code which does the >drawlines? I can definitely instrument the bottleneck C-code, but I >don't have time to ascend the wxPython learning curve. > > Let me second that. With a little nudge from Chris Barker, I managed to get wxPython to compile here and I have some changes that may speed up drawlines, but it'd probably be best if we were all using the same benchmark. -tim From tim.hochberg at cox.net Tue Jun 29 13:21:07 2004 From: tim.hochberg at cox.net (Tim Hochberg) Date: Tue Jun 29 13:21:07 2004 Subject: [off topic] Re: [Numpy-discussion] numarray speed - PySequence_GetItem In-Reply-To: <1088520761.17789.184.camel@halloween.stsci.edu> References: <200406250949.44897.haase@msg.ucsf.edu> <200406281414.25400.haase@msg.ucsf.edu> <1088464671.3744.300.camel@localhost.localdomain> <200406281638.03922.haase@msg.ucsf.edu> <1088520761.17789.184.camel@halloween.stsci.edu> Message-ID: <40E1CEE6.3050108@cox.net> I'd bet a case of beer (or cash equivalent) that one of the main bottlenecks is the path PySequence_GetItem->_ndarray_item->_universalIndexing->_simpleIndexing->_simpleIndexingCore. The path through _universalIndexing in particular, if I deciphered it correctly, looks very slow. I don't think it needs to be that way though, _universalIndexing could probably be sped up, but more promising I think _ndarray_item could be made to call _simpleIndexingCore without all that much work. It appears that this would save the creation of several intermediate objects and it also looks like a couple of calls back to python! I'm not familiar with this code though, so I could easily be missing something that makes calling _simpleIndexingCore harder than it looks. -tim From Chris.Barker at noaa.gov Tue Jun 29 14:08:06 2004 From: Chris.Barker at noaa.gov (Chris Barker) Date: Tue Jun 29 14:08:06 2004 Subject: [off topic] Re: [Numpy-discussion] numarray speed - PySequence_GetItem In-Reply-To: <40E1C12E.4060102@cox.net> References: <200406250949.44897.haase@msg.ucsf.edu> <200406281414.25400.haase@msg.ucsf.edu> <1088464671.3744.300.camel@localhost.localdomain> <200406281638.03922.haase@msg.ucsf.edu> <1088520761.17789.184.camel@halloween.stsci.edu> <40E1C12E.4060102@cox.net> Message-ID: <40E1D998.8010204@noaa.gov> Tim Hochberg wrote: > Todd Miller wrote: >> Me too. Can you (or somebody) post the application code which does the >> drawlines? I can definitely instrument the bottleneck C-code, but I >> don't have time to ascend the wxPython learning curve. > Let me second that. With a little nudge from Chris Barker, I managed to > get wxPython to compile here and I have some changes that may speed up > drawlines, but it'd probably be best if we were all using the same > benchmark. Well, if I can't code, at least I can nudge! Perhaps I can also help get Todd what he's asking for, except that I'm not sure what you mean by "application code". Do you mean a small wxPython application that uses DC.DrawLines (and friends)? If so, yes, I can do that. What version of wxPython should I target? CVS head? 2.5.1? (The latest "released" version) there are some slight incompatibilities between versions, particularly with DC calls, unfortunately. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From gerard.vermeulen at grenoble.cnrs.fr Tue Jun 29 14:33:03 2004 From: gerard.vermeulen at grenoble.cnrs.fr (gerard.vermeulen at grenoble.cnrs.fr) Date: Tue Jun 29 14:33:03 2004 Subject: [Numpy-discussion] Numarray header PEP In-Reply-To: <1088536183.17789.346.camel@halloween.stsci.edu> References: <1088451653.3744.200.camel@localhost.localdomain> <20040629194456.44a1fa7f.gerard.vermeulen@grenoble.cnrs.fr> <1088536183.17789.346.camel@halloween.stsci.edu> Message-ID: <20040629211800.M55753@grenoble.cnrs.fr> On 29 Jun 2004 15:09:43 -0400, Todd Miller wrote > On Tue, 2004-06-29 at 13:44, Gerard Vermeulen wrote: > > > > > > The PEP is attached. It is formatted using the docutils package which > > > can be used to generate HTML or PDF. Comments and corrections would be > > > appreciated. > > > > > PyQwt is a Python extension which can be conditionally compiled against > > Numeric and/or numarray (both, one of them or none). > > Well that's cool! I'll have to keep the PyQwt guys in mind as potential > first users. > > > Your PEP excludes importing of Numeric and numarray in the same C-extension. > > This is true but I don't understand your solution so I'll blather on > below. > > > All you need to do is to hide the macros PyArray_Present(), PyArray_isArray() > > and import_array() into a few functions with numarray specific names, so > > that the following becomes possible: > > > > #include > > /* defines the functions (no macros) > > int Numeric_Present(); > > int Numeric_isArray(); > > void import_numeric(); > > to hide the Numeric C-API stuff in a small Numeric/meta.c file. > > */ > > #include > > /* defines the functions (no macros) > > int numarray_Present(); > > int numarray_isArray(); > > void import_numarray(); > > to hide the numarray C-API stuff in a small numarray/meta.c file. > > */ > > > > I may have come up with the wrong scheme for the Present() and > isArray(). With my scheme, they *have* to be macros because the API > functions are unsafe: when numarray or Numeric is not present, the API > function table pointers are NULL so calling through them results in > either a fatal error or a segfault. > The macros can be hidden from the module file scope by wrapping them in a function (see attached demo) > > There is an additional problem that the "same functions" need to be > called through different API pointers depending on whether numarray > or Numeric is being supported. Redefinition of typedefs and enumerations > > (or perhaps conditional compilation short-circuited re-definitions) may > also present a problem with compiling (or worse, running). > Tested and works. > > I certainly like the idea of supporting both in the same extension > module, but don't see how to get there, other than with separate > compilation units. With seperate .c files, I'm not aware of a problem > other than lots of global symbols. I haven't demoed that yet so I am > interested if someone has made it work. > Yes, you cannot mix the numarray API and Numeric API in the same .c file, but nevertheless you can hide the macros in small functions so that the macros don't pollute. Here you have a litte extension module 'pep' which tries to import Numeric and numarray. It has a method 'whatsThere' that prints the names of the Numerical/numarray extension modules it has imported. It has also a method 'whatsThis' that prints if an object is a Numeric array, a numarray array or something else: Python 2.3.3 (#2, Feb 17 2004, 11:45:40) [GCC 3.3.2 (Mandrake Linux 10.0 3.3.2-6mdk)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import pep >>> pep.whatsThere() Numeric! Numarray! >>> import Numeric >>> a = Numeric.arange(10) >>> pep.whatsThis(a) Numeric array >>> import numarray >>> a = numarray.arange(10) >>> pep.whatsThis(a) Numarray array >>> a = 10 >>> pep.whatsThis(a) Something else >>> The MANIFEST reads: pepmodule.c setup.py Numeric/meta.c Numeric/meta.h numarray/meta.c numarray/meta.h My initial idea was to add the meta.* files to the Numeric/numarray include files, but I did not try yet to solve the problem of the PyArray_API PY_ARRAY_UNIQUE_SYMBOL defines in the same way (I am sure it can be done. Regards -- Gerard -------------- next part -------------- A non-text attachment was scrubbed... Name: pep-0.0.1.tar.gz Type: application/gzip Size: 1421 bytes Desc: not available URL: From jmiller at stsci.edu Tue Jun 29 14:35:04 2004 From: jmiller at stsci.edu (Todd Miller) Date: Tue Jun 29 14:35:04 2004 Subject: [off topic] Re: [Numpy-discussion] numarray speed - PySequence_GetItem In-Reply-To: <40E1D998.8010204@noaa.gov> References: <200406250949.44897.haase@msg.ucsf.edu> <200406281414.25400.haase@msg.ucsf.edu> <1088464671.3744.300.camel@localhost.localdomain> <200406281638.03922.haase@msg.ucsf.edu> <1088520761.17789.184.camel@halloween.stsci.edu> <40E1C12E.4060102@cox.net> <40E1D998.8010204@noaa.gov> Message-ID: <1088544847.17789.393.camel@halloween.stsci.edu> On Tue, 2004-06-29 at 17:05, Chris Barker wrote: > Tim Hochberg wrote: > > Todd Miller wrote: > >> Me too. Can you (or somebody) post the application code which does the > >> drawlines? I can definitely instrument the bottleneck C-code, but I > >> don't have time to ascend the wxPython learning curve. > > > Let me second that. With a little nudge from Chris Barker, I managed to > > get wxPython to compile here and I have some changes that may speed up > > drawlines, but it'd probably be best if we were all using the same > > benchmark. > > Well, if I can't code, at least I can nudge! > > Perhaps I can also help get Todd what he's asking for, except that I'm > not sure what you mean by "application code". Do you mean a small > wxPython application that uses DC.DrawLines (and friends)? Yes. What I most want to do is a 50000 point drawlines, similar to what you profiled. Friends are fine too. > If so, yes, I > can do that. What version of wxPython should I target? > CVS head? > 2.5.1? (The latest "released" version) I'd prefer 2.5.1 unless Tim says otherwise. > there are some slight incompatibilities between versions, particularly > with DC calls, unfortunately. I'm hoping this won't affect the profile. Regards, Todd From jmiller at stsci.edu Tue Jun 29 14:43:10 2004 From: jmiller at stsci.edu (Todd Miller) Date: Tue Jun 29 14:43:10 2004 Subject: [Numpy-discussion] Numarray header PEP In-Reply-To: <20040629211800.M55753@grenoble.cnrs.fr> References: <1088451653.3744.200.camel@localhost.localdomain> <20040629194456.44a1fa7f.gerard.vermeulen@grenoble.cnrs.fr> <1088536183.17789.346.camel@halloween.stsci.edu> <20040629211800.M55753@grenoble.cnrs.fr> Message-ID: <1088545351.17789.406.camel@halloween.stsci.edu> Awesome! Thanks Gerard. This certainly sounds like the way to do it. I'll take a closer look tomorrow. Regards, Todd On Tue, 2004-06-29 at 17:32, gerard.vermeulen at grenoble.cnrs.fr wrote: > On 29 Jun 2004 15:09:43 -0400, Todd Miller wrote > > On Tue, 2004-06-29 at 13:44, Gerard Vermeulen wrote: > > > > > > > > The PEP is attached. It is formatted using the docutils package which > > > > can be used to generate HTML or PDF. Comments and corrections would be > > > > appreciated. > > > > > > > PyQwt is a Python extension which can be conditionally compiled against > > > Numeric and/or numarray (both, one of them or none). > > > > Well that's cool! I'll have to keep the PyQwt guys in mind as potential > > first users. > > > > > Your PEP excludes importing of Numeric and numarray in the same C-extension. > > > > This is true but I don't understand your solution so I'll blather on > > below. > > > > > All you need to do is to hide the macros PyArray_Present(), PyArray_isArray() > > > and import_array() into a few functions with numarray specific names, so > > > that the following becomes possible: > > > > > > #include > > > /* defines the functions (no macros) > > > int Numeric_Present(); > > > int Numeric_isArray(); > > > void import_numeric(); > > > to hide the Numeric C-API stuff in a small Numeric/meta.c file. > > > */ > > > #include > > > /* defines the functions (no macros) > > > int numarray_Present(); > > > int numarray_isArray(); > > > void import_numarray(); > > > to hide the numarray C-API stuff in a small numarray/meta.c file. > > > */ > > > > > > > I may have come up with the wrong scheme for the Present() and > > isArray(). With my scheme, they *have* to be macros because the API > > functions are unsafe: when numarray or Numeric is not present, the API > > function table pointers are NULL so calling through them results in > > either a fatal error or a segfault. > > > The macros can be hidden from the module file scope by wrapping them > in a function (see attached demo) > > > > There is an additional problem that the "same functions" need to be > > called through different API pointers depending on whether numarray > > or Numeric is being supported. Redefinition of typedefs and enumerations > > > > (or perhaps conditional compilation short-circuited re-definitions) may > > also present a problem with compiling (or worse, running). > > > Tested and works. > > > > I certainly like the idea of supporting both in the same extension > > module, but don't see how to get there, other than with separate > > compilation units. With seperate .c files, I'm not aware of a problem > > other than lots of global symbols. I haven't demoed that yet so I am > > interested if someone has made it work. > > > Yes, you cannot mix the numarray API and Numeric API in the same .c > file, but nevertheless you can hide the macros in small functions so > that the macros don't pollute. > > Here you have a litte extension module 'pep' which tries to import Numeric > and numarray. It has a method 'whatsThere' that prints the names of > the Numerical/numarray extension modules it has imported. > > It has also a method 'whatsThis' that prints if an object is a Numeric > array, a numarray array or something else: > > Python 2.3.3 (#2, Feb 17 2004, 11:45:40) > [GCC 3.3.2 (Mandrake Linux 10.0 3.3.2-6mdk)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> import pep > >>> pep.whatsThere() > Numeric! > Numarray! > >>> import Numeric > >>> a = Numeric.arange(10) > >>> pep.whatsThis(a) > Numeric array > >>> import numarray > >>> a = numarray.arange(10) > >>> pep.whatsThis(a) > Numarray array > >>> a = 10 > >>> pep.whatsThis(a) > Something else > >>> > > The MANIFEST reads: > pepmodule.c > setup.py > Numeric/meta.c > Numeric/meta.h > numarray/meta.c > numarray/meta.h > > My initial idea was to add the meta.* files to the Numeric/numarray include > files, but I did not try yet to solve the problem of the > PyArray_API PY_ARRAY_UNIQUE_SYMBOL defines in the same way (I am sure > it can be done. > > Regards -- Gerard -- From haase at msg.ucsf.edu Tue Jun 29 14:53:00 2004 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue Jun 29 14:53:00 2004 Subject: [Numpy-discussion] bug ? in Records arrays in numarray In-Reply-To: <200406281700.37694.haase@msg.ucsf.edu> References: <200406281700.37694.haase@msg.ucsf.edu> Message-ID: <200406291452.24673.haase@msg.ucsf.edu> OK, I'm still trying to get a handle on these record arrays - because I think they are pretty cool, if I could get them to work... Following the code from yesterday (see that posting below) I discovered this: main.ring4ext[0][0] is not the same as main.ring4ext[0,0] is this intended ?? >>> main.ring4ext[0][0] (2308, 76, 272, 1088481152.0, 104.18000030517578, 1994.949951171875) >>> main.ring4ext[0,0] (array([2308, 2309]), array([76, 76]), array([272, 269]), array([ 1.08848115e +09, 1.08848115e+09], type=Float32), array([ 104.18000031, 104.45999908], type=Float32), array([ 1994.94995117, 1994.95996094], type=Float32)) >>> main.ring4ext.shape # yesterday I had this different !!! (20,1) (20, 2) Any comments are appreciated, Thanks Sebastian On Monday 28 June 2004 05:00 pm, Sebastian Haase wrote: > Hi, > I have two record arrays. I was trying to assign one item from one recarray > to > > the other: > >>> omx.zext[0] = main.ring4ext[0,0] > > Traceback (most recent call last): > File "", line 1, in ? > File "X:/PrWin\numarray\records.py", line 744, in _setitem > self.field(self._names[i])[row] = value.field(self._names[i]) > File "C:\Python22\Lib\site-packages\numarray\numarraycore.py", line 619, > in __tonumtype__ > ValueError: Can't use non-rank-0 array as a scalar. > > >>> `omx.zext[0]` > > '' > > >>> `main.ring4ext[0,0]` > > '' > > >>> q = omx.zext[0] > >>> w = main.ring4ext[0,0] > >>> q > > (2097152107, -595656700, 91141, 1.0634608642000868e+037, -1.14841804241843e > +018, 1.2771574333702815e-040) > > >>> w > > (array([428]), array([75]), array([124]), array([ 1.08846451e+09], > type=Float32), array([ 99.25], type=Float32), array([ 1996.82995605], > type=Float32)) > > >>> omx.zext._formats > > ['1Int32', '1Int32', '1Int32', '1Float32', '1Float32', '1Float32'] > > >>> main.ring4ext._formats > > ['1Int32', '1Int32', '1Int32', '1Float32', '1Float32', '1Float32'] > > > I can't track down why one (q) contains scalars and the other (w) constains > arrays (is array([428]) a 'rank-0 array'? ) ! > This is how I generate them: > main.ring4ext = rec.RecArray(main._extHdrRingBuffer, "i4,i4,i4,f4,f4,f4", > shape=(ts,nc), names=("num","min","max","time","mean","darkVal"), > aligned=1) omx.zext = rec.array(formats="i4,i4,i4,f4,f4,f4", > > names=("num","min","max","time","mean","darkVal"),shape=tuple(shapeZ),align >ed=1 ) > > [[BTW: (shapeZ is a list, if I say: ...,shape=shapeZ I get > NameError, "Illegal shape %s" % `shape` from "records.py" in line 462 > -- usually type(shape) == types.ListType should be OK, right ?) > ]] > > > Any ideas? > > Thanks, > Sebastian Haase > > > > ------------------------------------------------------- > This SF.Net email sponsored by Black Hat Briefings & Training. > Attend Black Hat Briefings & Training, Las Vegas July 24-29 - > digital self defense, top technical experts, no vendor pitches, > unmatched networking opportunities. Visit www.blackhat.com > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From jmiller at stsci.edu Tue Jun 29 15:03:02 2004 From: jmiller at stsci.edu (Todd Miller) Date: Tue Jun 29 15:03:02 2004 Subject: [off topic] Re: [Numpy-discussion] numarray speed - PySequence_GetItem In-Reply-To: <40E1CEE6.3050108@cox.net> References: <200406250949.44897.haase@msg.ucsf.edu> <200406281414.25400.haase@msg.ucsf.edu> <1088464671.3744.300.camel@localhost.localdomain> <200406281638.03922.haase@msg.ucsf.edu> <1088520761.17789.184.camel@halloween.stsci.edu> <40E1CEE6.3050108@cox.net> Message-ID: <1088546550.17789.428.camel@halloween.stsci.edu> On Tue, 2004-06-29 at 16:19, Tim Hochberg wrote: > I'd bet a case of beer (or cash equivalent) that one of the main > bottlenecks is the path > PySequence_GetItem->_ndarray_item->_universalIndexing->_simpleIndexing->_simpleIndexingCore. I won't take the bet but if this works out, you get the beer. If it doesn't, well, I don't drink anymore anyway. > The path through _universalIndexing in particular, if I deciphered it > correctly, looks very slow. I don't think it needs to be that way > though, _universalIndexing could probably be sped up, but more promising > I think _ndarray_item could be made to call _simpleIndexingCore without > all that much work. It appears that this would save the creation of > several intermediate objects and it also looks like a couple of calls > back to python! I'm not familiar with this code though, so I could > easily be missing something that makes calling _simpleIndexingCore > harder than it looks. This looks very promising. I'll take a look tomorrow. Regards, Todd From Chris.Barker at noaa.gov Tue Jun 29 16:28:05 2004 From: Chris.Barker at noaa.gov (Chris Barker) Date: Tue Jun 29 16:28:05 2004 Subject: [off topic] Re: [Numpy-discussion] numarray speed - PySequence_GetItem In-Reply-To: <1088544847.17789.393.camel@halloween.stsci.edu> References: <200406250949.44897.haase@msg.ucsf.edu> <200406281414.25400.haase@msg.ucsf.edu> <1088464671.3744.300.camel@localhost.localdomain> <200406281638.03922.haase@msg.ucsf.edu> <1088520761.17789.184.camel@halloween.stsci.edu> <40E1C12E.4060102@cox.net> <40E1D998.8010204@noaa.gov> <1088544847.17789.393.camel@halloween.stsci.edu> Message-ID: <40E1FA6F.9090407@noaa.gov> Todd Miller wrote: > Yes. What I most want to do is a 50000 point drawlines, similar to what > you profiled. Friends are fine too. Actually, it was someone else that did the profiling, but here is a sample, about as simple as I could make it. It draws an N point line and M points. At the moment, it is using Numeric for the points, and numarray for the lines. Numeric is MUCH faster (which is the whole point of this discussion). Otherwise, it takes about the same amount of time to draw the lines as the points. Another note: if use the tolist() method in the numarray first, it's much faster also: dc.DrawLines(LinesPoints.tolist()) Obviously, the tolist method is much faster than wxPython's sequence methods, as would be expected. I'm going to send a note to Robin Dunn about this as well, and see what he thinks. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: DrawLinesTest.py URL: From jmiller at stsci.edu Tue Jun 29 16:51:09 2004 From: jmiller at stsci.edu (Todd Miller) Date: Tue Jun 29 16:51:09 2004 Subject: [Numpy-discussion] bug ? in Records arrays in numarray In-Reply-To: <200406291452.24673.haase@msg.ucsf.edu> References: <200406281700.37694.haase@msg.ucsf.edu> <200406291452.24673.haase@msg.ucsf.edu> Message-ID: <1088552996.3750.4.camel@localhost.localdomain> On Tue, 2004-06-29 at 17:52, Sebastian Haase wrote: > OK, > I'm still trying to get a handle on these record arrays - because I think they > are pretty cool, if I could get them to work... > Following the code from yesterday (see that posting below) I discovered this: > main.ring4ext[0][0] is not the same as main.ring4ext[0,0] > is this intended ?? > > >>> main.ring4ext[0][0] > (2308, 76, 272, 1088481152.0, 104.18000030517578, 1994.949951171875) > >>> main.ring4ext[0,0] > (array([2308, 2309]), array([76, 76]), array([272, 269]), array([ 1.08848115e > +09, 1.08848115e+09], type=Float32), array([ 104.18000031, 104.45999908], > type=Float32), array([ 1994.94995117, 1994.95996094], type=Float32)) > >>> main.ring4ext.shape # yesterday I had this different !!! (20,1) > (20, 2) > > Any comments are appreciated, I talked to JC Hsu, the numarray.records author, and he explained that we're probably looking at a limitation of numarray.records: it doesn't yet handle multi-dimensional arrays of records. JC indicated he had replied to Sebastian, but for the benefit of everyone else, that's the deal. Regards, Todd > Thanks > Sebastian > > > > > On Monday 28 June 2004 05:00 pm, Sebastian Haase wrote: > > Hi, > > I have two record arrays. I was trying to assign one item from one recarray > > to > > > > the other: > > >>> omx.zext[0] = main.ring4ext[0,0] > > > > Traceback (most recent call last): > > File "", line 1, in ? > > File "X:/PrWin\numarray\records.py", line 744, in _setitem > > self.field(self._names[i])[row] = value.field(self._names[i]) > > File "C:\Python22\Lib\site-packages\numarray\numarraycore.py", line 619, > > in __tonumtype__ > > ValueError: Can't use non-rank-0 array as a scalar. > > > > >>> `omx.zext[0]` > > > > '' > > > > >>> `main.ring4ext[0,0]` > > > > '' > > > > >>> q = omx.zext[0] > > >>> w = main.ring4ext[0,0] > > >>> q > > > > (2097152107, -595656700, 91141, 1.0634608642000868e+037, -1.14841804241843e > > +018, 1.2771574333702815e-040) > > > > >>> w > > > > (array([428]), array([75]), array([124]), array([ 1.08846451e+09], > > type=Float32), array([ 99.25], type=Float32), array([ 1996.82995605], > > type=Float32)) > > > > >>> omx.zext._formats > > > > ['1Int32', '1Int32', '1Int32', '1Float32', '1Float32', '1Float32'] > > > > >>> main.ring4ext._formats > > > > ['1Int32', '1Int32', '1Int32', '1Float32', '1Float32', '1Float32'] > > > > > > I can't track down why one (q) contains scalars and the other (w) constains > > arrays (is array([428]) a 'rank-0 array'? ) ! > > This is how I generate them: > > main.ring4ext = rec.RecArray(main._extHdrRingBuffer, "i4,i4,i4,f4,f4,f4", > > shape=(ts,nc), names=("num","min","max","time","mean","darkVal"), > > aligned=1) omx.zext = rec.array(formats="i4,i4,i4,f4,f4,f4", > > > > names=("num","min","max","time","mean","darkVal"),shape=tuple(shapeZ),align > >ed=1 ) > > > > [[BTW: (shapeZ is a list, if I say: ...,shape=shapeZ I get > > NameError, "Illegal shape %s" % `shape` from "records.py" in line 462 > > -- usually type(shape) == types.ListType should be OK, right ?) > > ]] > > > > > > Any ideas? > > > > Thanks, > > Sebastian Haase > > > > > > > > ------------------------------------------------------- > > This SF.Net email sponsored by Black Hat Briefings & Training. > > Attend Black Hat Briefings & Training, Las Vegas July 24-29 - > > digital self defense, top technical experts, no vendor pitches, > > unmatched networking opportunities. Visit www.blackhat.com > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > ------------------------------------------------------- > This SF.Net email sponsored by Black Hat Briefings & Training. > Attend Black Hat Briefings & Training, Las Vegas July 24-29 - > digital self defense, top technical experts, no vendor pitches, > unmatched networking opportunities. Visit www.blackhat.com > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- From haase at msg.ucsf.edu Tue Jun 29 17:01:07 2004 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue Jun 29 17:01:07 2004 Subject: [Numpy-discussion] bug ? in Records arrays in numarray In-Reply-To: <1088552996.3750.4.camel@localhost.localdomain> References: <200406281700.37694.haase@msg.ucsf.edu> <200406291452.24673.haase@msg.ucsf.edu> <1088552996.3750.4.camel@localhost.localdomain> Message-ID: <200406291700.20613.haase@msg.ucsf.edu> On Tuesday 29 June 2004 04:49 pm, Todd Miller wrote: > On Tue, 2004-06-29 at 17:52, Sebastian Haase wrote: > > OK, > > I'm still trying to get a handle on these record arrays - because I think > > they are pretty cool, if I could get them to work... > > Following the code from yesterday (see that posting below) I discovered > > this: main.ring4ext[0][0] is not the same as main.ring4ext[0,0] > > is this intended ?? > > > > >>> main.ring4ext[0][0] > > > > (2308, 76, 272, 1088481152.0, 104.18000030517578, 1994.949951171875) > > > > >>> main.ring4ext[0,0] > > > > (array([2308, 2309]), array([76, 76]), array([272, 269]), array([ > > 1.08848115e +09, 1.08848115e+09], type=Float32), array([ 104.18000031, > > 104.45999908], type=Float32), array([ 1994.94995117, 1994.95996094], > > type=Float32)) > > > > >>> main.ring4ext.shape # yesterday I had this different !!! (20,1) > > > > (20, 2) > > > > Any comments are appreciated, > > I talked to JC Hsu, the numarray.records author, and he explained that > we're probably looking at a limitation of numarray.records: it doesn't > yet handle multi-dimensional arrays of records. JC indicated he had > replied to Sebastian, but for the benefit of everyone else, that's the > deal. > > Regards, > Todd > Thanks, but it actually seems to work well now, as long as I use myRecArray[i][j] instead of myRecArray[i,j]. So, it looks like there is not missing much. I'll keep your "official statement" in mind when I explore this further... Thanks again, Sebastian Haase From haase at msg.ucsf.edu Tue Jun 29 17:08:01 2004 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue Jun 29 17:08:01 2004 Subject: [Numpy-discussion] bug in numarray.maximum.reduce ? Message-ID: <200406291705.55454.haase@msg.ucsf.edu> Hi, Is this a bug?: >>> # (import numarray as na ; 'd' is a 3 dimensional array) >>> d.type() Float32 >>> d[80, 136, 122] 80.3997039795 >>> na.maximum.reduce(d[:,136, 122]) 85.8426361084 >>> na.maximum.reduce(d) [136, 122] 37.3658103943 >>> >>> >>> na.maximum.reduce(d,0)[136, 122] 37.3658103943 >>> na.maximum.reduce(d,1)[136, 122] Traceback (most recent call last): File "", line 1, in ? IndexError: Index out of range I was using na.maximum.reduce(d) to get a "pixelwise" maximum along Z (axis 0). But as seen above it does not get it right. I then tried to reproduce this with some simple arrays, but here it works just fine: >>> a = na.arange(4*4*4) >>> a.shape=(4,4,4) >>> na.maximum.reduce(a) [[48 49 50 51] [52 53 54 55] [56 57 58 59] [60 61 62 63]] >>> a = na.arange(4*4*4).astype(na.Float32) >>> a.shape=(4,4,4) >>> na.maximum.reduce(a) [[ 48. 49. 50. 51.] [ 52. 53. 54. 55.] [ 56. 57. 58. 59.] [ 60. 61. 62. 63.]] >>> Any hint ? Regards, Sebastian Haase From bsder at mail.allcaps.org Tue Jun 29 17:15:42 2004 From: bsder at mail.allcaps.org (Andrew P. Lentvorski, Jr.) Date: Tue Jun 29 17:15:42 2004 Subject: [off topic] Re: [Numpy-discussion] numarray speed - PySequence_GetItem In-Reply-To: <40E1FA6F.9090407@noaa.gov> References: <200406250949.44897.haase@msg.ucsf.edu> <200406281414.25400.haase@msg.ucsf.edu> <1088464671.3744.300.camel@localhost.localdomain> <200406281638.03922.haase@msg.ucsf.edu> <1088520761.17789.184.camel@halloween.stsci.edu> <40E1C12E.4060102@cox.net> <40E1D998.8010204@noaa.gov> <1088544847.17789.393.camel@halloween.stsci.edu> <40E1FA6F.9090407@noaa.gov> Message-ID: <4C65F30B-CA2A-11D8-B6A8-000A95C874EE@mail.allcaps.org> On Jun 29, 2004, at 4:25 PM, Chris Barker wrote: > Todd Miller wrote: >> Yes. What I most want to do is a 50000 point drawlines, similar to >> what >> you profiled. Friends are fine too. > > Actually, it was someone else that did the profiling, but here is a > sample, about as simple as I could make it. > > It draws an N point line and M points. At the moment, it is using > Numeric for the points, and numarray for the lines. Numeric is MUCH > faster (which is the whole point of this discussion). Otherwise, it > takes about the same amount of time to draw the lines as the points. > > Another note: if use the tolist() method in the numarray first, it's > much faster also: > > dc.DrawLines(LinesPoints.tolist()) > If you are going to move lots of lines and points, I would recommend pushing this stuff through PyOpenGL with array objects and vertex objects. Letting OpenGL handle the transformations, clipping, movement and iteration in hardware stomps all over even the best written C code. Most UI toolkits have some form of OpenGL widget. For lots of the code I have written, even Mesa (the open-souce software OpenGL renderer) was fast enough, and not having to write all of the display transformation code by hand was a huge win even when the speed was somewhat lagging. -a From rowen at u.washington.edu Wed Jun 30 08:57:06 2004 From: rowen at u.washington.edu (Russell E Owen) Date: Wed Jun 30 08:57:06 2004 Subject: [Numpy-discussion] bug ? in Records arrays in numarray In-Reply-To: <1088552996.3750.4.camel@localhost.localdomain> References: <200406281700.37694.haase@msg.ucsf.edu> <200406291452.24673.haase@msg.ucsf.edu> <1088552996.3750.4.camel@localhost.localdomain> Message-ID: At 7:49 PM -0400 2004-06-29, Todd Miller wrote: >On Tue, 2004-06-29 at 17:52, Sebastian Haase wrote: >> OK, >> I'm still trying to get a handle on these record arrays - because >>I think they >> are pretty cool, if I could get them to work... >> Following the code from yesterday (see that posting below) I >>discovered this: >> main.ring4ext[0][0] is not the same as main.ring4ext[0,0] >> is this intended ?? >> >> >>> main.ring4ext[0][0] >> (2308, 76, 272, 1088481152.0, 104.18000030517578, 1994.949951171875) >> >>> main.ring4ext[0,0] >> (array([2308, 2309]), array([76, 76]), array([272, 269]), array([ >>1.08848115e >> +09, 1.08848115e+09], type=Float32), array([ 104.18000031, 104.45999908], >> type=Float32), array([ 1994.94995117, 1994.95996094], type=Float32)) >> >>> main.ring4ext.shape # yesterday I had this different !!! (20,1) >> (20, 2) >> >> Any comments are appreciated, > >I talked to JC Hsu, the numarray.records author, and he explained that >we're probably looking at a limitation of numarray.records: it doesn't >yet handle multi-dimensional arrays of records. JC indicated he had >replied to Sebastian, but for the benefit of everyone else, that's the >deal. I agree. I have gotten numarray.records to handle multi-dimensional arrays, but it's a terrible pain to create them, str(arry) fails and setting elements of records arrays is painful. I hope at some point they get a major redesign, as they don't actually seem to have been designed to fit in with numarray. The resulting code was so ugly that I gave up and used multiple identically shaped arrays instead. -- Russell From perry at stsci.edu Wed Jun 30 09:08:16 2004 From: perry at stsci.edu (Perry Greenfield) Date: Wed Jun 30 09:08:16 2004 Subject: [Numpy-discussion] bug ? in Records arrays in numarray In-Reply-To: References: <200406281700.37694.haase@msg.ucsf.edu> <200406291452.24673.haase@msg.ucsf.edu> <1088552996.3750.4.camel@localhost.localdomain> Message-ID: <925A9DEB-CAAF-11D8-80D7-000393989D66@stsci.edu> On Jun 30, 2004, at 11:55 AM, Russell E Owen wrote: > I agree. I have gotten numarray.records to handle multi-dimensional > arrays, but it's a terrible pain to create them, str(arry) fails and > setting elements of records arrays is painful. I hope at some point > they get a major redesign, as they don't actually seem to have been > designed to fit in with numarray. The resulting code was so ugly that > I gave up and used multiple identically shaped arrays instead. > I think we will be taking a look at that soon. I agree that they could be generalized to work better with numarray. Hopefully we will be soliciting comments in the next few weeks about the best way to do that. Perry From tim.hochberg at cox.net Wed Jun 30 12:58:17 2004 From: tim.hochberg at cox.net (Tim Hochberg) Date: Wed Jun 30 12:58:17 2004 Subject: [Numpy-discussion] Speeding up wxPython/numarray Message-ID: <40E31B31.7040105@cox.net> I spend some time seeing what I could do in the way of speeding up wxPoint_LIST_helper by tweaking the numarray code. My first suspect was _universalIndexing by way of _ndarray_item. However, due to some new-style machinations, _ndarray_item was never getting called. Instead, _ndarray_subscript was being called. So, I added a special case to _ndarray_subscript. This sped things up by 50% or so (I don't recall exactly). The code for that is at the end of this message; it's not gauranteed to be 100% correct; it's all experimental. After futzing around some more I figured out a way to trick python into using _ndarray_item. I added "type->tp_as_sequence->sq_item = _ndarray_item;" to _ndarray new. I then optimized _ndarray_item (code at end). This halved the execution time of my arbitrary benchmark. This trick may have horrible, unforseen consequences so use at your own risk. Finally I commented out the __del__ method numarraycore. This resulted in an additional speedup of 64% for a total speed up of 240%. Still not close to 10x, but a large improvement. However, this is obviously not viable for real use, but it's enough of a speedup that I'll try to see if there's anyway to move the shadow stuff back to tp_dealloc. In summary: Version Time Rel Speedup Abs Speedup Stock 0.398 ---- ---- _naarray_item mod 0.192 107% 107% del __del__ 0.117 64% 240% There were a couple of other things I tried that resulted in additional small speedups, but the tactics I used were too horrible to reproduce here. The main one of interest is that all of the calls to NA_updateDataPtr seem to burn some time. However, I don't have any idea what one could do about that. That's all for now. -tim static PyObject* _ndarray_subscript(PyArrayObject* self, PyObject* key) { PyObject *result; #ifdef TAH if (PyInt_CheckExact(key)) { long ikey = PyInt_AsLong(key); long offset; if (NA_getByteOffset(self, 1, &ikey, &offset) < 0) return NULL; if (!NA_updateDataPtr(self)) return NULL; return _simpleIndexingCore(self, offset, 1, Py_None); } #endif #if _PYTHON_CALLBACKS result = PyObject_CallMethod( (PyObject *) self, "_universalIndexing", "(OO)", key, Py_None); #else result = _universalIndexing(self, key, Py_None); #endif return result; } static PyObject * _ndarray_item(PyArrayObject *self, int i) { #ifdef TAH long offset; if (NA_getByteOffset(self, 1, &i, &offset) < 0) return NULL; if (!NA_updateDataPtr(self)) return NULL; return _simpleIndexingCore(self, offset, 1, Py_None); #else PyObject *result; PyObject *key = PyInt_FromLong(i); if (!key) return NULL; result = _universalIndexing(self, key, Py_None); Py_DECREF(key); return result; #endif } From jmiller at stsci.edu Wed Jun 30 14:48:09 2004 From: jmiller at stsci.edu (Todd Miller) Date: Wed Jun 30 14:48:09 2004 Subject: [Numpy-discussion] Speeding up wxPython/numarray In-Reply-To: <40E31B31.7040105@cox.net> References: <40E31B31.7040105@cox.net> Message-ID: <1088632048.7526.204.camel@halloween.stsci.edu> On Wed, 2004-06-30 at 15:57, Tim Hochberg wrote: > I spend some time seeing what I could do in the way of speeding up > wxPoint_LIST_helper by tweaking the numarray code. My first suspect was > _universalIndexing by way of _ndarray_item. However, due to some > new-style machinations, _ndarray_item was never getting called. Instead, > _ndarray_subscript was being called. So, I added a special case to > _ndarray_subscript. This sped things up by 50% or so (I don't recall > exactly). The code for that is at the end of this message; it's not > gauranteed to be 100% correct; it's all experimental. > > After futzing around some more I figured out a way to trick python into > using _ndarray_item. I added "type->tp_as_sequence->sq_item = > _ndarray_item;" to _ndarray new. I'm puzzled why you had to do this. You're using Python-2.3.x, right? There's conditionally compiled code which should be doing this statically. (At least I thought so.) > I then optimized _ndarray_item (code > at end). This halved the execution time of my arbitrary benchmark. This > trick may have horrible, unforseen consequences so use at your own risk. Right now the sq_item hack strikes me as somewhere between completely unnecessary and too scary for me! Maybe if python-dev blessed it. This optimization looks good to me. > Finally I commented out the __del__ method numarraycore. This resulted > in an additional speedup of 64% for a total speed up of 240%. Still not > close to 10x, but a large improvement. However, this is obviously not > viable for real use, but it's enough of a speedup that I'll try to see > if there's anyway to move the shadow stuff back to tp_dealloc. FYI, the issue with tp_dealloc may have to do with which mode Python is compiled in, --with-pydebug, or not. One approach which seems like it ought to work (just thought of this!) is to add an extra reference in C to the NumArray instance __dict__ (from NumArray.__init__ and stashed via a new attribute in the PyArrayObject struct) and then DECREF it as the last part of the tp_dealloc. > In summary: > > Version Time Rel Speedup Abs Speedup > Stock 0.398 ---- ---- > _naarray_item mod 0.192 107% 107% > del __del__ 0.117 64% 240% > > There were a couple of other things I tried that resulted in additional > small speedups, but the tactics I used were too horrible to reproduce > here. The main one of interest is that all of the calls to > NA_updateDataPtr seem to burn some time. However, I don't have any idea > what one could do about that. Francesc Alted had the same comment about NA_updateDataPtr a while ago. I tried to optimize it then but didn't get anywhere. NA_updateDataPtr() should be called at most once per extension function (more is unnecessary but not harmful) but needs to be called at least once as a consequence of the way the buffer protocol doesn't give locked pointers. > That's all for now. > > -tim Well, be picking out your beer. Todd > > > > static PyObject* > _ndarray_subscript(PyArrayObject* self, PyObject* key) > > { > PyObject *result; > #ifdef TAH > if (PyInt_CheckExact(key)) { > long ikey = PyInt_AsLong(key); > long offset; > if (NA_getByteOffset(self, 1, &ikey, &offset) < 0) > return NULL; > if (!NA_updateDataPtr(self)) > return NULL; > return _simpleIndexingCore(self, offset, 1, Py_None); > } > #endif > #if _PYTHON_CALLBACKS > result = PyObject_CallMethod( > (PyObject *) self, "_universalIndexing", "(OO)", key, Py_None); > #else > result = _universalIndexing(self, key, Py_None); > #endif > return result; > } > > > > static PyObject * > _ndarray_item(PyArrayObject *self, int i) > { > #ifdef TAH > long offset; > if (NA_getByteOffset(self, 1, &i, &offset) < 0) > return NULL; > if (!NA_updateDataPtr(self)) > return NULL; > return _simpleIndexingCore(self, offset, 1, Py_None); > #else > PyObject *result; > PyObject *key = PyInt_FromLong(i); > if (!key) return NULL; > result = _universalIndexing(self, key, Py_None); > Py_DECREF(key); > return result; > #endif > } > > > > > > ------------------------------------------------------- > This SF.Net email sponsored by Black Hat Briefings & Training. > Attend Black Hat Briefings & Training, Las Vegas July 24-29 - > digital self defense, top technical experts, no vendor pitches, > unmatched networking opportunities. Visit www.blackhat.com > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- From jmiller at stsci.edu Wed Jun 30 14:55:04 2004 From: jmiller at stsci.edu (Todd Miller) Date: Wed Jun 30 14:55:04 2004 Subject: [Numpy-discussion] Numarray header PEP In-Reply-To: <20040629211800.M55753@grenoble.cnrs.fr> References: <1088451653.3744.200.camel@localhost.localdomain> <20040629194456.44a1fa7f.gerard.vermeulen@grenoble.cnrs.fr> <1088536183.17789.346.camel@halloween.stsci.edu> <20040629211800.M55753@grenoble.cnrs.fr> Message-ID: <1088632459.7526.213.camel@halloween.stsci.edu> On Tue, 2004-06-29 at 17:32, gerard.vermeulen at grenoble.cnrs.fr wrote: > On 29 Jun 2004 15:09:43 -0400, Todd Miller wrote > > On Tue, 2004-06-29 at 13:44, Gerard Vermeulen wrote: > > > > > > > > The PEP is attached. It is formatted using the docutils package which > > > > can be used to generate HTML or PDF. Comments and corrections would be > > > > appreciated. > > > > > > > PyQwt is a Python extension which can be conditionally compiled against > > > Numeric and/or numarray (both, one of them or none). > > > > Well that's cool! I'll have to keep the PyQwt guys in mind as potential > > first users. > > > > > Your PEP excludes importing of Numeric and numarray in the same C-extension. > > > > This is true but I don't understand your solution so I'll blather on > > below. > > > > > All you need to do is to hide the macros PyArray_Present(), PyArray_isArray() > > > and import_array() into a few functions with numarray specific names, so > > > that the following becomes possible: > > > > > > #include > > > /* defines the functions (no macros) > > > int Numeric_Present(); > > > int Numeric_isArray(); > > > void import_numeric(); > > > to hide the Numeric C-API stuff in a small Numeric/meta.c file. > > > */ > > > #include > > > /* defines the functions (no macros) > > > int numarray_Present(); > > > int numarray_isArray(); > > > void import_numarray(); > > > to hide the numarray C-API stuff in a small numarray/meta.c file. > > > */ > > > > > > > I may have come up with the wrong scheme for the Present() and > > isArray(). With my scheme, they *have* to be macros because the API > > functions are unsafe: when numarray or Numeric is not present, the API > > function table pointers are NULL so calling through them results in > > either a fatal error or a segfault. > > > The macros can be hidden from the module file scope by wrapping them > in a function (see attached demo) Your demo is very clear... nice! > > There is an additional problem that the "same functions" need to be > > called through different API pointers depending on whether numarray > > or Numeric is being supported. Redefinition of typedefs and enumerations > > > > (or perhaps conditional compilation short-circuited re-definitions) may > > also present a problem with compiling (or worse, running). > > > Tested and works. > > > > I certainly like the idea of supporting both in the same extension > > module, but don't see how to get there, other than with separate > > compilation units. With seperate .c files, I'm not aware of a problem > > other than lots of global symbols. I haven't demoed that yet so I am > > interested if someone has made it work. > > > Yes, you cannot mix the numarray API and Numeric API in the same .c > file, but nevertheless you can hide the macros in small functions so > that the macros don't pollute. So... you use the "meta" code to provide package specific ordinary (not-macro-fied) functions to keep the different versions of the Present() and isArray() macros from conflicting. It would be nice to have a standard approach for using the same "extension enhancement code" for both numarray and Numeric. The PEP should really be expanded to provide an example of dual support for one complete and real function, guts and all, so people can see the process end-to-end; Something like a simple arrayprint. That process needs to be refined to remove as much tedium and duplication of effort as possible. The idea is to make it as close to providing one implementation to support both array packages as possible. I think it's important to illustrate how to partition the extension module into separate compilation units which correctly navigate the dual implementation mine field in the easiest possible way. It would also be nice to add some logic to the meta-functions so that which array package gets used is configurable. We did something like that for the matplotlib plotting software at the Python level with the "numerix" layer, an idea I think we copied from Chaco. The kind of dispatch I think might be good to support configurability looks like this: PyObject * whatsThis(PyObject *dummy, PyObject *args) { PyObject *result, *what = NULL; if (!PyArg_ParseTuple(args, "O", &what)) return 0; switch(PyArray_Which(what)) { USE_NUMERIC: result = Numeric_whatsThis(what); break; USE_NUMARRAY: result = Numarray_whatsThis(what); break; USE_SEQUENCE: result = Sequence_whatsThis(what); break; } Py_INCREF(Py_None); return Py_None; } In the above, I'm picturing a separate .c file for Numeric_whatsThis and for Numarray_whatsThis. It would be nice to streamline that to one .c and a process which somehow (simply) produces both functions. Or, ideally, the above would be done more like this: PyObject * whatsThis(PyObject *dummy, PyObject *args) { PyObject *result, *what = NULL; if (!PyArg_ParseTuple(args, "O", &what)) return 0; switch(Numerix_Which(what)) { USE_NUMERIX: result = Numerix_whatsThis(what); break; USE_SEQUENCE: result = Sequence_whatsThis(what); break; } Py_INCREF(Py_None); return Py_None; } Here, a common Numerix implementation supports both numarray and Numeric from a single simple .c. The extension module would do "#include numerix/arrayobject.h" and "import_numerix()" and otherwise just call PyArray_* functions. The current stumbling block is that numarray is not binary compatible with Numeric... so numerix in C falls apart. I haven't analyzed every symbol and struct to see if it is really feasible... but it seems like it is *almost* feasible, at least for typical usage. So, in a nutshell, I think the dual implementation support you demoed is important and we should work up an example and kick it around to make sure it's the best way we can think of doing it. Then we should add a section to the PEP describing dual support as well. Regards, Todd From tim.hochberg at cox.net Wed Jun 30 16:02:02 2004 From: tim.hochberg at cox.net (Tim Hochberg) Date: Wed Jun 30 16:02:02 2004 Subject: [Numpy-discussion] Speeding up wxPython/numarray In-Reply-To: <1088632048.7526.204.camel@halloween.stsci.edu> References: <40E31B31.7040105@cox.net> <1088632048.7526.204.camel@halloween.stsci.edu> Message-ID: <40E3462A.9080303@cox.net> Todd Miller wrote: >On Wed, 2004-06-30 at 15:57, Tim Hochberg wrote: > > >>[SNIP] >> >>After futzing around some more I figured out a way to trick python into >>using _ndarray_item. I added "type->tp_as_sequence->sq_item = >>_ndarray_item;" to _ndarray new. >> >> > >I'm puzzled why you had to do this. You're using Python-2.3.x, right? >There's conditionally compiled code which should be doing this >statically. (At least I thought so.) > > By this do you mean the "#if PY_VERSION_HEX >= 0x02030000 " that is wrapped around _ndarray_item? If so, I believe that it *is* getting compiled, it's just never getting called. What I think is happening is that the class NumArray inherits its sq_item from PyClassObject. In particular, I think it picks up instance_item from Objects/classobject.c. This appears to be fairly expensive and, I think, ends up calling tp_as_mapping->mp_subscript. Thus, _ndarray's sq_item slot never gets called. All of this is pretty iffy since I don't know this stuff very well and I didn't trace it all the way through. However, it explains what I've seen thus far. This is why I ended up using the horrible hack. I'm resetting NumArray's sq_item to point to _ndarray_item instead of instance_item. I believe that access at the python level goes through mp_subscrip, so it shouldn't be affected, and only objects at the C level should notice and they should just get the faster sq_item. You, will notice that there are an awful lot of I thinks in the above paragraphs though... >>I then optimized _ndarray_item (code >>at end). This halved the execution time of my arbitrary benchmark. This >>trick may have horrible, unforseen consequences so use at your own risk. >> >> > >Right now the sq_item hack strikes me as somewhere between completely >unnecessary and too scary for me! Maybe if python-dev blessed it. > > Yes, very scary. And it occurs to me that it will break subclasses of NumArray if they override __getitem__. When these subclasses are accessed from C they will see nd_array's sq_item instead of the overridden getitem. However, I think I also know how to fix it. But it does point out that it is very dangerous and there are probably dark corners of which I'm unaware. Asking on Python-List or PyDev would probably be a good idea. The nonscary, but painful, fix would to rewrite NumArray in C. >This optimization looks good to me. > > Unfortunately, I don't think the optimization to sq_item will affect much since NumArray appears to override it with >>Finally I commented out the __del__ method numarraycore. This resulted >>in an additional speedup of 64% for a total speed up of 240%. Still not >>close to 10x, but a large improvement. However, this is obviously not >>viable for real use, but it's enough of a speedup that I'll try to see >>if there's anyway to move the shadow stuff back to tp_dealloc. >> >> > >FYI, the issue with tp_dealloc may have to do with which mode Python is >compiled in, --with-pydebug, or not. One approach which seems like it >ought to work (just thought of this!) is to add an extra reference in C >to the NumArray instance __dict__ (from NumArray.__init__ and stashed >via a new attribute in the PyArrayObject struct) and then DECREF it as >the last part of the tp_dealloc. > > That sounds promising. [SNIP] > >Well, be picking out your beer. > > I was only about half right, so I'm not sure I qualify... -tim From gerard.vermeulen at grenoble.cnrs.fr Wed Jun 30 23:34:01 2004 From: gerard.vermeulen at grenoble.cnrs.fr (gerard.vermeulen at grenoble.cnrs.fr) Date: Wed Jun 30 23:34:01 2004 Subject: [Numpy-discussion] Numarray header PEP In-Reply-To: <1088632459.7526.213.camel@halloween.stsci.edu> References: <1088451653.3744.200.camel@localhost.localdomain> <20040629194456.44a1fa7f.gerard.vermeulen@grenoble.cnrs.fr> <1088536183.17789.346.camel@halloween.stsci.edu> <20040629211800.M55753@grenoble.cnrs.fr> <1088632459.7526.213.camel@halloween.stsci.edu> Message-ID: <20040701053355.M99698@grenoble.cnrs.fr> On 30 Jun 2004 17:54:19 -0400, Todd Miller wrote > > So... you use the "meta" code to provide package specific ordinary > (not-macro-fied) functions to keep the different versions of the > Present() and isArray() macros from conflicting. > > It would be nice to have a standard approach for using the same > "extension enhancement code" for both numarray and Numeric. The PEP > should really be expanded to provide an example of dual support for one > complete and real function, guts and all, so people can see the process > end-to-end; Something like a simple arrayprint. That process needs > to be refined to remove as much tedium and duplication of effort as > possible. The idea is to make it as close to providing one > implementation to support both array packages as possible. I think it's > important to illustrate how to partition the extension module into > separate compilation units which correctly navigate the dual > implementation mine field in the easiest possible way. > > It would also be nice to add some logic to the meta-functions so that > which array package gets used is configurable. We did something like > that for the matplotlib plotting software at the Python level with > the "numerix" layer, an idea I think we copied from Chaco. The kind > of dispatch I think might be good to support configurability looks like > this: > > PyObject * > whatsThis(PyObject *dummy, PyObject *args) > { > PyObject *result, *what = NULL; > if (!PyArg_ParseTuple(args, "O", &what)) > return 0; > switch(PyArray_Which(what)) { > USE_NUMERIC: > result = Numeric_whatsThis(what); break; > USE_NUMARRAY: > result = Numarray_whatsThis(what); break; > USE_SEQUENCE: > result = Sequence_whatsThis(what); break; > } > Py_INCREF(Py_None); > return Py_None; > } > > In the above, I'm picturing a separate .c file for Numeric_whatsThis > and for Numarray_whatsThis. It would be nice to streamline that to one > .c and a process which somehow (simply) produces both functions. > > Or, ideally, the above would be done more like this: > > PyObject * > whatsThis(PyObject *dummy, PyObject *args) > { > PyObject *result, *what = NULL; > if (!PyArg_ParseTuple(args, "O", &what)) > return 0; > switch(Numerix_Which(what)) { > USE_NUMERIX: > result = Numerix_whatsThis(what); break; > USE_SEQUENCE: > result = Sequence_whatsThis(what); break; > } > Py_INCREF(Py_None); > return Py_None; > } > > Here, a common Numerix implementation supports both numarray and Numeric > from a single simple .c. The extension module would do "#include > numerix/arrayobject.h" and "import_numerix()" and otherwise just call > PyArray_* functions. > > The current stumbling block is that numarray is not binary compatible > with Numeric... so numerix in C falls apart. I haven't analyzed > every symbol and struct to see if it is really feasible... but it > seems like it is *almost* feasible, at least for typical usage. > > So, in a nutshell, I think the dual implementation support you > demoed is important and we should work up an example and kick it > around to make sure it's the best way we can think of doing it. > Then we should add a section to the PEP describing dual support as well. > I would never apply numarray code to Numeric arrays and the inverse. It looks dangerous and I do not know if it is possible. The first thing coming to mind is that numarray and Numeric arrays refer to different type objects (this is what my pep module uses to differentiate them). So, even if numarray and Numeric are binary compatible, any 'alien' code referring the the 'Python-standard part' of the type objects may lead to surprises. A PEP proposing hacks will raise eyebrows at least. Secondly, most people use Numeric *or* numarray and not both. So, I prefer: Numeric In => Numeric Out or Numarray In => Numarray Out (NINO) Of course, Numeric or numarray output can be a user option if NINO does not apply. (explicit safe conversion between Numeric and numarray is possible if really needed). I'll try to flesh out the demo with real functions in the way you indicated (going as far as I consider safe). The problem of coding the Numeric (or numarray) functions in more than a single source file has also be addressed. It may take 2 weeks because I am off to a conference next week. Regards -- Gerard