From edcjones at comcast.net Mon Jan 2 09:54:11 2006 From: edcjones at comcast.net (Edward C. Jones) Date: Mon Jan 2 09:54:11 2006 Subject: [Numpy-discussion] Clipping, casting, and maximum values Message-ID: <43B968AE.8090406@comcast.net> #! /usr/bin/env python import numarray from numarray.numerictypes import * # Is this program too complicated? # On my machine (AMD64 chip, Debian unstable i386, numarray 1.5.0): # float(2**63-1) == float(2**63) def clipcast(farr): if farr.type() != Float64: raise TypeError('input must be a Float64 array') farr = numarray.clip(farr, 0.0, float(2**63)) print 'L13 %30.4f' % farr[0] top = (farr == float(2**63)) numarray.Error.pushMode(invalid='ignore') arr = farr.astype(Int64) numarray.Error.popMode() return numarray.where(top, 2**63-1, arr) farr = numarray.array([1.0e300], Float64) print farr.type(), farr[0] arr = clipcast(farr) print arr.type(), arr[0] From jmiller at stsci.edu Tue Jan 3 04:18:12 2006 From: jmiller at stsci.edu (Todd Miller) Date: Tue Jan 3 04:18:12 2006 Subject: [Numpy-discussion] Clipping, casting, and maximum values In-Reply-To: <43B968AE.8090406@comcast.net> References: <43B968AE.8090406@comcast.net> Message-ID: <43BA6B5F.1030604@stsci.edu> On my machine running FC4 x86_64, I get: Float64 1e+300 L13 9223372036854775808.0000 Int64 9223372036854775807 What are you getting? What did you expect to get? At 19 digits, the numbers we're talking about are outside the precision of Float64 which is ~16 digits. I did see one (for me) surprise: >>> farr = numarray.clip(farr, 0.0, float(2**63)) >>> farr array([ 9.22337204e+18]) >>> numarray.Error.pushMode(invalid='ignore') >>> arr = farr.astype(Int64) >>> numarray.Error.popMode() >>> arr array([-9223372036854775808]) Removing the Error suppression, I also got: Warning: Encountered invalid numeric result(s) during type conversion That makes it (for me) much less surprising: since you can't stuff 2**63 into Int64, there is no right answer for the astype(). Everything else looked OK to me on Fedora Core 4. Regards, Todd Edward C. Jones wrote: > #! /usr/bin/env python > > import numarray > from numarray.numerictypes import * > > # Is this program too complicated? > # On my machine (AMD64 chip, Debian unstable i386, numarray 1.5.0): > # float(2**63-1) == float(2**63) > def clipcast(farr): > if farr.type() != Float64: > raise TypeError('input must be a Float64 array') > farr = numarray.clip(farr, 0.0, float(2**63)) > print 'L13 %30.4f' % farr[0] > top = (farr == float(2**63)) > numarray.Error.pushMode(invalid='ignore') > arr = farr.astype(Int64) > numarray.Error.popMode() > return numarray.where(top, 2**63-1, arr) > > farr = numarray.array([1.0e300], Float64) > print farr.type(), farr[0] > arr = clipcast(farr) > print arr.type(), arr[0] > > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log > files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From edcjones at comcast.net Tue Jan 3 14:40:06 2006 From: edcjones at comcast.net (Edward C. Jones) Date: Tue Jan 3 14:40:06 2006 Subject: [Numpy-discussion] numarray: Python crash Message-ID: <43BAFD05.6050306@comcast.net> #! /usr/bin/env python import numarray """I have an AMD64 PC with Debian unstable for i386 on it. I use Python 2.4.2 and numarray 1.5.0. I compile these myself. """ arr = numarray.array([-2000000000L], 'UInt32') arr = numarray.array([-2000000000L], 'UInt32') """If I execute one of the two statements I get: Exception exceptions.OverflowError: "can't convert negative value to unsigned long" in 'garbage collection' ignored Fatal Python error: unexpected exception during garbage collection Aborted If I execute both statements, I get: Traceback (most recent call last): File "./bug2.py", line 10, in ? arr = numarray.array([-2000000000L], 'UInt32') File "/usr/local/lib/python2.4/site-packages/numarray/numarraycore.py", line 354, in array type=_typeFromTypeAndTypecode(type,typecode,dtype) File "/usr/local/lib/python2.4/site-packages/numarray/numarraycore.py", line 278, in _typeFromTypeAndTypecode for a in [type, typecode, dtype]: OverflowError: can't convert negative value to unsigned long """ From jmiller at stsci.edu Tue Jan 3 15:36:18 2006 From: jmiller at stsci.edu (Todd Miller) Date: Tue Jan 3 15:36:18 2006 Subject: [Numpy-discussion] numarray: Python crash In-Reply-To: <43BAFD05.6050306@comcast.net> References: <43BAFD05.6050306@comcast.net> Message-ID: <43BB0A15.2090701@stsci.edu> Hi Edward, This is working OK for me on Fedora Core 4 x86_64 using numarray-1.5.0 and Python-2.4.2. I masked the overflow exception with try/except and did 10000 in a loop without problems. Are you getting a core dump file? If so, try: % gdb python core* # on FC4 cores are annotated with PID so you may need to rm a few (gdb) where .... C function traceback Let me know what you see. Todd Edward C. Jones wrote: > #! /usr/bin/env python > > import numarray > > """I have an AMD64 PC with Debian unstable for i386 on it. I use Python > 2.4.2 and numarray 1.5.0. I compile these myself. > """ > > arr = numarray.array([-2000000000L], 'UInt32') > arr = numarray.array([-2000000000L], 'UInt32') > > """If I execute one of the two statements I get: > > Exception exceptions.OverflowError: "can't convert negative value to > unsigned long" in 'garbage collection' ignored > Fatal Python error: unexpected exception during garbage collection > Aborted > > If I execute both statements, I get: > > Traceback (most recent call last): > File "./bug2.py", line 10, in ? > arr = numarray.array([-2000000000L], 'UInt32') > File > "/usr/local/lib/python2.4/site-packages/numarray/numarraycore.py", > line 354, in array > type=_typeFromTypeAndTypecode(type,typecode,dtype) > File > "/usr/local/lib/python2.4/site-packages/numarray/numarraycore.py", > line 278, in _typeFromTypeAndTypecode > for a in [type, typecode, dtype]: > OverflowError: can't convert negative value to unsigned long > """ > > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log > files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From faltet at carabos.com Tue Jan 3 19:20:15 2006 From: faltet at carabos.com (Francesc Altet) Date: Tue Jan 3 19:20:15 2006 Subject: [Numpy-discussion] Slow performance in array protocol with string arrays Message-ID: <200601040037.12663.faltet@carabos.com> Hi, Perhaps this is not very important because it only has effects at high dimensionality, but I think it would be good to send it here for the records. It seems that numarray implementation for the array protocol in string arrays is very slow for dimensionality > 10: In [258]: a=scicore.reshape(scicore.array((1,)), (1,)*15) In [259]: a Out[259]: array([[[[[[[[[[[[[[[1]]]]]]]]]]]]]]]) In [260]: t1=time(); c=numarray.array(a);print time()-t1 0.000355958938599 # numerical conversion is pretty fast: 0.3 ms In [261]: b=scicore.array(a, dtype="S1") In [262]: b Out[262]: array([[[[[[[[[[[[[[[1]]]]]]]]]]]]]]], dtype=(string,1)) In [263]: t1=time(); c=numarray.strings.array(b);print time()-t1 0.61981511116 # string conversion is more than 1000x slower In [264]: t1=time(); d=scicore.array(c);print time()-t1 0.000162839889526 # scipy_core speed seems normal In [266]: t1=time(); d=numarray.strings.array(c);print time()-t1 1.38820910454 # converting numarray strings into themselves is # the slowest! Using numarray 1.5.0 and scipy_core 0.9.2.1763. Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From jmiller at stsci.edu Wed Jan 4 04:23:18 2006 From: jmiller at stsci.edu (Todd Miller) Date: Wed Jan 4 04:23:18 2006 Subject: [Numpy-discussion] Slow performance in array protocol with string arrays In-Reply-To: <200601040037.12663.faltet@carabos.com> References: <200601040037.12663.faltet@carabos.com> Message-ID: <43BBBDE2.2040904@stsci.edu> Francesc Altet wrote: >Hi, > >Perhaps this is not very important because it only has effects at high >dimensionality, but I think it would be good to send it here for the >records. > >It seems that numarray implementation for the array protocol in string >arrays is very slow for dimensionality > 10: > >In [258]: a=scicore.reshape(scicore.array((1,)), (1,)*15) > >In [259]: a >Out[259]: array([[[[[[[[[[[[[[[1]]]]]]]]]]]]]]]) > >In [260]: t1=time(); c=numarray.array(a);print time()-t1 >0.000355958938599 # numerical conversion is pretty fast: 0.3 ms > >In [261]: b=scicore.array(a, dtype="S1") > >In [262]: b >Out[262]: array([[[[[[[[[[[[[[[1]]]]]]]]]]]]]]], dtype=(string,1)) > >In [263]: t1=time(); c=numarray.strings.array(b);print time()-t1 >0.61981511116 # string conversion is more than 1000x slower > >In [264]: t1=time(); d=scicore.array(c);print time()-t1 >0.000162839889526 # scipy_core speed seems normal > >In [266]: t1=time(); d=numarray.strings.array(c);print time()-t1 >1.38820910454 # converting numarray strings into themselves is > # the slowest! > >Using numarray 1.5.0 and scipy_core 0.9.2.1763. > >Cheers > > I logged this on Source Forge with the growing collection of numarray.strings issues. For now, strings.array() isn't taking advantage of the new array protocol and is implemented largely in Python. Todd From uiwjsngojfe at us.loreal.com Wed Jan 4 04:53:22 2006 From: uiwjsngojfe at us.loreal.com (Annie Robison) Date: Wed Jan 4 04:53:22 2006 Subject: [Numpy-discussion] This one will go crazy on Wend Message-ID: <0001$01a7d04a$0246b512@VLADISLAVSDD> St ock Alert iPackets International, Inc. Global Developer and Provider of a Wide Range of Wireless and Communications Solutions for Selected Enterprises Including Mine-Safety (Source: News 1/3/06) OTC: IPKL Price: .35 Huge PR For Wednesday is Underway on IPKL. Short/Day Trading Opportunity for You? Sometimes it is Bang-Zoom on These Small st ocks..As Many of You may Know Recent News: Go Read the Full Stories Now! 1)iPackets International Receives US$85,000 Down Payment for Its First iPMine Deployment in China 2)iPackets International Attends Several Mining Trade Shows and Receives Tremendous Response for Its iPMine Mine-Safety Product Watch This One Trade on Wednesday! Radar it Right Now.. _______________ Information within this email contains 4rward l00 king sta tements within meaning of Section 27A of the Sec urities Act of nineteen thirty three and Section 21B of the Se curities Exchange Act of nineteen thirty four. Any statements that express or involve discussions with respect to predictions, expectations, beliefs, plans, projections, objectives, goals, assumptions future events or performance are not statements of historical fact and may be 4rward 1o0king statements. 4rward looking statements are based on ex pectations, es timates and pr ojections at the time the statements are that involve a number of risks and uncertainties which could cause actual results or events to differ materially from those presently featured Com pany is not a reporting company under the SEC Act of 1934 and there is limited information available on the company. The Co-mpany has a nominal ca sh position.It is an operating company. The company is going to need financing. If that financing does not occur, the company may not be able to continue as a going concern in which case you could lose your in-vestment. The pu blisher of this new sletter does not represent that the information contained in this mes sage states all material facts or does not omit a material fact necessary to make the statements therein not misleading. All information provided within this e_ mail pertaining to in-vesting, st 0cks, se curities must be understood as information provided and not inv estment advice. Remember a thorough due diligence effort, including a review of a company's filings when available, should be completed prior to in_vesting. The pub lisher of this news letter advises all readers and subscribers to seek advice from a registered professional securities representative before deciding to trade in st0cks featured this e mail. None of the material within this report shall be construed as any kind of in_vestment advice or solicitation. Many of these com panies on the verge of bankruptcy. You can lose all your mony by inv esting in st 0ck. The publisher of this new sletter is not a registered in-vestment advis0r. Subscribers should not view information herein as legal, tax, accounting or inve stment advice. In com pliance with the Securities Act of nineteen thirty three, Section 17(b),The publisher of this newsletter is contracted to receive fifteen th0 usand d0 l1ars from a third party, not an off icer, dir ector or af filiate shar eh0lder for the ci rculation of this report. Be aware of an inherent conflict of interest resulting from such compensation due to the fact that this is a paid advertisement and is not without bias. All factual information in this report was gathered from public sources, including but not limited to Co mpany Press Releases. Use of the information in this e mail constitutes your acceptance of these terms. From Chris.Barker at noaa.gov Wed Jan 4 09:29:23 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed Jan 4 09:29:23 2006 Subject: [Numpy-discussion] Slow performance in array protocol with string arrays In-Reply-To: <43BBBDE2.2040904@stsci.edu> References: <200601040037.12663.faltet@carabos.com> <43BBBDE2.2040904@stsci.edu> Message-ID: <43BC0595.9080509@noaa.gov> > Francesc Altet wrote: >> It seems that numarray implementation for the array protocol in string >> arrays is very slow for dimensionality > 10: OK, I'll bite -- what in the world do you need an array of strings with dimensionality > 10 for ? Which brings up another curiosity: I'm all in favor of not having arbitrary limits on anything, but I'm curious what the largest rank NumPy array anyone has ever had a real use for is? I don't think I've ever used rank > 3, or maybe 4. Anyone have a use case for a very large rank array? -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From strang at nmr.mgh.harvard.edu Wed Jan 4 10:30:11 2006 From: strang at nmr.mgh.harvard.edu (Gary Strangman) Date: Wed Jan 4 10:30:11 2006 Subject: [Numpy-discussion] Slow performance in array protocol with string arrays In-Reply-To: <43BC0595.9080509@noaa.gov> References: <200601040037.12663.faltet@carabos.com> <43BBBDE2.2040904@stsci.edu> <43BC0595.9080509@noaa.gov> Message-ID: > Which brings up another curiosity: I'm all in favor of not having arbitrary > limits on anything, but I'm curious what the largest rank NumPy array anyone > has ever had a real use for is? I don't think I've ever used rank > 3, or > maybe 4. > > Anyone have a use case for a very large rank array? Depends on your definition of "very". In neuroimaging at least, rank 4 is a standard dataset "unit" (3D+time). If you then include subjects, replications (same day), and sessions (i.e., testing on different days), that's rank=7. Can't say as I've ever reached 10 though. ;-) -best Gary -------------------------------------------------------------- Gary Strangman, PhD | Director, Neural Systems Group Office: 617-724-0662 | Massachusetts General Hospital Fax: 617-726-4078 | 149 13th Street, Ste 10018 strang at nmr.mgh.harvard.edu | Charlestown, MA 02129 http://www.nmr.mgh.harvard.edu/NSG/ From jmiller at stsci.edu Wed Jan 4 10:42:14 2006 From: jmiller at stsci.edu (Todd Miller) Date: Wed Jan 4 10:42:14 2006 Subject: [Numpy-discussion] Slow performance in array protocol with string arrays In-Reply-To: References: <200601040037.12663.faltet@carabos.com> <43BBBDE2.2040904@stsci.edu> <43BC0595.9080509@noaa.gov> Message-ID: <43BC1631.6010000@stsci.edu> I fixed a performance bug in numarray.strings so the lion's share of this problem is now gone in CVS: scipy -> numarray Int32 0.000634908676147 scipy -> numarray S1 0.000502109527588 numarray -> scipy S1 0.000125885009766 numarray -> numarray S1 0.00110602378845 Things could be further improved by adding "import" support for the newcore array protocol to numarray.strings. Todd Gary Strangman wrote: > >> Which brings up another curiosity: I'm all in favor of not having >> arbitrary limits on anything, but I'm curious what the largest rank >> NumPy array anyone has ever had a real use for is? I don't think I've >> ever used rank > 3, or maybe 4. >> >> Anyone have a use case for a very large rank array? > > > Depends on your definition of "very". In neuroimaging at least, rank 4 > is a standard dataset "unit" (3D+time). If you then include subjects, > replications (same day), and sessions (i.e., testing on different > days), that's rank=7. Can't say as I've ever reached 10 though. ;-) > > -best > Gary > > -------------------------------------------------------------- > Gary Strangman, PhD | Director, Neural Systems Group > Office: 617-724-0662 | Massachusetts General Hospital > Fax: 617-726-4078 | 149 13th Street, Ste 10018 > strang at nmr.mgh.harvard.edu | Charlestown, MA 02129 > http://www.nmr.mgh.harvard.edu/NSG/ > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log > files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From faltet at carabos.com Wed Jan 4 13:36:16 2006 From: faltet at carabos.com (Francesc Altet) Date: Wed Jan 4 13:36:16 2006 Subject: [Numpy-discussion] Slow performance in array protocol with string arrays In-Reply-To: <43BC1631.6010000@stsci.edu> References: <200601040037.12663.faltet@carabos.com> <43BC1631.6010000@stsci.edu> Message-ID: <200601042235.37826.faltet@carabos.com> Good! Thanks Todd A Dimecres 04 Gener 2006 19:38, Todd Miller va escriure: > I fixed a performance bug in numarray.strings so the lion's share of > this problem is now gone in CVS: > > scipy -> numarray Int32 0.000634908676147 > scipy -> numarray S1 0.000502109527588 > numarray -> scipy S1 0.000125885009766 > numarray -> numarray S1 0.00110602378845 > > Things could be further improved by adding "import" support for the > newcore array protocol to numarray.strings. > > Todd -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From faltet at carabos.com Wed Jan 4 13:47:08 2006 From: faltet at carabos.com (Francesc Altet) Date: Wed Jan 4 13:47:08 2006 Subject: [Numpy-discussion] Slow performance in array protocol with string arrays In-Reply-To: <43BC0595.9080509@noaa.gov> References: <200601040037.12663.faltet@carabos.com> <43BBBDE2.2040904@stsci.edu> <43BC0595.9080509@noaa.gov> Message-ID: <200601042246.07038.faltet@carabos.com> A Dimecres 04 Gener 2006 18:27, Christopher Barker va escriure: > > Francesc Altet wrote: > >> It seems that numarray implementation for the array protocol in string > >> arrays is very slow for dimensionality > 10: > > OK, I'll bite -- what in the world do you need an array of strings with > dimensionality > 10 for ? Good question. The fact is that Numeric, numarray and scipy_core has all been designed to support objects up to 32 (and perhaps 40 in some cases) dimensions. Why? I must confess that I don't exactly know, but when your mission is to check every bit of the implementation and push your package to its limits, you may encounter very funny things that probably will never be a problem for real users. Somehow, this kind of issues with high dimensionalities are sometimes useful for isolating other potential problems. So, yeah, one can say that they are usally a loss of time, but sometimes they can save your life (well, kind of ;-). -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From focke at slac.stanford.edu Wed Jan 4 14:33:20 2006 From: focke at slac.stanford.edu (Warren Focke) Date: Wed Jan 4 14:33:20 2006 Subject: [Numpy-discussion] Slow performance in array protocol with string arrays In-Reply-To: <200601042246.07038.faltet@carabos.com> References: <200601040037.12663.faltet@carabos.com> <43BBBDE2.2040904@stsci.edu> <43BC0595.9080509@noaa.gov> <200601042246.07038.faltet@carabos.com> Message-ID: I ran up against the dimesion limit in Numeric back when it was 10. Or 20, it was actually defined inconsistently in different places, and you could crash the interpreter by creating and array w/ >10 dim in a part of the code that would let you do that and feeding it to a part that wouldn't. I didn't complain about the limit because the code I was working on was a toy^H^H^Hlearning exercise and I didn't complain about the inconsistency because I suck. I was trying to make an FFT routine that would reshape the input sequence to have one dimension per prime factor of its length, and then manipulate that. Warren Focke On Wed, 4 Jan 2006, Francesc Altet wrote: > A Dimecres 04 Gener 2006 18:27, Christopher Barker va escriure: > > > Francesc Altet wrote: > > >> It seems that numarray implementation for the array protocol in string > > >> arrays is very slow for dimensionality > 10: > > > > OK, I'll bite -- what in the world do you need an array of strings with > > dimensionality > 10 for ? > > Good question. The fact is that Numeric, numarray and scipy_core has > all been designed to support objects up to 32 (and perhaps 40 in some > cases) dimensions. Why? I must confess that I don't exactly know, but > when your mission is to check every bit of the implementation and push > your package to its limits, you may encounter very funny things that > probably will never be a problem for real users. > > Somehow, this kind of issues with high dimensionalities are sometimes > useful for isolating other potential problems. So, yeah, one can say > that they are usally a loss of time, but sometimes they can save your > life (well, kind of ;-). > > -- > >0,0< Francesc Altet ? ? http://www.carabos.com/ > V V C?rabos Coop. V. ??Enjoy Data > "-" > > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://ads.osdn.com/?ad_idv37&alloc_id865&op?k > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > From haase at msg.ucsf.edu Wed Jan 4 17:22:21 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Wed Jan 4 17:22:21 2006 Subject: [Numpy-discussion] dtype attribute missing in numarray Message-ID: <200601041721.15149.haase@msg.ucsf.edu> Hi, In an effort to follow the scipy_core development I just got the latest CVS version of numarray. I was mainly interested in changing my code (and the tutorial that I write) slowly but surely to use 'dtype' instead of 'type'. So I get this: >>> na.__version__ '1.5.1' >>> a = na.arange(10,dtype=na.float32) # great ! >>> a.dtypechar 'f' >>> a.type() Float32 >>> a.dtype Traceback (most recent call last): File "", line 1, in ? AttributeError: 'NumArray' object has no attribute 'dtype' Is there a reason for not having the 'dtype' attribute !? I *really* never liked the need for parenthesis when using 'a.type()' ... Thanks, Sebastian Haase From jmiller at stsci.edu Thu Jan 5 03:13:14 2006 From: jmiller at stsci.edu (Todd Miller) Date: Thu Jan 5 03:13:14 2006 Subject: [Numpy-discussion] dtype attribute missing in numarray In-Reply-To: <200601041721.15149.haase@msg.ucsf.edu> References: <200601041721.15149.haase@msg.ucsf.edu> Message-ID: <43BCFE89.8040209@stsci.edu> Sebastian Haase wrote: >Hi, >In an effort to follow the scipy_core development I just got the latest CVS >version of numarray. >I was mainly interested in changing my code (and the tutorial that I write) >slowly but surely to use 'dtype' instead of 'type'. >So I get this: > > >>>>na.__version__ >>>> >>>> >'1.5.1' > > >>>>a = na.arange(10,dtype=na.float32) # great ! >>>>a.dtypechar >>>> >>>> >'f' > > >>>>a.type() >>>> >>>> >Float32 > > >>>>a.dtype >>>> >>>> >Traceback (most recent call last): > File "", line 1, in ? >AttributeError: 'NumArray' object has no attribute 'dtype' > >Is there a reason for not having the 'dtype' attribute !? I *really* never >liked the need for parenthesis when using 'a.type()' ... > >Thanks, >Sebastian Haase > > I added .dtype returning the numarray numerical type object corresponding to the array. Todd From jh at oobleck.astro.cornell.edu Thu Jan 5 08:27:14 2006 From: jh at oobleck.astro.cornell.edu (Joe Harrington) Date: Thu Jan 5 08:27:14 2006 Subject: [Numpy-discussion] Re: [SciPy-dev] Long live to numpy (and its list as well) In-Reply-To: (scipy-dev-request@scipy.net) References: Message-ID: <200601051625.k05GPixf021407@oobleck.astro.cornell.edu> Francesc Altet on scipy-dev: > Now that numpy is dead and we all want a long life to numpy, what > about moving discussions in this list into the venerable: > "numpy-discussion " We're one community (or want to be). Could we just have one mailing list per type of discussion (dev and user)? All the cross-posting and cross-monitoring is tedious. I don't care whether we keep scipy-* or numpy-* lists, but is there a functional reason to have four lists? Consider that soon we may have *-doc, *-newbie, *-announce, and others as well, if this takes off like we all hope. If the developers want separate lists because some are only working on one of the two packages, I can see that argument (in the future if not now). I don't see a need for two user lists, unless perhaps sorted by high and low traffic. I propose we either drop the numpy-* lists (if subscribers there agree), or leave them for ongoing discussion of the legacy packages, and discourage discussion of the new numpy/scipy there. Ok, flame me. --jh-- From haase at msg.ucsf.edu Thu Jan 5 09:29:13 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Thu Jan 5 09:29:13 2006 Subject: [Numpy-discussion] dtype attribute missing in numarray In-Reply-To: <43BCFE89.8040209@stsci.edu> References: <200601041721.15149.haase@msg.ucsf.edu> <43BCFE89.8040209@stsci.edu> Message-ID: <200601050928.33471.haase@msg.ucsf.edu> Thanks Todd, I like this a lot. Maybe it could be mentioned in the documention - I don't have any good suggestions on where, I just noticed that 'dtype' is not in the index. I assume the following assignment to 'dtype' is not that important !? >>> na.__version__ '1.5.1' >>> a = na.arange(10,dtype=na.float32) >>> a.dtype Float32 >>> a.dtype = na.int32 Traceback (most recent call last): File "", line 1, in ? AttributeError: can't set attribute Also: when / why was it decided that dtypes (float32, int32, ...) should be lowercase !? Aren't all python types usually uppercase ... Thanks for all your work. Sebastian Haase On Thursday 05 January 2006 03:10, Todd Miller wrote: > Sebastian Haase wrote: > >Hi, > >In an effort to follow the scipy_core development I just got the latest > > CVS version of numarray. > >I was mainly interested in changing my code (and the tutorial that I > > write) slowly but surely to use 'dtype' instead of 'type'. > > > >So I get this: > >>>>na.__version__ > > > >'1.5.1' > > > >>>>a = na.arange(10,dtype=na.float32) # great ! > >>>>a.dtypechar > > > >'f' > > > >>>>a.type() > > > >Float32 > > > >>>>a.dtype > > > >Traceback (most recent call last): > > File "", line 1, in ? > >AttributeError: 'NumArray' object has no attribute 'dtype' > > > >Is there a reason for not having the 'dtype' attribute !? I *really* never > >liked the need for parenthesis when using 'a.type()' ... > > > >Thanks, > >Sebastian Haase > > I added .dtype returning the numarray numerical type object > corresponding to the array. > > Todd From Fernando.Perez at colorado.edu Thu Jan 5 09:58:22 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Thu Jan 5 09:58:22 2006 Subject: [Numpy-discussion] Re: [SciPy-user] [SciPy-dev] Long live to numpy (and its list as well) In-Reply-To: <200601051625.k05GPixf021407@oobleck.astro.cornell.edu> References: <200601051625.k05GPixf021407@oobleck.astro.cornell.edu> Message-ID: <43BD5DFA.20906@colorado.edu> Joe Harrington wrote: > Francesc Altet on scipy-dev: > > >>Now that numpy is dead and we all want a long life to numpy, what >>about moving discussions in this list into the venerable: > > >>"numpy-discussion " > > > We're one community (or want to be). Could we just have one mailing > list per type of discussion (dev and user)? All the cross-posting and > cross-monitoring is tedious. I don't care whether we keep scipy-* or > numpy-* lists, but is there a functional reason to have four lists? > Consider that soon we may have *-doc, *-newbie, *-announce, and others > as well, if this takes off like we all hope. If the developers want > separate lists because some are only working on one of the two > packages, I can see that argument (in the future if not now). I don't > see a need for two user lists, unless perhaps sorted by high and low > traffic. > > I propose we either drop the numpy-* lists (if subscribers there > agree), or leave them for ongoing discussion of the legacy packages, > and discourage discussion of the new numpy/scipy there. > > Ok, flame me. Uh, no. I'm actually with you on this one: I just don't think we are a large enough group to warrant the existence of separate numpy- and scipy- lists, especially when the overlap in topics is so high. Every scipy user is, by necessity, a numpy user as well. I think that, IF in the future: 1. the traffic on the scipy- lists becomes enormous, AND 2. a significant portion of that traffic is for users of numpy as a pure array library with no scientific concerns (if it really becomes a popular 'data exchange' system for Python-and-C libraries), THEN we can consider resuscitating the numpy lists. For now, I vote on leaving them dormant, and moving all numeric(abandoned), numarray(maintenance-transition) and numpy/scipy (new development) discussion to the scipy-* lists. I don't think the occasional post about Numeric or numarray is a major concern (at least it doesn't bother me). It is an issue also of friendliness to newbies: I'd like to tell newcomers "for information and discussion, just join scipy-user and matplotlib-user, and you should be set on all numerics and plotting in python". Telling them to subscribe to, or monitor via gmane, 8 different lists is annoying. Cheers, f From cjw at sympatico.ca Thu Jan 5 10:11:12 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Thu Jan 5 10:11:12 2006 Subject: [Numpy-discussion] Re: [SciPy-user] [SciPy-dev] Long live to numpy (and its list as well) In-Reply-To: <43BD5DFA.20906@colorado.edu> References: <200601051625.k05GPixf021407@oobleck.astro.cornell.edu> <43BD5DFA.20906@colorado.edu> Message-ID: <43BD6109.9030402@sympatico.ca> Fernando Perez wrote: > Joe Harrington wrote: > >> Francesc Altet on scipy-dev: >> >> >>> Now that numpy is dead and we all want a long life to numpy, what >>> about moving discussions in this list into the venerable: >> >> >> >>> "numpy-discussion " >> >> >> >> We're one community (or want to be). Could we just have one mailing >> list per type of discussion (dev and user)? All the cross-posting and >> cross-monitoring is tedious. I don't care whether we keep scipy-* or >> numpy-* lists, but is there a functional reason to have four lists? >> Consider that soon we may have *-doc, *-newbie, *-announce, and others >> as well, if this takes off like we all hope. If the developers want >> separate lists because some are only working on one of the two >> packages, I can see that argument (in the future if not now). I don't >> see a need for two user lists, unless perhaps sorted by high and low >> traffic. >> >> I propose we either drop the numpy-* lists (if subscribers there >> agree), or leave them for ongoing discussion of the legacy packages, >> and discourage discussion of the new numpy/scipy there. >> >> Ok, flame me. > > > Uh, no. I'm actually with you on this one: I just don't think we are > a large enough group to warrant the existence of separate numpy- and > scipy- lists, especially when the overlap in topics is so high. Every > scipy user is, by necessity, a numpy user as well. > > I think that, IF in the future: > > 1. the traffic on the scipy- lists becomes enormous, AND > > 2. a significant portion of that traffic is for users of numpy as a > pure array library with no scientific concerns (if it really becomes a > popular 'data exchange' system for Python-and-C libraries), > > THEN we can consider resuscitating the numpy lists. > > For now, I vote on leaving them dormant, and moving all > numeric(abandoned), numarray(maintenance-transition) and numpy/scipy > (new development) discussion to the scipy-* lists. I don't think the > occasional post about Numeric or numarray is a major concern (at least > it doesn't bother me). > > It is an issue also of friendliness to newbies: I'd like to tell > newcomers "for information and discussion, just join scipy-user and > matplotlib-user, and you should be set on all numerics and plotting in > python". Telling them to subscribe to, or monitor via gmane, 8 > different lists is annoying. > > Cheers, > > f > It seems to me that NumPy is the better way to go, the archives and downloads are more readily available there, but it's up to Todd Miller and the folk who have been maintaining NumPy. NumPy has been around for a while, probably longer than SciPy. Colin W. From cjw at sympatico.ca Thu Jan 5 10:12:09 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Thu Jan 5 10:12:09 2006 Subject: [Numpy-discussion] dtype attribute missing in numarray In-Reply-To: <200601050928.33471.haase@msg.ucsf.edu> References: <200601041721.15149.haase@msg.ucsf.edu> <43BCFE89.8040209@stsci.edu> <200601050928.33471.haase@msg.ucsf.edu> Message-ID: <43BD6171.1020609@sympatico.ca> Sebastian Haase wrote: >Thanks Todd, >I like this a lot. Maybe it could be mentioned in the documention - I don't >have any good suggestions on where, I just noticed that 'dtype' is not in the >index. >I assume the following assignment to 'dtype' is not that important !? > > >>>>na.__version__ >>>> >>>> >'1.5.1' > > >>>>a = na.arange(10,dtype=na.float32) >>>>a.dtype >>>> >>>> >Float32 > > >>>>a.dtype = na.int32 >>>> >>>> >Traceback (most recent call last): > File "", line 1, in ? >AttributeError: can't set attribute > >Also: when / why was it decided that dtypes (float32, int32, ...) should be >lowercase !? Aren't all python types usually uppercase ... > >Thanks for all your work. >Sebastian Haase > > > > >On Thursday 05 January 2006 03:10, Todd Miller wrote: > > >>Sebastian Haase wrote: >> >> >>>Hi, >>>In an effort to follow the scipy_core development I just got the latest >>>CVS version of numarray. >>>I was mainly interested in changing my code (and the tutorial that I >>>write) slowly but surely to use 'dtype' instead of 'type'. >>> >>>So I get this: >>> >>> >>>>>>na.__version__ >>>>>> >>>>>> >>>'1.5.1' >>> >>> >>> >>>>>>a = na.arange(10,dtype=na.float32) # great ! >>>>>>a.dtypechar >>>>>> >>>>>> >>>'f' >>> >>> >>> >>>>>>a.type() >>>>>> >>>>>> >>>Float32 >>> >>> >>> >>>>>>a.dtype >>>>>> >>>>>> >>>Traceback (most recent call last): >>> File "", line 1, in ? >>>AttributeError: 'NumArray' object has no attribute 'dtype' >>> >>>Is there a reason for not having the 'dtype' attribute !? I *really* never >>>liked the need for parenthesis when using 'a.type()' ... >>> >>>Thanks, >>>Sebastian Haase >>> >>> >>I added .dtype returning the numarray numerical type object >>corresponding to the array. >> >>Todd >> >> > > >------------------------------------------------------- >This SF.net email is sponsored by: Splunk Inc. Do you grep through log files >for problems? Stop! Download the new AJAX search engine that makes >searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! >http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click >_______________________________________________ >Numpy-discussion mailing list >Numpy-discussion at lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > dtype is an improvement. The use of type created confusion. Colin W. From jmiller at stsci.edu Thu Jan 5 10:26:20 2006 From: jmiller at stsci.edu (Todd Miller) Date: Thu Jan 5 10:26:20 2006 Subject: [Numpy-discussion] dtype attribute missing in numarray In-Reply-To: <200601050928.33471.haase@msg.ucsf.edu> References: <200601041721.15149.haase@msg.ucsf.edu> <43BCFE89.8040209@stsci.edu> <200601050928.33471.haase@msg.ucsf.edu> Message-ID: <43BD6482.6090707@stsci.edu> Sebastian Haase wrote: >Thanks Todd, >I like this a lot. Maybe it could be mentioned in the documention - I don't >have any good suggestions on where, I just noticed that 'dtype' is not in the >index. >I assume the following assignment to 'dtype' is not that important !? > > >>>>na.__version__ >>>> >>>> >'1.5.1' > > >>>>a = na.arange(10,dtype=na.float32) >>>>a.dtype >>>> >>>> >Float32 > > >>>>a.dtype = na.int32 >>>> >>>> >Traceback (most recent call last): > File "", line 1, in ? >AttributeError: can't set attribute > > Things are changing fast but I don't think this is supposed to work: >>> import scipy >>> a = scipy.arange(10) >>> a array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) >>> a.dtype >>> a.dtype = scipy.int16 Traceback (most recent call last): File "", line 1, in ? TypeError: attribute 'dtype' of 'scipy.bigndarray' objects is not writable My thinking was that you'd need a.astype() which returns a new array object. Since types vary in size, it's not clear what it means to assign type or how it would affect shape. >Also: when / why was it decided that dtypes (float32, int32, ...) should be >lowercase !? Aren't all python types usually uppercase ... > > The naming looks clean to me. I don't know how the decision was made. From haase at msg.ucsf.edu Thu Jan 5 11:03:12 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Thu Jan 5 11:03:12 2006 Subject: [Numpy-discussion] dtype attribute missing in numarray In-Reply-To: <43BD6482.6090707@stsci.edu> References: <200601041721.15149.haase@msg.ucsf.edu> <200601050928.33471.haase@msg.ucsf.edu> <43BD6482.6090707@stsci.edu> Message-ID: <200601051053.38540.haase@msg.ucsf.edu> On Thursday 05 January 2006 10:25, you wrote: > Sebastian Haase wrote: > >Thanks Todd, > >I like this a lot. Maybe it could be mentioned in the documention - I > > don't have any good suggestions on where, I just noticed that 'dtype' is > > not in the index. > >I assume the following assignment to 'dtype' is not that important !? > > > >>>>na.__version__ > > > >'1.5.1' > > > >>>>a = na.arange(10,dtype=na.float32) > >>>>a.dtype > > > >Float32 > > > >>>>a.dtype = na.int32 > > > >Traceback (most recent call last): > > File "", line 1, in ? > >AttributeError: can't set attribute > > Things are changing fast but I don't think this is supposed to work: > >>> import scipy > >>> > >>> a = scipy.arange(10) > >>> > >>> a > > array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) > > >>> a.dtype > > > > >>> a.dtype = scipy.int16 > > Traceback (most recent call last): > > File "", line 1, in ? > > TypeError: attribute 'dtype' of 'scipy.bigndarray' objects is not writable > > My thinking was that you'd need a.astype() which returns a new array > object. Since types vary in size, it's not clear what it means to > assign type or how it would affect shape. > No, only assignment with a type of same size was supposed to work - that's what I remember having read somewhere !.... BUT I think having that restriction would be weird to say the least ... Event thought being able to say 'a.dtype = ' just like one can say 'a.shape = ' would look nice . Then again only includes shapes that are "compatible" are accepted -> so that "compatible types" _would_ really be the consequent restriction !?!? > >Also: when / why was it decided that dtypes (float32, int32, ...) should > > be lowercase !? Aren't all python types usually uppercase ... > > The naming looks clean to me. I don't know how the decision was made. Is numarray going to change to this ( i.e. a.dtype would return 'uint32' and 'UInt32' would be the alias ) ? From jmiller at stsci.edu Thu Jan 5 11:32:14 2006 From: jmiller at stsci.edu (Todd Miller) Date: Thu Jan 5 11:32:14 2006 Subject: [Numpy-discussion] dtype attribute missing in numarray In-Reply-To: <200601051053.38540.haase@msg.ucsf.edu> References: <200601041721.15149.haase@msg.ucsf.edu> <200601050928.33471.haase@msg.ucsf.edu> <43BD6482.6090707@stsci.edu> <200601051053.38540.haase@msg.ucsf.edu> Message-ID: <43BD7408.8010906@stsci.edu> Sebastian Haase wrote: >On Thursday 05 January 2006 10:25, you wrote: > > >>Sebastian Haase wrote: >> >> >>>Thanks Todd, >>>I like this a lot. Maybe it could be mentioned in the documention - I >>>don't have any good suggestions on where, I just noticed that 'dtype' is >>>not in the index. >>>I assume the following assignment to 'dtype' is not that important !? >>> >>> >>> >>>>>>na.__version__ >>>>>> >>>>>> >>>'1.5.1' >>> >>> >>> >>>>>>a = na.arange(10,dtype=na.float32) >>>>>>a.dtype >>>>>> >>>>>> >>>Float32 >>> >>> >>> >>>>>>a.dtype = na.int32 >>>>>> >>>>>> >>>Traceback (most recent call last): >>> File "", line 1, in ? >>>AttributeError: can't set attribute >>> >>> >>Things are changing fast but I don't think this is supposed to work: >> >> >>>>>import scipy >>>>> >>>>>a = scipy.arange(10) >>>>> >>>>>a >>>>> >>>>> >>array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) >> >> >> >>>>>a.dtype >>>>> >>>>> >> >> >> >> >>>>>a.dtype = scipy.int16 >>>>> >>>>> >>Traceback (most recent call last): >> >> File "", line 1, in ? >> >>TypeError: attribute 'dtype' of 'scipy.bigndarray' objects is not writable >> >>My thinking was that you'd need a.astype() which returns a new array >>object. Since types vary in size, it's not clear what it means to >>assign type or how it would affect shape. >> >> >> >No, only assignment with a type of same size was supposed to work - that's >what I remember having read somewhere !.... BUT I think having that >restriction would be weird to say the least ... > >Event thought being able to say 'a.dtype = ' just like one can say >'a.shape = ' would look nice . Then again only >includes shapes that are "compatible" are accepted -> so that "compatible >types" _would_ really be the consequent restriction !?!? > > > > >>>Also: when / why was it decided that dtypes (float32, int32, ...) should >>>be lowercase !? Aren't all python types usually uppercase ... >>> >>> >>The naming looks clean to me. I don't know how the decision was made. >> >> > >Is numarray going to change to this ( i.e. a.dtype would return 'uint32' and >'UInt32' would be the alias ) > > I think we're close to the point where numarray/scipy compatibility breaks down / won't be built up. numarray's .dtype is returning a numarray type object, not a string. Most notably, the type object is not a numpy/scipy type object. Also, it has a different repr which spells the type in mixed case. I returned numarray's type object because AFAIK newcore's .dtype returns a type object. I also considered returning a string but that wouldn't really be compatible either. Todd From Chris.Barker at noaa.gov Thu Jan 5 12:00:11 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu Jan 5 12:00:11 2006 Subject: [Numpy-discussion] Re: [SciPy-user] [SciPy-dev] Long live to numpy (and its list as well) In-Reply-To: <43BD6109.9030402@sympatico.ca> References: <200601051625.k05GPixf021407@oobleck.astro.cornell.edu> <43BD5DFA.20906@colorado.edu> <43BD6109.9030402@sympatico.ca> Message-ID: <43BD7A99.8070002@noaa.gov> Colin J. Williams wrote: > It seems to me that NumPy is the better way to go, the archives and > downloads are more readily available there, but it's up to Todd Miller > and the folk who have been maintaining NumPy. > > NumPy has been around for a while, probably longer than SciPy. I, for one, am only subscribed to the NumPy list, and have been for years. I don't know how much non-NumPy stuff there is on the SciPy list, but while I do use NumPy for Scientific stuff, I don't use the rest of SciPy (because, frankly, it's always been a pain in the @#! to install) and I don't need another high-traffic list. +1 numpy-discussion -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From jmfnonldrz at mib.com Thu Jan 5 13:32:10 2006 From: jmfnonldrz at mib.com (Demetrius Wolff) Date: Thu Jan 5 13:32:10 2006 Subject: [Numpy-discussion] 8.57% UP! Message-ID: <000d$019fc857$0c032e72@mikes> St ock Alert iPackets International, Inc. Global Developer and Provider of a Wide Range of Wireless and Communications Solutions for Selected Enterprises Including Mine-Safety (Source: News 1/3/06) OTC: IPKL Price: .40 Huge PR For Thursday is Underway on IPKL. Short/Day Trading Opportunity for You? Sometimes it is Bang-Zoom on These Small st ocks..As Many of You may Know Recent News: Go Read the Full Stories Now! 1)iPackets International Receives US$85,000 Down Payment for Its First iPMine Deployment in China 2)iPackets International Attends Several Mining Trade Shows and Receives Tremendous Response for Its iPMine Mine-Safety Product Watch This One Trade on Wednesday! Radar it Right Now.. _______________ Information within this email contains 4rward l00 king sta tements within meaning of Section 27A of the Sec urities Act of nineteen thirty three and Section 21B of the Se curities Exchange Act of nineteen thirty four. Any statements that express or involve discussions with respect to predictions, expectations, beliefs, plans, projections, objectives, goals, assumptions future events or performance are not statements of historical fact and may be 4rward 1o0king statements. 4rward looking statements are based on ex pectations, es timates and pr ojections at the time the statements are that involve a number of risks and uncertainties which could cause actual results or events to differ materially from those presently featured Com pany is not a reporting company under the SEC Act of 1934 and there is limited information available on the company. The Co-mpany has a nominal ca sh position.It is an operating company. The company is going to need financing. If that financing does not occur, the company may not be able to continue as a going concern in which case you could lose your in-vestment. The pu blisher of this new sletter does not represent that the information contained in this mes sage states all material facts or does not omit a material fact necessary to make the statements therein not misleading. All information provided within this e_ mail pertaining to in-vesting, st 0cks, se curities must be understood as information provided and not inv estment advice. Remember a thorough due diligence effort, including a review of a company's filings when available, should be completed prior to in_vesting. The pub lisher of this news letter advises all readers and subscribers to seek advice from a registered professional securities representative before deciding to trade in st0cks featured this e mail. None of the material within this report shall be construed as any kind of in_vestment advice or solicitation. Many of these com panies on the verge of bankruptcy. You can lose all your mony by inv esting in st 0ck. The publisher of this new sletter is not a registered in-vestment advis0r. Subscribers should not view information herein as legal, tax, accounting or inve stment advice. In com pliance with the Securities Act of nineteen thirty three, Section 17(b),The publisher of this newsletter is contracted to receive fifteen th0 usand d0 l1ars from a third party, not an off icer, dir ector or af filiate shar eh0lder for the ci rculation of this report. Be aware of an inherent conflict of interest resulting from such compensation due to the fact that this is a paid advertisement and is not without bias. All factual information in this report was gathered from public sources, including but not limited to Co mpany Press Releases. Use of the information in this e mail constitutes your acceptance of these terms. From faltet at carabos.com Thu Jan 5 13:42:08 2006 From: faltet at carabos.com (Francesc Altet) Date: Thu Jan 5 13:42:08 2006 Subject: [Numpy-discussion] A proposal for dtype/dtypedescr in numpy objects Message-ID: <200601052240.58099.faltet@carabos.com> Hi, In my struggle for getting consistent behaviours with data types, I've ended with a new proposal for treating them. The basic thing is that I suggest to deprecate .dtype as being a first-class attribute and replace it instead by the descriptor type container, which I find quite more useful for end users. The current .dtype type will be still accessible (mainly for developers) but buried in .dtype.type. Briefly stated: current proposed ======= ======== .dtypedescr --> moved into .dtype .dtype --> moved into .dtype.type .dtype.dtypestr --> moved into .dtype.str new .dtype.name What is achieved with that? Well, not much, except easy of use and type comparison correctness. For example, with the next setup: >>> import numpy >>> a=numpy.arange(10,dtype='i') >>> b=numpy.arange(10,dtype='l') we have currently: >>> a.dtype >>> a.dtypedescr dtypedescr('>> a.dtypedescr.dtypestr '>> a.dtype.__name__[:-8] 'int32' >>> a.dtype == b.dtype False With the new proposal, we would have: >>> a.dtype.type >>> a.dtype dtype('>> a.dtype.str '>> a.dtype.name 'int32' >>> a.dtype == b.dtype True The advantages of the new proposal are: - No more .dtype and .dtypedescr lying together, just current .dtypedescr renamed to .dtype. I think that current .dtype does not provide more useful information than current .dtypedesc, and giving it a shorter name than .dtypedescr seems to indicate that it is more useful to users (and in my opinion, it isn't). - Current .dtype is still accessible, but specifying and extra name in path: .dtype.type (can be changed into .dtype.type_ or whatever). This should be useful mainly for developers. - Added a useful dtype(descr).name so that one can quickly access to the type name. - Comparison between data types works as it should now (without having to create a metaclass for PyType_Type). Drawbacks: - Backward incompatible change. However, provided the advantages are desirable, I think it is better changing now than later. - I don't specially like the string representation for the new .dtype class. For example, I'd find dtype('Int32') much better than dtype('0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" -------------- next part -------------- A non-text attachment was scrubbed... Name: dtype-dtypedescr.patch Type: text/x-diff Size: 28690 bytes Desc: not available URL: From faltet at carabos.com Thu Jan 5 13:52:12 2006 From: faltet at carabos.com (Francesc Altet) Date: Thu Jan 5 13:52:12 2006 Subject: [Numpy-discussion] Re: [SciPy-dev] A proposal for dtype/dtypedescr in numpy objects In-Reply-To: <200601052240.58099.faltet@carabos.com> References: <200601052240.58099.faltet@carabos.com> Message-ID: <200601052249.57811.faltet@carabos.com> Oops, re-reading my last message, I discovered a small errata: A Dijous 05 Gener 2006 22:40, Francesc Altet va escriure: > - I don't specially like the string representation for the new .dtype > class. For example, I'd find dtype('Int32') much better than ^^^^^ should read 'int32' (to follow numpy conventions) > dtype(' code, but they can be made later on (much less disruptive than the > proposed change). -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From faltet at carabos.com Thu Jan 5 14:04:17 2006 From: faltet at carabos.com (Francesc Altet) Date: Thu Jan 5 14:04:17 2006 Subject: [Numpy-discussion] Re: [SciPy-user] [SciPy-dev] Long live to numpy (and its list as well) In-Reply-To: <43BD7A99.8070002@noaa.gov> References: <43BD6109.9030402@sympatico.ca> <43BD7A99.8070002@noaa.gov> Message-ID: <200601052303.04502.faltet@carabos.com> A Dijous 05 Gener 2006 20:59, Christopher Barker va escriure: > Colin J. Williams wrote: > > It seems to me that NumPy is the better way to go, the archives and > > downloads are more readily available there, but it's up to Todd Miller > > and the folk who have been maintaining NumPy. > > > > NumPy has been around for a while, probably longer than SciPy. > > I, for one, am only subscribed to the NumPy list, and have been for > years. I don't know how much non-NumPy stuff there is on the SciPy list, > but while I do use NumPy for Scientific stuff, I don't use the rest of > SciPy (because, frankly, it's always been a pain in the @#! to install) > and I don't need another high-traffic list. Yeah, I was thinking in people like you. In fact, I'm myself in the same case than you: I'm very interested in a basic array module (Numeric/numarray/numpy) and not so much in more powerful (but complex) packages like scipy. And I think there can be a lot of people out there that can be in the same position. Accordingly, my vote is also: +1 numpy-discussion -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From oliphant.travis at ieee.org Thu Jan 5 15:26:12 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu Jan 5 15:26:12 2006 Subject: [Numpy-discussion] ANN: Release of NumPy 0.9.2 Message-ID: <43BDAAD3.6070705@ieee.org> Numpy 0.9.2 is the successor to both Numeric and Numarray and builds and uses code from both. This release marks the first release using the new (but historical) Numpy name. The release notes are included below: Best regards, -Travis Oliphant Release Notes ================== NumPy 0.9.2 marks the first release of the new array package under its new name. This new name should reflect that the new package is a hybrid of the Numeric and Numarray packages. This release adds many more features and speed-enhancements from Numarray. Changes from (SciPy Core) 0.8.4: - Namespace and Python package name is now "numpy" and "numpy" instead of "scipy" and "scipy_core" respectively. This should help packagers and egg-builders. - The NumPy arrayobject now both exports and consumes the full array_descr protocol (including field information). - Removed NOTSWAPPED flag. The byteswapping information is handled by the data-type descriptor. - The faster sorting functions were brought over from numarray leading to a factor of 2-3 speed increase in sorting. Also changed .sort() method to be in-place like numarray and lists. - Polynomial division has been fixed. - basic.fft, basic.linalg, basic.random have been moved to dft, linalg, and random respectively (no extra basic sub-package layer). - Introduced numpy.dual to allow calling of functions that are both in SciPy and NumPy when it is desired to use the SciPy function if the user has it but otherwise use the NumPy function. - The "rtype" keyword used in a couple of places has been changed to "dtype" for consistency. - Fixes so that the standard array constructor can be used to construct record-arrays with fields. - Changed method .toscalar() to .item() (added to convertcode.py) - Added numpy.lib.mlab to be fully compatible with old MLab including the addition of a kaiser window even when full SciPy is not installed. - Arrays of nested records should behave better. - Fixed masked arrays buglets. - Added code so that strings can be converted to numbers using .astype() - Added a lexsort (lexigraphic) function so that sorting on multiple keys can be done -- very useful for record-arrays - Speed ups and bug-fixes for 1-d "fancy" indexing by going through the flattened array iterator when possible. - Added the ability to add docstrings to builtin objects "on-the-fly". Allows adding docstrings without re-compiling C-code. - Moved the weave subpackage to SciPy. - Changed the fields attribute of the dtypedescr object to return a "read-only" dictionary when accessed from Python. - Added a typeNA dictionary for the numarray types and added a compare function for dtypedescr objects so that equivalent types can be detected. Please not that all modules are imported using lower-case letters (so don't let the NumPy marketing name confuse you, the package to import is "numpy"). From edcjones at comcast.net Thu Jan 5 16:49:13 2006 From: edcjones at comcast.net (Edward C. Jones) Date: Thu Jan 5 16:49:13 2006 Subject: [Numpy-discussion] numarray: Two edge case buglets for clip Message-ID: <43BDBE52.90606@comcast.net> #! /usr/bin/env python import numarray from numarray.numerictypes import * # I have a PC with an AMD Athlon 64 3500+ processor. The operating # system is up-to-date Debian unstable Linux. I use the "i386" # distribution, not the "amd64" distribution. The "i386" distribution # works on both 32 and 64 bit machines. def bug1(): # Here is an edge case. Should the following work or not: arr = numarray.array([100], Int8) arrout = numarray.clip(arr, -1000, 1000) # The documentation doesn't quite give the answer: # "The clip function creates an array with the same shape and # type as m, but where every entry in m that is less than m_min # is replaced by m_min, and every entry greater than m_max is # replaced by m_max. Entries within the range [m_min, m_max] are # left unchanged." # In image processing, I can easily imagine a loop where # numarrays of various types are all clipped to [0, 255]. # Therefore I think that the "clip" above should return a copy of # arr. def bug2(): # In this case, 2**63 _is_ representable as a UInt64. arr = numarray.array([100], UInt64) print arr.type(), arr arrout = numarray.clip(arr, 0, 2**63-1) print arr.type(), arrout arrout = numarray.clip(arr, 0, 2**63) print arrout bug1() bug2() From alexander.belopolsky at gmail.com Thu Jan 5 20:51:09 2006 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu Jan 5 20:51:09 2006 Subject: [Numpy-discussion] return type of ndarray + ma.array Message-ID: the following session demonstrates the difference between numpy.ma and old MA behavior. Sum ndarray and ma.array is ndarray >>> type(numeric.array([1])+ma.array([1])) However, taken in the different order, >>> type(ma.array([1])+numeric.array([1])) In the old MA module the sum is MaskedArray in any order: >>> type(MA.array([1])+Numeric.array([1])) >>> type(Numeric.array([1])+MA.array([1])) I suspected that new ma module may not have __radd__ defined, but it does. Can anyone explain this? Thanks. -- sasha PS: >>> import numpy >>> numpy.__version__ '0.9.3.1831' From archibald at web.de Fri Jan 6 03:23:29 2006 From: archibald at web.de (antioch) Date: Fri Jan 6 03:23:29 2006 Subject: [Numpy-discussion] angle Message-ID: <0a4d01c61232$eb97990d$9e6e7fc6@web.de> ancestral -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Fri Jan 6 08:27:15 2006 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Fri Jan 6 08:27:15 2006 Subject: [Numpy-discussion] Properties of ma.masked singleton Message-ID: <2988C214-E82E-4604-8697-989C4F6E3AB1@gmail.com> In the current numpy version ma.masked singleton has the following properties: >>> len(ma.masked) 0 which is a change from old MA, where >>> len(MA.masked) 1 Similarly: >>> ma.masked.shape (0,) >>> MA.masked.shape () It changes shape when filled: >>> ma.masked.filled() array([999999]) and so on. The code contains numerous "if x is masked" statements to support all this logic. I would like to suggest a somewhat radical change for your review. There are two main uses of ma.masked: 1. To set mask: >>> x = ma.zeros(5) >>> x[2:4] = ma.masked >>> print x [0 0 -- -- 0] 2. To check if an element is masked: >>> x[2] is ma.masked True The second property allows a cute looking idiom "x[i] is masked", but only works for single element access: >>> x[2:4] is ma.masked False It also leads to strange results: >>> x[2].shape (0,) My proposal is to eliminate the property #2 from ma and redefine masked as None. Single element getitem will return a rank zero MaskedArray. We can also add "is_masked" property, which will allow a convenient check in the form "x[i].is_masked". The main benefit of this proposal is that x[i] will be duck typing compatible with numpy scalars. -- sasha From jmiller at stsci.edu Fri Jan 6 09:24:10 2006 From: jmiller at stsci.edu (Todd Miller) Date: Fri Jan 6 09:24:10 2006 Subject: [Numpy-discussion] numarray: Two edge case buglets for clip In-Reply-To: <43BDBE52.90606@comcast.net> References: <43BDBE52.90606@comcast.net> Message-ID: <43BEA76D.6000406@stsci.edu> Edward C. Jones wrote: > #! /usr/bin/env python > > import numarray > from numarray.numerictypes import * > > # I have a PC with an AMD Athlon 64 3500+ processor. The operating > # system is up-to-date Debian unstable Linux. I use the "i386" > # distribution, not the "amd64" distribution. The "i386" distribution > # works on both 32 and 64 bit machines. > > def bug1(): > # Here is an edge case. Should the following work or not: > > arr = numarray.array([100], Int8) > arrout = numarray.clip(arr, -1000, 1000) I don't think it's a bug that clip() doesn't silently turn itself into a no-op when given out of range limits. If you want to submit a patch to make clip() smarter, we'll consider it, but you should probably be pouring your energy into newcore/numpy instead. > > # The documentation doesn't quite give the answer: > > # "The clip function creates an array with the same shape and > # type as m, but where every entry in m that is less than m_min > # is replaced by m_min, and every entry greater than m_max is > # replaced by m_max. Entries within the range [m_min, m_max] are > # left unchanged." > > # In image processing, I can easily imagine a loop where > # numarrays of various types are all clipped to [0, 255]. > # Therefore I think that the "clip" above should return a copy of > # arr. > > def bug2(): > # In this case, 2**63 _is_ representable as a UInt64. > arr = numarray.array([100], UInt64) > print arr.type(), arr > arrout = numarray.clip(arr, 0, 2**63-1) > print arr.type(), arrout > arrout = numarray.clip(arr, 0, 2**63) > print arrout I agree that bug2 is a problem and logged this on Source Forge. Is bug2 impacting you or can you work around it? Regards, Todd From perry at stsci.edu Fri Jan 6 09:34:12 2006 From: perry at stsci.edu (Perry Greenfield) Date: Fri Jan 6 09:34:12 2006 Subject: [Numpy-discussion] Re: [SciPy-user] [SciPy-dev] Long live to numpy (and its list as well) In-Reply-To: <200601052303.04502.faltet@carabos.com> References: <43BD6109.9030402@sympatico.ca> <43BD7A99.8070002@noaa.gov> <200601052303.04502.faltet@carabos.com> Message-ID: Perhaps all discussions of the core stuff could go to numpy-discussion and the applications and libraries go to scipy-discussion. That segregation often doesn't work and you still get all the cross-posting anyway. But I would think that it would be important to have the core element highlighted in the name since that user base will be bigger than the scipy one by necessity. Perry On Jan 5, 2006, at 5:03 PM, Francesc Altet wrote: > A Dijous 05 Gener 2006 20:59, Christopher Barker va escriure: >> Colin J. Williams wrote: >>> It seems to me that NumPy is the better way to go, the archives and >>> downloads are more readily available there, but it's up to Todd >>> Miller >>> and the folk who have been maintaining NumPy. >>> >>> NumPy has been around for a while, probably longer than SciPy. >> >> I, for one, am only subscribed to the NumPy list, and have been for >> years. I don't know how much non-NumPy stuff there is on the SciPy >> list, >> but while I do use NumPy for Scientific stuff, I don't use the rest of >> SciPy (because, frankly, it's always been a pain in the @#! to >> install) >> and I don't need another high-traffic list. > > Yeah, I was thinking in people like you. In fact, I'm myself in the > same case than you: I'm very interested in a basic array module > (Numeric/numarray/numpy) and not so much in more powerful (but > complex) packages like scipy. And I think there can be a lot of people > out there that can be in the same position. > > Accordingly, my vote is also: > > +1 numpy-discussion From haase at msg.ucsf.edu Fri Jan 6 10:15:09 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Fri Jan 6 10:15:09 2006 Subject: [Numpy-discussion] Re: [SciPy-user] [SciPy-dev] Long live to numpy (and its list as well) In-Reply-To: References: <200601052303.04502.faltet@carabos.com> Message-ID: <200601061014.22379.haase@msg.ucsf.edu> On Friday 06 January 2006 09:34, Perry Greenfield wrote: > Perhaps all discussions of the core stuff could go to numpy-discussion > and the applications and libraries go to scipy-discussion. That > segregation often doesn't work and you still get all the cross-posting > anyway. But I would think that it would be important to have the core > element highlighted in the name since that user base will be bigger > than the scipy one by necessity. > > Perry > Reading this I would second the idea of keeping two lists: if and only if the total number of people exceeds a certain limit (which I think was the original statement that that is not the case -- is there some statistics of how many (active) people are subscribed to each list) Also I would like to point out that the way Perry wrote his posting the "scipy"(!) list should soon be the larger one, because "applications and libraries" is the reason we all do this in the first place and the underlying "numpy" would "just be working" - at least as most people will concern/presume. - Sebastian Haase > On Jan 5, 2006, at 5:03 PM, Francesc Altet wrote: > > A Dijous 05 Gener 2006 20:59, Christopher Barker va escriure: > >> Colin J. Williams wrote: > >>> It seems to me that NumPy is the better way to go, the archives and > >>> downloads are more readily available there, but it's up to Todd > >>> Miller > >>> and the folk who have been maintaining NumPy. > >>> > >>> NumPy has been around for a while, probably longer than SciPy. > >> > >> I, for one, am only subscribed to the NumPy list, and have been for > >> years. I don't know how much non-NumPy stuff there is on the SciPy > >> list, > >> but while I do use NumPy for Scientific stuff, I don't use the rest of > >> SciPy (because, frankly, it's always been a pain in the @#! to > >> install) > >> and I don't need another high-traffic list. > > > > Yeah, I was thinking in people like you. In fact, I'm myself in the > > same case than you: I'm very interested in a basic array module > > (Numeric/numarray/numpy) and not so much in more powerful (but > > complex) packages like scipy. And I think there can be a lot of people > > out there that can be in the same position. > > > > Accordingly, my vote is also: > > > > +1 numpy-discussion From alexander.belopolsky at gmail.com Fri Jan 6 12:33:14 2006 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Fri Jan 6 12:33:14 2006 Subject: [Numpy-discussion] Properties of ma.masked singleton In-Reply-To: <43BEB19F.6010107@pfdubois.com> References: <2988C214-E82E-4604-8697-989C4F6E3AB1@gmail.com> <43BEB19F.6010107@pfdubois.com> Message-ID: Yes, you are right. I did not realize that x[i] = None is legitimate for dtype=object arrays. This can easily be fixed by using an instance of an empty class instead of None. My main concern was that since now x[i] provides full array inteface in numpy regardless of the value of i, using a singleton value for the same construct in ma leads to all kinds of surprizing results. For example: >>> ma.array([1.0],mask=[1])[0].dtype >>> ma.array([1.0],mask=[0])[0].dtype While I appreciate the utility of "x[i] = masked", particularly for fancy i, I believe "x[i] is masked" is of limited usefulness. Note that the first construct does not require a singleton - any masked scalar can be used instead of the singleton. The second construct on the other hand would not work without a singleton. I would disagree that "x[2:4] is masked is a red herring". Consider the following session: >>> i = slice(2,4) >>> x = ma.arange(5) >>> x[i] = ma.masked >>> x[i] is ma.masked False Note that the result would be true if i was an integer. It is not obvious that "x[2:4] is masked" should be false is all elements in 2:4 range are masked. Numpy deprecated conversion of size > 1 arrays to bool in favor of explicit any or all: >>> bool(array([1,1])) Traceback (most recent call last): File "", line 1, in ? ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() In my view "x[i] is masked" is ambiguous as well and x.mask.any() or x.mask.all() seems to be a better choice (this would require use of bool_(0) instead of None to indicate no mask). How much code out there relies on "x[i] is masked" construct? I've used MA for many years (thanks for an excellent product, Paul!) but never used this construct in my code. Thanks. -- sasha On 1/6/06, Paul F. Dubois wrote: > I'll look into your complaints. Your proposed solution does not work: > if x[2] is masked: > and > if x[2] is None: > become indistinguishable. Likewise for assignment. > > Your example if x[2:4] is masked is a red herring. Obviously this > statement is false no matter how you define 'masked'. > > I redid ma in a bit of a hurry and didn't pay any attention to the > issues you raise as to len(masked), which I agree should be 1. > > Alexander Belopolsky wrote: > > In the current numpy version ma.masked singleton has the following > > properties: > > > > >>> len(ma.masked) > > 0 > > > > which is a change from old MA, where > > >>> len(MA.masked) > > 1 > > > > Similarly: > > > > >>> ma.masked.shape > > (0,) > > >>> MA.masked.shape > > () > > > > It changes shape when filled: > > > > >>> ma.masked.filled() > > array([999999]) > > > > and so on. > > > > The code contains numerous "if x is masked" statements to support all > > this logic. > > > > I would like to suggest a somewhat radical change for your review. > > > > There are two main uses of ma.masked: > > > > 1. To set mask: > > > > >>> x = ma.zeros(5) > > >>> x[2:4] = ma.masked > > >>> print x > > [0 0 -- -- 0] > > > > > > 2. To check if an element is masked: > > > > >>> x[2] is ma.masked > > True > > > > > > The second property allows a cute looking idiom "x[i] is masked", but > > only works for single element access: > > > > >>> x[2:4] is ma.masked > > False > > > > It also leads to strange results: > > > > >>> x[2].shape > > (0,) > > > > > > My proposal is to eliminate the property #2 from ma and redefine masked > > as None. Single element getitem will return a rank zero MaskedArray. We > > can also add "is_masked" property, which will allow a convenient check > > in the form "x[i].is_masked". > > > > The main benefit of this proposal is that x[i] will be duck typing > > compatible with numpy scalars. > > > > -- sasha > > > > > > ------------------------------------------------------- > > This SF.net email is sponsored by: Splunk Inc. Do you grep through log > > files > > for problems? Stop! Download the new AJAX search engine that makes > > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > > http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > From gruben at bigpond.net.au Fri Jan 6 14:46:36 2006 From: gruben at bigpond.net.au (Gary Ruben) Date: Fri Jan 6 14:46:36 2006 Subject: [Numpy-discussion] test() failures in 0.9.2 In-Reply-To: <43BDAAD3.6070705@ieee.org> References: <43BDAAD3.6070705@ieee.org> Message-ID: <43BE7B46.3030908@bigpond.net.au> Note: Running numpy.test() with the python23 installer version results in 2 errors on both my Win98 and my Win2k box: ERROR: check_basic (numpy.core.defmatrix.test_defmatrix.test_algebra) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\PYTHON23\Lib\site-packages\numpy\core\tests\test_defmatrix.py", line 111, in check_basic assert allclose((mA ** -i).A, B) File "C:\PYTHON23\Lib\site-packages\numpy\core\defmatrix.py", line 148, in __p ow__ x = self.I File "C:\PYTHON23\Lib\site-packages\numpy\core\defmatrix.py", line 203, in get I return matrix(inv(self)) File "C:\PYTHON23\Lib\site-packages\scipy\linalg\basic.py", line 176, in inv a1 = asarray_chkfinite(a) File "C:\PYTHON23\Lib\site-packages\scipy_base\function_base.py", line 32, in asarray_chkfinite if not all(isfinite(x)): TypeError: function not supported for these types, and can't coerce to supported types ====================================================================== ERROR: check_basic (numpy.core.defmatrix.test_defmatrix.test_properties) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\PYTHON23\Lib\site-packages\numpy\core\tests\test_defmatrix.py", line 34, in check_basic assert allclose(linalg.inv(A), mA.I) File "C:\PYTHON23\Lib\site-packages\numpy\core\defmatrix.py", line 203, in get I return matrix(inv(self)) File "C:\PYTHON23\Lib\site-packages\scipy\linalg\basic.py", line 176, in inv a1 = asarray_chkfinite(a) File "C:\PYTHON23\Lib\site-packages\scipy_base\function_base.py", line 32, in asarray_chkfinite if not all(isfinite(x)): TypeError: function not supported for these types, and can't coerce to supported types ---------------------------------------------------------------------- Ran 177 tests in 6.100s FAILED (errors=2) Gary R. From bsouthey at gmail.com Fri Jan 6 14:49:41 2006 From: bsouthey at gmail.com (Bruce Southey) Date: Fri Jan 6 14:49:41 2006 Subject: [Numpy-discussion] Re: [SciPy-user] [SciPy-dev] Long live to numpy (and its list as well) In-Reply-To: <43BD7A99.8070002@noaa.gov> References: <200601051625.k05GPixf021407@oobleck.astro.cornell.edu> <43BD5DFA.20906@colorado.edu> <43BD6109.9030402@sympatico.ca> <43BD7A99.8070002@noaa.gov> Message-ID: I am also +1 to keep numpy for the same reasons as Colin and Chris. So far there has been nothing in alpha SciPy versions that offer any advantage over Numarray for what I use or develop. Bruce On 1/5/06, Christopher Barker wrote: > > Colin J. Williams wrote: > > It seems to me that NumPy is the better way to go, the archives and > > downloads are more readily available there, but it's up to Todd Miller > > and the folk who have been maintaining NumPy. > > > > NumPy has been around for a while, probably longer than SciPy. > > I, for one, am only subscribed to the NumPy list, and have been for > years. I don't know how much non-NumPy stuff there is on the SciPy list, > but while I do use NumPy for Scientific stuff, I don't use the rest of > SciPy (because, frankly, it's always been a pain in the @#! to install) > and I don't need another high-traffic list. > > +1 numpy-discussion > > -Chris > > > -- > Christopher Barker, Ph.D. > Oceanographer > > NOAA/OR&R/HAZMAT (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log > files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Fri Jan 6 19:58:52 2006 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Fri Jan 6 19:58:52 2006 Subject: [Numpy-discussion] Installation of numpy header files Message-ID: Numpy include files get installed in /lib/pythonX.X/site- packages/numpy/base/include/numpy instead of /include/ pythonX.X/numpy. The attached patch fixes the problem. Does anyone rely on the current behavior? -- sasha PS: Using scipy core svn r1847 on a Linux system. -------------- next part -------------- A non-text attachment was scrubbed... Name: test_ma.py-patch Type: application/octet-stream Size: 688 bytes Desc: not available URL: From oliphant.travis at ieee.org Fri Jan 6 19:59:12 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri Jan 6 19:59:12 2006 Subject: [Numpy-discussion] numpy lists In-Reply-To: References: <200601051625.k05GPixf021407@oobleck.astro.cornell.edu> <43BD5DFA.20906@colorado.edu> <43BD6109.9030402@sympatico.ca> <43BD7A99.8070002@noaa.gov> Message-ID: <43BEFA6D.7090305@ieee.org> Bruce Southey wrote: > I am also +1 to keep numpy for the same reasons as Colin and Chris. So > far there has been nothing in alpha SciPy versions that offer any > advantage over Numarray for what I use or develop. There are two new mailing lists for NumPy numpy-devel at lists.sourceforge.net numpy-user at lists.sourceforge.net These are for developers and users to talk about only NumPy The SciPy lists can be for SciPy itself. Two packages deserve separate lists. -Travis From strawman at astraw.com Fri Jan 6 21:13:21 2006 From: strawman at astraw.com (Andrew Straw) Date: Fri Jan 6 21:13:21 2006 Subject: [Numpy-discussion] Installation of numpy header files In-Reply-To: References: Message-ID: <43BF4DD1.2050908@astraw.com> Alexander Belopolsky wrote: >Numpy include files get installed in /lib/pythonX.X/site- >packages/numpy/base/include/numpy instead of /include/ >pythonX.X/numpy. > > >Does anyone rely on the current behavior? > > I do. There was discussion about this a couple of months ago. To find the headers, use the following: import numpy numpy.get_numpy_include() >PS: Using scipy core svn r1847 on a Linux system. > > I guess you're not actually using scipy core, but numpy. > The attached patch fixes the problem. Actually, the attached patch seems entire unrelated. :) From ndarray at mac.com Fri Jan 6 21:24:45 2006 From: ndarray at mac.com (Sasha) Date: Fri Jan 6 21:24:45 2006 Subject: [Numpy-discussion] Re: Installation of numpy header files In-Reply-To: References: Message-ID: Sorry for the wrong attachment. Installing include files under lib breaks modules that use python distutils. Is it recommended that modules that depend on numpy use numpy distutils instead? Thanks. -- sasha On 1/6/06, Alexander Belopolsky wrote: > Numpy include files get installed in /lib/pythonX.X/site- > packages/numpy/base/include/numpy instead of /include/ > pythonX.X/numpy. > > The attached patch fixes the problem. Does anyone rely on the current behavior? > > -- sasha > > PS: Using scipy core svn r1847 on a Linux system. > > > -------------- next part -------------- A non-text attachment was scrubbed... Name: numpy-core-setup-patch Type: application/octet-stream Size: 1505 bytes Desc: not available URL: From ndarray at mac.com Fri Jan 6 21:46:49 2006 From: ndarray at mac.com (Sasha) Date: Fri Jan 6 21:46:49 2006 Subject: [Numpy-discussion] Installation of numpy header files In-Reply-To: <43BF4DD1.2050908@astraw.com> References: <43BF4DD1.2050908@astraw.com> Message-ID: On 1/7/06, Andrew Straw wrote: > ... There was discussion about this a couple of months ago. Could you, please give me some keywords for this discussion? I searched the archives and the only relevant thread was from 2002 (http://aspn.activestate.com/ASPN/Mail/Message/scipy-dev/1592700). I understand that at that time there was a solution involving sitecustomize.py, but that was replaced with site.cfg some time ago. The message that I cited above has a much better description of the problem that I am experiencing than what I wrote: > (1) include directories. > Distutils knows to include files from /usr/include/python2.2 (or > wherever it is installed) whenever building extension modules. > Numeric installs its header files inside this directory when installed > as root. However, when I install Numeric in /home/eric/linux, the > header files are installed in /home/eric/linux/python2.2. Distutils > doesn't know to look in hear for headers. To solve this, I'd have to > hack all the setup.py files for modules that rely on Numeric to use my > include_dirs. This isn't so nice. Thanks. -- sasha From alexander.belopolsky at gmail.com Fri Jan 6 21:47:27 2006 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Fri Jan 6 21:47:27 2006 Subject: [Numpy-discussion] Properties of ma.masked singleton In-Reply-To: <43BEB19F.6010107@pfdubois.com> References: <2988C214-E82E-4604-8697-989C4F6E3AB1@gmail.com> <43BEB19F.6010107@pfdubois.com> Message-ID: On 1/6/06, Paul F. Dubois wrote: > ... > I redid ma in a bit of a hurry and didn't pay any attention to the > issues you raise as to len(masked), which I agree should be 1. I did not suggest that len(masked) should be 1. It was 1 in Numeric MA because scallars had len 1 in Numeric, but in numpy len(scalar) is an error consistent with scalar.shape==() and len(x) == x.shape[0]. My proposal was not to return masked singleton from x[i] and return ma.array(x.data[i],mask=x.mask[i]), this will result in len(x[i]) being an error for rank 1 x and integer i, which would be consistent with numpy. -- sasha From ndarray at mac.com Fri Jan 6 22:05:31 2006 From: ndarray at mac.com (Sasha) Date: Fri Jan 6 22:05:31 2006 Subject: [Numpy-discussion] Installation of numpy header files In-Reply-To: References: <43BF4DD1.2050908@astraw.com> Message-ID: Ok, I've found it: http://www.scipy.org/documentation/mailman?fn=scipy-dev/2005-September/003238.html Sorry for the extra traffic. Let me paraphrase the solution here to hopefully make it more discoverable: """ Extension modules that need to compile against numpy should use get_numpy_include function to locate the appropriate include directory. Using distutils: import numpy Extension('extension_name', ... include_dirs=[numpy.get_numpy_include()]) """ -- sasha On 1/7/06, Sasha wrote: > On 1/7/06, Andrew Straw wrote: > > ... There was discussion about this a couple of months ago. > > Could you, please give me some keywords for this discussion? I > searched the archives and the only relevant thread was from 2002 > (http://aspn.activestate.com/ASPN/Mail/Message/scipy-dev/1592700). I > understand that at that time there was a solution involving > sitecustomize.py, but that was replaced with site.cfg some time ago. > > The message that I cited above has a much better description of the > problem that I am experiencing than what I wrote: > > > (1) include directories. > > Distutils knows to include files from /usr/include/python2.2 (or > > wherever it is installed) whenever building extension modules. > > Numeric installs its header files inside this directory when installed > > as root. However, when I install Numeric in /home/eric/linux, the > > header files are installed in /home/eric/linux/python2.2. Distutils > > doesn't know to look in hear for headers. To solve this, I'd have to > > hack all the setup.py files for modules that rely on Numeric to use my > > include_dirs. This isn't so nice. > > Thanks. > > -- sasha > From strawman at astraw.com Fri Jan 6 22:28:26 2006 From: strawman at astraw.com (Andrew Straw) Date: Fri Jan 6 22:28:26 2006 Subject: [Numpy-discussion] Re: [SciPy-user] numpy lists In-Reply-To: <43BEFA6D.7090305@ieee.org> References: <200601051625.k05GPixf021407@oobleck.astro.cornell.edu> <43BD5DFA.20906@colorado.edu> <43BD6109.9030402@sympatico.ca> <43BD7A99.8070002@noaa.gov> <43BEFA6D.7090305@ieee.org> Message-ID: <43BF14E6.8070507@astraw.com> Travis Oliphant wrote: >There are two new mailing lists for NumPy > >numpy-devel at lists.sourceforge.net >numpy-user at lists.sourceforge.net > >These are for developers and users to talk about only NumPy > > You can subscribe to these lists from http://sourceforge.net/mail/?group_id=1369 From Chris.Barker at noaa.gov Fri Jan 6 22:29:02 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri Jan 6 22:29:02 2006 Subject: [Numpy-discussion] Re: [SciPy-user] [SciPy-dev] Long live to numpy (and its list as well) In-Reply-To: References: <200601051625.k05GPixf021407@oobleck.astro.cornell.edu> <43BD5DFA.20906@colorado.edu> <43BD6109.9030402@sympatico.ca> <43BD7A99.8070002@noaa.gov> Message-ID: <43BF155C.20005@noaa.gov> Bruce Southey wrote: > I am also +1 to keep numpy for the same reasons as Colin and Chris. So > far there has been nothing in alpha SciPy versions that offer any > advantage over Numarray for what I use or develop. Just so it's clear: I was advocating that the numpy-discussion group be the primary place for NumPy-related ( as opposed to other scipy stuff) discussion. I was not advocating for or against the adaptation of the new scipy_base - scipy_core - now-numpy. As it happens, I am very excited by the new numpy, am really looking forward to its rapid adoption. -Chris > On 1/5/06, *Christopher Barker* +1 numpy-discussion -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From paul at pfdubois.com Fri Jan 6 23:19:08 2006 From: paul at pfdubois.com (Paul F. Dubois) Date: Fri Jan 6 23:19:08 2006 Subject: [Numpy-discussion] Properties of ma.masked singleton In-Reply-To: <2988C214-E82E-4604-8697-989C4F6E3AB1@gmail.com> References: <2988C214-E82E-4604-8697-989C4F6E3AB1@gmail.com> Message-ID: <43BEB19F.6010107@pfdubois.com> I'll look into your complaints. Your proposed solution does not work: if x[2] is masked: and if x[2] is None: become indistinguishable. Likewise for assignment. Your example if x[2:4] is masked is a red herring. Obviously this statement is false no matter how you define 'masked'. I redid ma in a bit of a hurry and didn't pay any attention to the issues you raise as to len(masked), which I agree should be 1. Alexander Belopolsky wrote: > In the current numpy version ma.masked singleton has the following > properties: > > >>> len(ma.masked) > 0 > > which is a change from old MA, where > >>> len(MA.masked) > 1 > > Similarly: > > >>> ma.masked.shape > (0,) > >>> MA.masked.shape > () > > It changes shape when filled: > > >>> ma.masked.filled() > array([999999]) > > and so on. > > The code contains numerous "if x is masked" statements to support all > this logic. > > I would like to suggest a somewhat radical change for your review. > > There are two main uses of ma.masked: > > 1. To set mask: > > >>> x = ma.zeros(5) > >>> x[2:4] = ma.masked > >>> print x > [0 0 -- -- 0] > > > 2. To check if an element is masked: > > >>> x[2] is ma.masked > True > > > The second property allows a cute looking idiom "x[i] is masked", but > only works for single element access: > > >>> x[2:4] is ma.masked > False > > It also leads to strange results: > > >>> x[2].shape > (0,) > > > My proposal is to eliminate the property #2 from ma and redefine masked > as None. Single element getitem will return a rank zero MaskedArray. We > can also add "is_masked" property, which will allow a convenient check > in the form "x[i].is_masked". > > The main benefit of this proposal is that x[i] will be duck typing > compatible with numpy scalars. > > -- sasha > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log > files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From vukweer at conservatek.com Sun Jan 8 13:47:45 2006 From: vukweer at conservatek.com (Adkins Sherrie) Date: Sun Jan 8 13:47:45 2006 Subject: [Numpy-discussion] wanted info Message-ID: <001001c6149c$ff8d93c0$72acc756@bleuazurb2nx0k> Current Price: 1.75 Expected 5 day: 2.30 COMPANY OVERVIEW Aggressive and energetic, Infinex boasts a dynamic and diversified portfolio of operations across North America, with an eye on international expansion. Grounded in natural resource exploration, Inifinex also offers investors access to exciting new developments in the high-tech sector and the booming international real estate market. Our market based experience, tenacious research techniques, and razor sharp analytical skills allow us to leverage opportunities in emerging markets and developing technologies. Identifying these opportunities in the earliest stages allows us to accelerate business development and fully realize the company??s true potential. Maximizing overall profitability and in turn enhancing shareholder value. Current Press Release Infinex Ventures Inc. (IFNX - News) is pleased to announce the appointment of Mr. Stefano Masullo, to its Board of Directors. Mr. Michael De Rosa the President says, "Mr. Masullo's varied background in finance, engineering and economics, as well as his experience of over 10 years as a Board member of a vast number of International companies, will make him a valuable addition to the Infinex Board. His appointment will show our commitment to the financial, engineering and business structure of our Company." Mr. Masullo attended the University of Luigi Bocconi, in Milan Italy, where he graduated in industrial, economic and financial sciences. Mr. Masullo first began his well rounded career during one of his years at University (1986-1987), where he assisted the Director of Faculty of Finance in finance and investment. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: yvybu.gif Type: image/gif Size: 2036 bytes Desc: not available URL: From cjw at sympatico.ca Sun Jan 8 14:43:56 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Sun Jan 8 14:43:56 2006 Subject: [Numpy-discussion] Re: [SciPy-user] numpy lists In-Reply-To: <43BF14E6.8070507@astraw.com> References: <200601051625.k05GPixf021407@oobleck.astro.cornell.edu> <43BD5DFA.20906@colorado.edu> <43BD6109.9030402@sympatico.ca> <43BD7A99.8070002@noaa.gov> <43BEFA6D.7090305@ieee.org> <43BF14E6.8070507@astraw.com> Message-ID: <43C15693.2060601@sympatico.ca> Andrew Straw wrote: > Travis Oliphant wrote: > >> There are two new mailing lists for NumPy >> >> numpy-devel at lists.sourceforge.net >> numpy-user at lists.sourceforge.net >> >> These are for developers and users to talk about only NumPy >> >> > You can subscribe to these lists from > http://sourceforge.net/mail/?group_id=1369 I send my postings to numpy-discussion at lists.sourceforge.net. Colin W. From lefeil at email.arizona.edu Sun Jan 8 20:19:10 2006 From: lefeil at email.arizona.edu (lefeil at email.arizona.edu) Date: Sun Jan 8 20:19:10 2006 Subject: [Numpy-discussion] Why I cannot import NumPy? Message-ID: <20060108211842.phbmjs0oggogcckg@www.email.arizona.edu> After installation, I cannot import NumPy. I tried to run setup.py in the numpy directory. But the following error occurs: IDLE 1.1.2 >>> ================================ RESTART ================================ >>> Assuming default configuration (distutils\command/{setup_command,setup}.py was not found) Appending numpy.distutils.command configuration to numpy.distutils Assuming default configuration (distutils\fcompiler/{setup_fcompiler,setup}.py was not found) Appending numpy.distutils.fcompiler configuration to numpy.distutils Appending numpy.distutils configuration to numpy Appending numpy.testing configuration to numpy F2PY Version 2_1830 Appending numpy.f2py configuration to numpy Traceback (most recent call last): File "C:\Python24\lib\site-packages\numpy\setup.py", line 27, in -toplevel- setup(**configuration(top_path='')) File "C:\Python24\lib\site-packages\numpy\setup.py", line 10, in configuration config.add_subpackage('core') File "C:\Python24\Lib\site-packages\numpy\distutils\misc_util.py", line 399, in add_subpackage config = self.get_subpackage(subpackage_name,subpackage_path) File "C:\Python24\Lib\site-packages\numpy\distutils\misc_util.py", line 389, in get_subpackage config = setup_module.configuration(*args) File "C:\Python24\Lib\site-packages\numpy\core\setup.py", line 19, in configuration open(generate_umath_py,'U'),generate_umath_py, IOError: [Errno 2] No such file or directory: 'core\\code_generators\\generate_umath.py' Can someone tell me how to solve this problem? There is no directory names code_generators in the code directory. Thanks Lefei From faltet at carabos.com Mon Jan 9 03:55:02 2006 From: faltet at carabos.com (Francesc Altet) Date: Mon Jan 9 03:55:02 2006 Subject: [Numpy-discussion] Re: [SciPy-dev] How to handle a[...] in numpy? In-Reply-To: References: Message-ID: <200601091254.10861.faltet@carabos.com> A Diumenge 08 Gener 2006 05:10, Sasha va escriure: > In Numeric a[...] would return an array unless a was 0-rank and a > python type othewise. What is the right way to do the same in numpy? [snip] > Proposal: Although I like a lot that 0-rank arrays and numpy scalar > types non-iterable, it may be reasonable to allow a[...]. This way > ellipsis can be interpereted as any number of ":"s including zero. > Another subscript operation that makes sense for scalars would be > a[...,newaxis] or even a[{newaxis, }* ..., {newaxis,}*], where > {newaxis,}* stands for any number of comma-separated newaxis tokens. > This will allow one to use ellipsis in generic code that would work on > any numpy type. I will contribute code if there is any interest. +1 More specifically, provided that: In [65]: type(numpy.array([0])[...]) Out[65]: In [66]: type(numpy.array([0])[0]) Out[66]: In [67]: type(numpy.array([0]).item()) Out[67]: I'd propose the next behaviour for 0-rank arrays: In [65]: type(numpy.array(0)[...]) Out[65]: In [66]: type(numpy.array(0)[()]) # Indexing a la numarray Out[66]: In [67]: type(numpy.array(0).item()) # already works Out[67]: -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From ndarray at mac.com Mon Jan 9 10:20:02 2006 From: ndarray at mac.com (Sasha) Date: Mon Jan 9 10:20:02 2006 Subject: [Numpy-discussion] Re: [SciPy-dev] How to handle a[...] in numpy? In-Reply-To: <200601091254.10861.faltet@carabos.com> References: <200601091254.10861.faltet@carabos.com> Message-ID: On 1/9/06, Francesc Altet wrote: > ... > I'd propose the next behaviour for 0-rank arrays: > > In [65]: type(numpy.array(0)[...]) > Out[65]: > > In [66]: type(numpy.array(0)[()]) # Indexing a la numarray > Out[66]: > I like the idea of supporting [()] on zero rank ndarrays, but I think it should be equivalent to [...]. I view [...] as [(slice(None),) * rank], and thus for rank=0, [...] is the same as [()]. Furthermore, I don't see any use for [...] that always returns the same array. As far as I remember in some old version of Numeric, [...] was a way to make a contiguous copy, but in numpy this is not the case (one needs to use copy method for that). >>> x[::2][...].flags {'WRITEABLE': True, 'UPDATEIFCOPY': False, 'CONTIGUOUS': False, 'FORTRAN': False, 'ALIGNED': True, 'OWNDATA': False} -- sasha From Chris.Barker at noaa.gov Mon Jan 9 10:51:06 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon Jan 9 10:51:06 2006 Subject: [Numpy-discussion] Re: [SciPy-user] numpy lists In-Reply-To: <43BF14E6.8070507@astraw.com> References: <200601051625.k05GPixf021407@oobleck.astro.cornell.edu> <43BD5DFA.20906@colorado.edu> <43BD6109.9030402@sympatico.ca> <43BD7A99.8070002@noaa.gov> <43BEFA6D.7090305@ieee.org> <43BF14E6.8070507@astraw.com> Message-ID: <43C2B09F.9040205@noaa.gov> Andrew Straw wrote: > Travis Oliphant wrote: >> numpy-devel at lists.sourceforge.net >> numpy-user at lists.sourceforge.net > You can subscribe to these lists from > http://sourceforge.net/mail/?group_id=1369 I don't see numpy-devel there. At least it's not obvious. I see: numpy-discussion (which is the list I'm posting too now, and has been around a long time for Numeric, then numarray, and now numpy) and: numpy-user (which does look like a new one) where can I find numpy-devel ? -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From strawman at astraw.com Mon Jan 9 10:55:02 2006 From: strawman at astraw.com (Andrew Straw) Date: Mon Jan 9 10:55:02 2006 Subject: [Numpy-discussion] Re: [SciPy-user] numpy lists In-Reply-To: <43C2B09F.9040205@noaa.gov> References: <200601051625.k05GPixf021407@oobleck.astro.cornell.edu> <43BD5DFA.20906@colorado.edu> <43BD6109.9030402@sympatico.ca> <43BD7A99.8070002@noaa.gov> <43BEFA6D.7090305@ieee.org> <43BF14E6.8070507@astraw.com> <43C2B09F.9040205@noaa.gov> Message-ID: <43C2B161.10009@astraw.com> Christopher Barker wrote: > Andrew Straw wrote: > >> Travis Oliphant wrote: >> >>> numpy-devel at lists.sourceforge.net >>> numpy-user at lists.sourceforge.net >> > >> You can subscribe to these lists from >> http://sourceforge.net/mail/?group_id=1369 > > > I don't see numpy-devel there. At least it's not obvious. I see: > > numpy-discussion (which is the list I'm posting too now, and has been > around a long time for Numeric, then numarray, and now numpy) > > and: > > numpy-user (which does look like a new one) > > where can I find numpy-devel ? It seems to be that numpy-devel has been (silently) deleted. I subscribed immediately after it was announced, tried posting a few days later, and got an error message. I can't remember exactly what the error was, but it seemed to be directly from sourceforge indicating problems with the numpy-devel list. So, I assume the list has simply been deleted. Is this so? From faltet at carabos.com Mon Jan 9 11:04:08 2006 From: faltet at carabos.com (Francesc Altet) Date: Mon Jan 9 11:04:08 2006 Subject: [Numpy-discussion] Re: [SciPy-dev] How to handle a[...] in numpy? In-Reply-To: References: <200601091254.10861.faltet@carabos.com> Message-ID: <200601092003.35687.faltet@carabos.com> A Dilluns 09 Gener 2006 19:19, Sasha va escriure: > On 1/9/06, Francesc Altet wrote: > > ... > > I'd propose the next behaviour for 0-rank arrays: > > > > In [65]: type(numpy.array(0)[...]) > > Out[65]: > > > > In [66]: type(numpy.array(0)[()]) # Indexing a la numarray > > Out[66]: > > I like the idea of supporting [()] on zero rank ndarrays, but I think > it should be equivalent to [...]. I view [...] as [(slice(None),) * > rank], and thus for rank=0, [...] is the same as [()]. However, the original aim of the "..." (ellipsis) operator is [taken for the numarray manual]: """ One final way of slicing arrays is with the keyword `...' This keyword is somewhat complicated. It stands for "however many `:' I need depending on the rank of the object I'm indexing, so that the indices I do specify are at the end of the index list as opposed to the usual beginning". So, if one has a rank-3 array A, then A[...,0] is the same thing as A[:,:,0], but if B is rank-4, then B[...,0] is the same thing as: B[:,:,:,0]. Only one `...' is expanded in an index expression, so if one has a rank-5 array C, then C[...,0,...] is the same thing as C[:,:,:,0,:]. """ > Furthermore, I don't see any use for [...] that always returns the > same array. As far as I remember in some old version of Numeric, > [...] was a way to make a contiguous copy, but in numpy this is not > the case (one needs to use copy method for that). I don't know for old versions of Numeric, but my impression is that the ellipsis meaning is clearly stated above. In fact, in a 4-dimensional array, say a, a[...] should be equivalent to a[:,:,:,:] and this does not necessarily implies a copy. Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From oliphant at ee.byu.edu Mon Jan 9 11:10:05 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Jan 9 11:10:05 2006 Subject: [Numpy-discussion] Re: [SciPy-user] numpy lists In-Reply-To: <43C2B09F.9040205@noaa.gov> References: <200601051625.k05GPixf021407@oobleck.astro.cornell.edu> <43BD5DFA.20906@colorado.edu> <43BD6109.9030402@sympatico.ca> <43BD7A99.8070002@noaa.gov> <43BEFA6D.7090305@ieee.org> <43BF14E6.8070507@astraw.com> <43C2B09F.9040205@noaa.gov> Message-ID: <43C2B4D9.8010307@ee.byu.edu> Christopher Barker wrote: > Andrew Straw wrote: > >> Travis Oliphant wrote: >> >>> numpy-devel at lists.sourceforge.net >>> numpy-user at lists.sourceforge.net >> > >> You can subscribe to these lists from >> http://sourceforge.net/mail/?group_id=1369 > > > I don't see numpy-devel there. At least it's not obvious. I see: > > numpy-discussion (which is the list I'm posting too now, and has been > around a long time for Numeric, then numarray, and now numpy) > > and: > > numpy-user (which does look like a new one) > > where can I find numpy-devel ? Sorry for the noise. I decided one numpy list was enough for now and hid the numpy-devel list. -Travis From alexander.belopolsky at gmail.com Mon Jan 9 12:37:05 2006 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Mon Jan 9 12:37:05 2006 Subject: [SciPy-dev] [Numpy-discussion] Re: How to handle a[...] in numpy? In-Reply-To: <200601092003.35687.faltet@carabos.com> References: <200601091254.10861.faltet@carabos.com> <200601092003.35687.faltet@carabos.com> Message-ID: On 1/9/06, Francesc Altet wrote: > However, the original aim of the "..." (ellipsis) operator is [taken > for the numarray manual]: > > """ > One final way of slicing arrays is with the keyword `...' This keyword > is somewhat complicated. It stands for "however many `:' I need This is precisely my motivation for making a[...] the same as a[()] for zero rank a. In this case "however many" is zero. In other words, a[...] is a short-hand for a[(slice(None),)*len(a.shape)]. Specifically, if a.shape = (), then a[...] = a[()]. > I don't know for old versions of Numeric, but my impression is that > the ellipsis meaning is clearly stated above. In fact, in a > 4-dimensional array, say a, a[...] should be equivalent to a[:,:,:,:] > and this does not necessarily implies a copy. I am not proposing any change for rank > 0 arrays, nor for the new numpy scalars. For a = array(0), why would you want a[...] have different type from a[()]? If as for rank-4 array a, a[...] should be equivalent to a[:,:,:,:] why would you expect a[...] for a rank-0 a be different from a[()]? -- sasha PS: There seems to be a terminological difficulty discussing this type of things. You call an array that takes 4 indices a 4-dimensional array, but in algebra 4-dimensional vector is a sequence of 4 numbers (array of shape (4,)). An object that is indexed by 4 numbers is a tensor of rank 4 (array of shape (n1, n2, n3, n4)). From ndarray at mac.com Mon Jan 9 19:23:03 2006 From: ndarray at mac.com (Sasha) Date: Mon Jan 9 19:23:03 2006 Subject: [SciPy-dev] [Numpy-discussion] Re: How to handle a[...] in numpy? In-Reply-To: References: <200601091254.10861.faltet@carabos.com> <200601092003.35687.faltet@carabos.com> Message-ID: Attached patch implements indexing of zero rank arrays: >>> x = array(42) >>> x[...] 42 >>> x[()] 42 >>> x[()].dtype >>> type(x[()]) Ellipsis and empty tuple are the only two valid indices. So far there was only one +1 vote for the feature (from Francesc Altet, who also suggested that x[()] be allowed). Does anyone object to this feature? I have also proposed to support x[...,newaxis] (see comment by sdementen at http://www.scipy.org/wikis/numdesign/). I will add this feature if there is interest. Finally, if we make x[...] valid for rank-0 arrays, should x[...] = value be allowed as well? I think it should, applying the principle of least surprized. An additional consideration is that rank-0 arrays are unhashable, but there is no obvious way to change their values. The following example show that rank-0 arrays are in fact mutable (in a non-obvious way): >>> x = array(1) >>> x.shape=(1,); x[0] = 42; x.shape=() >>> x Please comment on the following proposals: 1. x[...] 2. x[...] = value 3. x[..., newaxis] -- sasha From alexander.belopolsky at gmail.com Mon Jan 9 19:25:01 2006 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Mon Jan 9 19:25:01 2006 Subject: [SciPy-dev] [Numpy-discussion] Re: How to handle a[...] in numpy? In-Reply-To: References: <200601091254.10861.faltet@carabos.com> <200601092003.35687.faltet@carabos.com> Message-ID: Sorry forgot to attach. On 1/9/06, Sasha wrote: > Attached patch implements indexing of zero rank arrays: > > >>> x = array(42) > >>> x[...] > 42 > >>> x[()] > 42 > >>> x[()].dtype > > >>> type(x[()]) > > > Ellipsis and empty tuple are the only two valid indices. > > So far there was only one +1 vote for the feature (from Francesc > Altet, who also suggested that x[()] be allowed). Does anyone object > to this feature? > > I have also proposed to support x[...,newaxis] (see comment by > sdementen at http://www.scipy.org/wikis/numdesign/). I will add this > feature if there is interest. > > Finally, if we make x[...] valid for rank-0 arrays, should x[...] = > value be allowed as well? I think it should, applying the principle of > least surprized. An additional consideration is that rank-0 arrays > are unhashable, but there is no obvious way to change their values. > > The following example show that rank-0 arrays are in fact mutable (in > a non-obvious way): > > >>> x = array(1) > >>> x.shape=(1,); x[0] = 42; x.shape=() > >>> x > > Please comment on the following proposals: > > 1. x[...] > 2. x[...] = value > 3. x[..., newaxis] > > > -- sasha > -------------- next part -------------- Index: numpy/core/src/arrayobject.c =================================================================== --- numpy/core/src/arrayobject.c (revision 1863) +++ numpy/core/src/arrayobject.c (working copy) @@ -1796,6 +1796,9 @@ return NULL; } if (self->nd == 0) { + if (op == Py_Ellipsis || + PyTuple_Check(op) && 0 == PyTuple_GET_SIZE(op)) + return PyArray_ToScalar(self->data, self); PyErr_SetString(PyExc_IndexError, "0-d arrays can't be indexed."); return NULL; Index: numpy/core/tests/test_multiarray.py =================================================================== --- numpy/core/tests/test_multiarray.py (revision 1863) +++ numpy/core/tests/test_multiarray.py (working copy) @@ -69,5 +69,32 @@ d2 = dtypedescr('f8') assert_equal(d2, dtypedescr(float64)) +class test_zero_rank(ScipyTestCase): + def setUp(self): + self.d = array(0), array('x', object) - + def check_ellipsis_subscript(self): + a,b = self.d + + self.failUnlessEqual(a[...], 0) + self.failUnlessEqual(b[...].item(), 'x') + self.failUnless(type(a[...]) is a.dtype) + self.failUnless(type(b[...]) is b.dtype) + + def check_empty_subscript(self): + a,b = self.d + + self.failUnlessEqual(a[()], 0) + self.failUnlessEqual(b[()].item(), 'x') + self.failUnless(type(a[()]) is a.dtype) + self.failUnless(type(b[()]) is b.dtype) + + def check_invalid_subscript(self): + a,b = self.d + self.failUnlessRaises(IndexError, lambda x: x[0], a) + self.failUnlessRaises(IndexError, lambda x: x[0], b) + self.failUnlessRaises(IndexError, lambda x: x[array([], int)], a) + self.failUnlessRaises(IndexError, lambda x: x[array([], int)], b) + +if __name__ == "__main__": + ScipyTest('numpy.core.multiarray').run() From ndarray at mac.com Mon Jan 9 19:29:02 2006 From: ndarray at mac.com (Sasha) Date: Mon Jan 9 19:29:02 2006 Subject: [SciPy-dev] [Numpy-discussion] Re: How to handle a[...] in numpy? In-Reply-To: References: <200601091254.10861.faltet@carabos.com> <200601092003.35687.faltet@carabos.com> Message-ID: Sorry, forgot to attach. On 1/9/06, Sasha wrote: > Attached patch implements indexing of zero rank arrays: > > >>> x = array(42) > >>> x[...] > 42 > >>> x[()] > 42 > >>> x[()].dtype > > >>> type(x[()]) > > > Ellipsis and empty tuple are the only two valid indices. > > So far there was only one +1 vote for the feature (from Francesc > Altet, who also suggested that x[()] be allowed). Does anyone object > to this feature? > > I have also proposed to support x[...,newaxis] (see comment by > sdementen at http://www.scipy.org/wikis/numdesign/). I will add this > feature if there is interest. > > Finally, if we make x[...] valid for rank-0 arrays, should x[...] = > value be allowed as well? I think it should, applying the principle of > least surprized. An additional consideration is that rank-0 arrays > are unhashable, but there is no obvious way to change their values. > > The following example show that rank-0 arrays are in fact mutable (in > a non-obvious way): > > >>> x = array(1) > >>> x.shape=(1,); x[0] = 42; x.shape=() > >>> x > > Please comment on the following proposals: > > 1. x[...] > 2. x[...] = value > 3. x[..., newaxis] > > > -- sasha > -------------- next part -------------- Index: numpy/core/src/arrayobject.c =================================================================== --- numpy/core/src/arrayobject.c (revision 1863) +++ numpy/core/src/arrayobject.c (working copy) @@ -1796,6 +1796,9 @@ return NULL; } if (self->nd == 0) { + if (op == Py_Ellipsis || + PyTuple_Check(op) && 0 == PyTuple_GET_SIZE(op)) + return PyArray_ToScalar(self->data, self); PyErr_SetString(PyExc_IndexError, "0-d arrays can't be indexed."); return NULL; Index: numpy/core/tests/test_multiarray.py =================================================================== --- numpy/core/tests/test_multiarray.py (revision 1863) +++ numpy/core/tests/test_multiarray.py (working copy) @@ -69,5 +69,32 @@ d2 = dtypedescr('f8') assert_equal(d2, dtypedescr(float64)) +class test_zero_rank(ScipyTestCase): + def setUp(self): + self.d = array(0), array('x', object) - + def check_ellipsis_subscript(self): + a,b = self.d + + self.failUnlessEqual(a[...], 0) + self.failUnlessEqual(b[...].item(), 'x') + self.failUnless(type(a[...]) is a.dtype) + self.failUnless(type(b[...]) is b.dtype) + + def check_empty_subscript(self): + a,b = self.d + + self.failUnlessEqual(a[()], 0) + self.failUnlessEqual(b[()].item(), 'x') + self.failUnless(type(a[()]) is a.dtype) + self.failUnless(type(b[()]) is b.dtype) + + def check_invalid_subscript(self): + a,b = self.d + self.failUnlessRaises(IndexError, lambda x: x[0], a) + self.failUnlessRaises(IndexError, lambda x: x[0], b) + self.failUnlessRaises(IndexError, lambda x: x[array([], int)], a) + self.failUnlessRaises(IndexError, lambda x: x[array([], int)], b) + +if __name__ == "__main__": + ScipyTest('numpy.core.multiarray').run() From faltet at carabos.com Tue Jan 10 00:02:00 2006 From: faltet at carabos.com (Francesc Altet) Date: Tue Jan 10 00:02:00 2006 Subject: [SciPy-dev] [Numpy-discussion] Re: How to handle a[...] in numpy? In-Reply-To: References: <200601092003.35687.faltet@carabos.com> Message-ID: <200601100900.50025.faltet@carabos.com> A Dilluns 09 Gener 2006 21:36, Alexander Belopolsky va escriure: > On 1/9/06, Francesc Altet wrote: > > However, the original aim of the "..." (ellipsis) operator is [taken > > for the numarray manual]: > > > > """ > > One final way of slicing arrays is with the keyword `...' This keyword > > is somewhat complicated. It stands for "however many `:' I need > > This is precisely my motivation for making a[...] the same as a[()] for > zero rank a. In this case "however many" is zero. In other words, a[...] > is a short-hand > for a[(slice(None),)*len(a.shape)]. Specifically, if a.shape = (), > then a[...] = a[()]. I see your point. I was just proposing a way to return different objects for [...] and [()] that were consistent with higher dimensionality. So if one has a non-0 dimensional array (tensor) called a, then a[...] will return exactly the same object, so the same should be done with 0-d arrays (tensors) IMO. Now, the () index in 0d objects is just a way to substitute integer indexes in higher dimensional objects, and should return a scalar type, as it does in the later case. I recognize that this is my interpretation of the facts and chances are that I'm completely wrong with this. Anyway, sometimes I tend to think that this kind of issues are not very important, but from time to time they will certainly bit us, so perhaps this discussion would be useful for the community in the future. > > I don't know for old versions of Numeric, but my impression is that > > the ellipsis meaning is clearly stated above. In fact, in a > > 4-dimensional array, say a, a[...] should be equivalent to a[:,:,:,:] > > and this does not necessarily implies a copy. > > I am not proposing any change for rank > 0 arrays, nor for the new numpy > scalars. For a = array(0), why would you want a[...] have different > type from a[()]? > If as for rank-4 array a, a[...] should be equivalent to a[:,:,:,:] > why would you expect > a[...] for a rank-0 a be different from a[()]? For the reasons mentioned above: just to be consistent with higher dimensions (although as it turns out to be, the interpretation of "consistency" is different between both of us) and to have two different objects returned from [...] and [()]. > PS: There seems to be a terminological difficulty discussing this type > of things. You call an array that takes 4 indices a 4-dimensional > array, but in algebra 4-dimensional vector is a sequence of 4 numbers > (array of shape (4,)). An object that is indexed by 4 numbers is a > tensor of rank 4 (array of shape (n1, n2, n3, n4)). That's true. However, in the Numeric/numarray/numpy literature I think N-dimensional arrays are generally understood as tensors of rank-N. From the scipy manual ("Object Essentials" chapter): """ NumPy provides two fundamental objects: an N-dimensional array object (ndarray) and... """ But anyway, I think that, provided the context is clearly defined, the probability of being misled is (hopefully) quite low. Regards, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From humufr at yahoo.fr Tue Jan 10 07:22:01 2006 From: humufr at yahoo.fr (Humufr) Date: Tue Jan 10 07:22:01 2006 Subject: [Numpy-discussion] initialise an array Message-ID: <43C3D080.7060507@yahoo.fr> Hello, I have a function like this: def test(x,xall=None): if xall == None: xall =x I obtain this error message when I'm doing this with numpy (I don't have this problem with numarray and Numeric). ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all How can I do something similar? Do I have to use the function suggest in the error message? The function is something inside a long script with a lot of condition similar to this one so it's why I ask if it's normal to not have the special None parameter take in count. Thanks, N. From tim.hochberg at cox.net Tue Jan 10 08:47:05 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Tue Jan 10 08:47:05 2006 Subject: [Numpy-discussion] initialise an array In-Reply-To: <43C3D080.7060507@yahoo.fr> References: <43C3D080.7060507@yahoo.fr> Message-ID: <43C3E4CF.10206@cox.net> Humufr wrote: > Hello, > > I have a function like this: > > def test(x,xall=None): > if xall == None: xall =x > > I obtain this error message when I'm doing this with numpy (I don't > have this problem with numarray and Numeric). > > ValueError: The truth value of an array with more than one element is > ambiguous. Use a.any() or a.all > > How can I do something similar? Do I have to use the function suggest > in the error message? The function is something inside a long script > with a lot of condition similar to this one so it's why I ask if it's > normal to not have the special None parameter take in count. Use 'is', for example: if xall is None: xall = x -tim > > Thanks, > > N. > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log > files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > From faltet at carabos.com Tue Jan 10 10:59:13 2006 From: faltet at carabos.com (Francesc Altet) Date: Tue Jan 10 10:59:13 2006 Subject: [Numpy-discussion] Conversion of heterogeneous arrays in numpy<-->numarray Message-ID: <200601101958.32245.faltet@carabos.com> Hi, Is there any plan to provide a conversion between recarrays in numpy and numarray?. For the moment it does not seem to work: In [83]: r=numarray.records.array([(1,11,'a'),(2,22,'b')], formats='u1,f4,a1') In [84]: a=numpy.array(r, dtype='u1,f4,a1') In [85]: numarray.records.array(a) --------------------------------------------------------------------------- exceptions.ValueError Traceback (most recent call last) /tmp/ /usr/lib/python2.3/site-packages/numarray-1.5.0-py2.3-linux-i686.egg/numarray/records.py in array(buffer, formats, shape, names, byteorder, aligned) 406 byteorder=byteorder, aligned=aligned) 407 else: --> 408 raise ValueError("Unknown input type") 409 410 def _RecGetType(name): ValueError: Unknown input type In [86]: numpy.array(r) Out[86]: array([0Aa, ?Ab], dtype=(void,6)) However, if one specifies the format in for the numarray-->numpy it does seem to work: In [87]: numpy.array(r, dtype='u1,f4,a1') Out[87]: array([(1, 11.0, 'a'), (2, 22.0, 'b')], dtype=(void,6)) But not in the sense numpy-->numarray: In [88]: numarray.records.array(a, formats='u1,f4,a1') --------------------------------------------------------------------------- exceptions.ValueError Traceback (most recent call last) /tmp/ /usr/lib/python2.3/site-packages/numarray-1.5.0-py2.3-linux-i686.egg/numarray/records.py in array(buffer, formats, shape, names, byteorder, aligned) 406 byteorder=byteorder, aligned=aligned) 407 else: --> 408 raise ValueError("Unknown input type") 409 410 def _RecGetType(name): ValueError: Unknown input type Having this would be great for supporting heterogeneous arrays in PyTables. Regards, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From perry at stsci.edu Tue Jan 10 11:21:06 2006 From: perry at stsci.edu (Perry Greenfield) Date: Tue Jan 10 11:21:06 2006 Subject: [Numpy-discussion] Conversion of heterogeneous arrays in numpy<-->numarray In-Reply-To: <200601101958.32245.faltet@carabos.com> References: <200601101958.32245.faltet@carabos.com> Message-ID: <7d1c2d9b042dbfd73ac3e7ac273545b5@stsci.edu> I think it would be a good thing, but right now we are more focused on seeing what is involved in porting some of our libraries first. I'll try to find out how much work it would be though. Perry On Jan 10, 2006, at 1:58 PM, Francesc Altet wrote: > Hi, > > Is there any plan to provide a conversion between recarrays in numpy > and numarray?. For the moment it does not seem to work: > > In [83]: r=numarray.records.array([(1,11,'a'),(2,22,'b')], > formats='u1,f4,a1') > > In [84]: a=numpy.array(r, dtype='u1,f4,a1') > > In [85]: numarray.records.array(a) > ----------------------------------------------------------------------- > ---- > exceptions.ValueError Traceback (most > recent > call last) > > /tmp/ > > /usr/lib/python2.3/site-packages/numarray-1.5.0-py2.3-linux-i686.egg/ > numarray/records.py > in array(buffer, formats, shape, names, byteorder, aligned) > 406 byteorder=byteorder, aligned=aligned) > 407 else: > --> 408 raise ValueError("Unknown input type") > 409 > 410 def _RecGetType(name): > > ValueError: Unknown input type > > In [86]: numpy.array(r) > Out[86]: array([0Aa, ?Ab], dtype=(void,6)) > > However, if one specifies the format in for the numarray-->numpy it > does seem to work: > > In [87]: numpy.array(r, dtype='u1,f4,a1') > Out[87]: array([(1, 11.0, 'a'), (2, 22.0, 'b')], dtype=(void,6)) > > But not in the sense numpy-->numarray: > > In [88]: numarray.records.array(a, formats='u1,f4,a1') > ----------------------------------------------------------------------- > ---- > exceptions.ValueError Traceback (most > recent > call last) > > /tmp/ > > /usr/lib/python2.3/site-packages/numarray-1.5.0-py2.3-linux-i686.egg/ > numarray/records.py > in array(buffer, formats, shape, names, byteorder, aligned) > 406 byteorder=byteorder, aligned=aligned) > 407 else: > --> 408 raise ValueError("Unknown input type") > 409 > 410 def _RecGetType(name): > > ValueError: Unknown input type > > Having this would be great for supporting heterogeneous arrays in > PyTables. > > Regards, > > -- >> 0,0< Francesc Altet ? ? http://www.carabos.com/ > V V C?rabos Coop. V. ??Enjoy Data > "-" > > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log > files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://ads.osdn.com/?ad_idv37&alloc_id865&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From ndarray at mac.com Wed Jan 11 10:50:05 2006 From: ndarray at mac.com (Sasha) Date: Wed Jan 11 10:50:05 2006 Subject: [Numpy-discussion] Who will use numpy.ma? Message-ID: MA is intended to be a drop-in replacement for Numeric arrays that can explicitely handle missing observations. With the recent improvements to the array object in NumPy, the MA library has fallen behind. There are more than 50 methods in the ndarray object that are not present in ma.array. I would like to hear from people who work with datasets with missing observations? Do you use MA? Do you think with the support for nan's and replaceable mathematical operations, should missing observations be handled in numpy using special values rather than an array of masks? Thanks. -- sasha From konrad.hinsen at laposte.net Thu Jan 12 02:53:05 2006 From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net) Date: Thu Jan 12 02:53:05 2006 Subject: [Numpy-discussion] Re: [SciPy-user] [SciPy-dev] Long live to numpy (and its list as well) In-Reply-To: <200601052303.04502.faltet@carabos.com> References: <43BD6109.9030402@sympatico.ca> <43BD7A99.8070002@noaa.gov> <200601052303.04502.faltet@carabos.com> Message-ID: <6E760C38-F0F8-4137-B2A0-1FCF42F79FD3@laposte.net> On 05.01.2006, at 23:03, Francesc Altet wrote: > Accordingly, my vote is also: > > +1 numpy-discussion +1 from me as well. Same reason. Konrad. -- ------------------------------------------------------------------------ ------- Konrad Hinsen Laboratoire Leon Brillouin (CEA-CNRS), CEA Saclay, 91191 Gif-sur-Yvette Cedex, France Tel.: +33-1 69 08 79 25 Fax: +33-1 69 08 82 61 E-Mail: khinsen at cea.fr ------------------------------------------------------------------------ ------- From faltet at carabos.com Thu Jan 12 03:23:01 2006 From: faltet at carabos.com (Francesc Altet) Date: Thu Jan 12 03:23:01 2006 Subject: [Numpy-discussion] Re: [SciPy-user] [SciPy-dev] Long live to numpy (and its list as well) In-Reply-To: <6E760C38-F0F8-4137-B2A0-1FCF42F79FD3@laposte.net> References: <200601052303.04502.faltet@carabos.com> <6E760C38-F0F8-4137-B2A0-1FCF42F79FD3@laposte.net> Message-ID: <200601121221.55759.faltet@carabos.com> A Dijous 12 Gener 2006 11:52, konrad.hinsen at laposte.net va escriure: > On 05.01.2006, at 23:03, Francesc Altet wrote: > > Accordingly, my vote is also: > > > > +1 numpy-discussion > > +1 from me as well. Same reason. The outcome of the discussion is that it has been created a new list called numpy-user at lists.sf.net for discussing the new NumPy package. Apparently, the numpy-discussion will still be the place for discussing Numeric/numarray issues. Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From pjssilva at ime.usp.br Thu Jan 12 03:39:01 2006 From: pjssilva at ime.usp.br (Paulo J. S. Silva) Date: Thu Jan 12 03:39:01 2006 Subject: [Numpy-discussion] Two bugs in numpy 0.9.2 and a suggestion. Message-ID: <1137065869.16214.15.camel@localhost.localdomain> Hello, I have just installed numpy 0.9.2 in my system. Thanks for the good work. I think I got two bugs: 1) In the numpy.linalg method the cholesky decomposition doesn't work because the function triu is not available from numpy.core. If you add a line "from numpy import triu" on the top of the linalg.py file (with the other imports), it starts working. 2) If you try to instantiate a ndarray with weird parameters it can segfault and abort the interpreter. For example try to do the following: a = arange(10.) b = ndarray(a) Now the suggestion. What about adding a "transpose" property (in the Python parlance) to ndarray objects? I would call it "t". The following Python code explains what I mean: from numpy import * class Myndarray(ndarray): t = property(transpose, None, None) If you add this 3 lines to a file, import the Myndarray class, and create one of such arrays you will get transpose(a) by simply typing a.t. This allow for the following type of code (which I prefer): cost = dot(c.t, x) instead of cost = dot(transpose(c), x) or cost = dot(c.transpose(), x) Such a compact notation can be a blessing in code that needs lots of transpose (like the BFGS formula for example). If I knew more about writing C extensions for Python I would try to do it myself, but I don't. Is it possible to define properties in a C extension? Best, Paulo -- Paulo Jos? da Silva e Silva Professor Assistente do Dep. de Ci?ncia da Computa??o (Assistant Professor of the Computer Science Dept.) Universidade de S?o Paulo - Brazil e-mail: rsilva at ime.usp.br Web: http://www.ime.usp.br/~rsilva Teoria ? o que n?o entendemos o (Theory is something we don't) suficiente para chamar de pr?tica. (understand well enough to call practice) From joris at ster.kuleuven.be Thu Jan 12 04:44:03 2006 From: joris at ster.kuleuven.be (Joris De Ridder) Date: Thu Jan 12 04:44:03 2006 Subject: [Numpy-discussion] Who will use numpy.ma? Message-ID: <200601121343.13487.joris@ster.kuleuven.be> > With the recent improvements to the array object in NumPy, > the MA library has fallen behind. There are more than 50 methods > in the ndarray object that are not present in ma.array. I guess maintaining MA means double work, doesn't it? So, even if MA is updated now, in the future it's likely to be always somewhat behind. This is perhaps an argument against the use of this library? I wouldn't like to extensively use a library that will be phased out in a couple of years because it turns out to be somewhat redundant and/or behind w.r.t. the rest of numpy. The fact that with numpy you can do things like >>> from numpy import * >>> x = sin(1./arange(3.)) # arbitrary example >>> x[1] = nan # 'masking' an element >>> x = where(isnan(x), 10., x) # replacing masks by a number >>> x = where(x >= 10, nan, x) # clipping with nan replacement >>> y = sin(x) # sin(nan) = nan fulfils most of my needs. However, I haven't compared execution times for large arrays. Note that using NaNs, we don't distinguish between NaN and NA (not available). I am not sure this won't bite us somewhere in the future. I have a related question. numpy introduces the functions nanargmax(), nanargmin(), nanmax(), nanmin(), and nansum(). Was there a special reason to introduce extra nan... functions rather than adding an option like ignore_nan = True to the normal functions? Is this a performance issue? Will similar nan... equivalents be introduced for the functions mean() and reduce()? A side remark: >>> y = array([1, nan, 0.47]) >>> sort(y) array([ 1. , nan, 0.47 ]) No exception, no sorting. Is this a feature or a bug? :) J. Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm From bsouthey at gmail.com Thu Jan 12 06:01:10 2006 From: bsouthey at gmail.com (Bruce Southey) Date: Thu Jan 12 06:01:10 2006 Subject: [Numpy-discussion] Who will use numpy.ma? In-Reply-To: References: Message-ID: Hi, For any data collection in the real world, actual missing values occur very frequently - almost a certainity. For various operations there is probably no difference in what is really used, the main thing that comes to mind is the ability to separate those values that are actually missing i.e. unobserved from those that are obtained from mathematical functions like division by zero. However, it has been some time since I looked at the options so I am out-of-date. Perhaps the approach of the R language ( http://wwwr-project.org) may provide suitable approach to this. A second aspect of the masked arrays that is very neat is to be able to choose a masking value and it can be changed. This is a really feature that you don't realize how great it really is unless you have it! . It is very easy to identify and work with elements of the array that meet changing criteria just by changing the mask rather than a series of complex boolean operations and steps to get the same results. Regards Bruce On 1/11/06, Sasha wrote: > > MA is intended to be a drop-in replacement for Numeric arrays that can > explicitely handle missing observations. With the recent improvements > to the array object in NumPy, the MA library has fallen behind. There > are more than 50 methods in the ndarray object that are not present in > ma.array. > > I would like to hear from people who work with datasets with missing > observations? Do you use MA? Do you think with the support for nan's > and replaceable mathematical operations, should missing observations > be handled in numpy using special values rather than an array of > masks? > > Thanks. > > -- sasha > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log > files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://ads.osdn.com/?ad_idv37&alloc_id865&opclick > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cjw at sympatico.ca Thu Jan 12 06:30:04 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Thu Jan 12 06:30:04 2006 Subject: [Numpy-discussion] Two bugs in numpy 0.9.2 and a suggestion. In-Reply-To: <1137065869.16214.15.camel@localhost.localdomain> References: <1137065869.16214.15.camel@localhost.localdomain> Message-ID: <43C667AE.1010506@sympatico.ca> Paulo J. S. Silva wrote: >Hello, > >I have just installed numpy 0.9.2 in my system. Thanks for the good >work. I think I got two bugs: > >1) In the numpy.linalg method the cholesky decomposition doesn't work >because the function triu is not available from numpy.core. If you add a >line "from numpy import triu" on the top of the linalg.py file (with the >other imports), it starts working. > >2) If you try to instantiate a ndarray with weird parameters it can >segfault and abort the interpreter. For example try to do the following: > >a = arange(10.) >b = ndarray(a) > >Now the suggestion. What about adding a "transpose" property (in the >Python parlance) to ndarray objects? I would call it "t". > Or "T" to distinguish properties, which require no parentheses, from methods, whch do require parentheses? Colin W. From tim.hochberg at cox.net Thu Jan 12 06:51:09 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu Jan 12 06:51:09 2006 Subject: [Numpy-discussion] Two bugs in numpy 0.9.2 and a suggestion. In-Reply-To: <1137065869.16214.15.camel@localhost.localdomain> References: <1137065869.16214.15.camel@localhost.localdomain> Message-ID: <43C66CA5.9050005@cox.net> Paulo J. S. Silva wrote: [SNIP] >Now the suggestion. What about adding a "transpose" property (in the >Python parlance) to ndarray objects? I would call it "t". The following >Python code explains what I mean: > >from numpy import * >class Myndarray(ndarray): > t = property(transpose, None, None) > >If you add this 3 lines to a file, import the Myndarray class, and >create one of such arrays you will get transpose(a) by simply typing >a.t. This allow for the following type of code (which I prefer): > >cost = dot(c.t, x) > >instead of > >cost = dot(transpose(c), x) > >or > >cost = dot(c.transpose(), x) > >Such a compact notation can be a blessing in code that needs lots of >transpose (like the BFGS formula for example). > >If I knew more about writing C extensions for Python I would try to do >it myself, but I don't. Is it possible to define properties in a C >extension? > > Wouldn't this be more appropriate in Matrix (I'm assuming numpy has an equivalent to Numeric/Numarray's Matrix class)? For general N-D arrays I rarely find transpose without axes arguments to be useful, so for me this would would just amount to extra cruft. -tim >Best, > >Paulo > > From aisaac at american.edu Thu Jan 12 06:54:03 2006 From: aisaac at american.edu (Alan G Isaac) Date: Thu Jan 12 06:54:03 2006 Subject: [Numpy-discussion] Two bugs in numpy 0.9.2 and a suggestion. In-Reply-To: <43C667AE.1010506@sympatico.ca> References: <1137065869.16214.15.camel@localhost.localdomain> <43C667AE.1010506@sympatico.ca> Message-ID: Paulo wrote: >> Now the suggestion. What about adding a "transpose" property (in the >> Python parlance) to ndarray objects? I would call it "t". On Thu, 12 Jan 2006, "Colin J. Williams" apparently wrote: > Or "T" to distinguish properties, You mean like this? Or something else? Alan Isaac >>> import numpy as N >>> x = N.mat([[1,2,3],[4,5,6]]) >>> y = x.T >>> y matrix([[1, 4], [2, 5], [3, 6]]) From service at inl.ebay.com Thu Jan 12 08:39:10 2006 From: service at inl.ebay.com (eBay) Date: Thu Jan 12 08:39:10 2006 Subject: [Numpy-discussion] Important eBay! Account Information. Message-ID: An HTML attachment was scrubbed... URL: From haase at msg.ucsf.edu Thu Jan 12 09:14:02 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Thu Jan 12 09:14:02 2006 Subject: [Numpy-discussion] Re: [SciPy-user] [SciPy-dev] Long live to numpy (and its list as well) In-Reply-To: <200601121221.55759.faltet@carabos.com> References: <6E760C38-F0F8-4137-B2A0-1FCF42F79FD3@laposte.net> <200601121221.55759.faltet@carabos.com> Message-ID: <200601120913.25890.haase@msg.ucsf.edu> On Thursday 12 January 2006 03:21, Francesc Altet wrote: > A Dijous 12 Gener 2006 11:52, konrad.hinsen at laposte.net va escriure: > > On 05.01.2006, at 23:03, Francesc Altet wrote: > > > Accordingly, my vote is also: > > > > > > +1 numpy-discussion > > > > +1 from me as well. Same reason. > > The outcome of the discussion is that it has been created a new list > called numpy-user at lists.sf.net for discussing the new NumPy package. > Apparently, the numpy-discussion will still be the place for > discussing Numeric/numarray issues. > I'm sorry to disagree, I really think there are to many lists now ! The start of this "lists discussion" was about _reducing_ the total number of scipy/numpy lists, now we have two more ! Obviously there are going to be lots of posts regarding "transition from Numeric/numarray to NumPy" -- are they all supposed to cross post !? My vote (and that's how I read Konrad's - correct my if I'm wrong) is to continue using numpy-discussion and NOT have another lists (numpy-user) . My two cents, Sebastian Haase From konrad.hinsen at laposte.net Thu Jan 12 09:41:04 2006 From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net) Date: Thu Jan 12 09:41:04 2006 Subject: [Numpy-discussion] Re: [SciPy-user] [SciPy-dev] Long live to numpy (and its list as well) In-Reply-To: <200601120913.25890.haase@msg.ucsf.edu> References: <6E760C38-F0F8-4137-B2A0-1FCF42F79FD3@laposte.net> <200601121221.55759.faltet@carabos.com> <200601120913.25890.haase@msg.ucsf.edu> Message-ID: <24C944F6-95F8-4160-B3FD-4C3709591D2C@laposte.net> On 12.01.2006, at 18:13, Sebastian Haase wrote: > Obviously there are going to be lots of posts regarding "transition > from > Numeric/numarray to NumPy" -- are they all supposed to cross post !? > > My vote (and that's how I read Konrad's - correct my if I'm wrong) > is to > continue using numpy-discussion and NOT have another lists (numpy- > user) . I agree. It makes sense to have a separate developer list for those who work *on* NumPy, but please let's stick to a single list for users of all Python array packages. Separate lists will encourage a split in the community, and create lots of crosspostings. Konrad. -- ------------------------------------------------------------------------ ------- Konrad Hinsen Laboratoire Leon Brillouin (CEA-CNRS), CEA Saclay, 91191 Gif-sur-Yvette Cedex, France Tel.: +33-1 69 08 79 25 Fax: +33-1 69 08 82 61 E-Mail: khinsen at cea.fr ------------------------------------------------------------------------ ------- From cjw at sympatico.ca Thu Jan 12 09:52:11 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Thu Jan 12 09:52:11 2006 Subject: [Numpy-discussion] Two bugs in numpy 0.9.2 and a suggestion. In-Reply-To: References: <1137065869.16214.15.camel@localhost.localdomain> <43C667AE.1010506@sympatico.ca> Message-ID: <43C6970F.4080209@sympatico.ca> Alan G Isaac wrote: >Paulo wrote: > > >>>Now the suggestion. What about adding a "transpose" property (in the >>>Python parlance) to ndarray objects? I would call it "t". >>> >>> > > >On Thu, 12 Jan 2006, "Colin J. Williams" apparently wrote: > > >>Or "T" to distinguish properties, >> >> > >You mean like this? Or something else? >Alan Isaac > > > >>>>import numpy as N >>>>x = N.mat([[1,2,3],[4,5,6]]) >>>>y = x.T >>>>y >>>> >>>> >matrix([[1, 4], > [2, 5], > [3, 6]]) > > > > Thanks. Yes, exactly. Colin W. > >------------------------------------------------------- >This SF.net email is sponsored by: Splunk Inc. Do you grep through log files >for problems? Stop! Download the new AJAX search engine that makes >searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! >http://ads.osdn.com/?ad_idv37&alloc_id865&op=click >_______________________________________________ >Numpy-discussion mailing list >Numpy-discussion at lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > From strawman at astraw.com Thu Jan 12 09:56:21 2006 From: strawman at astraw.com (Andrew Straw) Date: Thu Jan 12 09:56:21 2006 Subject: [Numpy-discussion] Re: [SciPy-user] [SciPy-dev] Long live to numpy (and its list as well) In-Reply-To: <24C944F6-95F8-4160-B3FD-4C3709591D2C@laposte.net> References: <6E760C38-F0F8-4137-B2A0-1FCF42F79FD3@laposte.net> <200601121221.55759.faltet@carabos.com> <200601120913.25890.haase@msg.ucsf.edu> <24C944F6-95F8-4160-B3FD-4C3709591D2C@laposte.net> Message-ID: <43C6982B.2040407@astraw.com> konrad.hinsen at laposte.net wrote: > On 12.01.2006, at 18:13, Sebastian Haase wrote: > >> Obviously there are going to be lots of posts regarding "transition >> from >> Numeric/numarray to NumPy" -- are they all supposed to cross post !? >> >> My vote (and that's how I read Konrad's - correct my if I'm wrong) >> is to >> continue using numpy-discussion and NOT have another lists (numpy- >> user) . > > > I agree. It makes sense to have a separate developer list for those > who work *on* NumPy, but please let's stick to a single list for > users of all Python array packages. Separate lists will encourage a > split in the community, and create lots of crosspostings. I agree as well. In the last days, there have been several posts by authors who clearly missed ongoing discussions following from an original cross-post because further discussion took place exclusively on an unsubscribed list. Can we shut down numpy-user? It's only been around a couple weeks, and I bet everyone subscribed there is also on numpy-discussion. Cheers! Andrew From Chris.Barker at noaa.gov Thu Jan 12 09:59:02 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu Jan 12 09:59:02 2006 Subject: [Numpy-discussion] Re: [SciPy-user] [SciPy-dev] Long live to numpy (and its list as well) In-Reply-To: <200601120913.25890.haase@msg.ucsf.edu> References: <6E760C38-F0F8-4137-B2A0-1FCF42F79FD3@laposte.net> <200601121221.55759.faltet@carabos.com> <200601120913.25890.haase@msg.ucsf.edu> Message-ID: <43C69895.4050102@noaa.gov> Sebastian Haase wrote: > now we have two more ! I think it's only one: numpy-dev got dropped. > Obviously there are going to be lots of posts regarding "transition from > Numeric/numarray to NumPy" -- are they all supposed to cross post !? No, they should post to numpy-user. If it's about numpy, post there. Period. If it's about Numeric or numarray, and NOT numpy, then post to the old list. As I imagine there are a lot of folks that are not interested in the latest developments, and are "just using" one of the older packages, then the old list has a valuable place. Personally, I filter them both to the same place, so it makes little difference to me, though all that cross-posting gets annoying. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Fernando.Perez at colorado.edu Thu Jan 12 10:20:06 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Thu Jan 12 10:20:06 2006 Subject: [Numpy-discussion] Re: [SciPy-user] [SciPy-dev] Long live to numpy (and its list as well) In-Reply-To: <43C69895.4050102@noaa.gov> References: <6E760C38-F0F8-4137-B2A0-1FCF42F79FD3@laposte.net> <200601121221.55759.faltet@carabos.com> <200601120913.25890.haase@msg.ucsf.edu> <43C69895.4050102@noaa.gov> Message-ID: <43C69D96.603@colorado.edu> Christopher Barker wrote: > Sebastian Haase wrote: > >>now we have two more ! > > > I think it's only one: numpy-dev got dropped. > > >>Obviously there are going to be lots of posts regarding "transition from >>Numeric/numarray to NumPy" -- are they all supposed to cross post !? > > > No, they should post to numpy-user. If it's about numpy, post there. > Period. If it's about Numeric or numarray, and NOT numpy, then post to > the old list. As I imagine there are a lot of folks that are not > interested in the latest developments, and are "just using" one of the > older packages, then the old list has a valuable place. I don't really have an opinion on this matter, and have run out of mental bandwith to follow it. But I'd like to ask that, when the dust settles, could the powers-that-be please put up a clear notice on the new wiki as to what the lists are, and the purpose of each, with links to the relevant subscription pages? Cheers, f From pjssilva at ime.usp.br Thu Jan 12 10:41:07 2006 From: pjssilva at ime.usp.br (Paulo J. S. Silva) Date: Thu Jan 12 10:41:07 2006 Subject: [Numpy-discussion] Two bugs in numpy 0.9.2 and a suggestion. In-Reply-To: References: <1137065869.16214.15.camel@localhost.localdomain> <43C667AE.1010506@sympatico.ca> Message-ID: <1137091204.16214.26.camel@localhost.localdomain> Thank Alan and David for calling my attention back to the matrix object in numpy/numarray. I usually avoided using it because my first contact comes from Numeric where the Matrix objects don't play nicely with ufunctions: In [1]:from Numeric import * In [2]:from Matrix import Matrix In [3]:a = Matrix([1.,2,3]) In [4]:sin(a) Out[4]:array([ [ 0.84147098, 0.90929743, 0.14112001]]) See... The sine of a Matrix was not a Matrix anymore. I didn't realize this was fixed in numarray: In [1]:from numarray import * In [2]:from numarray.matrix import Matrix In [3]:a = Matrix([1.,2,3]) In [4]:sin(a) Out[4]:_Matrix([ 0.84147098, 0.90929743, 0.14112001]) And numpy keeps the nice behavior: In [1]:from numpy import * In [2]:a = matrix([1.,2,3]) In [3]:sin(a) Out[3]:matrix([[ 0.84147098, 0.90929743, 0.14112001]]) Great! Best, Paulo From cookedm at physics.mcmaster.ca Thu Jan 12 12:41:07 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Thu Jan 12 12:41:07 2006 Subject: [Numpy-discussion] Long live to numpy (and its list as well) In-Reply-To: <43C6982B.2040407@astraw.com> References: <6E760C38-F0F8-4137-B2A0-1FCF42F79FD3@laposte.net> <200601121221.55759.faltet@carabos.com> <200601120913.25890.haase@msg.ucsf.edu> <24C944F6-95F8-4160-B3FD-4C3709591D2C@laposte.net> <43C6982B.2040407@astraw.com> Message-ID: On Jan 12, 2006, at 12:55 , Andrew Straw wrote: > konrad.hinsen at laposte.net wrote: > >> On 12.01.2006, at 18:13, Sebastian Haase wrote: >> >>> Obviously there are going to be lots of posts regarding "transition >>> from >>> Numeric/numarray to NumPy" -- are they all supposed to cross post !? >>> >>> My vote (and that's how I read Konrad's - correct my if I'm wrong) >>> is to >>> continue using numpy-discussion and NOT have another lists (numpy- >>> user) . >> >> >> I agree. It makes sense to have a separate developer list for those >> who work *on* NumPy, but please let's stick to a single list for >> users of all Python array packages. Separate lists will encourage a >> split in the community, and create lots of crosspostings. > > I agree as well. In the last days, there have been several posts by > authors who clearly missed ongoing discussions following from an > original cross-post because further discussion took place > exclusively on > an unsubscribed list. Can we shut down numpy-user? It's only been > around > a couple weeks, and I bet everyone subscribed there is also on > numpy-discussion. I'd have to agree too. For one thing, the names are too similar: what really is the difference between -discussion and -user? numpy- discussion has a long history: this includes archives on sourceforge.net, gmane.org, and elsewhere, which gives a one-stop shop for searching archives, instead of having to remember to look up (or forgetting to look up) in several lists. It's also better for newcomers to only have one obvious list to post to. At the moment, I'd also say numpy is still closely linked with scipy in terms of developers, so a separate numpy-dev list would be premature. -- |>|\/|< /------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From oliphant.travis at ieee.org Thu Jan 12 13:00:06 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu Jan 12 13:00:06 2006 Subject: [Numpy-discussion] Re: [SciPy-user] [SciPy-dev] Long live to numpy (and its list as well) In-Reply-To: <24C944F6-95F8-4160-B3FD-4C3709591D2C@laposte.net> References: <6E760C38-F0F8-4137-B2A0-1FCF42F79FD3@laposte.net> <200601121221.55759.faltet@carabos.com> <200601120913.25890.haase@msg.ucsf.edu> <24C944F6-95F8-4160-B3FD-4C3709591D2C@laposte.net> Message-ID: <43C6C335.3030903@ieee.org> konrad.hinsen at laposte.net wrote: > Separate lists will encourage a split in the community, and create > lots of crosspostings. This convinced me... I closed the numpy-user list (already closed the numpy-devel list). Please make all numpy-related posts to numpy-discussion at lists.sourceforge.net It would help if users would make clear in the subject if they are not posting regarding current NumPy, but that their post is specific to Numarray and/or Numeric. -Travis From oliphant.travis at ieee.org Thu Jan 12 13:23:05 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu Jan 12 13:23:05 2006 Subject: [Numpy-discussion] Re: [Numpy-user] Optional shape in array description In-Reply-To: <200601121945.06205.faltet@carabos.com> References: <200601121945.06205.faltet@carabos.com> Message-ID: <43C6C8A5.1050208@ieee.org> Francesc Altet wrote: >Hi, > >I went over a weirdness on interpretation of the optional shape in the >descriptor of types when declaring records. > >In [176]: numpy.zeros((2,), dtype=[('c1', 'f4')]) >Out[176]: array([(0.0,), (0.0,)], dtype=(void,4)) > >So far, so good. However: > >In [177]: numpy.zeros((2,), dtype=[('c1', 'f4', 1)]) >Out[177]: array([(array([ 0.], dtype=float32),), (array([ 0.], >dtype=float32),)], dtype=(void,4)) > >And: > >In [178]: numpy.zeros((2,), dtype=[('c1', 'f4', (1,))]) >Out[178]: array([(array([ 0.], dtype=float32),), (array([ 0.], >dtype=float32),)], dtype=(void,4)) > >i.e. the very same value is obtained specifying shape=1 and shape=(1,). > >OTOH, in the point 3 of the description of the __array_descr__ of the >array protocol (http://numeric.scipy.org/array_interface.html), one >can read: > >""" >An optional shape tuple providing how many times this part of the >record should be repeated. Assumed (1,) if not given. >""" > >But this is clearly not what it is implemented. > >IMO, the correct thing (and what the numarray does) would be: > >In [176]: numpy.zeros((2,), dtype=[('c1', 'f4')]) >Out[176]: array([(0.0,), (0.0,)], dtype=(void,4)) >In [192]: numpy.zeros((2,), dtype=[('c1', 'f4')])[0][0] >Out[192]: 0.0 > >In [177]: numpy.zeros((2,), dtype=[('c1', 'f4', 1)]) >Out[177]: array([(0.0,), (0.0,)], dtype=(void,4)) >In [193]: numpy.zeros((2,), dtype=[('c1', 'f4', 1)])[0][0] >Out[193]: 0.0 > >In [178]: numpy.zeros((2,), dtype=[('c1', 'f4', (1,))]) >Out[178]: array([(array([ 0.], dtype=float32),), (array([ 0.], >dtype=float32),)], dtype=(void,4)) >In [194]: numpy.zeros((2,), dtype=[('c1', 'f4', (1,))])[0][0] >Out[194]: array([ 0.], dtype=float32) > >i.e. that shape=1 would be equivalent to don't specify it (scalar >case), and shape=(1,) would be equivalent to declare the field as >unidimensional but with one element. > > This is what is now done in SVN... Thanks for catching the corner case. We should also change the verbage on the array_protocol page. -Travis From ndarray at mac.com Thu Jan 12 13:30:05 2006 From: ndarray at mac.com (Sasha) Date: Thu Jan 12 13:30:05 2006 Subject: [Numpy-discussion] Who will use numpy.ma? In-Reply-To: References: <43C5E722.2040007@pfdubois.com> Message-ID: With Paul's permission I am posting his arguments and my responses. Numpy.ma will follow Paul's design and there is now a wiki page dedicated to the effort to make ma work better in numpy. (See http://projects.scipy.org/scipy/numpy/wiki/MaskedArray). -- sasha On 1/12/06, Sasha wrote: > Paul, > > Thank you very much for your insights and once again thanks for all > the great work that you've done. I've noticed that your reply was not > posted on any list, do you mind if I forward it to numpy-user? Please > see more below. > > On 1/12/06, Paul F. Dubois wrote: > > What special values? Are you sure this works on any platform? What, for > > example, is the special value for integer arrays? For arrays of objects? > > > Yes, these are hard questions. For floats nan is an obvious choice and > IEEE support is getting better on the new hardware. For objects None is > a fine choice. For integers some may argue for sys.maxint, and given that > numpy integer arithmetics is already handling overflow a check for maxint will > not add much complexity. Yet don't get me wrong: I don't see any replacement > for ma myself. > > > How do replaceable mathematical operations make any difference? The > > fundamental problem is that if array x has special values in some places > > and array y has them in some other places, how do you create a result > > that has special values in the correct places AND is of a type for which > > those special values are still treated as 'missing'. How do you do this? > > > > Replaceable operations would allow one to redefine all operations on integer > arrays to treat say sys.maxint as invariant and cast it to nan in floating point > conversions without changing the logic of main line numpy. > > > I converted MA to ma but did not have time to flesh out all the > > differences with the new ndarray. I was hoping the community would do > > that. > > Me too. That's was the point of my post - to find out the size of the comunity > rather than to suggest an alternative. > > > I am retired. > > > You deserve it. > > > It is my belief that the approach you outline is not workable, but > > perhaps I am not understanding it properly. > > > I don't have any workable approach other than enchancing ma to work > better with numpy. This is what I am doing right now. > > > If I, who have thought about this a lot, do not know for sure, what > > information can you derive from a poll of the general public, who will > > not think through these issues very carefully? > > > I was trying to poll numpy community to find out how many people actually > use ma in real projects. This would determine how well tested the new features > will be and how quickly any bugs will be discovered and fixed. > Unfortunately, I have > not seen a single response saying - I've been using MA for X years on > Y projects and > plan on using it as we upgrade to numpy. There was a lot of > theoretical discussions > and a pointer to a plotting library that has recently added MA > support, but no testimony > from end users. > > > I am close to absolutely positive that subclassing won't particularly > > ease the task. > > > I thought about this a little, and I think you are right. Subclassing > may improve speed a little, but all methods will need to be adapted > the same ways as it is done without subclassing. > > > For the reason I indicated, I don't care to engage in public discussions > > of complex technical issues so I have not cc'd this to the group. > > > I respect that, but please allow me to forward at least portions of > this correspondence > to the community. Your insights are invaluable. > > -- sasha > > > > > Sasha wrote: > > > MA is intended to be a drop-in replacement for Numeric arrays that can > > > explicitely handle missing observations. With the recent improvements > > > to the array object in NumPy, the MA library has fallen behind. There > > > are more than 50 methods in the ndarray object that are not present in > > > ma.array. > > > > > > I would like to hear from people who work with datasets with missing > > > observations? Do you use MA? Do you think with the support for nan's > > > and replaceable mathematical operations, should missing observations > > > be handled in numpy using special values rather than an array of > > > masks? > > > > > > Thanks. > > > > > > -- sasha > > > > > > > > > ------------------------------------------------------- > > > This SF.net email is sponsored by: Splunk Inc. Do you grep through log files > > > for problems? Stop! Download the new AJAX search engine that makes > > > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > > > http://ads.osdn.com/?ad_idv37&alloc_id865&op=click > > > _______________________________________________ > > > Numpy-discussion mailing list > > > Numpy-discussion at lists.sourceforge.net > > > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > > > > From archetype at virtualcountries.com Thu Jan 12 16:36:05 2006 From: archetype at virtualcountries.com (=?windows-1251?B?MjYtMjcg/+3i4PD/IDIwMDYg4+7k4A==?=) Date: Thu Jan 12 16:36:05 2006 Subject: [Numpy-discussion] =?windows-1251?B?1Ojt4O3x+yDk6/8g8u7vLezl7eXk5uXw7uI=?= Message-ID: ??????? ??? ???-?????????? 26-27 ?????? 2006 ???? ?????? ???????????? ???????? ? ?? ????????? ????????????? ?????????? ? ?????????? ???????? ??????? ?????? ?????, ????? ????????? ??????. ????? ???, ??? ?????????? ?????????? ????????? ??????? ?????? ? ????? ??????? ??????????? ??????????. ?????? ??????? ???????? ???????? ????? ??????????? ???????????, ? ????? ??? ????????????? ????????????, ??????? ????????? ?????????? ????????? ?????????? ????????? ????????, ??????????? ?????????? ??????, ????????? ?????????? ??????? ? - ??? ????????? - ????? ???????? ? ???????? ??????. ? ????????? ??????? ???????? ????? ??????????? ?????? ?????? ??????? ??????????? ??????? ????????, ?????? ??????? ?????????? ???????????; ?????????? ??????? ??????????????? ?????, ?????????? ??????????? ???????? ?? ???? ?????? ???????? ????????. ????????? ???????????? ? ???????? ?????????? ????????? ? ???????, ???????? ?????? ?????????? ?????????, ?????? ???????? ?????? ??????????????, ??????????? ???????????? ? ??????? ??? ???????? ????????? ?????????????? ???????. ?????????? ?????? ????? ?????????? ?? ??????? ???????????? ???????? (??????) ? ? ??????? ?????. ????????? ???????? ?????????? ??????: - ?????????? ?????????? ????? ???????, ?????????? ?????????? ?????????, - ??????????? ? ???????????? ???????? ???? ???????, - ?????? ? ???????? ????????, ??????? ? ???????? ? ???????, - ?????????? ????????? ???????? ???????, - ???????????? ? ??????? ?????????? ???????????, - ???????? ?????????????? ??????? ? ???????? ???????????????? ?????, - ????????? ???????????? ???????? ?? ???????? ?????????? ??????????, - ????????????? ???????? ?????????? ??? ???????? ???????? ??????? ???????, - ????????????? ????????????? ????? ????? ??????????, - ??????????????? ??????? ?????? ????? ?????????? ??????? ? ?????????????? ?? ??????????. ?????????? ?????????: 1. ??????? ?????????????? ??????????? ???????????. 2. ?????????????? ???? ? ????? ? ????????. 3. ???????????? ? ?????????? ????????? ?????. 4. ??????????? ?????????? ??????????. 5. ?????????? ????????? ????????. 6. ?????? ??????????? ????????? ?? ?????? ???????. 7. ???????? ?????? ???????. 8. ?????????? ????? ??????????? ???????. 9. ?????????? ? ????????????? ??????????. 10. ?????????? ????????? ????????. 11. ?????????? ?????????? ????????. 12. ????????? ????????????? ????????????? ???????? ?????????????. 13. ?????????? ??????? ? ????????. 14. ?????????????? ??? ???????? ???????? ??????????? ?????????? ? ????????????. ?????? ????????? ?? ?????? ???????? ? ????????????? ????????, ? ?????? ????????? ?????? ??????? ? ?????? ?????????? ??????. ???????? ????????? ?????????? ?? ?????? ?? ?????????: (095) 980-65-?9, 980-65-?6 -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Thu Jan 12 16:47:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu Jan 12 16:47:02 2006 Subject: [Numpy-discussion] Who will use numpy.ma? In-Reply-To: <200601121343.13487.joris@ster.kuleuven.be> References: <200601121343.13487.joris@ster.kuleuven.be> Message-ID: <43C6F860.20701@ieee.org> Joris De Ridder wrote: >>>>y = array([1, nan, 0.47]) >>>>sort(y) >>>> >>>> >array([ 1. , nan, 0.47 ]) > >No exception, no sorting. Is this a feature or a bug? :) > > It's a "problem" that the user needs to be aware of, but I'm not sure how to fix. Notice that the same problem exists with python try: b =y.tolist() b.sort() Same effect. The problem may be that nans are always False on comparison with another object and this ends up confusing the sort functions. To fix it we would have to write a special nan-sort or else remove the nans entirely, but I don't think we should check for nans for every sort of floating-point values, because it would be slower. So, the user needs to do that. -Travis From cjw at sympatico.ca Thu Jan 12 18:30:01 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Thu Jan 12 18:30:01 2006 Subject: [Numpy-discussion] Overlapping websites? Message-ID: <43C7106E.1070103@sympatico.ca> There are now: http://www.numpy.org and http://sourceforge.net/projects/numpy/ These seem to be synonyms. Are two URI's needed? Numeric 24, not that I use it, seems to have been lost. Colin W. From ariciputi at pito.com Fri Jan 13 01:04:02 2006 From: ariciputi at pito.com (Andrea Riciputi) Date: Fri Jan 13 01:04:02 2006 Subject: [Numpy-discussion] Numpy and the old scipy Message-ID: <7875CF3B-37D2-4BC1-9757-DAFFA9DF8B9A@pito.com> Hi all, I'm a long time Numeric and SciPy user and I've been following with interest the development of the new Numpy package since last year when Travis started rewriting Numeric (thanks a lot Travis!). Now that Numpy seems in a quite-ready state I'd like to give it a try (and hopefully help the developer in the bug fixing cycle). However I remember that in its early days (when it was still named scipy_core) it was not possible to install both the old scipy version (I mean version 0.3.2) and the new one. Since I use both Numeric and SciPy (the old one) on a daily basis for my job, I'd like to know if it's still the case, or if it is now possible to install the new numpy package side by side with the old scipy (and Numeric as well). Thanks, Andrea. From oliphant.travis at ieee.org Fri Jan 13 01:37:00 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri Jan 13 01:37:00 2006 Subject: [Numpy-discussion] Numpy and the old scipy In-Reply-To: <7875CF3B-37D2-4BC1-9757-DAFFA9DF8B9A@pito.com> References: <7875CF3B-37D2-4BC1-9757-DAFFA9DF8B9A@pito.com> Message-ID: <43C7747B.50302@ieee.org> Andrea Riciputi wrote: > > Since I use both Numeric and SciPy (the old one) on a daily basis for > my job, I'd like to know if it's still the case, or if it is now > possible to install the new numpy package side by side with the old > scipy (and Numeric as well). You can install NumPy and Numeric side-by-side just fine. They use different name spaces. In order to install new and old scipy you will definitely need to install one of them to another location besides site-packages (probably new scipy). Then you will need to make sure your sys.path is set up properly to find the scipy you are interested in using for that session. In principle new scipy is almost identical to old scipy except internally it uses NumPy instead of Numeric. So, once you convert from Numeric to NumPy, converting to the new scipy is done (it depends on if you regularly imported scipy_base or scipy_distutils. If you did those are gone. scipy_base--->numpy and scipy_distutils--->scipy.distutils. Keep us posted as to the difficulties as they arise... -Travis From annale at wp.pl Fri Jan 13 03:03:05 2006 From: annale at wp.pl (=?windows-1251?B?x+D35ewg6/7k6CDw4OHu8uD+8g==?=) Date: Fri Jan 13 03:03:05 2006 Subject: [Numpy-discussion] =?windows-1251?B?yuDqIOL75/vi4PL8IPMg6/7k5ekg5uXr4O3o5SDw4OHu8uDy/D8=?= Message-ID: "??? ?? ????? - ???? ???????, ??? ????? - ???? ???????????:" (???????? ????????) ?????? ???? ????????? ????? ???? ????????? ?????? ???? ?????? - ? ??????, ? ?????? - ????? ????? ??? ???????? ??????? ? ??? ?????? ???????????? ????????? ? ?????????? ? ???????? ?????????? ??? ???????? ??? ???????? ?????????? ?????? ??? ?????????? ??? ??????? "?????????????? ?????????"? ?????? ???? "?? ????? ????????????"? ?????? ????????? ?? ?????? ?????? ??????? ?????????? ????? ???????? ?? ???? ??????????? ???? ? ??? ???????? ?? ?? ??????. ?? ?????? ??? ???????? ? ???????? - ?????? ???????? ??? ??????? ? ? ?????? ?????. ? ?????? ??? ???????? ?????? ? ???? ??? ????????? ????????? ? ??????????? ??? ???????? ? ????? ??????? ????????? ????????? "?????? ????" ????????????? ? ?????????? ???? ? ????????? ?????: ??? ????? ????????? ??????????? ??? ?? ????????????? ??? ????????? ????? ? ??????? ??????????? ??? ????? ?????????? ????????? ???? ????????? ??????????? ? ?????? - ???????, ??????? ??????? ??? "????????? ??????", ? ??? ?????????? ? ??????? ????? ????? "?????? - ?? ??????" - ? ???????? ??? ??????????? ???????????. ??? ????? ?????????? ??????????? ??? ???????? ? ???????? ???????? ??? ??????? ? ???????? ??????? ?????????? ??? ?????????? ?????????????, ?????????????? ? ?????????? ?????? ? ??????? ??? ?????????? ????????????? ? ????????????? ???????????? ????????? - ???????????????????? ???????. ? ??? ?????????? ????????? ??????? ???????????????????? ????????? ?????? ??????????? ? ??????????? ??????? ???????? a. ?????????? ???????????, b. ????????????, c. ????????? ?????????????? ??????? ????????????????? ????????: 8 ????? ????????? ???????: 7700 ??????. ??? ?? ??????????. ???????? ????????? ?????????? ? ?????????? ?? ??????? ?? ?????? ?? ???????? (095) 518-05-75, 544-88-14 -------------- next part -------------- An HTML attachment was scrubbed... URL: From pjssilva at ime.usp.br Fri Jan 13 03:49:05 2006 From: pjssilva at ime.usp.br (Paulo J. S. Silva) Date: Fri Jan 13 03:49:05 2006 Subject: [Numpy-discussion] Should dot return 1x1 matrix as scalar? Message-ID: <1137152864.9521.30.camel@localhost.localdomain> Hello, I was playing with the matrix type in numpy and I felt the "need" to have the "dot" function returning a 1x1 matrix as a scalar. This would allow for expressions like: x = matrix(arange(10.)) versorX = x / sqrt(x.T*x) Right now, I have to write: versorX = x / sqrt(x.T*x).item() Actually, a.T*b can not always be used as the inner product with the current behavior. Note that dot already returns scalars when we multiply rank-1 arrays, but matrices are always rank-2. Best, Paulo From aisaac at american.edu Fri Jan 13 06:41:05 2006 From: aisaac at american.edu (Alan G Isaac) Date: Fri Jan 13 06:41:05 2006 Subject: [Numpy-discussion] Should dot return 1x1 matrix as scalar? In-Reply-To: <1137152864.9521.30.camel@localhost.localdomain> References: <1137152864.9521.30.camel@localhost.localdomain> Message-ID: On Fri, 13 Jan 2006, "Paulo J. S. Silva" apparently wrote: > I was playing with the matrix type in numpy and I felt the "need" to > have the "dot" function returning a 1x1 matrix as a scalar. Since a 1x1 matrix is not a scalar, the current behavior is desirable, IMO. Specifically, I want an error in the last example below, since otherwise matrix multiplication will not be associative. Cheers, Alan Isaac PS Current behavior: Python 2.4.1 (#65, Mar 30 2005, 09:13:57) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy as N >>> x = N.mat(range(10)) >>> x matrix([[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]]) >>> x*x.T matrix([[285]]) >>> 285*x matrix([[ 0, 285, 570, 855, 1140, 1425, 1710, 1995, 2280, 2565]]) >>> (x*x.T)*(x.T*x) Traceback (most recent call last): File "", line 1, in ? File "C:\Python24\Lib\site-packages\numpy\core\defmatrix.py", line 128, in __m ul__ return N.dot(self, other) ValueError: objects are not aligned From chanley at stsci.edu Fri Jan 13 07:34:02 2006 From: chanley at stsci.edu (Christopher Hanley) Date: Fri Jan 13 07:34:02 2006 Subject: [Numpy-discussion] numpy revision 1894 fails to build on Solaris Message-ID: <43C7C82C.4070002@stsci.edu> Hi Travis, Numpy failed to build in this morning's regression test. A cursory look seems to indicate that the build failure is related to the umathmodule changes. I'm looking into this now. I've included the build log in case you can recognize the problem right away. Chris -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: build.log URL: From pjssilva at ime.usp.br Fri Jan 13 07:46:01 2006 From: pjssilva at ime.usp.br (Paulo J. S. Silva) Date: Fri Jan 13 07:46:01 2006 Subject: [Numpy-discussion] Should dot return 1x1 matrix as scalar? In-Reply-To: References: <1137152864.9521.30.camel@localhost.localdomain> Message-ID: <1137167107.12551.41.camel@localhost.localdomain> I am sorry, but I don't agree with you. The matrix class seems to be there to help people with numerical linear algebra algorithms "a la matlab". Is this an incorrect assumption? The behavior I am asking can be restricted to the matrix class objects only. However as dot already makes a scalar out the multiplication of two rank-1 arrays (in which case it computes the inner product), I thought that this behavior could be extended to matrix objects. Another possibility (without touching dot) would be to change the functions __mul__, __rmul__, and __imul__ in the matrix class to deal with 1x1 matrices as scalars. That would be OK to me too. The current behavior is odd when writing numerical algorithms. Take a look at these two examples: 1) Computing the norm: The natural way: def norm(x): return sqrt(x.T*x) But in this case you can't write norm(v)*v! Then, you have then to define norm as def norm(x): return sqrt(x.T*x).item() 2) The conjugate gradient iteration: alpha = (r.T*r) / (p.T*A*p) x += alpha*p rold = rnew r += alpha*(A*p) beta = (r.T*r) / (rold.T*rold) p = -r + beta*p This returns an error on second, forth, and the last equation! In all these cases, you'll need a call to the "item" method. Finally, the example you gave (x*x.T)*(x.T*x) does not make sense as matrix multiplication formula unless you interpret the first part as a scalar. You seem to be happy that the compiler got "the mistake". But in any Mathematics book the formula above would be considered OK as the first part would be interpreted as an inner product (and the second an outer product). In my opinion, it is not a mistake in a linear algebra setting. My 2 cents, Paulo -- Paulo Jos? da Silva e Silva Professor Assistente do Dep. de Ci?ncia da Computa??o (Assistant Professor of the Computer Science Dept.) Universidade de S?o Paulo - Brazil e-mail: rsilva at ime.usp.br Web: http://www.ime.usp.br/~rsilva Teoria ? o que n?o entendemos o (Theory is something we don't) suficiente para chamar de pr?tica. (understand well enough to call practice) From oliphant at ee.byu.edu Fri Jan 13 09:07:02 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri Jan 13 09:07:02 2006 Subject: [Numpy-discussion] Should dot return 1x1 matrix as scalar? In-Reply-To: <1137155787.9521.43.camel@localhost.localdomain> References: <1137152864.9521.30.camel@localhost.localdomain> <43C798C1.2070401@ieee.org> <1137155787.9521.43.camel@localhost.localdomain> Message-ID: <43C7DDF9.3010409@ee.byu.edu> Paulo J. S. Silva wrote: >Ops... You are right. The example is not good. Look this though (now I >am actually copying my ipython session, instead of "remembering" it): > >--- Session copy here --- > >In [5]:x = matrix(arange(10.)).T >In [6]:x = matrix(arange(3.)).T >In [7]:A = matrix([[1.,2,3],[4,5,6],[7,8,9]]) >In [8]:b = x.T*x*A >--------------------------------------------------------------------------- >exceptions.ValueError Traceback (most >recent call last) > >/home/pjssilva/ > >/usr/local/lib/python2.4/site-packages/numpy/core/defmatrix.py in >__mul__(self, other) > 126 return N.multiply(self, other) > 127 else: >--> 128 return N.dot(self, other) > 129 > 130 def __rmul__(self, other): > >ValueError: matrices are not aligned > >--- End of copy --- > >You see, the inner product can not be used to mutiply by a matrix, which >is very odd in linear algebra. As the matrix class is supposed to >represent the linear algebra object we I see two options: > >1) Change the __mul__, __rmul__, __imul__ to deal with 1x1 matrices as >scalars. > >2) Change dot to convert 1x1 matrix to scalar at return. > > or 3) Return scalars instead of 1x1 matrices inside of __array_finalize__ (where the magic of ensuring matrices are rank-2 arrays is actually done). From oliphant at ee.byu.edu Fri Jan 13 09:10:01 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri Jan 13 09:10:01 2006 Subject: [Numpy-discussion] Should dot return 1x1 matrix as scalar? In-Reply-To: <1137155787.9521.43.camel@localhost.localdomain> References: <1137152864.9521.30.camel@localhost.localdomain> <43C798C1.2070401@ieee.org> <1137155787.9521.43.camel@localhost.localdomain> Message-ID: <43C7DEAB.8080806@ee.byu.edu> Paulo J. S. Silva wrote: >Obs: Actually I found the "problem" when implementing a QR decomposition >based on Householder reflections and comparing it to a Matlab code. >Numpy is only 10% slower than Matlab in this code. Man, I'll love to >give my numerical linear algebra course this year using Python instead >of Matlab/Octave. > > I would be interested to see how a raw-array solution compares. There is going to be some overhead of using a subclass, because of the attribute lookups that occur on all array creations and because the subclass is written in Python that could be on the order of 10%. Quantifying the subclass slow-down would be useful... Also, which BLAS are you using? -Travis From haase at msg.ucsf.edu Fri Jan 13 09:20:09 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Fri Jan 13 09:20:09 2006 Subject: [Numpy-discussion] A proposal for dtype/dtypedescr in numpy objects In-Reply-To: <200601052240.58099.faltet@carabos.com> References: <200601052240.58099.faltet@carabos.com> Message-ID: <200601130919.41420.haase@msg.ucsf.edu> Hi, Following up: There was never any response to Francesc proposal ! I thought it sounded pretty good - as he argued: Still a good (late but acceptable) time to clean things up ! (I like just the fact that it removes the "ugly" doubling of having two: arr.dtype and arr.dtypecode ) Is this still on the table !? - Sebastian Haase On Thursday 05 January 2006 13:40, Francesc Altet wrote: > Hi, > > In my struggle for getting consistent behaviours with data types, I've > ended with a new proposal for treating them. The basic thing is that I > suggest to deprecate .dtype as being a first-class attribute and > replace it instead by the descriptor type container, which I find > quite more useful for end users. The current .dtype type will be still > accessible (mainly for developers) but buried in .dtype.type. > > Briefly stated: > > current proposed > ======= ======== > .dtypedescr --> moved into .dtype > .dtype --> moved into .dtype.type > .dtype.dtypestr --> moved into .dtype.str > new .dtype.name > > What is achieved with that? Well, not much, except easy of use and > > type comparison correctness. For example, with the next setup: > >>> import numpy > >>> a=numpy.arange(10,dtype='i') > >>> b=numpy.arange(10,dtype='l') > > we have currently: > >>> a.dtype > > > > >>> a.dtypedescr > > dtypedescr(' > >>> a.dtypedescr.dtypestr > > ' > >>> a.dtype.__name__[:-8] > > 'int32' > > >>> a.dtype == b.dtype > > False > > With the new proposal, we would have: > >>> a.dtype.type > > > > >>> a.dtype > > dtype(' > >>> a.dtype.str > > ' > >>> a.dtype.name > > 'int32' > > >>> a.dtype == b.dtype > > True > > The advantages of the new proposal are: > > - No more .dtype and .dtypedescr lying together, just current > .dtypedescr renamed to .dtype. I think that current .dtype does not > provide more useful information than current .dtypedesc, and giving > it a shorter name than .dtypedescr seems to indicate that it is more > useful to users (and in my opinion, it isn't). > > - Current .dtype is still accessible, but specifying and extra name in > path: .dtype.type (can be changed into .dtype.type_ or > whatever). This should be useful mainly for developers. > > - Added a useful dtype(descr).name so that one can quickly access to > the type name. > > - Comparison between data types works as it should now (without having > to create a metaclass for PyType_Type). > > Drawbacks: > > - Backward incompatible change. However, provided the advantages are > desirable, I think it is better changing now than later. > > - I don't specially like the string representation for the new .dtype > class. For example, I'd find dtype('Int32') much better than > dtype(' code, but they can be made later on (much less disruptive than the > proposed change). > > - Some other issues that I'm not aware of. > > > I'm attaching the patch for latest SVN. Once applied (please, pay > attention to the "XXX" signs in patched code), it passes all tests. > However, it may remain some gotchas (specially those cases that are > not checked in current tests). In case you are considering this change > to check in, please, tell me and I will revise much more carefully the > patch. If don't, never mind, it has been a good learning experience > anyway. > > Uh, sorry for proposing this sort of things in the hours previous to a > public release of numpy. From oliphant at ee.byu.edu Fri Jan 13 09:38:03 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri Jan 13 09:38:03 2006 Subject: [Numpy-discussion] A proposal for dtype/dtypedescr in numpy objects In-Reply-To: <200601130919.41420.haase@msg.ucsf.edu> References: <200601052240.58099.faltet@carabos.com> <200601130919.41420.haase@msg.ucsf.edu> Message-ID: <43C7E566.5090701@ee.byu.edu> Sebastian Haase wrote: >Hi, >Following up: There was never any response to Francesc proposal ! >I thought it sounded pretty good - as he argued: Still a good (late but >acceptable) time to clean things up ! >(I like just the fact that it removes the "ugly" doubling of having two: >arr.dtype and arr.dtypecode ) > > I think this proposal came during busy times and was not able to be looked at seriously. Times are still busy and so it is difficult to know what to do wih it. I think there is validity to what he is saying. The dtypedescr was only added in December while the dtype was there in March, so the reason for it is historical. I would not mind changing it so that .dtype actually returned the type-descriptor object. This would actually make things easier. It's only historical that it's not that way. One issue is that .dtypechar is a simple replacement for .typecode() but .dtype.char would involve two attribute lookups which may not be a good thing. But, this might not be a big deal because they should probably be using .dtype anyway. >Is this still on the table !? > > I'm willing to look at it, especially since I like the concept of the dtypedescr much better. >>In my struggle for getting consistent behaviours with data types, I've >>ended with a new proposal for treating them. The basic thing is that I >>suggest to deprecate .dtype as being a first-class attribute and >>replace it instead by the descriptor type container, which I find >>quite more useful for end users. >> I think this is true... I was just nervous to change it. But, prior to a 1.0 release I think we still could, if we do it quickly... >>The current .dtype type will be still >>accessible (mainly for developers) but buried in .dtype.type. >> >>Briefly stated: >> >>current proposed >>======= ======== >>.dtypedescr --> moved into .dtype >>.dtype --> moved into .dtype.type >>.dtype.dtypestr --> moved into .dtype.str >> new .dtype.name >> >> >> I actually like this proposal a lot as I think it gives proper place to the data-type descriptors. I say we do it, very soon, and put out another release quickly. -Travis From aisaac at american.edu Fri Jan 13 11:08:03 2006 From: aisaac at american.edu (Alan Isaac) Date: Fri Jan 13 11:08:03 2006 Subject: [Numpy-discussion] Should dot return 1x1 matrix as scalar? In-Reply-To: <1137167107.12551.41.camel@localhost.localdomain> References: <1137152864.9521.30.camel@localhost.localdomain><1137167107.12551.41.camel@localhost.localdomain> Message-ID: On Fri, 13 Jan 2006, "Paulo J. S. Silva" wrote: > as dot already makes a scalar out the multiplication of > two rank-1 arrays (in which case it computes the inner > product), I thought that this behavior could be extended > to matrix objects. Well, 'dot' for matrices is just a synonym for 'matrixmultiply', which it cannot be (in the same sense) for arrays. But I grant that I find it odd to enforce conformability for multiplication and not for addition. (See below.) I will also grant that GAUSS and Matlab behave as you wish, which might reflect a natural convenience and might reflect their impoverished types. Finally, I grant that I have not been able to quickly think up a use case where I want a 1x1 matrix for anything except error checking. Just to be clear, you do not want to get rid of 1x1 matrices, you just want to get rid of them as the result of *multiplication*, right. So [[1]]*[[2]]=2 but [[1]]+[[2]]=[[3]]. Right? And you would, I presume, find a 'scalar' function to be too clumsy. Cheers, Alan Isaac PS Conformability details: >>> t matrix([[2]]) >>> z matrix([[0, 0, 0], [0, 1, 2], [0, 2, 4]]) >>> t+z matrix([[2, 2, 2], [2, 3, 4], [2, 4, 6]]) >>> t*z Traceback (most recent call last): File "", line 1, in ? File "C:\Python24\Lib\site-packages\numpy\core\defmatrix.py", line 128, in __m ul__ return N.dot(self, other) ValueError: objects are not aligned >>> t.A*z.A array([[0, 0, 0], [0, 2, 4], [0, 4, 8]]) >>> From russel at appliedminds.com Fri Jan 13 11:08:13 2006 From: russel at appliedminds.com (Russel Howe) Date: Fri Jan 13 11:08:13 2006 Subject: [Numpy-discussion] question about index array behavior Message-ID: In the session below, I expected the for loop and the index array to have the same behavior. Is this behavior by design? Is there some other way to get the behavior of the for loop? The loop is too slow for my application ( len(ar1) == 18000). Russel Python 2.4.2 (#1, Nov 29 2005, 08:43:33) [GCC 4.0.1 (Apple Computer, Inc. build 5247)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> from numarray import * >>> import numarray.random_array as ra >>> print libnumarray.__version__ 1.5.0 >>> ar1=ra.random(10) >>> ar2=zeros(5, type=Float32) >>> ind=array([0,0,1,1,2,2,3,3,4,4]) >>> ar2[ind]+=ar1 >>> ar2 array([ 0.09791247, 0.26159889, 0.89386773, 0.32572687, 0.86001897], type=Float32) >>> ar1 array([ 0.49895534, 0.09791247, 0.424059 , 0.26159889, 0.29791802, 0.89386773, 0.44290054, 0.32572687, 0.53337622, 0.86001897]) >>> ar2*=0.0 >>> for x in xrange(len(ind)): ... ar2[ind[x]]+=ar1[x] ... >>> ar2 array([ 0.5968678 , 0.68565786, 1.19178581, 0.76862741, 1.39339519], type=Float32) >>> From aisaac at american.edu Fri Jan 13 11:09:04 2006 From: aisaac at american.edu (Alan Isaac) Date: Fri Jan 13 11:09:04 2006 Subject: [Numpy-discussion] Should dot return 1x1 matrix as scalar? In-Reply-To: <43C7DDF9.3010409@ee.byu.edu> References: <1137152864.9521.30.camel@localhost.localdomain> <43C798C1.2070401@ieee.org> <1137155787.9521.43.camel@localhost.localdomain><43C7DDF9.3010409@ee.byu.edu> Message-ID: On Fri, 13 Jan 2006, Travis Oliphant wrote: > 3) Return scalars instead of 1x1 matrices inside of __array_finalize__ > (where the magic of ensuring matrices are rank-2 arrays is actually done). Just for multiplication, or also for addition etc? Alan Isaac From Chris.Barker at noaa.gov Fri Jan 13 11:15:05 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri Jan 13 11:15:05 2006 Subject: [Numpy-discussion] Numpy and the old scipy In-Reply-To: <43C7747B.50302@ieee.org> References: <7875CF3B-37D2-4BC1-9757-DAFFA9DF8B9A@pito.com> <43C7747B.50302@ieee.org> Message-ID: <43C7FC25.9080607@noaa.gov> Travis Oliphant wrote: > In order to install new and old scipy you will definitely need to > install one of them to another location besides site-packages (probably > new scipy). > Then you will need to make sure your sys.path is set up properly to find > the scipy you are interested in using for that session. It sounds like SciPy could use a versioning scheme much like wxPythons: import wxversion wxversion.select("2.6") import wx See: http://wiki.wxpython.org/index.cgi/MultiVersionInstalls In fact, you could probably just grab the wxPython code and tweek it a little for SciPy. This was debated a lot on the wxPython lists before being implemented. After all you could "Just" write a few start-up scripts that manipulate PYTHONPATH, or re-name some directories, or put in a few sym links, or, or, or... Also, ideally wxPython major versions are compatible, etc, etc. However, when all was said and Dunn, this is a very nice system that works the same way on all platforms, and doesn't get in the way of anything. New versions get installed as the default, so if you never use wxversion, you never know it's there. If you do, then you can test new versions without having to break any old, running utilities, etc. Also, the infrastructure is in place for future major version changes, etc. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From pjssilva at ime.usp.br Fri Jan 13 11:18:01 2006 From: pjssilva at ime.usp.br (Paulo J. S. Silva) Date: Fri Jan 13 11:18:01 2006 Subject: [Numpy-discussion] Should dot return 1x1 matrix as scalar? In-Reply-To: <43C7DEAB.8080806@ee.byu.edu> References: <1137152864.9521.30.camel@localhost.localdomain> <43C798C1.2070401@ieee.org> <1137155787.9521.43.camel@localhost.localdomain> <43C7DEAB.8080806@ee.byu.edu> Message-ID: <1137179742.19206.9.camel@localhost.localdomain> > I would be interested to see how a raw-array solution compares. > There > is going to be some overhead of using a subclass, because of the > attribute lookups that occur on all array creations and because the > subclass is written in Python that could be on the order of 10%. > > Quantifying the subclass slow-down would be useful... Also, which > BLAS > are you using? I am using matlab 6.5.0 release 13 in a Athlon tbird 1.2Ghz Blas with numpy is ATLAS optimized for 3dnow. I m getting very interesting results. First the time for a QR factorization of a 1000x1000 random matrix: Matlab: >> t = cputime; R = houseqr(A); cputime - t ans = 67.1000 Numpy: In [9]:i = time.clock(); R = houseqr.houseqr(A); time.clock() - i Out[9]:79.009999999999991 If I code numpy naively making an extra matrix slice this drops to: In [14]:i = time.clock(); R = houseqr.houseqr(A); time.clock() - i Out[14]:114.34999999999999 (See the code below). I have then decided to campare the blas and here things get funny: Matrix multiplication: Matlab: >> t = cputime; A*A; cputime - t ans = 2.0600 Numpy: In [32]:i = time.clock(); C=A*A; time.clock() - i Out[32]:1.3600000000000136 It looks like numpy (ATLAS) is much better.... However in the QR code the most important part is a matrix times vector multiplication and an outer product. If I benchmark this operations I get: Matlab: >> t = cputime; bench(A,b); cputime - t ans = 5.7500 Numpy: In [27]:i = time.clock(); bench(A,b); time.clock() - i Out[27]:10.610000000000014 Why is numpy so slow?????? Paulo code --- houseqr.py --- from numpy import * def norm(x): return sqrt(x.T*x).item() def sinal(x): if x < 0.0: return -1.0 else: return 1.0 def houseqr(A): m, n = A.shape R = A.copy() bigE1 = matrix(zeros((m, 1), float)) bigE1[0] = 1.0 for j in range(n): x = R[j:,j] v = sinal(x[0])*norm(x)*bigE1[:m-j] + x v = v / norm(v) # Slower version. #R[j:,j:] -= 2.0*v*(v.T*R[j:,j:]) # Faster version: avoids the extra slicing. upRight = R[j:,j:] upRight -= 2.0*v*(v.T*upRight) return R --- houseqr.m --- function [A] = houseqr(A) [m, n] = size(A); bigE1 = zeros(m,1); bigE1(1) = 1.0; for j = 1:n, x = A(j:m,j); v = sinal(x(1))*norm(x)*bigE1(1:m-j+1) + x; v = v / (norm(v)); upRight = A(j:m,j:n); A(j:m,j:n) = upRight - 2*v*(v'*upRight); end --- bench.py --- from numpy import * def bench(A, b): for i in range(100): c = b.T*A d = b*c --- bench.m --- function bench(A, b) for i=1:100, c = b'*A; d = b*c; end From perry at stsci.edu Fri Jan 13 11:39:01 2006 From: perry at stsci.edu (Perry Greenfield) Date: Fri Jan 13 11:39:01 2006 Subject: [Numpy-discussion] question about index array behavior In-Reply-To: References: Message-ID: <1619fd20730e3d62f2b91aa3a28753f1@stsci.edu> On Jan 13, 2006, at 2:07 PM, Russel Howe wrote: > In the session below, I expected the for loop and the index array to > have the same behavior. Is this behavior by design? Is there some > other way to get the behavior of the for loop? The loop is too slow > for my application ( len(ar1) == 18000). > Russel This sort of usage of index arrays is always going to be a bit confusing and this is a common example of that. Anytime you are using repeated indices for index assignment, you are not going to get what you would naively think. It's useful to think of what is going on in a little more detail. Your use of index arrays is resulting in the elements you selected generating a 10 element array which is added to the random elements. Initially it is a 10 element array with all zero elements, and after the addition, it equals the random array elements. Then, the index assignment takes place. First, the first element of the summed array is assigned to 0, then the second element of the summed array is assigned to 0, and that is the problem. The summing is done before the assignment. Generally the last index of a repeated set is what is assigned as the final value. It is possible to do what you want without a for loop, but perhaps not as fast as it would be in C. One way to do it is to sort the indices in increasing order, generate the corresponding selected value array and then use accumulated sums to derive the sums corresponding to each index. It's a bit complicated, but can be much faster than a for loop. See example 3.7.4 to see the details of how this is done in our tutorial: http://www.scipy.org/wikis/topical_software/Tutorial Maybe someone has a more elegant, faster or clever way to do this that I've overlooked. I've seen this come up enough that it may be useful to provide a special function to make this easier to do. Perry Greenfield > Python 2.4.2 (#1, Nov 29 2005, 08:43:33) > [GCC 4.0.1 (Apple Computer, Inc. build 5247)] on darwin > Type "help", "copyright", "credits" or "license" for more information. > >>> from numarray import * > >>> import numarray.random_array as ra > >>> print libnumarray.__version__ > 1.5.0 > >>> ar1=ra.random(10) > >>> ar2=zeros(5, type=Float32) > >>> ind=array([0,0,1,1,2,2,3,3,4,4]) > >>> ar2[ind]+=ar1 > >>> ar2 > array([ 0.09791247, 0.26159889, 0.89386773, 0.32572687, > 0.86001897], type=Float32) > >>> ar1 > array([ 0.49895534, 0.09791247, 0.424059 , 0.26159889, 0.29791802, > 0.89386773, 0.44290054, 0.32572687, 0.53337622, > 0.86001897]) > >>> ar2*=0.0 > >>> for x in xrange(len(ind)): > ... ar2[ind[x]]+=ar1[x] > ... > >>> ar2 > array([ 0.5968678 , 0.68565786, 1.19178581, 0.76862741, > 1.39339519], type=Float32) > >>> > > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log > files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From strawman at astraw.com Fri Jan 13 19:17:01 2006 From: strawman at astraw.com (Andrew Straw) Date: Fri Jan 13 19:17:01 2006 Subject: [Numpy-discussion] Numpy and the old scipy In-Reply-To: <43C7FC25.9080607@noaa.gov> References: <7875CF3B-37D2-4BC1-9757-DAFFA9DF8B9A@pito.com> <43C7747B.50302@ieee.org> <43C7FC25.9080607@noaa.gov> Message-ID: <1497.4.240.165.38.1137208614.squirrel@webmail.astraw.com> Python eggs also support runtime version selection. It's possible we may need to backport egg support to old scipy, but it already works great with new numpy/scipy. I'd suggest this over re-implementing another wheel. > Travis Oliphant wrote: > >> In order to install new and old scipy you will definitely need to >> install one of them to another location besides site-packages (probably >> new scipy). >> Then you will need to make sure your sys.path is set up properly to find >> the scipy you are interested in using for that session. > > It sounds like SciPy could use a versioning scheme much like wxPythons: > > import wxversion > wxversion.select("2.6") > import wx > > See: > > http://wiki.wxpython.org/index.cgi/MultiVersionInstalls > > > In fact, you could probably just grab the wxPython code and tweek it a > little for SciPy. > > This was debated a lot on the wxPython lists before being implemented. > After all you could "Just" write a few start-up scripts that manipulate > PYTHONPATH, or re-name some directories, or put in a few sym links, > or, or, or... Also, ideally wxPython major versions are compatible, etc, > etc. > > However, when all was said and Dunn, this is a very nice system that > works the same way on all platforms, and doesn't get in the way of > anything. New versions get installed as the default, so if you never use > wxversion, you never know it's there. If you do, then you can test new > versions without having to break any old, running utilities, etc. > > Also, the infrastructure is in place for future major version changes, > etc. > > -Chris > > > -- > Christopher Barker, Ph.D. > Oceanographer > > NOAA/OR&R/HAZMAT (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log > files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From oliphant.travis at ieee.org Fri Jan 13 20:41:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri Jan 13 20:41:02 2006 Subject: [Numpy-discussion] Should dot return 1x1 matrix as scalar? In-Reply-To: <1137179742.19206.9.camel@localhost.localdomain> References: <1137152864.9521.30.camel@localhost.localdomain> <43C798C1.2070401@ieee.org> <1137155787.9521.43.camel@localhost.localdomain> <43C7DEAB.8080806@ee.byu.edu> <1137179742.19206.9.camel@localhost.localdomain> Message-ID: <43C880B2.1050002@ieee.org> Paulo J. S. Silva wrote: >Numpy: > >In [27]:i = time.clock(); bench(A,b); time.clock() - i >Out[27]:10.610000000000014 > > >Why is numpy so slow?????? > > > I think the problem here is that using the properties here to take advantage of the nice matrix math stuff is slower than just computing the dot product in the fastest possible way with raw arrays. I've been concerned about this for awhile. The benchmark below makes my point. While a matrix is a nice thing, I think it will always be slower.... It might be possible to speed it up and I'm open to suggestions... To see what I mean, try this.... import timeit t1 = timeit.Timer('c = b.T*A; d=c*b','from numpy import rand,mat; A = mat(rand(1000,1000));b = mat(rand(1000,1))') t2 = timeit.Timer('c = dot(b,A); d=dot(b,c)','from numpy import rand, dot; A = rand(1000,1000);b = rand(1000)') >>> t1.timeit(100) 6.0398328304290771 >>> t2.timeit(100) 1.2430641651153564 So, using raw arrays and dot product is 5x faster in this case..... -Travis From pjssilva at ime.usp.br Sat Jan 14 05:52:06 2006 From: pjssilva at ime.usp.br (Paulo J. S. Silva) Date: Sat Jan 14 05:52:06 2006 Subject: [Numpy-discussion] Should dot return 1x1 matrix as scalar? In-Reply-To: <43C880B2.1050002@ieee.org> References: <1137152864.9521.30.camel@localhost.localdomain> <43C798C1.2070401@ieee.org> <1137155787.9521.43.camel@localhost.localdomain> <43C7DEAB.8080806@ee.byu.edu> <1137179742.19206.9.camel@localhost.localdomain> <43C880B2.1050002@ieee.org> Message-ID: <1137246317.7490.15.camel@localhost.localdomain> Em Sex, 2006-01-13 ?s 21:40 -0700, Travis Oliphant escreveu: > Paulo J. S. Silva wrote: > > >Numpy: > > > >In [27]:i = time.clock(); bench(A,b); time.clock() - i > >Out[27]:10.610000000000014 > > > > > >Why is numpy so slow?????? > > > > > > > > I think the problem here is that using the properties here to take > advantage of the nice matrix math stuff is slower than just computing > the dot product in the fastest possible way with raw arrays. I've been > concerned about this for awhile. The benchmark below makes my point. > > While a matrix is a nice thing, I think it will always be slower.... It > might be possible to speed it up and I'm open to suggestions... > > To see what I mean, try this.... > Travis, I think I got the gotcha! The problem is the in the dot function. I have only skimmed on the _dotblas code, but I assume it try to find the right blas function based on the arguments ranks. If both arguments are matrices then both are rank two and the matrix-multiply blas function is called. This is "very" suboptimal if some of the matrices objects actually represent a vector or a scalar. The same problem would appear with pure arrays if one insists on using a column vector: In [3]:t1 = timeit.Timer('c = dot(A,b)','from numpy import rand, dot; A = rand(1000,1000);b = rand(1000)') In [4]:t2 = timeit.Timer('c = A*b','from numpy import rand, dot, mat; A = mat(rand(1000,1000));b = mat(rand(1000,1))') In [5]:t3 = timeit.Timer('c = dot(A,b)','from numpy import rand, dot, transpose; A = rand(1000,1000);b = rand(1000,1)') In [6]:t1.timeit(100) Out[6]:0.69448995590209961 In [7]:t2.timeit(100) Out[7]:1.1080958843231201 You see? The third test with pure arrays is as slow as the matrix based test. The problem is even more dramatic if you are trying to compute an inner product: In [13]:t1 = timeit.Timer('c = dot(b,b)','from numpy import rand, dot; b = rand(1000)') In [14]:t2 = timeit.Timer('c = b.T*b','from numpy import rand, mat; b = mat(rand(1000,1))') In [15]:t3 = timeit.Timer('c = dot(transpose(b),b)','from numpy import rand, dot, transpose; b = rand(1000,1)') In [16]:t1.timeit(10000) Out[16]:0.053219079971313477 In [17]:t2.timeit(10000) Out[17]:0.65550899505615234 In [18]:t3.timeit(10000) Out[18]:0.62446498870849609 Note that this is a very serious drawback for matrices objects as the most usual operation, matrix-vector multiplication, in numerical linear algebra algorithms is always suboptimal. The solution may be a "smarter" analysis in _dotblas.c that not only looks at the rank but that also "collapses" dimensions with only one element to decide which blas function to call. Note that this approach may be used to solve the problem with 1x1 matrices if properly coded. Some special care has to be taken to identify outer products to. I may try to look at this myself if you want. But certainly only in three weeks. I am preparing for an exam to confirm (make permanent) my position at the university that will take place in the last days of January. Best, Paulo From baccarat at albawaba.com Sat Jan 14 16:07:00 2006 From: baccarat at albawaba.com (=?windows-1251?B?MjQgLSAyNSD/7eLg8P8gMjAwNiDj7uTg?=) Date: Sat Jan 14 16:07:00 2006 Subject: [Numpy-discussion] =?windows-1251?B?y87DyNHSyMrAINHKy8DEwDogwcjHzcXRLc/LwM3I0M7CwM3IxSwgz9DOxcrSyNDO?= =?windows-1251?B?wsDNyMUgyCDQxc7Qw8DNyMfA1sjf?= Message-ID: <9d5001c618ea$0e435f2c$4e8f2a6e@albawaba.com> ?????????? ????????????? ????????????? ?????, ??????????? ???????, ??????? ?????????? ??????? ??????? ? ????? ?????????? ????????: ????????? ??????: ??????-????????????, ?????????????? ? ?????????????24 - 25 ?????? 2006 ???? ?? ????????? ???????? ?? ???????? ???? ???????????? ? ???????? ?????????????? ? ????????????? ??????, ??????????? ????????? ?????? ? ??????????, ??????? ?????? ????????????? ??????????? ???????????? ? ????? ? ???????? ?????????????? ? ?????????????, ? ????? ??????? ??????????? ?????????? ??????-????? ???????? ?????? ? ??????? ??????, ??????????? ??? ???????? ??????? ? ?????????? ? ???????????. ????????? ????????: ??????-???? - ?????? ???????? ?????????????? ??????. ??????? ??????????????, ????????????? ? ??????? ??????. ??????-???? - ?????? ???????? ????????????. ????????? ??????????? ?????????. ???????? ????????????? ?????????. ????????? ? ??????????? ?????? ???????. ??????????? ?????? - ?? ????? ? ?????????. ???? ?????????? ????????? ????????????? ?? ??????????? ???????. ???? ?????????? ????????? ????????????. ????? ???????????? ??? ?????? ? ??????????? ?? ???? ??????????. ??????????????? ???? ??????. ????? ??? ? ??????????? ?? ???? ???????????????? ???????? ???????????? ??? ???????? ???????. ????????????? ????? ??????????? ????????????. ??????????? ???????????? ??? ??????. ?????????????????? ?????????? ??????? ????????, ???????? ? ?????????????? ????? ???? ROTOMAT, LEANLIFT. ??????????? ?????????? ??????? ??????? ???????. ??????????? ???????? ?????? ? ?????????. ????????-??????????? ? ?????????????????? ???????. ????????????? ? ?????????????? ????????-????????????? ????????????. ????? ?????? ? ?????????????? ??????. ?????? ???????? ? ????? ?????? ??????????????. ??????????????????. ??????????? ??????? ? ??? ??????? ?? ??????. ??????? ???????? ??????? ? ??? ??????? ?? ??????? ?????????????? ??????. ????? ????????? ??????? ????????? ??????? ?? ?????? 3 - 5 ???. ??????????? ???????. ?????????? ???? ???????? ???????. ??????? ??????????????? ??????. ??????????????? ??????? ?? ??????. ?????????????? ?????????? ???????? ??????????????? ????????. ???????. ?????????????. ?????????? ?? ????????. ?????. ???????????? ???????. ???????? ??????? ???????. ??????????????. ??????????? ?????????? ?????????????? ??????. ???????????? ????????????? - ?????? ??????? ???????????. ?????????????? ????????????? ??????????????? ?????????. ????? ????????? ???????????. ?????? ????????????? ????????????? ?????? ? ?????? ??????? ???????? ??????????. ??????????? ? ???????????????? ??????? ?? ???????? ???????. ?????? ????????????? ????????????? ??????. ??????? ??? ????????? ????? ??????????? ??????. ?????????????? ????????????? ? ??????????? ???????? ??????????. ????????? ???????????. ?????? ? ??????????. ?????? ?????? ? ???????? ???????? ??????? ?? ???????????. ??????? ?????????? ???????????????? ?????????? ?? ?????? ????? ???????? - ????????? 7 ??? ?????????? ??????????? ??????????????? ???????? ?????????? ????????? ???????????????? ??????????? ? ????????? ?????? ????????? ???????. ? ????????? ????? ????? ?????????? ???????? ????? ????????? ?????????? ??? ???? ??????????, ????? ??? AVON, ???????? ???, ?????, ???????? Computerized Storage Systems Corp (??????? ???????? ??????? - ?????????), ???????? ???????. ?????? ????????? ?? ?????? ???????? ? ????????????? ????????,? ?????? ????????? ?????? ??????? ? ?????? ?????????? ??????. ???????? ????????? ?????????? ?? ?????? ?? ?????????: (495) 980-65-?9, 98?-65-36 -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Sat Jan 14 19:47:00 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sat Jan 14 19:47:00 2006 Subject: [Numpy-discussion] Changed attributes .dtypedescr, .dtypechar, .dtypestr and .dtype Message-ID: <43C9C573.2020809@ieee.org> There was some cruft left over from the change to making data-type descriptors real Python objects. This left lots of .dtype related attributes on the array object --- too many as Francesc Altet graciously pointed out. In the latest SVN, I've cleaned things up (thanks to a nice patch from Francesc to get it started). Basically, there is now only one attribute on the array object dealing with the data-type (arr.dtype). This attribute returns the data-type descriptor object for the array. This object itself has the attributes .char, .str, and .type (among others). I think this will lead to less confusion long term. The cruft was due to the fact that my understanding of the data-type descriptor came in December while seriously looking at records module. This will have some backward-compatibility issues (we are still pre-1.0 and early enough that I hope this is not too difficult to deal with). The compatibility to numpy-0.9.2 issues I can see are: 1) Replacing attributes that are now gone: .dtypechar --> .dtype.char .dtypestr --> .dtype.str .dtypedescr --> .dtype 2) Changing old .dtype -> .dtype.type This is only necessary if you were using a.dtype as a *typeobject* as in issubclass(a.dtype, ) If you were using .dtype as a parameter to dtype= then that usage will still work great (in fact a little faster) because now .dtype returns a "descriptor object" 3) The dtypedescr constructor is now called dtype. This change should have gone into the 0.9.2 release, but things got too hectic with all the name changes. I will quickly release 0.9.4 with these changes unless I hear strong disagreements within the next few days. -Travis P.S. SciPy SVN has been updated and fixed with the changes. Numeric compatibility now implies that .typecode() --> .dtype.char although if .typecode() was used as an argument to a function, then .dtype will very likely work. -Travis From oliphant.travis at ieee.org Sat Jan 14 21:59:01 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sat Jan 14 21:59:01 2006 Subject: [Numpy-discussion] Speed ups to matrix-vector multiplication In-Reply-To: <43C880B2.1050002@ieee.org> References: <1137152864.9521.30.camel@localhost.localdomain> <43C798C1.2070401@ieee.org> <1137155787.9521.43.camel@localhost.localdomain> <43C7DEAB.8080806@ee.byu.edu> <1137179742.19206.9.camel@localhost.localdomain> <43C880B2.1050002@ieee.org> Message-ID: <43C9E457.3020109@ieee.org> Travis Oliphant wrote: > Paulo J. S. Silva wrote: > >> Numpy: >> >> In [27]:i = time.clock(); bench(A,b); time.clock() - i >> Out[27]:10.610000000000014 >> >> >> Why is numpy so slow?????? >> >> >> > > I think the problem here is that using the properties here to take > advantage of the nice matrix math stuff is slower than just computing > the dot product in the fastest possible way with raw arrays. I've > been concerned about this for awhile. The benchmark below makes my > point. > > While a matrix is a nice thing, I think it will always be slower.... > It might be possible to speed it up and I'm open to suggestions... Indeed it was possible to speed things up as Paulo pointed out, by using the correct blas calls for the real size of the array. With some relatively simple modifications, I was able to get significant speed-ups in time using matrices: import timeit t1 = timeit.Timer('c = b.T*A; d=c*b','from numpy import rand,mat; A = mat(rand(1000,1000));b = mat(rand(1000,1))') t2 = timeit.Timer('c = dot(b,A); d=dot(b,c)','from numpy import rand, dot; A = rand(1000,1000);b = rand(1000)') t1.timeit(100) 1.4369449615478516 t2.timeit(100) 1.2983191013336182 Now, that looks more like a 10% overhead for the matrix class. -Travis From zpincus at stanford.edu Sun Jan 15 00:24:01 2006 From: zpincus at stanford.edu (Zachary Pincus) Date: Sun Jan 15 00:24:01 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location Message-ID: <936E82E6-E8C0-44A5-AB78-22FAEC4F4622@stanford.edu> Hello folks, I'm quite excited about the new release of numpy -- thanks for all the work. One question: where should I expect the arrayobject.h file to be installed? My old installation of Numeric placed the arrayobject.h (etc.) header files here: /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/ Numeric/ I'm on OS X, using the standard 'framework' build of python, which accounts for the particular install path. However, this is the proper include path for python-related C headers. When I installed numpy, arrayobject.h was placed here, in the site- packages directory: /Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site- packages/numpy/core/include/numpy/arrayobject.h My question: is this an error (possibly specific to OS X) and the numpy headers should have been placed with the other C headers? Or is this a policy change which puts the headers in the site-packages directory? I'm writing some code which I hope to be compatible with Numeric, numarray, and numpy; being able to reliably find arrayobject.h is pretty important for this. Thus, it would be good to know where, in general, this file will be installed by numpy. Thanks, Zach Pincus Program in Biomedical Informatics and Department of Biochemistry Stanford University School of Medicine From pearu at cens.ioc.ee Sun Jan 15 04:24:01 2006 From: pearu at cens.ioc.ee (pearu at cens.ioc.ee) Date: Sun Jan 15 04:24:01 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: <936E82E6-E8C0-44A5-AB78-22FAEC4F4622@stanford.edu> Message-ID: On Sun, 15 Jan 2006, Zachary Pincus wrote: > One question: where should I expect the arrayobject.h file to be > installed? numpy/arrayobject.h is installed to numpy.get_numpy_include(). > My old installation of Numeric placed the arrayobject.h (etc.) header > files here: > /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/ > Numeric/ > > I'm on OS X, using the standard 'framework' build of python, which > accounts for the particular install path. However, this is the proper > include path for python-related C headers. > > When I installed numpy, arrayobject.h was placed here, in the site- > packages directory: > /Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site- > packages/numpy/core/include/numpy/arrayobject.h > > My question: is this an error (possibly specific to OS X) and the > numpy headers should have been placed with the other C headers? Or is > this a policy change which puts the headers in the site-packages > directory? This is not an error. Use numpy.get_numpy_include() to retrive the directory of numpy header files. See numpy.get_numpy_include.__doc__ for more information. HTH, Pearu From aisaac at american.edu Sun Jan 15 05:14:02 2006 From: aisaac at american.edu (Alan G Isaac) Date: Sun Jan 15 05:14:02 2006 Subject: [Numpy-discussion] round_ vs round Message-ID: numpy's use of `round_` rather than `round` feels like a blemish. Was this done just to protect the built-in? If so, is that a good enough reason? fwiw, Alan Isaac From zpincus at stanford.edu Sun Jan 15 11:25:03 2006 From: zpincus at stanford.edu (Zachary Pincus) Date: Sun Jan 15 11:25:03 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: References: Message-ID: <2D771DF0-1A60-463B-A643-AD6E2980AB94@stanford.edu> >> Or is this a policy change which puts the headers in the site- >> packages >> directory? > > This is not an error. Use numpy.get_numpy_include() to retrive the > directory of numpy header files. See numpy.get_numpy_include.__doc__ > for more information. Thanks for the information. Unfortunately, the package I'm working on can't be built with distutils, so it may wind up being something of a contortion to call numpy.get_numpy_include() from the build system I need to use. Understanding that the optimal, preferred method of finding the include dir will always be numpy.get_numpy_include(), if I must resort to a suboptimal method of guessing likely locations, what will those locations be? Are they generally within the site-packages directory? Sometimes in the python include directory? Will they be ever-changing? I will endeavor to use the more proper methods in my system, but this information would be useful as a fallback. Thanks, Zach From xkgbpbxwo at amtote.com Sun Jan 15 13:00:03 2006 From: xkgbpbxwo at amtote.com (Ali Hattie) Date: Sun Jan 15 13:00:03 2006 Subject: [Numpy-discussion] Top news from Kristie Message-ID: <001001c61a16$8adf23c0$402c1c53@xr2xz8y0ujs4zq> Palmer Sara -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: dbdctphdmw.gif Type: image/gif Size: 11162 bytes Desc: not available URL: From ndarray at mac.com Sun Jan 15 19:17:01 2006 From: ndarray at mac.com (Sasha) Date: Sun Jan 15 19:17:01 2006 Subject: [Numpy-discussion] NumPy's integer arithmetics differs from Numeric Message-ID: It looks like NumPy and Numeric handle invalid integer operations differently: >>> from numpy import * >>> import Numeric as n >>> array(2)**40 2147483647 >>> array(2)/0 0 In numeric both fail: >>> n.array(2)**40 Traceback (most recent call last): File "", line 1, in ? ArithmeticError: Integer overflow in power. >>> n.array(2)/0 Traceback (most recent call last): File "", line 1, in ? ZeroDivisionError: divide by zero I don't know if the change was intentional, but if it was, I think it should be advertised. -- sasha From ndarray at mac.com Sun Jan 15 21:40:03 2006 From: ndarray at mac.com (Sasha) Date: Sun Jan 15 21:40:03 2006 Subject: [Numpy-discussion] NumPy's integer arithmetics differs from Numeric In-Reply-To: References: Message-ID: It looks like NumPy and Numeric handle invalid integer operations differently: >>> from numpy import * >>> import Numeric as n >>> array(2)**40 2147483647 >>> array(2)/0 0 In numeric both fail: >>> n.array(2)**40 Traceback (most recent call last): File "", line 1, in ? ArithmeticError: Integer overflow in power. >>> n.array(2)/0 Traceback (most recent call last): File "", line 1, in ? ZeroDivisionError: divide by zero I don't know if the change was intentional, but if it was, I think it should be advertised. -- sasha From cookedm at physics.mcmaster.ca Sun Jan 15 23:08:02 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Sun Jan 15 23:08:02 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: <2D771DF0-1A60-463B-A643-AD6E2980AB94@stanford.edu> References: <2D771DF0-1A60-463B-A643-AD6E2980AB94@stanford.edu> Message-ID: On Jan 15, 2006, at 14:23 , Zachary Pincus wrote: >>> Or is this a policy change which puts the headers in the site- >>> packages >>> directory? >> >> This is not an error. Use numpy.get_numpy_include() to retrive the >> directory of numpy header files. See numpy.get_numpy_include.__doc__ >> for more information. > > Thanks for the information. Unfortunately, the package I'm working > on can't be built with distutils, so it may wind up being something > of a contortion to call numpy.get_numpy_include() from the build > system I need to use. You don't say what type of build system it is. If it's autoconf or Makefile based, you could do something like NUMPY_HEADERS=$(python -c 'import numpy; print numpy.get_numpy_include ()') > Understanding that the optimal, preferred method of finding the > include dir will always be numpy.get_numpy_include(), if I must > resort to a suboptimal method of guessing likely locations, what > will those locations be? > > Are they generally within the site-packages directory? Sometimes in > the python include directory? Will they be ever-changing? For now, they're in the site-packages directory, under numpy/core/ include. That position might change (it used to be numpy/base/ include, for instance), if we rename modules again. They probably won't migrate to the python include directory for the reason they're not in there now (non-root users can't install there). -- |>|\/|< /------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From arnd.baecker at web.de Sun Jan 15 23:25:00 2006 From: arnd.baecker at web.de (Arnd Baecker) Date: Sun Jan 15 23:25:00 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: References: <2D771DF0-1A60-463B-A643-AD6E2980AB94@stanford.edu> Message-ID: On Mon, 16 Jan 2006, David M. Cooke wrote: > On Jan 15, 2006, at 14:23 , Zachary Pincus wrote: > > >>> Or is this a policy change which puts the headers in the site- > >>> packages directory? > >> > >> This is not an error. Use numpy.get_numpy_include() to retrive the > >> directory of numpy header files. See numpy.get_numpy_include.__doc__ > >> for more information. > > > > Thanks for the information. Unfortunately, the package I'm working > > on can't be built with distutils, It seems that distutils is not loved that much (E.g. Wax no longer uses distutils http://zephyrfalcon.org/weblog2/arch_e10_00870.html#e878 ) > > so it may wind up being something > > of a contortion to call numpy.get_numpy_include() from the build > > system I need to use. > > You don't say what type of build system it is. If it's autoconf or > Makefile based, you could do something like > > NUMPY_HEADERS=$(python -c 'import numpy; print numpy.get_numpy_include > ()') > > > Understanding that the optimal, preferred method of finding the > > include dir will always be numpy.get_numpy_include(), if I must > > resort to a suboptimal method of guessing likely locations, what > > will those locations be? > > > > Are they generally within the site-packages directory? Sometimes in > > the python include directory? Will they be ever-changing? > > For now, they're in the site-packages directory, under numpy/core/ > include. That position might change (it used to be numpy/base/ > include, for instance), if we rename modules again. They probably > won't migrate to the python include directory for the reason they're > not in there now (non-root users can't install there). If I do a `python setup.py --prefix=$HOME/my_numpy install`, this leads to ~/my_numpy/bin/f2py ~/my_numpy/lib/python2.3/site-packages/numpy/* Wouldn't then a ~/my_numpy/include/numpy/* be reasonable? Presumably I am overlooking something, and I know that this has been discussed in much detail before, but still ... Best, Arnd From faltet at carabos.com Mon Jan 16 03:38:06 2006 From: faltet at carabos.com (Francesc Altet) Date: Mon Jan 16 03:38:06 2006 Subject: [Numpy-discussion] Maximum available dimensions in numpy Message-ID: <1137411430.3273.5.camel@localhost.localdomain> Hi, I've downloaded a fresh SVN version of numpy (0.9.4.1914) and got a bit surprised becuase of this: >>> numpy.reshape(numpy.array((22,)), (1,)*20) array([[[[[[[[[[[[[[[[[[[[22]]]]]]]]]]]]]]]]]]]]) >>> numpy.reshape(numpy.array((22,)), (1,)*21) Traceback (most recent call last): File "", line 1, in ? File "/usr/lib/python2.3/site-packages/numpy/core/oldnumeric.py", line 164, in reshape return asarray(a).reshape(newshape) ValueError: sequence too large; must be smaller than 20 Before, I think numpy supported up to 32 dimensions. Is there any reason for this new limit? Just curious. Cheers, -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From faltet at carabos.com Mon Jan 16 04:44:12 2006 From: faltet at carabos.com (Francesc Altet) Date: Mon Jan 16 04:44:12 2006 Subject: [Numpy-discussion] Changed attributes .dtypedescr, .dtypechar, .dtypestr and .dtype In-Reply-To: <43C9C573.2020809@ieee.org> References: <43C9C573.2020809@ieee.org> Message-ID: <1137415387.10345.4.camel@localhost.localdomain> Hi Travis, El ds 14 de 01 del 2006 a les 20:45 -0700, en/na Travis Oliphant va escriure: > 1) Replacing attributes that are now gone: > > .dtypechar --> .dtype.char > .dtypestr --> .dtype.str > .dtypedescr --> .dtype > > 2) Changing old .dtype -> .dtype.type > > This is only necessary if you were using a.dtype as a *typeobject* > as in > issubclass(a.dtype, ) > > If you were using .dtype as a parameter to dtype= then that usage > will still work > great (in fact a little faster) because now .dtype returns a > "descriptor object" > > 3) The dtypedescr constructor is now called dtype. > > This change should have gone into the 0.9.2 release, but things got too > hectic with all the name changes. I will quickly release 0.9.4 with > these changes unless I hear strong disagreements within the next few days. Glad that you liked the suggestion :-). BTW, if you are going to release 0.9.4 soon, do not forget to correct this typo: --- python.nobackup/numpy/trunk/numpy/core/numerictypes.py 2006-01-16 11:07:13.000000000 +0100 +++ /usr/lib/python2.4/site-packages/numpy/core/numerictypes.py 2006-01-16 11:59:28.000000000 +0100 @@ -167,7 +167,7 @@ typeDict[tmpstr] = typeobj na_name = tmpstr elif base == 'complex': - na_num = '%s%d' % (base.capitalize(), bit/2) + na_name = '%s%d' % (base.capitalize(), bit/2) elif base == 'bool': na_name = base.capitalize() typeDict[na_name] = typeobj Thanks, -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From faltet at carabos.com Mon Jan 16 08:57:24 2006 From: faltet at carabos.com (Francesc Altet) Date: Mon Jan 16 08:57:24 2006 Subject: [Numpy-discussion] Inconsistency in dtype.arrdescr? Message-ID: <1137430595.10345.55.camel@localhost.localdomain> Hi, I find the next a bit misleading: >>> numpy.array((3,), dtype='u4').dtype.arrdescr [('', '>> numpy.zeros((3,), dtype='u4,u2').dtype.arrdescr [('f1', '0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From efiring at hawaii.edu Mon Jan 16 10:51:02 2006 From: efiring at hawaii.edu (Eric Firing) Date: Mon Jan 16 10:51:02 2006 Subject: [Numpy-discussion] bug in shape() Message-ID: <43CBEAEB.9000402@hawaii.edu> Travis et al., Jeff Whitaker pointed out to me that masked array support in matplotlib contouring does not work with numpy. The problem is that the shape function does not work with masked arrays. One easy solution is to return to the older Numeric version of shape via the attached patch against svn (edited to remove irrelevant chunks). Thanks. Eric -------------- next part -------------- A non-text attachment was scrubbed... Name: shape.patch Type: text/x-patch Size: 402 bytes Desc: not available URL: From gerard.vermeulen at grenoble.cnrs.fr Mon Jan 16 12:07:17 2006 From: gerard.vermeulen at grenoble.cnrs.fr (Gerard Vermeulen) Date: Mon Jan 16 12:07:17 2006 Subject: [Numpy-discussion] buglet in numpy-0.9.4.1915 Message-ID: <20060116210641.225a13fa.gerard.vermeulen@grenoble.cnrs.fr> PyChecker found this: /home/packer/usr/lib/python2.4/site-packages/numpy/core/numeric.py:311: No global (math) found math must be probably replaced by umath. I did not verify all of PyChecker's noise, but this one is obviously a bug. Gerard From oliphant.travis at ieee.org Mon Jan 16 12:08:01 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Jan 16 12:08:01 2006 Subject: [Numpy-discussion] Inconsistency in dtype.arrdescr? In-Reply-To: <1137430595.10345.55.camel@localhost.localdomain> References: <1137430595.10345.55.camel@localhost.localdomain> Message-ID: <43CBFCD2.1070602@ieee.org> Francesc Altet wrote: >Hi, > >I find the next a bit misleading: > > >>>>numpy.array((3,), dtype='u4').dtype.arrdescr >>>> >>>> >[('', ' >but... > > > >>>>numpy.zeros((3,), dtype='u4,u2').dtype.arrdescr >>>> >>>> >[('f1', ' >I think it would be more consistent if the former expression would give >as output: > >[('f1', ' > > The problem is these are two separate circumstances. In the first case you have no fields defined and so no field names. In the second case, you have two fields which receive default field names. -Travis From ndarray at mac.com Mon Jan 16 13:16:06 2006 From: ndarray at mac.com (Sasha) Date: Mon Jan 16 13:16:06 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: References: <2D771DF0-1A60-463B-A643-AD6E2980AB94@stanford.edu> Message-ID: > Wouldn't then a > ~/my_numpy/include/numpy/* > be reasonable? > I've asked the same question before: http://www.scipy.net/pipermail/scipy-dev/2006-January/004952.html I understand that this was done for the benefit of people who have permissions to write to site-packages, but not to the include directory. I don't see why such users cannot just provide --prefix and why would any administrator allow users without administrative privileges to write to the system site-packages directory. For what it's worth, my +1 on keeping includes in include. -- sasha From pearu at cens.ioc.ee Mon Jan 16 13:45:01 2006 From: pearu at cens.ioc.ee (pearu at cens.ioc.ee) Date: Mon Jan 16 13:45:01 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: <2D771DF0-1A60-463B-A643-AD6E2980AB94@stanford.edu> Message-ID: On Sun, 15 Jan 2006, Zachary Pincus wrote: > >> Or is this a policy change which puts the headers in the site- > >> packages > >> directory? > > > > This is not an error. Use numpy.get_numpy_include() to retrive the > > directory of numpy header files. See numpy.get_numpy_include.__doc__ > > for more information. > > Thanks for the information. Unfortunately, the package I'm working on > can't be built with distutils, so it may wind up being something of a > contortion to call numpy.get_numpy_include() from the build system I > need to use. > > Understanding that the optimal, preferred method of finding the > include dir will always be numpy.get_numpy_include(), if I must > resort to a suboptimal method of guessing likely locations, what will > those locations be? > > Are they generally within the site-packages directory? Sometimes in > the python include directory? Will they be ever-changing? If you install numpy, it's header files, e.g. numpy/arrayobject.h, always endup in numpy/core/include directory. So, using import numpy os.path.join(os.path.dirname(numpy.__file__),'core','include') should always return correct path to numpy header files. Pearu From pearu at cens.ioc.ee Mon Jan 16 13:57:01 2006 From: pearu at cens.ioc.ee (pearu at cens.ioc.ee) Date: Mon Jan 16 13:57:01 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: Message-ID: On Mon, 16 Jan 2006, Arnd Baecker wrote: > On Mon, 16 Jan 2006, David M. Cooke wrote: > > > On Jan 15, 2006, at 14:23 , Zachary Pincus wrote: > > > > >>> Or is this a policy change which puts the headers in the site- > > >>> packages directory? > > >> > > >> This is not an error. Use numpy.get_numpy_include() to retrive the > > >> directory of numpy header files. See numpy.get_numpy_include.__doc__ > > >> for more information. > > > > > > Thanks for the information. Unfortunately, the package I'm working > > > on can't be built with distutils, > > It seems that distutils is not loved that much > (E.g. Wax no longer uses distutils > http://zephyrfalcon.org/weblog2/arch_e10_00870.html#e878 ) What Wax does, works well with pure Python packages, so I guess Wax is a pure Python package. As soon as it would need extension modules, one should need to use distutils or some other extension building utility. > > > Understanding that the optimal, preferred method of finding the > > > include dir will always be numpy.get_numpy_include(), if I must > > > resort to a suboptimal method of guessing likely locations, what > > > will those locations be? > > > > > > Are they generally within the site-packages directory? Sometimes in > > > the python include directory? Will they be ever-changing? > > > > For now, they're in the site-packages directory, under numpy/core/ > > include. That position might change (it used to be numpy/base/ > > include, for instance), if we rename modules again. They probably > > won't migrate to the python include directory for the reason they're > > not in there now (non-root users can't install there). The location of numpy header files will not change in future, at least I don't see any reason for doing so.. > If I do a `python setup.py --prefix=$HOME/my_numpy install`, > this leads to > > ~/my_numpy/bin/f2py > ~/my_numpy/lib/python2.3/site-packages/numpy/* > > Wouldn't then a > ~/my_numpy/include/numpy/* > be reasonable? > > Presumably I am overlooking something, and I know that this > has been discussed in much detail before, but still ... Yes, this has been discussed before and the discussion is in scipy-dev archives somewhere.. The location of numpy header files is well-documented (we have numpy/doc/CAPI.txt and numpy.get_numpy_include()), so I don't understand what is the problem with the current situation. Pearu From oliphant.travis at ieee.org Mon Jan 16 13:57:04 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Jan 16 13:57:04 2006 Subject: [Numpy-discussion] Re: [SciPy-user] numpy.dft.real+fft problem? In-Reply-To: <43CBBD45.4050306@gmail.com> References: <43CBBD45.4050306@gmail.com> Message-ID: <43CC1686.3000401@ieee.org> Andrew Jaffe wrote: >Hi All, > >[This has already appeared on python-scientific-devel; apologies to >those of you seeing it twice...] > >There seems to be a problem with the real_fft routine; starting with a >length n array, it should give a length n/2+1 array with real numbers >in the first and last positions (for n even). However, note that the >last entry on the first row has a real part, as opposed to the >complex routine fft, which has it correct. > >The weirdness of the real part seems to imply it's due to uninitialized >memory. > > Thanks for the catch. Yes it was due to uninitalized memory. PyArray_FromDims always initializes the memory to zero (but it uses int * arguments). The new routine was using PyArray_SimpleNew (using intp* arguments) which does not initialize the memory to zero. Changing PyArray_SimpleNew to PyArray_Zeros fixed the problem. -Travis From alexander.belopolsky at gmail.com Mon Jan 16 14:14:03 2006 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Mon Jan 16 14:14:03 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: References: Message-ID: > Yes, this has been discussed before and the discussion is in scipy-dev > archives somewhere.. The location of numpy header files is well-documented > (we have numpy/doc/CAPI.txt and numpy.get_numpy_include()), so I don't > understand what is the problem with the current situation. The problem with the current situation is that the include files end up in a rather obscure place. Is there any other python extension package that installs its include files in site-packages? Can someone summarize the reasons for the change so that this problem can find its resting place in a FAQ somewhere? What problem does the new location solve that cannot be solved with the --prefix option? -- sasha From ndarray at mac.com Mon Jan 16 14:15:02 2006 From: ndarray at mac.com (Sasha) Date: Mon Jan 16 14:15:02 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: References: Message-ID: > Yes, this has been discussed before and the discussion is in scipy-dev > archives somewhere.. The location of numpy header files is well-documented > (we have numpy/doc/CAPI.txt and numpy.get_numpy_include()), so I don't > understand what is the problem with the current situation. The problem with the current situation is that the include files end up in a rather obscure place. Is there any other python extension package that installs its include files in site-packages? Can someone summarize the reasons for the change so that this problem can find its resting place in a FAQ somewhere? What problem does the new location solve that cannot be solved with the --prefix option? -- sasha From cjw at sympatico.ca Mon Jan 16 14:23:04 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Mon Jan 16 14:23:04 2006 Subject: [Numpy-discussion] Inconsistency in dtype.arrdescr? In-Reply-To: <1137430595.10345.55.camel@localhost.localdomain> References: <1137430595.10345.55.camel@localhost.localdomain> Message-ID: <43CC1C9E.40009@sympatico.ca> Francesc Altet wrote: >Hi, > >I find the next a bit misleading: > > > >>>>numpy.array((3,), dtype='u4').dtype.arrdescr >>>> >>>> >[('', ' >but... > > > >>>>numpy.zeros((3,), dtype='u4,u2').dtype.arrdescr >>>> >>>> >[('f1', ' >I think it would be more consistent if the former expression would give >as output: > >[('f1', ' >Cheers, > > > Why not respond with a simple character string, such as f1, References: <1137430595.10345.55.camel@localhost.localdomain> <43CC1C9E.40009@sympatico.ca> Message-ID: <43CC1D7A.5030404@ieee.org> Colin J. Williams wrote: > Francesc Altet wrote: > >> Hi, >> >> I find the next a bit misleading: >> >> >> >>>>> numpy.array((3,), dtype='u4').dtype.arrdescr >>>>> >>>> >> [('', '> >> but... >> >> >> >>>>> numpy.zeros((3,), dtype='u4,u2').dtype.arrdescr >>>>> >>>> >> [('f1', '> >> I think it would be more consistent if the former expression would give >> as output: >> >> [('f1', '> >> Cheers, >> >> >> > Why not respond with a simple character string, such as f1, References: <1137411430.3273.5.camel@localhost.localdomain> Message-ID: > Before, I think numpy supported up to 32 dimensions. Is there any reason > for this new limit? Just curious. It was actually 40 until recently. I don't know the answer to your question (Travis?), but I am curious why would anyone need more than say 4? Solving PDEs by finite differences method in more than 4 dimensional spaces, anyone? I know I sound like some very well known person, but really 20 "ought to be enough for everyone" (TM) :-). -- sasha From oliphant.travis at ieee.org Mon Jan 16 14:48:05 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Jan 16 14:48:05 2006 Subject: [Numpy-discussion] Maximum available dimensions in numpy In-Reply-To: <1137411430.3273.5.camel@localhost.localdomain> References: <1137411430.3273.5.camel@localhost.localdomain> Message-ID: <43CC228B.6020703@ieee.org> Francesc Altet wrote: >Hi, > >I've downloaded a fresh SVN version of numpy (0.9.4.1914) and got a bit >surprised becuase of this: > > > >>>>numpy.reshape(numpy.array((22,)), (1,)*20) >>>> >>>> >array([[[[[[[[[[[[[[[[[[[[22]]]]]]]]]]]]]]]]]]]]) > > >>>>numpy.reshape(numpy.array((22,)), (1,)*21) >>>> >>>> >Traceback (most recent call last): > File "", line 1, in ? > File "/usr/lib/python2.3/site-packages/numpy/core/oldnumeric.py", line >164, in reshape > return asarray(a).reshape(newshape) >ValueError: sequence too large; must be smaller than 20 > >Before, I think numpy supported up to 32 dimensions. Is there any reason >for this new limit? Just curious. > > No, just trying to be a bit more conservative. If there is really a need for larger dimensions, it can be changed using a simple define in arrayobject.h -Travis From pebarrett at gmail.com Mon Jan 16 14:52:02 2006 From: pebarrett at gmail.com (Paul Barrett) Date: Mon Jan 16 14:52:02 2006 Subject: [Numpy-discussion] Maximum available dimensions in numpy In-Reply-To: References: <1137411430.3273.5.camel@localhost.localdomain> Message-ID: <40e64fa20601161451w659a1c1tfbe8c8afdb453610@mail.gmail.com> On 1/16/06, Sasha wrote: > > > Before, I think numpy supported up to 32 dimensions. Is there any reason > > for this new limit? Just curious. > > It was actually 40 until recently. I don't know the answer to your > question (Travis?), but I am curious why would anyone need more than > say 4? Solving PDEs by finite differences method in more than 4 > dimensional spaces, anyone? I know I sound like some very well known > person, but really 20 "ought to be enough for everyone" (TM) :-). > How about setting the default case to 3 or 4 dimensions and then special casing the rare higher dimensional arrays, i.e. using malloc for these situations. The default dimension size could be a compile time option for those who routinely exceed the default size of 3 or 4. -- Paul -- Paul Barrett, PhD Johns Hopkins University Assoc. Research Scientist Dept of Physics and Astronomy Phone: 410-516-5190 Baltimore, MD 21218 -------------- next part -------------- An HTML attachment was scrubbed... URL: From pearu at cens.ioc.ee Mon Jan 16 15:14:00 2006 From: pearu at cens.ioc.ee (pearu at cens.ioc.ee) Date: Mon Jan 16 15:14:00 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: Message-ID: On Mon, 16 Jan 2006, Sasha wrote: > > Yes, this has been discussed before and the discussion is in scipy-dev > > archives somewhere.. The location of numpy header files is well-documented > > (we have numpy/doc/CAPI.txt and numpy.get_numpy_include()), so I don't > > understand what is the problem with the current situation. > > The problem with the current situation is that the include files end > up in a rather obscure > place. Is there any other python extension package that installs its > include files in site-packages? Can someone summarize the reasons for > the change so that this problem > can find its resting place in a FAQ somewhere? What problem does the > new location solve that cannot be solved with the --prefix option? See the threads starting at http://www.scipy.org/mailinglists/mailman?fn=scipy-dev/2005-October/003420.html http://www.scipy.org/mailinglists/mailman?fn=scipy-user/2005-October/005454.html Basically, the main argument behind the current decision is that there are systems where people don't have permissions to install header files to std python include directory but they can install to python site-packages (e.g. when installing prebuilt distributions). Also note that not all potential numpy users are able/willing/.. to use --prefix option. Why we should scare them away when we can provide defaults that work for all situations, though the defaults are not standard and require some tolerance from developers to follow numpy docs.. Pearu From wbaxter at gmail.com Mon Jan 16 15:43:01 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Mon Jan 16 15:43:01 2006 Subject: [Numpy-discussion] The Low-down on numpy/scipy? Message-ID: Just this past week I thought I'd check out NumPy/SciPy to see if I might be able to port some of my matlab code over to that, easier than I was finding porting it to C++. But I find the current situation somewhat confusing. Travis' page says that scipy_core is now the word, and numarray/numeric are dead, but I can't find any links for a scipy_core download anywhere. Where should someone new to numpy and scipy start, if they are just beginning to look into it as of today? Would I be better off just waiting a month or two for things to settle down? Thanks, Bill Baxter -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Mon Jan 16 16:14:09 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Jan 16 16:14:09 2006 Subject: [Numpy-discussion] The Low-down on numpy/scipy? In-Reply-To: References: Message-ID: <43CC36B2.5050806@ieee.org> Bill Baxter wrote: > Just this past week I thought I'd check out NumPy/SciPy to see if I > might be able to port some of my matlab code over to that, easier than > I was finding porting it to C++. But I find the current situation > somewhat confusing. Travis' page says that scipy_core is now the word, Which page is that? Please help me make it less confusing... The word is numpy. > > Where should someone new to numpy and scipy start, if they are just > beginning to look into it as of today? Go to http://numeric.scipy.org (which is the homepage of the numpy project page at sourceforge) For scipy, go to http://new.scipy.org which should be moved over to http://www.scipy.org in the near future. -Travis From ndarray at mac.com Mon Jan 16 16:20:01 2006 From: ndarray at mac.com (Sasha) Date: Mon Jan 16 16:20:01 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: References: Message-ID: On 1/16/06, pearu at cens.ioc.ee wrote: > ... > Basically, the main argument behind the current decision is that there are > systems where people don't have permissions to install header files to std > python include directory but they can install to python site-packages > (e.g. when installing prebuilt distributions). > That's the argument that I mentioned. It also seems to be not just the "main", but the *only* argument. I am still not convinced. What kind of systems do you have in mind? If you are talking about Linux distributions that prepackage python, there is no reason for a sysadimin to let users write to /usr/lib/pythonX.Y/site-packages and not to /usr/include/pythonX.Y/. It is much easier to mess up a say Red Hat system by placing wrong files into /usr/lib/pythonX.Y/site-packages than by messing with /usr/include/pythonX.Y/. A site that would like to allow users to install python packages should either recommend them to use --prefix or provide an installation in say /usr/local that has more liberal permissions to pythonX.Y directories. > Also note that not all potential numpy users are able/willing/.. to use > --prefix option. Why we should scare them away when we can provide > defaults that work for all situations, though the defaults are not > standard and require some tolerance from developers to follow numpy docs. Personally, I can tolerate any location less that 10 levels deep under install prefix and current location is only 8 :-). However, if there is a valid reason to change the location of include files, this functionality should be implemented as a (possibly default) option to distutils, not by treating includes as "data_files" rather than "headers". If the alternative location is specified as an option in site.cfg and is accompanied by a comment describing how to disable it, I am sure there will be fewer posts asking where did the headers go in the future. -- sasha From ndarray at mac.com Mon Jan 16 16:24:04 2006 From: ndarray at mac.com (Sasha) Date: Mon Jan 16 16:24:04 2006 Subject: [Numpy-discussion] The Low-down on numpy/scipy? In-Reply-To: <43CC36B2.5050806@ieee.org> References: <43CC36B2.5050806@ieee.org> Message-ID: On 1/16/06, Travis Oliphant wrote: > For scipy, go to http://new.scipy.org ... That should be http://new.scipy.org/Wiki -- sasha From wbaxter at gmail.com Mon Jan 16 19:38:01 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Mon Jan 16 19:38:01 2006 Subject: [Numpy-discussion] The Low-down on numpy/scipy? In-Reply-To: References: <43CC36B2.5050806@ieee.org> Message-ID: Ok. Thanks for trying to clear this up. Top hit on a google for numpy is here: http://numeric.scipy.org/ In the text it says that "the new scipy used to be called scipy core but not anymore". However, right below that in big bold letters it says "New Features of SciPy Core". Then the page goes on to consistently refer to it as scipy core, despite that apparently no longer being the name. So it's easy to miss that the name has changed. Also the only file download link on that page is for the sourceforge site http://sourceforge.net/project/showfiles.php?group_id=1369&package_id=175103 Which has downloads for NumPy, but also has a prominent news item announcing the release of scipy_core. The presence of the news release there led me to belive that the download for scipy_core would also be located there. So it would be helpful if the numeric.scipy.org page had a link for downloading scipy. The links to the scipy_core presentations just make it seem all the more like you should be able to download scipy_core from there. This page: http://new.scipy.org/Wiki/Download could perhaps mention that the SciPy they are offering is the package formerly known as scipy_core, to additionally eliminate confusion. There was another page I was able to get to previously, perhaps www.scipy.org, but it's not responding now. Anyway, there was one scipy page that said basically "information on these pages may be out of date, there's something new afoot" but I don't seem to recall any handy link taking me directly to the proper downloads. Or maybe I was still thinking I was looking for something called scipy_core, rather than scipy. Hope that information can help you make the web pages clearer. Thanks for your help. --Bill On 1/17/06, Sasha wrote: > > On 1/16/06, Travis Oliphant wrote: > > For scipy, go to http://new.scipy.org ... > > That should be http://new.scipy.org/Wiki > > -- sasha > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.hochberg at cox.net Mon Jan 16 20:00:03 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Mon Jan 16 20:00:03 2006 Subject: [Numpy-discussion] Maximum available dimensions in numpy In-Reply-To: <40e64fa20601161451w659a1c1tfbe8c8afdb453610@mail.gmail.com> References: <1137411430.3273.5.camel@localhost.localdomain> <40e64fa20601161451w659a1c1tfbe8c8afdb453610@mail.gmail.com> Message-ID: <43CC6B58.80302@cox.net> Paul Barrett wrote: > On 1/16/06, *Sasha* > wrote: > > > Before, I think numpy supported up to 32 dimensions. Is there > any reason > > for this new limit? Just curious. > > It was actually 40 until recently. I don't know the answer to your > question (Travis?), but I am curious why would anyone need more than > say 4? Solving PDEs by finite differences method in more than 4 > dimensional spaces, anyone? I know I sound like some very well known > person, but really 20 "ought to be enough for everyone" (TM) :-). > > > How about setting the default case to 3 or 4 dimensions and then > special casing the rare higher dimensional arrays, i.e. using malloc > for these situations. The default dimension size could be a compile > time option for those who routinely exceed the default size of 3 or 4. This seems like premature optimization. In most cases, if you're in a situation where the dimensional overhead matters (lot's of small arrays) you are using Numeric/Numarray/NumPy poorly and your code is going to be slow and bloated anyway. The way to get good efficiency with these extensions is to do block operations on large matrices. This often involves a little trickery and several extra dimensions. Reducing the default matrix size down to 3 or 4 makes efficient code slower since going through malloc will involve an extra dereference and probably some extra branches. There's also no point in setting the default max matrix size to if the typical default allignment (8 byte, IIRC) is going to leave some bytes unused due to allignment. If one were to pursue some sort of hybrid scheme as proposed above, I'd say at a minimum a default dimension size would be 6; larger depending on how the allignment works out. I am also leary of this sort of hybrid scheme since the code for dimensions larger the the code would be little tested and would thus be a good place for bugs to lurk. I don't see anything wrong with making the maximum dimension size a compile time option though. However, since in the common case the extra dimensions are unlikely to affect performance in any meaningful, I'd recomend that the maximum number of default dimensions stay large by default. Thus people who need to conserve bytes, which I'd consider the rare case, have the option of reducing the max dimensions while the arrays behave in an unsuprising manner when compiled in the normal manner. If someone has the extra time, it would be interesting to see some data about how always mallocing the extra dimensions, so there was no maximum dimensions limit, affects performance. I'd also be interested in seeing cases where the extra dimensions actually affect performance before doing to stuff complicate^H^H^H^H^H^H^H^H fix things. -tim > > -- Paul > > -- > Paul Barrett, PhD Johns Hopkins University > Assoc. Research Scientist Dept of Physics and Astronomy > Phone: 410-516-5190 Baltimore, MD 21218 From pebarrett at gmail.com Mon Jan 16 20:31:02 2006 From: pebarrett at gmail.com (Paul Barrett) Date: Mon Jan 16 20:31:02 2006 Subject: [Numpy-discussion] Maximum available dimensions in numpy In-Reply-To: <43CC6B58.80302@cox.net> References: <1137411430.3273.5.camel@localhost.localdomain> <40e64fa20601161451w659a1c1tfbe8c8afdb453610@mail.gmail.com> <43CC6B58.80302@cox.net> Message-ID: <40e64fa20601162030v67941d24t62288147507c9822@mail.gmail.com> On 1/16/06, Tim Hochberg wrote: > > Paul Barrett wrote: > > > On 1/16/06, *Sasha* > wrote: > > > > > Before, I think numpy supported up to 32 dimensions. Is there > > any reason > > > for this new limit? Just curious. > > > > It was actually 40 until recently. I don't know the answer to your > > question (Travis?), but I am curious why would anyone need more than > > say 4? Solving PDEs by finite differences method in more than 4 > > dimensional spaces, anyone? I know I sound like some very well known > > person, but really 20 "ought to be enough for everyone" (TM) :-). > > > > > > How about setting the default case to 3 or 4 dimensions and then > > special casing the rare higher dimensional arrays, i.e. using malloc > > for these situations. The default dimension size could be a compile > > time option for those who routinely exceed the default size of 3 or 4. > > This seems like premature optimization. In most cases, if you're in a > situation where the dimensional overhead matters (lot's of small arrays) > you are using Numeric/Numarray/NumPy poorly and your code is going to be > slow and bloated anyway. The way to get good efficiency with these > extensions is to do block operations on large matrices. This often > involves a little trickery and several extra dimensions. Reducing the > default matrix size down to 3 or 4 makes efficient code slower since > going through malloc will involve an extra dereference and probably some > extra branches. It also avoids the possibility of running up against the maximum number of dimensions, while conserving memory. For those users that create a multitude of small arrays, the wasted memory might become important. I only suggested 3 or 4 dimensions, because it would appear to cover 99% of the cases. I hope those users creating the other 1%, know what they are doing. There's also no point in setting the default max matrix size to if > the typical default allignment (8 byte, IIRC) is going to leave some > bytes unused due to allignment. If one were to pursue some sort of > hybrid scheme as proposed above, I'd say at a minimum a default > dimension size would be 6; larger depending on how the allignment works > out. I am also leary of this sort of hybrid scheme since the code for > dimensions larger the the code would be little tested and would thus > be a good place for bugs to lurk. Agreed. But 20 or 30 extra dimensions also seems rather a waste and ad hoc. I don't see anything wrong with making the maximum dimension size a > compile time option though. However, since in the common case the extra > dimensions are unlikely to affect performance in any meaningful, I'd > recomend that the maximum number of default dimensions stay large by > default. Thus people who need to conserve bytes, which I'd consider the > rare case, have the option of reducing the max dimensions while the > arrays behave in an unsuprising manner when compiled in the normal manner. > > If someone has the extra time, it would be interesting to see some data > about how always mallocing the extra dimensions, so there was no maximum > dimensions limit, affects performance. I'd also be interested in seeing > cases where the extra dimensions actually affect performance before > doing to stuff complicate^H^H^H^H^H^H^H^H fix things. > I'd venture to guess that mallocing the extra dimensions would give a small, though noticeable, decrease in performance. Of course, as the array becomes larger, this overhead will decrease in relation to the time it takes to perform the operation, much like the overhead seen in numarray, though not as large. As you mentioned, arrays with many dimensions are often large arrays, so the malloc overhead will probably not be too significant. -- Paul -- Paul Barrett, PhD Johns Hopkins University Assoc. Research Scientist Dept of Physics and Astronomy Phone: 410-516-5190 Baltimore, MD 21218 -------------- next part -------------- An HTML attachment was scrubbed... URL: From Fernando.Perez at colorado.edu Mon Jan 16 20:52:14 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Mon Jan 16 20:52:14 2006 Subject: [Numpy-discussion] Maximum available dimensions in numpy In-Reply-To: <40e64fa20601162030v67941d24t62288147507c9822@mail.gmail.com> References: <1137411430.3273.5.camel@localhost.localdomain> <40e64fa20601161451w659a1c1tfbe8c8afdb453610@mail.gmail.com> <43CC6B58.80302@cox.net> <40e64fa20601162030v67941d24t62288147507c9822@mail.gmail.com> Message-ID: <43CC77B4.6050503@colorado.edu> Paul Barrett wrote: >>>How about setting the default case to 3 or 4 dimensions and then >>>special casing the rare higher dimensional arrays, i.e. using malloc >>>for these situations. The default dimension size could be a compile >>>time option for those who routinely exceed the default size of 3 or 4. >> >>This seems like premature optimization. In most cases, if you're in a >>situation where the dimensional overhead matters (lot's of small arrays) >>you are using Numeric/Numarray/NumPy poorly and your code is going to be >>slow and bloated anyway. The way to get good efficiency with these >>extensions is to do block operations on large matrices. This often >>involves a little trickery and several extra dimensions. Reducing the >>default matrix size down to 3 or 4 makes efficient code slower since >>going through malloc will involve an extra dereference and probably some >>extra branches. > > > > It also avoids the possibility of running up against the maximum number of > dimensions, while conserving memory. For those users that create a multitude > of small arrays, the wasted memory might become important. I only suggested > 3 or 4 dimensions, because it would appear to cover 99% of the cases. I > hope those users creating the other 1%, know what they are doing. How about following blitz convention (since we use blitz for looping in C++ fast via weave)? Blitz sets the max at 11 (I don't know if this comes from a Spinal Tap reference, string theory, or pure chance :). I agree that 32 is perhaps excessive, but 4 is certainly too low a number, I think. Given that most regular numerical algorithms scale exponentially with the number of dimensions, it's hard to think of anything useful you can do with a 32-dimensional array. But there are plenty of algorithms that brute-force their way through problems with k^d scaling, for d=10 or so. Cheers, f From pearu at cens.ioc.ee Mon Jan 16 21:07:03 2006 From: pearu at cens.ioc.ee (pearu at cens.ioc.ee) Date: Mon Jan 16 21:07:03 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: Message-ID: On Mon, 16 Jan 2006, Sasha wrote: > On 1/16/06, pearu at cens.ioc.ee wrote: > > ... > > Basically, the main argument behind the current decision is that there are > > systems where people don't have permissions to install header files to std > > python include directory but they can install to python site-packages > > (e.g. when installing prebuilt distributions). > > > > That's the argument that I mentioned. It also seems to be not just > the "main", but the *only* argument. I am still not convinced. What > kind of systems do you have in mind? If you are talking about Linux > distributions that prepackage python, ... No, not Linux. It was more like OSX issue, the exact platform should be mentioned somewhere in the discussion that I sent. Hopefully someone whom the header location change to std location would affect, can step in and give better arguments here than me.. > > Also note that not all potential numpy users are able/willing/.. to use > > --prefix option. Why we should scare them away when we can provide > > defaults that work for all situations, though the defaults are not > > standard and require some tolerance from developers to follow numpy docs. > > Personally, I can tolerate any location less that 10 levels deep under > install prefix and current location is only 8 :-). However, if there > is a valid reason to change the location of include files, this > functionality should be implemented as a (possibly default) option to > distutils, not by treating includes as "data_files" rather than > "headers". > > If the alternative location is specified as an option in site.cfg and > is accompanied by a comment describing how to disable it, I am sure > there will be fewer posts asking where did the headers go in the > future. I'm note sure that giving more options would be better. For me the exact location of numpy header files is unimportant as long as I have get_numpy_include(), or when using numpy.distutils that would use correct numpy include_dirs automatically when needed. Installing headers as data files has worked well sofar... except those posts:) Enhancing distutils would be another topic.. but the process would be too slow for the scipy project, we need numpy.distutils features now. If someone could start PEP'ing std distutils with numpy.distutils features, that would be great. Pearu From rudolphv at gmail.com Tue Jan 17 04:25:02 2006 From: rudolphv at gmail.com (Rudolph van der Merwe) Date: Tue Jan 17 04:25:02 2006 Subject: [Numpy-discussion] Use of numpy, scipy and matplotlib (pylab) together? Message-ID: <97670e910601170424w206793c7qa7c84e957cbb4882@mail.gmail.com> What is the correct importing order if one wants to use the full capabilities of numpy+scipy+matplotlib? I have all three packages installed and all of them import correctly. For Python code that makes use of all of these, should the import order be: from numpy import * from scipy import * from pylab import * ? I assume a certain amount of re-importing will occur since both scipy and matplotlib (pylab) uses numpy. How does one ensure that the correct objects don't get overwritten? -- Rudolph van der Merwe From list at jsaul.de Tue Jan 17 04:49:01 2006 From: list at jsaul.de (Joachim Saul) Date: Tue Jan 17 04:49:01 2006 Subject: [Numpy-discussion] Probable bug Message-ID: <20060117124739.GA23620@st18.gfz-potsdam.de> Hello all, I have observed a weird behaviour with 24.2, which actually can break existing code: >>> import Numeric >>> Numeric.__version__ '23.8' >>> x=Numeric.zeros(5) >>> type(x[0]) >>> x=Numeric.zeros(5,'f') >>> type(x[0]) >>> This is what I would expect. However, using 24.2: >>> import Numeric >>> Numeric.__version__ '24.2' >>> x=Numeric.zeros(5) >>> type(x[0]) >>> x=Numeric.zeros(5, 'f') >>> type(x[0]) >>> print type(1*x[0]) Strange, isn't it? Is there any rationale behind this inconsistent behaviour or is it just a bug? Cheers, Joachim From konrad.hinsen at laposte.net Tue Jan 17 06:45:11 2006 From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net) Date: Tue Jan 17 06:45:11 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: References: Message-ID: On Jan 17, 2006, at 1:18, Sasha wrote: >> Also note that not all potential numpy users are able/willing/.. >> to use >> --prefix option. Why we should scare them away when we can provide >> defaults that work for all situations, though the defaults are not >> standard and require some tolerance from developers to follow >> numpy docs. > > Personally, I can tolerate any location less that 10 levels deep under > install prefix > and current location is only 8 :-). However, if there is a valid > reason to change the location of include files, this functionality > should be implemented as a (possibly default) option to distutils, not > by treating includes as "data_files" rather than "headers". I tend to agree, and I am a bit less tolerant about playing with header file location. The argument "just follow the NumPy docs, it's not difficult" ignores that NumPy, like many other Python library, is a software component that many people install without reading any documentation, just because some other component requires it. If NumPy header files end up in a non-standard place, the authors of all Python packages that depend on NumPy, no matter by how many levels of indirection, have to add obscure code to their installation scrips. If every popular Python library did the same, we'd soon have a big mess. Moreover, it is possible that other libraries implement the same API as NumPy, but use different header conventions (in fact, Numeric and Numarray are candidates). Then every author of a package that indirectly depends on this API must not only add NumPy-specific code to his setup.py, but also check which of the multiple implementations is actually installed. Standard locations, be it for Python code or for header files, have been created to avoid this kind of mess. The standard for Python is defined by distutils, and at the moment it's putting header files into $prefix/include/pythonX.Y. If that turns out to be a bad decision on some platform, then distutils should be modified. If individual packages start trying to outsmart distutils, we lose the advantages of a standard installation system. Konrad. From hetland at tamu.edu Tue Jan 17 06:55:14 2006 From: hetland at tamu.edu (Robert Hetland) Date: Tue Jan 17 06:55:14 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: References: Message-ID: <22F30CE0-7362-44A7-8DA1-79C6FE9E70D0@tamu.edu> On Jan 16, 2006, at 5:12 PM, wrote: > Also note that not all potential numpy users are able/willing/.. to > use > --prefix option. Why we should scare them away when we can provide > defaults that work for all situations, though the defaults are not > standard and require some tolerance from developers to follow numpy > docs.. Why not just have the install script 'try:' to put a symlink in the standard place. That way, the files will be installed in any case, but (if successful) they will also be accessible from the standard places. If the symlinks are not created, a warning could be issued, but the install would not fail. I have used this solution -- putting the simlink in by hand -- and it seems to work fine. -Rob ----- Rob Hetland, Assistant Professor Dept of Oceanography, Texas A&M University p: 979-458-0096, f: 979-845-6331 e: hetland at tamu.edu, w: http://pong.tamu.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From konrad.hinsen at laposte.net Tue Jan 17 07:08:01 2006 From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net) Date: Tue Jan 17 07:08:01 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: <22F30CE0-7362-44A7-8DA1-79C6FE9E70D0@tamu.edu> References: <22F30CE0-7362-44A7-8DA1-79C6FE9E70D0@tamu.edu> Message-ID: <96D6BEBF-8A8D-4B69-9A6C-BC2A86BDA9B0@laposte.net> On Jan 17, 2006, at 15:55, Robert Hetland wrote: > Why not just have the install script 'try:' to put a symlink in the > standard place. That way, the files will be installed in any case, > but (if successful) they will also be accessible from the standard > places. If the symlinks are not created, a warning could be > issued, but the install would not fail. What good is that symlink if no one can rely on its presence? Konrad. -- --------------------------------------------------------------------- Konrad Hinsen Laboratoire L?on Brillouin, CEA Saclay, 91191 Gif-sur-Yvette Cedex, France Tel.: +33-1 69 08 79 25 Fax: +33-1 69 08 82 61 E-Mail: khinsen at cea.fr --------------------------------------------------------------------- From hetland at tamu.edu Tue Jan 17 07:15:08 2006 From: hetland at tamu.edu (Robert Hetland) Date: Tue Jan 17 07:15:08 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: <96D6BEBF-8A8D-4B69-9A6C-BC2A86BDA9B0@laposte.net> References: <22F30CE0-7362-44A7-8DA1-79C6FE9E70D0@tamu.edu> <96D6BEBF-8A8D-4B69-9A6C-BC2A86BDA9B0@laposte.net> Message-ID: <89D76E1D-BE44-43FF-AD68-D0A8FACD795D@tamu.edu> On Jan 17, 2006, at 9:08 AM, konrad.hinsen at laposte.net wrote: > What good is that symlink if no one can rely on its presence? True. I think putting the include files in a non-standard place should be the exception, rather than the rule. The standard install should behave itself, and put things in the right place. -Rob ----- Rob Hetland, Assistant Professor Dept of Oceanography, Texas A&M University p: 979-458-0096, f: 979-845-6331 e: hetland at tamu.edu, w: http://pong.tamu.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From byrnes at bu.edu Tue Jan 17 07:55:03 2006 From: byrnes at bu.edu (John Byrnes) Date: Tue Jan 17 07:55:03 2006 Subject: [Numpy-discussion] Use of numpy, scipy and matplotlib (pylab) together? In-Reply-To: <97670e910601170424w206793c7qa7c84e957cbb4882@mail.gmail.com> References: <97670e910601170424w206793c7qa7c84e957cbb4882@mail.gmail.com> Message-ID: <20060117155407.GA10790@localhost.localdomain> On Tue, Jan 17, 2006 at 02:24:33PM +0200, Rudolph van der Merwe wrote: > What is the correct importing order if one wants to use the full > capabilities of numpy+scipy+matplotlib? I believe the correct way to import the packages is to not use from blah import * but rather to use: import numpy as np import scipy as S import pylab as pl This keeps the namespace clean and prevents function collisions. HTH, John -- Courage and perseverance have a magical talisman, before which difficulties disappear and obstacles vanish into air. -- John Quincy Adams From hetland at tamu.edu Tue Jan 17 08:07:01 2006 From: hetland at tamu.edu (Robert Hetland) Date: Tue Jan 17 08:07:01 2006 Subject: [Numpy-discussion] Use of numpy, scipy and matplotlib (pylab) together? In-Reply-To: <20060117155407.GA10790@localhost.localdomain> References: <97670e910601170424w206793c7qa7c84e957cbb4882@mail.gmail.com> <20060117155407.GA10790@localhost.localdomain> Message-ID: <6C42166F-0E8E-4A3A-B951-D12F8A82680E@tamu.edu> On Jan 17, 2006, at 9:54 AM, John Byrnes wrote: > but rather to use: > import numpy as np > import scipy as S > import pylab as pl This may indeed be better (although I am generally lazy, and just import * them). I would certainly be willing to make the effort not to be lazy if there was a standard convention. Are there standard names for importing these packages? I have seen quite a few different conventions in various peoples code. E..g, why not import scipy as sp (instead of S)? From the Numeric days, people often import NumPy (et al.) as N, matplotlib as mpl, etc. -Rob ----- Rob Hetland, Assistant Professor Dept of Oceanography, Texas A&M University p: 979-458-0096, f: 979-845-6331 e: hetland at tamu.edu, w: http://pong.tamu.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From faltet at carabos.com Tue Jan 17 08:21:01 2006 From: faltet at carabos.com (Francesc Altet) Date: Tue Jan 17 08:21:01 2006 Subject: [Numpy-discussion] Some inconsistency in numarray.records format support Message-ID: <200601171719.40974.faltet@carabos.com> Hi, When building recarrays in numpy following the numarray convention: In [2]: numpy.zeros((3,), dtype='u1') Out[2]: array([0, 0, 0], dtype=uint8) In [3]: numpy.zeros((3,), dtype='u1,f4') Out[3]: array([(0, 0.0), (0, 0.0), (0, 0.0)], dtype=(void,5)) In [4]: numpy.zeros((3,), dtype='2u1,f4') Out[4]: array([(array([0, 0], dtype=uint8), 0.0), (array([0, 0], dtype=uint8), 0.0), (array([0, 0], dtype=uint8), 0.0)], dtype=(void,6)) In [7]: numpy.zeros((3,), dtype='(2,)u1,f4') Out[7]: array([(array([0, 0], dtype=uint8), 0.0), (array([0, 0], dtype=uint8), 0.0), (array([0, 0], dtype=uint8), 0.0)], dtype=(void,6)) So far so good, but In [5]: numpy.zeros((3,), dtype='2u1') --------------------------------------------------------------------------- exceptions.TypeError Traceback (most recent call last) /home/faltet/python.nobackup/numpy/ TypeError: data type not understood Also: In [6]: numpy.zeros((3,), dtype='(2,)u1') --------------------------------------------------------------------------- exceptions.ValueError Traceback (most recent call last) /home/faltet/python.nobackup/numpy/ /usr/lib/python2.4/site-packages/numpy/core/_internal.py in _commastring(astr) 296 res = _split(astr) 297 if (len(res)) == 1: --> 298 raise ValueError, "no commas present" 299 result = [] 300 for k,item in enumerate(res): ValueError: no commas present Perhaps this is the same case as defining tables with just one column? -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From jh at oobleck.astro.cornell.edu Tue Jan 17 08:26:06 2006 From: jh at oobleck.astro.cornell.edu (Joe Harrington) Date: Tue Jan 17 08:26:06 2006 Subject: [Numpy-discussion] Maximum available dimensions in numpy In-Reply-To: <20060117145603.DABE013765@sc8-sf-spam2.sourceforge.net> (numpy-discussion-request@lists.sourceforge.net) References: <20060117145603.DABE013765@sc8-sf-spam2.sourceforge.net> Message-ID: <200601171625.k0HGPde7007309@oobleck.astro.cornell.edu> For reasons that were laid out when this topic came up a month or so ago (perhaps on another of our infinitude of email lists), I'm opposed to reducing the number of dimensions to even 10. A sizeable chunk of the community does use many dimensions. I routinely use more than 5 dimensions, and I'm pretty good at reducing the number for performance where it's practical. There's an atmospheric sciences tool called vis5d (x,y,z,t,field parameter vector), which shows just how commonly meteorologists use 5 dimensions. I don't think of the atmospheric sciences as being a particularly dimension-heavy group, but you can easily think of adding a few more dimensions that let you choose model runs with varied initial parameters. Try tracking the variables of a complex medical trial in just 5 dimensions. IDL's limit is 7 dimensions, and many don't use it as a result. Having more in our default binary installs is a selling point. This is a pretty limiting change to be suggesting, so if there's a change to be made, it should come only after implementing and benchmarking to see what the actual performance benefits are, and then polling the community. If there is a really big improvement in having 4 or 8 dimensions max, and if "malloc and unlimit everything" isn't fast, then it may be worth supporting low-dim and high-dim versions of the binary installs. But, let's wait until we have more volunteers before doing that. --jh-- From cjw at sympatico.ca Tue Jan 17 08:38:01 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Tue Jan 17 08:38:01 2006 Subject: [Numpy-discussion] Use of numpy, scipy and matplotlib (pylab) together? In-Reply-To: <20060117155407.GA10790@localhost.localdomain> References: <97670e910601170424w206793c7qa7c84e957cbb4882@mail.gmail.com> <20060117155407.GA10790@localhost.localdomain> Message-ID: <43CD1D14.4080309@sympatico.ca> John Byrnes wrote: >On Tue, Jan 17, 2006 at 02:24:33PM +0200, Rudolph van der Merwe wrote: > > >>What is the correct importing order if one wants to use the full >>capabilities of numpy+scipy+matplotlib? >> >> >I believe the correct way to import the packages is to not use > >from blah import * > >but rather to use: >import numpy as np >import scipy as S >import pylab as pl > >This keeps the namespace clean and prevents function collisions. > >HTH, >John > > +1 From ndarray at mac.com Tue Jan 17 09:10:07 2006 From: ndarray at mac.com (Sasha) Date: Tue Jan 17 09:10:07 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: <89D76E1D-BE44-43FF-AD68-D0A8FACD795D@tamu.edu> References: <22F30CE0-7362-44A7-8DA1-79C6FE9E70D0@tamu.edu> <96D6BEBF-8A8D-4B69-9A6C-BC2A86BDA9B0@laposte.net> <89D76E1D-BE44-43FF-AD68-D0A8FACD795D@tamu.edu> Message-ID: My main concern is not with the non-standard default location, but with the way distutils are tricked into placing headers into that location. Currently this is done by mislabeling the header files as "data files" in setup.py as follows: config.add_data_files(join('include','numpy','*.h')) This looks like a hack that just happens to work on platforms where headers do not require any special treatement such as end-of-line conversion or pre-compilation. This is not guaranteed to work on every python platform now and may certainly break in the future. -- sasha On 1/17/06, Robert Hetland wrote: > > > On Jan 17, 2006, at 9:08 AM, konrad.hinsen at laposte.net wrote: > > > What good is that symlink if no one can rely on its presence? > > True. > > I think putting the include files in a non-standard place should be the > exception, rather than the rule. The standard install should behave itself, > and put things in the right place. > > -Rob > > > ----- > Rob Hetland, Assistant Professor > Dept of Oceanography, Texas A&M University > p: 979-458-0096, f: 979-845-6331 > e: hetland at tamu.edu, w: http://pong.tamu.edu > From strawman at astraw.com Tue Jan 17 09:30:14 2006 From: strawman at astraw.com (Andrew Straw) Date: Tue Jan 17 09:30:14 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: References: <22F30CE0-7362-44A7-8DA1-79C6FE9E70D0@tamu.edu> <96D6BEBF-8A8D-4B69-9A6C-BC2A86BDA9B0@laposte.net> <89D76E1D-BE44-43FF-AD68-D0A8FACD795D@tamu.edu> Message-ID: <43CD296C.7000700@astraw.com> The main advantage of the current situation for me is that it's easy to use with .eggs (for the reasons given). The symlink idea won't break this, but the "let's make it like it was" solution would break it (which was why it was changed). Can anyone think of a solution that would please both the old-way-preferers and the .egg users? Also, as for as I know, the only packages that install things into /usr/include/python2.x/packagename are Numeric and numarray, so I would argue it's not "standard", (although it may have lots of history). Sasha wrote: >My main concern is not with the non-standard default location, but >with the way distutils are tricked into placing headers into that >location. Currently this is done by mislabeling the header files as >"data files" in setup.py as follows: > > config.add_data_files(join('include','numpy','*.h')) > >This looks like a hack that just happens to work on platforms where >headers do not require any special treatement such as end-of-line >conversion or pre-compilation. This is not guaranteed to work on every >python platform now and may certainly break in the future. > >-- sasha > >On 1/17/06, Robert Hetland wrote: > > >>On Jan 17, 2006, at 9:08 AM, konrad.hinsen at laposte.net wrote: >> >> >>What good is that symlink if no one can rely on its presence? >> >>True. >> >>I think putting the include files in a non-standard place should be the >>exception, rather than the rule. The standard install should behave itself, >>and put things in the right place. >> >>-Rob >> >> >>----- >>Rob Hetland, Assistant Professor >>Dept of Oceanography, Texas A&M University >>p: 979-458-0096, f: 979-845-6331 >>e: hetland at tamu.edu, w: http://pong.tamu.edu >> >> >> > > >------------------------------------------------------- >This SF.net email is sponsored by: Splunk Inc. Do you grep through log files >for problems? Stop! Download the new AJAX search engine that makes >searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! >http://sel.as-us.falkag.net/sel?cmd=k&kid3432&bid#0486&dat1642 >_______________________________________________ >Numpy-discussion mailing list >Numpy-discussion at lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > From sj at cs.umb.edu Tue Jan 17 10:24:10 2006 From: sj at cs.umb.edu (Szymon Jaroszewicz) Date: Tue Jan 17 10:24:10 2006 Subject: [Numpy-discussion] Re: Maximum available dimensions in numpy Message-ID: <1137522194.9752.18.camel@localhost> I'm using multidimensional arrays (numarray currently) for storing conditional probability distributions in Bayesian networks, in which case a number of dimensions in the range of 20 or more can happen easily. Obviously for this reason I think its not a good idea to limit the dimensionality, especially that its not clear if the gain is worth it. Also I don't think my application to be particularly unusual, see for example Python Bayesian Network toolkit http://elliotpbnt.blogspot.com/ funded by Google summer of code. -- Szymon Jaroszewicz From alexander.belopolsky at gmail.com Tue Jan 17 10:27:12 2006 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Tue Jan 17 10:27:12 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: <43CD296C.7000700@astraw.com> References: <22F30CE0-7362-44A7-8DA1-79C6FE9E70D0@tamu.edu> <96D6BEBF-8A8D-4B69-9A6C-BC2A86BDA9B0@laposte.net> <89D76E1D-BE44-43FF-AD68-D0A8FACD795D@tamu.edu> <43CD296C.7000700@astraw.com> Message-ID: On 1/17/06, Andrew Straw wrote: > ... > Also, as for as I know, the only packages that install things into > /usr/include/python2.x/packagename are Numeric and numarray, so I would > argue it's not "standard", (although it may have lots of history). I don't know what would qualify as "standard", but Numeric's header location is specifically mentioned in the distutils manual: """ If you need to include header files from some other Python extension, you can take advantage of the fact that header files are installed in a consistent way by the Distutils install_header command. For example, the Numerical Python header files are installed (on a standard Unix installation) to /usr/local/include/python1.5/Numerical. (The exact location will differ according to your platform and Python installation.) Since the Python include directory--/usr/local/include/python1.5 in this case--is always included in the search path when building Python extensions, the best approach is to write C code like #include """ (see http://docs.python.org/dist/describing-extensions.html#SECTION002320000000000000000) The same section also criticises the idea of specifying include path explicitely: """ If you must put the Numerical include directory right into your header search path, though, you can find that directory using the Distutils distutils.sysconfig module: from distutils.sysconfig import get_python_inc incdir = os.path.join(get_python_inc(plat_specific=1), 'Numerical') setup(..., Extension(..., include_dirs=[incdir]), ) Even though this is quite portable--it will work on any Python installation, regardless of platform--it's probably easier to just write your C code in the sensible way. """ -- sasha From faltet at carabos.com Tue Jan 17 10:50:04 2006 From: faltet at carabos.com (Francesc Altet) Date: Tue Jan 17 10:50:04 2006 Subject: [Numpy-discussion] Re: Maximum available dimensions in numpy In-Reply-To: <1137522194.9752.18.camel@localhost> References: <1137522194.9752.18.camel@localhost> Message-ID: <200601171949.27382.faltet@carabos.com> A Dimarts 17 Gener 2006 19:23, Szymon Jaroszewicz va escriure: > I'm using multidimensional arrays (numarray currently) for storing > conditional probability distributions in Bayesian networks, in which > case a number of dimensions in the range of 20 or more can happen > easily. Obviously for this reason I think its not a good idea to limit > the dimensionality, especially that its not clear if the gain is worth > it. Also I don't think my application to be particularly unusual, see > for example Python Bayesian Network toolkit > http://elliotpbnt.blogspot.com/ funded by Google summer of code. Provided your example and that the minimum dimension size is 2 (1 does not increase the space requeriments) for such arrays, that a boolean would be saved on each element of the array, and that 2**32 = 4 GB (an amount of memory that is getting common in modern machines) I would say that allowing 32 dimensions (at least) would sound reasonable. Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From pearu at cens.ioc.ee Tue Jan 17 11:01:11 2006 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Tue Jan 17 11:01:11 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: References: <22F30CE0-7362-44A7-8DA1-79C6FE9E70D0@tamu.edu> <96D6BEBF-8A8D-4B69-9A6C-BC2A86BDA9B0@laposte.net> <89D76E1D-BE44-43FF-AD68-D0A8FACD795D@tamu.edu> <43CD296C.7000700@astraw.com> Message-ID: On Tue, 17 Jan 2006, Alexander Belopolsky wrote: > On 1/17/06, Andrew Straw wrote: >> ... >> Also, as for as I know, the only packages that install things into >> /usr/include/python2.x/packagename are Numeric and numarray, so I would >> argue it's not "standard", (although it may have lots of history). > > I don't know what would qualify as "standard", but Numeric's header > location is specifically mentioned in the distutils manual: > """ > If you need to include header files from some other Python extension, > you can take advantage of the fact that header files are installed in > a consistent way by the Distutils install_header command. For example, > the Numerical Python header files are installed (on a standard Unix > installation) to /usr/local/include/python1.5/Numerical. (The exact > location will differ according to your platform and Python > installation.) Since the Python include > directory--/usr/local/include/python1.5 in this case--is always > included in the search path when building Python extensions, the best > approach is to write C code like > > #include > """ As I read this section, I have two thoughts: (i) it was written in Python 1.5 days when distutils was first introduced, and back then I doubt that anyone thought about emergence of setuptools kind of tools - maybe future distutils will support setuptools better so that we treat header files like above, until then we can use the current working solution; and (ii) it all sounds like a recommendation, not a must-follow-or-something-will-break. Pearu From pearu at cens.ioc.ee Tue Jan 17 11:10:03 2006 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Tue Jan 17 11:10:03 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: References: <22F30CE0-7362-44A7-8DA1-79C6FE9E70D0@tamu.edu> <96D6BEBF-8A8D-4B69-9A6C-BC2A86BDA9B0@laposte.net> <89D76E1D-BE44-43FF-AD68-D0A8FACD795D@tamu.edu> Message-ID: On Tue, 17 Jan 2006, Sasha wrote: > My main concern is not with the non-standard default location, but > with the way distutils are tricked into placing headers into that > location. Currently this is done by mislabeling the header files as > "data files" in setup.py as follows: > > config.add_data_files(join('include','numpy','*.h')) > > This looks like a hack that just happens to work on platforms where > headers do not require any special treatement such as end-of-line > conversion or pre-compilation. This is not guaranteed to work on every > python platform now and may certainly break in the future. This is new to me. Sofar I know that numpy has been succesfully built on various unix systems, on Windows, and on OSX. The end-of-lines of numpy headers have never caused any breakage afaik and numpy header files do not require pre-compilation. Could you give more specific examples of python platforms where building numpy extension modules could fail? Pearu From ndarray at mac.com Tue Jan 17 12:19:07 2006 From: ndarray at mac.com (Sasha) Date: Tue Jan 17 12:19:07 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: References: <22F30CE0-7362-44A7-8DA1-79C6FE9E70D0@tamu.edu> <96D6BEBF-8A8D-4B69-9A6C-BC2A86BDA9B0@laposte.net> <89D76E1D-BE44-43FF-AD68-D0A8FACD795D@tamu.edu> Message-ID: > This is new to me. Sofar I know that numpy has been succesfully built on > various unix systems, on Windows, and on OSX. The end-of-lines of numpy > headers have never caused any breakage afaik and numpy header files do not > require pre-compilation. Could you give more specific examples of python > platforms where building numpy extension modules could fail? > No, I cannot and as I've just checked as of python 2.4 distutils do not process headers in any way. In fact install_data and install_headers commands are almost identical (both just copy files) and that's why your code happens to work. I did not say your code does not work on all platforms, I said it is not guaranteed to work. Any change to distutils that makes install_data and install_headers commands differ will break your code. When I first installled numpy (it was still scipy_core back then) my build scripts stopped working once I've changed #includes in my source. I tried to restore the old include location by changing install_headers setting, but that had no effect. Then I've dicovered that numpy does not use install_headers command at all. Surely everything was fixed by adding "include_dirs = [numpy.get_numpy_include()]" to my setup scripts, but it took several posts to various forums to discover that option. (I started by filing a bug report on SF.) Given that distutils manual specifically discouraged setting custom include_dirs, many future users of numpy are bound to relive my experience. Also, imagine if more third party packages follow numpy lead --- will I have to append an entry for each package to include_dirs? -- sasha From oliphant.travis at ieee.org Tue Jan 17 13:11:01 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue Jan 17 13:11:01 2006 Subject: ***[Possible UCE]*** Re: [Numpy-discussion] Speed ups to matrix-vector multiplication In-Reply-To: <1137508539.18730.18.camel@localhost.localdomain> References: <1137152864.9521.30.camel@localhost.localdomain> <43C798C1.2070401@ieee.org> <1137155787.9521.43.camel@localhost.localdomain> <43C7DEAB.8080806@ee.byu.edu> <1137179742.19206.9.camel@localhost.localdomain> <43C880B2.1050002@ieee.org> <43C9E457.3020109@ieee.org> <1137454543.14286.13.camel@localhost.localdomain> <43CC5F06.3060608@ieee.org> <1137508539.18730.18.camel@localhost.localdomain> Message-ID: <43CD5D22.3030506@ieee.org> Paulo J. S. Silva wrote: >Travis, > >It is good that my code was helpful. > >I have just tested your code and it seems to be OK now. I have written a >script to test the dot function. I am sending it attached. Think of the >script as a unit test for dot. (It tests correctness of the computations >and verify the dimensional integrity check). > > Thanks for the tests. I'll make them unit tests and put them in the test suite.. >I am using the svn version of numpy now (revision 1923). I am still >seeing slow matrix times vector multiplications if the matrix is >transposed: > >What is going on? > > Hmm.. I'm not sure. What I expect to get called is the cblas_Xgemv function with Order CblasColMajor if A has been transposed... Perhaps this is a slower operation? -Travis From chanley at stsci.edu Tue Jan 17 13:25:02 2006 From: chanley at stsci.edu (Christopher Hanley) Date: Tue Jan 17 13:25:02 2006 Subject: [Numpy-discussion] numpy.fromfile method modification? Message-ID: <43CD6086.7070204@stsci.edu> Hi Travis, Would it be possible to allow numpy's fromfile method to accept a shape parameter? This would make it similar to numarray's fromfile method and be one less thing for people making the switch to worry about. Chris From ndarray at mac.com Tue Jan 17 13:34:01 2006 From: ndarray at mac.com (Sasha) Date: Tue Jan 17 13:34:01 2006 Subject: [Numpy-discussion] Changing "nomask" in numpy.ma Message-ID: I would like to propose to change the default value of the mask array of the ma objects from None to bool_(0). This will allow ma to take advantage of numpy scalars providing array interface. For example a check that now has to be written as "a.mask is not None and a.mask.any()" can become just a.mask.any(). Ma will still use a singleton value for an empty mask so that the code that relies on "a.mask is None" check can be changed to "a.mask is nomask". Any objections? -- sasha PS: Somewhat related: >>> from numpy import * >>> bool_(0) is bool_(0) False >>> bool(0) is bool(0) True Is there any reason not to intern numpy's bool_'s? -- sasha From oliphant.travis at ieee.org Tue Jan 17 13:46:30 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue Jan 17 13:46:30 2006 Subject: [Numpy-discussion] Changing "nomask" in numpy.ma In-Reply-To: References: Message-ID: <43CD6573.2030305@ieee.org> Sasha wrote: >>>>bool_(0) is bool_(0) >>>> >>>> >False > > >>>>bool(0) is bool(0) >>>> >>>> >True > >Is there any reason not to intern numpy's bool_'s? > > No, it just hasn't been done. It would be a good idea... but require some code in PyArray_Scalar and in the bool_new method.. -Travis From oliphant.travis at ieee.org Tue Jan 17 13:49:01 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue Jan 17 13:49:01 2006 Subject: [Numpy-discussion] numpy.fromfile method modification? In-Reply-To: <43CD6442.4080303@stsci.edu> References: <43CD6086.7070204@stsci.edu> <43CD6375.3050008@ieee.org> <43CD6442.4080303@stsci.edu> Message-ID: <43CD6613.3060904@ieee.org> Christopher Hanley wrote: > Travis Oliphant wrote: > >> Yes, I think it would be possible. Please keep me informed of other >> possible compatibility improvements. >> >> Does the shape parameter just control the size of the resulting >> array? Is it equivalent to a >> .reshape(newshape) at the end? >> >> -Travis >> > > The shape parameter does control the size of the resulting array (so > you wouldn't need any information from the existing count parameter). > > The resulting array is the equivalent of what you would get if you > were to use the reshape method on the array currently returned by > fromfile. I prefer to us the array.shape = (newshape) since it is an > inplace operation instead of reshape. At least this is what I am > doing in the pyfits port. > What do you think about the sizing paramter of numpy? Is that important? If so, I would suggest a wrapper function in Python that uses the current fromfile function. We could rename the builtin to _fromfile in that case.. -Travis From oliphant.travis at ieee.org Tue Jan 17 13:49:03 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue Jan 17 13:49:03 2006 Subject: [Numpy-discussion] numpy.fromfile method modification? In-Reply-To: <43CD6442.4080303@stsci.edu> References: <43CD6086.7070204@stsci.edu> <43CD6375.3050008@ieee.org> <43CD6442.4080303@stsci.edu> Message-ID: <43CD661E.6020204@ieee.org> Christopher Hanley wrote: > Travis Oliphant wrote: > >> Yes, I think it would be possible. Please keep me informed of other >> possible compatibility improvements. >> >> Does the shape parameter just control the size of the resulting >> array? Is it equivalent to a >> .reshape(newshape) at the end? >> >> -Travis >> > > The shape parameter does control the size of the resulting array (so > you wouldn't need any information from the existing count parameter). > > The resulting array is the equivalent of what you would get if you > were to use the reshape method on the array currently returned by > fromfile. I prefer to us the array.shape = (newshape) since it is an > inplace operation instead of reshape. At least this is what I am > doing in the pyfits port. > What do you think about the sizing paramter of Numarray? Is that important? If so, I would suggest a wrapper function in Python that uses the current fromfile function. We could rename the builtin to _fromfile in that case.. -Travis From ndarray at mac.com Tue Jan 17 14:27:05 2006 From: ndarray at mac.com (Sasha) Date: Tue Jan 17 14:27:05 2006 Subject: [Numpy-discussion] Interesting timing results Message-ID: > python -m timeit -s "x=bool(0)" "x or x" 10000000 loops, best of 3: 0.111 usec per loop > python -m timeit -s "from numpy import bool_, logical_or as or_; x=bool_(0)" "or_(x, x)" 100000 loops, best of 3: 3.25 usec per loop Numpy is 32x slower than python -- not very good. Interestingly, > python -m timeit -s "from numpy import bool_; x=bool_(0)" "x or x" 1000000 loops, best of 3: 0.388 usec per loop -- sasha From chanley at stsci.edu Tue Jan 17 14:51:02 2006 From: chanley at stsci.edu (Christopher Hanley) Date: Tue Jan 17 14:51:02 2006 Subject: [Numpy-discussion] numpy.fromfile method modification? In-Reply-To: <43CD661E.6020204@ieee.org> References: <43CD6086.7070204@stsci.edu> <43CD6375.3050008@ieee.org> <43CD6442.4080303@stsci.edu> <43CD661E.6020204@ieee.org> Message-ID: <43CD73E7.1020909@stsci.edu> Travis Oliphant wrote: > What do you think about the sizing paramter of Numarray? Is that > important? The sizing parameter is apparently a recently contributed patch to numarray. We currently have no dependencies on it. I do not know how widely it is used outside of STScI. Chris From oliphant.travis at ieee.org Tue Jan 17 15:18:01 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue Jan 17 15:18:01 2006 Subject: [Numpy-discussion] Re: [Fwd: Re: _byteorder question] In-Reply-To: <43CD6F38.8070001@stsci.edu> References: <43CD6F38.8070001@stsci.edu> Message-ID: <43CD7AE4.3010700@ieee.org> Christopher Hanley wrote: > Hi Travis, > > This message is completely unrelated to the fromfile discussion. I > have forwarded your previous message to Todd for clarification of our > needs. > > I have a question regarding byteorder in numpy. In pyfits, I have a > need to change the "sense" of a byteorder flag on an array without > actually performing the byteswap operation. I cannot find a mechanism > in numpy to actually do this. Does one exist? > Yes, you want to change the array data-type object (which holds byte-order information). Previously NOTSWAPPED was a flag on the array itself which could have been toggled similarly to Numarray. Now, however, byte-order information is a property of the data-type object itself. On SVN version of NumPy (where the data-type object is now a data-type descriptor --- instead of the typeobject of the array scalar), you use either a.dtype = a.dtype.newbyteorder() # swaps byteorder (you can also specify 'big', 'little', or 'native') or if you want a new view with the new byteorder then, b = a.view(a.dtype.newbyteorder()) Notice: a = arange(5) b = a.view(a.dtype.newbyteorder('swap')) print a print b print a.tostring() == b.tostring() This results in [0 1 2 3 4] [ 0 16777216 33554432 50331648 67108864] True Notice that the actual data is exactly the same... For fun, try adding 1 to the b array and notice how the a array is changed :-) -Travis From aisaac at american.edu Tue Jan 17 15:28:04 2006 From: aisaac at american.edu (Alan G Isaac) Date: Tue Jan 17 15:28:04 2006 Subject: [Numpy-discussion] Re: Maximum available dimensions in numpy In-Reply-To: <1137522194.9752.18.camel@localhost> References: <1137522194.9752.18.camel@localhost> Message-ID: On Tue, 17 Jan 2006, Szymon Jaroszewicz apparently wrote: > see for example Python Bayesian Network toolkit > http://elliotpbnt.blogspot.com/ funded by Google summer of > code. Where is the code? Cheers, Alan Isaac From oliphant.travis at ieee.org Tue Jan 17 15:32:03 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue Jan 17 15:32:03 2006 Subject: [Numpy-discussion] Interesting timing results In-Reply-To: References: Message-ID: <43CD7E3A.8070404@ieee.org> Sasha wrote: >>python -m timeit -s "x=bool(0)" "x or x" >> >> >10000000 loops, best of 3: 0.111 usec per loop > > >>python -m timeit -s "from numpy import bool_, logical_or as or_; x=bool_(0)" "or_(x, x)" >> >> >100000 loops, best of 3: 3.25 usec per loop > >Numpy is 32x slower than python -- not very good. > > This is a known fact. Currently all array scalar math goes through the entire ufunc machinery. This is clearly sub-optimal. It is the reason for the scalarmath module that I've started in the src directory. Eventually, we should be able to have scalar math as fast as Python scalars. Anybody wanting to help out. This is a good place. One issue is how to handle scalar math division-by-zero, and overflow errors. Ideally these should be handled the same way that ufuncs do. But, this will necessarily cause some slow-down to look up the local and-or module-level attribute that can alter the behavior. -Travis >Interestingly, > > > >>python -m timeit -s "from numpy import bool_; x=bool_(0)" "x or x" >> >> >1000000 loops, best of 3: 0.388 usec per loop > > Again, not too surprising given that logical_or goes through the ufunc machinery... -Travis From oliphant.travis at ieee.org Tue Jan 17 15:37:07 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue Jan 17 15:37:07 2006 Subject: [Numpy-discussion] Maximum available dimensions in numpy In-Reply-To: <43CC6B58.80302@cox.net> References: <1137411430.3273.5.camel@localhost.localdomain> <40e64fa20601161451w659a1c1tfbe8c8afdb453610@mail.gmail.com> <43CC6B58.80302@cox.net> Message-ID: <43CD7F87.1010003@ieee.org> Tim Hochberg wrote: > > I don't see anything wrong with making the maximum dimension size a > compile time option though. However, since in the common case the > extra dimensions are unlikely to affect performance in any meaningful, > I'd recomend that the maximum number of default dimensions stay large > by default. Thus people who need to conserve bytes, which I'd > consider the rare case, have the option of reducing the max dimensions > while the arrays behave in an unsuprising manner when compiled in the > normal manner. > > If someone has the extra time, it would be interesting to see some > data about how always mallocing the extra dimensions, so there was no > maximum dimensions limit, affects performance. I'd also be interested > in seeing cases where the extra dimensions actually affect performance > before doing to stuff complicate^H^H^H^H^H^H^H^H fix things. Right now, the array has dimensions and strides arrays malloced as necessary. So, as far as the array-object itself is concerned there is no upper limit. The dimension limit only comes from the fact that for ease in coding, static arrays for dimensions and/or strides are used in quite a few places in the code. If these were all converted to use dynamic memory from the heap, then the dimension limit would go away. If anybody wants to try and do this, they are certainly welcome... -Travis From stephen.walton at csun.edu Tue Jan 17 16:02:01 2006 From: stephen.walton at csun.edu (Stephen Walton) Date: Tue Jan 17 16:02:01 2006 Subject: [Numpy-discussion] tracing bdist_rpm Message-ID: <43CD8524.6080307@csun.edu> I am still trying, fitfully, to figure out why numpy and scipy setup.py do not respect an alternative fortran compiler with config_fc when bdist_rpm is used as the build command, but do if the normal build command is used. Is there any easy way to get a trace of what Python is actually being called as a result of "python setup.py config_fc --fcompiler=absoft bdist_rpm"? From Chris.Barker at noaa.gov Tue Jan 17 17:20:03 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue Jan 17 17:20:03 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: <89D76E1D-BE44-43FF-AD68-D0A8FACD795D@tamu.edu> References: <22F30CE0-7362-44A7-8DA1-79C6FE9E70D0@tamu.edu> <96D6BEBF-8A8D-4B69-9A6C-BC2A86BDA9B0@laposte.net> <89D76E1D-BE44-43FF-AD68-D0A8FACD795D@tamu.edu> Message-ID: <43CD9776.4050002@noaa.gov> > On Jan 17, 2006, at 9:08 AM, konrad.hinsen at laposte.net wrote: > >> What good is that symlink if no one can rely on its presence? That way, on OS-X (or whatever) when someone has a problem compiling an extension, all they have to do is get their sysadmin to put in the symlink, rather than re-install the whole numpy package, or have the numpy install fail in the first place. A small improvemnet, I suppose. As a Linux+OS-X user, I've run into a number of these Apple "Think Different" issues, but I don't think creating an ugly, non-standard system to accommodate the fact that Apple like to do things strangely is the right solution. Apple doesn't maintain their python distribution well, or really play well with the python community, so many of us end up installing a new Python anyway. Maybe we could even get Apple to fix this. As someone pointed out it doesn't really make any sense to let regular users write to site-packages, but not to include. +1 the standard include location. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From wbaxter at gmail.com Tue Jan 17 17:48:04 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Tue Jan 17 17:48:04 2006 Subject: [Numpy-discussion] The Low-down on numpy/scipy? In-Reply-To: References: <43CC36B2.5050806@ieee.org> Message-ID: I see the numeric.scipy.org page has been updated. Looks much less confusing now. Great. One point that I think might still be slightly confusing is that scipy_core was never intending to replace functionality in scipy at all. Is that right? I wasn't sure for a while there whether "scipy_core is now NumPy" meant that SciPy functionality had been subsumed into NumPy. --bb On 1/17/06, Bill Baxter wrote: > > Ok. Thanks for trying to clear this up. > > Top hit on a google for numpy is here: http://numeric.scipy.org/ > > In the text it says that "the new scipy used to be called scipy core but > not anymore". However, right below that in big bold letters it says "New > Features of SciPy Core". Then the page goes on to consistently refer to it > as scipy core, despite that apparently no longer being the name. So it's > easy to miss that the name has changed. > > Also the only file download link on that page is for the sourceforge site http://sourceforge.net/project/showfiles.php?group_id=1369&package_id=175103 > > Which has downloads for NumPy, but also has a prominent news item > announcing the release of scipy_core. The presence of the news release > there led me to belive that the download for scipy_core would also be > located there. > > So it would be helpful if the numeric.scipy.org page had a link for > downloading scipy. The links to the scipy_core presentations just make it > seem all the more like you should be able to download scipy_core from there. > > > This page: http://new.scipy.org/Wiki/Download could perhaps mention that > the SciPy they are offering is the package formerly known as scipy_core, to > additionally eliminate confusion. > > There was another page I was able to get to previously, perhaps > www.scipy.org, but it's not responding now. Anyway, there was one scipy > page that said basically "information on these pages may be out of date, > there's something new afoot" but I don't seem to recall any handy link > taking me directly to the proper downloads. Or maybe I was still thinking I > was looking for something called scipy_core, rather than scipy. > > Hope that information can help you make the web pages clearer. > > Thanks for your help. > > --Bill > > > On 1/17/06, Sasha < ndarray at mac.com> wrote: > > > > On 1/16/06, Travis Oliphant < oliphant.travis at ieee.org> wrote: > > > For scipy, go to http://new.scipy.org ... > > > > That should be http://new.scipy.org/Wiki > > > > -- sasha > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndarray at mac.com Tue Jan 17 17:51:04 2006 From: ndarray at mac.com (Sasha) Date: Tue Jan 17 17:51:04 2006 Subject: [Numpy-discussion] Interesting timing results In-Reply-To: <43CD7E3A.8070404@ieee.org> References: <43CD7E3A.8070404@ieee.org> Message-ID: On 1/17/06, Travis Oliphant wrote: > Anybody wanting to help out. This is a good place. One issue is how to > handle scalar math division-by-zero, and overflow errors. Ideally these > should be handled the same way that ufuncs do. But, this will > necessarily cause some slow-down to look up the local and-or > module-level attribute that can alter the behavior. > Let me try to fill in the functions that take and return bools. Since this is the only type with a finite domain, it deserves to be handled separately. Once the two values are interned, the ops can be implemented as real fast table lookups. -- sasha From oliphant.travis at ieee.org Tue Jan 17 18:23:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue Jan 17 18:23:02 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: <43CD296C.7000700@astraw.com> References: <22F30CE0-7362-44A7-8DA1-79C6FE9E70D0@tamu.edu> <96D6BEBF-8A8D-4B69-9A6C-BC2A86BDA9B0@laposte.net> <89D76E1D-BE44-43FF-AD68-D0A8FACD795D@tamu.edu> <43CD296C.7000700@astraw.com> Message-ID: <43CDA64E.8090902@ieee.org> Andrew Straw wrote: >The main advantage of the current situation for me is that it's easy to >use with .eggs (for the reasons given). The symlink idea won't break >this, but the "let's make it like it was" solution would break it (which >was why it was changed). Can anyone think of a solution that would >please both the old-way-preferers and the .egg users? > > This is the important point in this discussion. There was a reason it was changed. I remember opposing the change, originally. However, it solved a real problem and only required people to give up their arbitrary notions of where include files *should* be. I don't see this as a real issue since you can get where the include files are very easily. But more than that, I don't see how we can change the current behavior until a solution is found that will also work for the people who forged the current solution in the first place. With that, however, I think it would be helpful for one of those people who use .eggs to clarify why it solves their problem to have the include files in their current location. -Travis From alexander.belopolsky at gmail.com Tue Jan 17 18:55:01 2006 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Tue Jan 17 18:55:01 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: <43CDA64E.8090902@ieee.org> References: <22F30CE0-7362-44A7-8DA1-79C6FE9E70D0@tamu.edu> <96D6BEBF-8A8D-4B69-9A6C-BC2A86BDA9B0@laposte.net> <89D76E1D-BE44-43FF-AD68-D0A8FACD795D@tamu.edu> <43CD296C.7000700@astraw.com> <43CDA64E.8090902@ieee.org> Message-ID: On 1/17/06, Travis Oliphant wrote: > Andrew Straw wrote: > ... > This is the important point in this discussion. There was a reason it > was changed. I remember opposing the change, originally. However, it > solved a real problem and only required people to give up their > arbitrary notions of where include files *should* be. I don't see this > as a real issue since you can get where the include files are very easily. > I don't have any "arbitrary notions of where include files *should* be" and I have no opposition to the current location. The only thing I dislike is that the needs of a particular site situation were addressed by misdiscribing the package structure to distutils rather than by adding a default site.cfg file to the distribution that users with different needs can easily edit. Another concern is that if numpy will ever be considered for inclusion in the standard "batteries included" python distribution, the same issue will rise again. -- sasha From pearu at scipy.org Wed Jan 18 00:42:00 2006 From: pearu at scipy.org (Pearu Peterson) Date: Wed Jan 18 00:42:00 2006 Subject: [Numpy-discussion] tracing bdist_rpm In-Reply-To: <43CD8524.6080307@csun.edu> References: <43CD8524.6080307@csun.edu> Message-ID: On Tue, 17 Jan 2006, Stephen Walton wrote: > I am still trying, fitfully, to figure out why numpy and scipy setup.py do > not respect an alternative fortran compiler with config_fc when bdist_rpm is > used as the build command, but do if the normal build command is used. Is > there any easy way to get a trace of what Python is actually being called as > a result of "python setup.py config_fc --fcompiler=absoft bdist_rpm"? This is not scipy/numpy setup.py specific issue. bdist_rpm calls `python build` and ignores all other commands in command line. See the code in std distutils/commands/bdist_rpm.py around lines #411, def_build variable.. To resolve this issue, numpy.distutils.command.bdist_rpm could add a wrapper around _make_spec_file that fixes bdist_rpm build command. Pearu From sj at cs.umb.edu Wed Jan 18 01:26:04 2006 From: sj at cs.umb.edu (Szymon Jaroszewicz) Date: Wed Jan 18 01:26:04 2006 Subject: [Numpy-discussion] Re: Re: Maximum available dimensions in numpy Message-ID: <1137576326.4048.6.camel@localhost> It's in https://developer.berlios.de/projects/pbnt/. Cheers, Szymon > On Tue, 17 Jan 2006, Szymon Jaroszewicz apparently wrote: > > see for example Python Bayesian Network toolkit > > http://elliotpbnt.blogspot.com/ funded by Google summer of > > code. > > Where is the code? > > Cheers, > Alan Isaac > -- Szymon Jaroszewicz From pjssilva at ime.usp.br Wed Jan 18 03:45:02 2006 From: pjssilva at ime.usp.br (Paulo Jose da Silva e Silva) Date: Wed Jan 18 03:45:02 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks Message-ID: <1137584647.19613.17.camel@localhost.localdomain> Hello, Travis asked me to benchmark numpy versus matlab in some basic linear algebra operations. Here are the resuts for matrices/vectors of dimensions 5, 50 and 500: Operation x'*y x*y' A*x A*B A'*x Half 2in2 Dimension 5 Array 0.94 0.7 0.22 0.28 1.12 0.98 1.1 Matrix 7.06 1.57 0.66 0.79 1.6 3.11 4.56 Matlab 1.88 0.44 0.41 0.35 0.37 1.2 0.98 Dimension 50 Array 9.74 3.09 0.56 18.12 13.93 4.2 4.33 Matrix 81.99 3.81 1.04 19.13 14.58 6.3 7.88 Matlab 16.98 1.94 1.07 17.86 0.73 1.57 1.77 Dimension 500 Array 1.2 8.97 2.03 166.59 20.34 3.99 4.31 Matrix 17.95 9.09 2.07 166.62 20.67 4.11 4.45 Matlab 2.09 6.07 2.17 169.45 2.1 2.56 3.06 Obs: The operation Half is actually A*x using only the lower half of the matrix and vector. The operation 2in2 is A*x using only the even indexes. Of course there are many repetitions of the same operation: 100000 for dim 5 and 50 and 1000 for dim 500. The inner product is number of repetitions is multiplied by dimension (it is very fast). The software is numpy svn version 1926 Matlab 6.5.0.180913a Release 13 (Jun 18 2002) Both softwares are using the *same* BLAS and LAPACK (ATLAS for sse). As you can see, numpy array looks very competitive. The matrix class in numpy has too much overhead for small dimension though. This overhead is very small for medium size arrays. Looking at the results above (specially the small dimensions ones, for higher dimensions the main computations are being performed by the same BLAS) I believe we can say: 1) Numpy array is faster on usual operations but outerproduct (I believe the reason is that the dot function uses the regular matrix multiplication to compute outer-products, instead of using a special function. This can "easily" changes). In particular numpy was faster in matrix times vector operations, which is the most usual in numerical linear algebra. 2) Any operation that involves transpose suffers a very big penalty in numpy. Compare A'*x and A*x, it is 10 times slower. In contrast Matlab deals with transpose quite well. Travis is already aware of this and it can be probably solved. 3) When using subarrays, numpy is a slower. The difference seems acceptable. Travis, can this be improved? Best, Paulo Obs: Latter on (in a couple of days) I may present less synthetic benchmarks (a QR factorization and a Modified Cholesky). From bsouthey at gmail.com Wed Jan 18 05:57:03 2006 From: bsouthey at gmail.com (Bruce Southey) Date: Wed Jan 18 05:57:03 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: <1137584647.19613.17.camel@localhost.localdomain> References: <1137584647.19613.17.camel@localhost.localdomain> Message-ID: Hi, Thanks for doing this as it helps determine which approach to take when coding problems. Could you add the Numeric and numarray to these benchmarks? If for no other reason to show the advantage of the new numpy. I am curious in your code because you get very different results for matrix class depending on whether x or y is transposed. Do you first transpose the x or y first before the multiplication or is the multiplication done in place by just switching the indices? Also, for x'*y, is the results for Dimension 50 and Dimension 500 switched? Thanks Bruce On 1/18/06, Paulo Jose da Silva e Silva wrote: > > Hello, > > Travis asked me to benchmark numpy versus matlab in some basic linear > algebra operations. Here are the resuts for matrices/vectors of > dimensions 5, 50 and 500: > > Operation x'*y x*y' A*x A*B A'*x Half 2in2 > > Dimension 5 > Array 0.94 0.7 0.22 0.28 1.12 0.98 1.1 > Matrix 7.06 1.57 0.66 0.79 1.6 3.11 4.56 > Matlab 1.88 0.44 0.41 0.35 0.37 1.2 0.98 > > Dimension 50 > Array 9.74 3.09 0.56 18.12 13.93 4.2 4.33 > Matrix 81.99 3.81 1.04 19.13 14.58 6.3 7.88 > Matlab 16.98 1.94 1.07 17.86 0.73 1.57 1.77 > > Dimension 500 > Array 1.2 8.97 2.03 166.59 20.34 3.99 4.31 > Matrix 17.95 9.09 2.07 166.62 20.67 4.11 4.45 > Matlab 2.09 6.07 2.17 169.45 2.1 2.56 3.06 > > Obs: The operation Half is actually A*x using only the lower half of the > matrix and vector. The operation 2in2 is A*x using only the even > indexes. > > Of course there are many repetitions of the same operation: 100000 for > dim 5 and 50 and 1000 for dim 500. The inner product is number of > repetitions is multiplied by dimension (it is very fast). > > The software is > > numpy svn version 1926 > Matlab 6.5.0.180913a Release 13 (Jun 18 2002) > > Both softwares are using the *same* BLAS and LAPACK (ATLAS for sse). > > As you can see, numpy array looks very competitive. The matrix class in > numpy has too much overhead for small dimension though. This overhead is > very small for medium size arrays. Looking at the results above > (specially the small dimensions ones, for higher dimensions the main > computations are being performed by the same BLAS) I believe we can say: > > 1) Numpy array is faster on usual operations but outerproduct (I believe > the reason is that the dot function uses the regular matrix > multiplication to compute outer-products, instead of using a special > function. This can "easily" changes). In particular numpy was faster in > matrix times vector operations, which is the most usual in numerical > linear algebra. > > 2) Any operation that involves transpose suffers a very big penalty in > numpy. Compare A'*x and A*x, it is 10 times slower. In contrast Matlab > deals with transpose quite well. Travis is already aware of this and it > can be probably solved. > > 3) When using subarrays, numpy is a slower. The difference seems > acceptable. Travis, can this be improved? > > Best, > > Paulo > > Obs: Latter on (in a couple of days) I may present less synthetic > benchmarks (a QR factorization and a Modified Cholesky). > > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log > files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From svetosch at gmx.net Wed Jan 18 07:47:08 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Wed Jan 18 07:47:08 2006 Subject: [Numpy-discussion] array vs. matrix for matlab-compatible stuff Message-ID: <43CE631D.1000006@gmx.net> Hi, I've spent a couple of weeks with scipy/numpy and the old-to-new transition; now that the transition is over (?) but some confusion is remaining (on my side) I feel the need to ask a basic question about matlab compatibility in terms of matrix (linear algebra) programming. Take "eye" and "identity" for example; given that "eye" supposedly exists to facilitate transiton from matlab to numpy/scipy (correct?), I expected eye to be/return a matrix. However (as you guys know better than I do), it's not a matrix: >>> import numpy >>> type(numpy.eye(2)) >>> type(numpy.identity(2)) >>> type(numpy.mat(numpy.eye(2))) Why is that so? There doesn't seem to be any value added of eye over identity, and the downside is inconvenience of having to explicitly convert those arrays to matrices (to use them in multiplication, inverting them, avoid unexpected broadcasting behavior if I mistakenly write down a non-defined matrix product etc.). Surely I'm missing something here, but what? Thanks for your help, Sven From matthew.brett at gmail.com Wed Jan 18 08:08:03 2006 From: matthew.brett at gmail.com (Matthew Brett) Date: Wed Jan 18 08:08:03 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: <1137584647.19613.17.camel@localhost.localdomain> References: <1137584647.19613.17.camel@localhost.localdomain> Message-ID: <1e2af89e0601180738x3979fa9dyed6616c116333680@mail.gmail.com> Hi, > Travis asked me to benchmark numpy versus matlab in some basic linear > algebra operations. Here are the resuts for matrices/vectors of > dimensions 5, 50 and 500: This is really excellent, thanks. Is there any chance we can make these and other benchmarks part of the pre-release testing? Apart from testing for bottlenecks, if we could show that we were in the ballpark of matlab for speed for each release, this would be very helpful for those us trying to persuade our matlab colleagues to switch. Best, Matthew From aghast at thirdage.com Wed Jan 18 08:37:03 2006 From: aghast at thirdage.com (=?windows-1251?B?NiAtIDExINTFwtDAy98gMjAwNiDj7uTg?=) Date: Wed Jan 18 08:37:03 2006 Subject: [Numpy-discussion] =?windows-1251?B?z9DAytLI18XRysjJIMrT0NEgxMvfINTIzcDN0c7CzsPOIMTI0MXK0s7QwA==?= Message-ID: <926601c61c4c$21d57af4$87c47dbb@thirdage.com> ? ??? ????? ?? ??????? ??????????? ?????????: ? ???? ??! ?????????? ??????????? ?? ???? ?????? ???????? ?????? ??????????? ???????????? ? ???????????? ?????????? ????????? ????????????? ?????????? ???????????? ???????? ? ??? ???? ??????????. ???????????? ???? ??? ??????????? ????????? 6 - 11 ??????? 2006 ???? ? ????? ?????????? ???????????? ???????? ?????, ?????????? ???????? SRC ??????? ????? ?????????? ??????????, ??????? ???????? ??????? ????? ??? ???? ?????????????, ?????????? ??? ?????????, ??????? ? ???????????. ?????? ??? ??????? ?????????? ?????, ??????? ????? ?????? ?????????? ??????????, ?? ??????? ?????????? ?????????? ? ???????????, ????????? ?? ? ????????? ?????????? ??????????????????? ????????-??????????: ? ????????? ?????: ???? ? ????? ??????????? ????????? ? ????????? ??????????? ??????????? ?????????? ????????? ???????? ?????????? ?????????? ???????? ???????? ??????? ??????????????? ????? ?????????? ? ???????? ?????????? ???????? ????????? ???? ?????????? ???????? ?????? ??????????? ? ???????? ???????? ?????? ????????-????????????? ???????????? ????????? ???????? ???????? ?????????? ?????? ? ???????? ?????? ?????????? ?????? ?????? ???? ???? ????????, ?????????? ?????????? ? ????????, ??????????? ??? ??????????? ?????????? ? ??????????????? ???????????? ????????. ?????? ????????? ?? ?????? ???????? ? ????????????? ????????, ? ?????? ????????? ?????? ??????? ? ?????? ?????????? ??????. ????? ??????? ??????? ? ????? ??? ???????? ????????? ??????????, ????????? ??? ?? ?????????: (095) 98O-65-16, 98O-65-49 -------------- next part -------------- An HTML attachment was scrubbed... URL: From pjssilva at ime.usp.br Wed Jan 18 09:04:00 2006 From: pjssilva at ime.usp.br (Paulo J. S. Silva) Date: Wed Jan 18 09:04:00 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: References: <1137584647.19613.17.camel@localhost.localdomain> Message-ID: <1137603780.19613.44.camel@localhost.localdomain> Em Qua, 2006-01-18 ?s 07:56 -0600, Bruce Southey escreveu: > Hi, > Thanks for doing this as it helps determine which approach to take > when coding problems. Could you add the Numeric and numarray to these > benchmarks? If for no other reason to show the advantage of the new > numpy. I add a new table with the requested benchmarks below. As you can see, numpy is always faster then numarray but slower than Numeric in indexing. Numeric also seems to suffer less from the transpose phenomenon. > > I am curious in your code because you get very different results for > matrix class depending on whether x or y is transposed. Do you first > transpose the x or y first before the multiplication or is the > multiplication done in place by just switching the indices? I think the transpose problem is that the code makes a copy of the array before calling blas (to make it fit the blas call). Actually if you change the code from dot(transpose(A), b) to M = transpose(A).copy() dot(M, b) the time spend in the operation doesn't change. I am also sending my Python benchmark code attached to this message. Anyone can used as you want (it requires numpy, Numeric and numarray installed). > Also, for x'*y, is the results for Dimension 50 and Dimension 500 > switched? No the inner product results have different number of calls for all dimensions. Hence you can't compare time between dimensions. Paulo --- New table --- Obs: I have fixed a error in my old code that made the outer product for numpy look bad. The way it was coded it was forcing an extra array copy before the BLAS call. Tests x.T*y x*y.T A*x A*B A.T*x half 2in2 Dimension: 5 Array 0.94 0.25 0.22 0.27 1.08 0.93 1.10 Matrix 6.93 1.56 0.64 0.75 1.64 3.08 4.48 NumArr 2.87 0.63 0.62 0.68 8.24 7.22 1 1.37 Numeri 1.15 0.33 0.29 0.36 0.68 0.61 0.72 Matlab 1.88 0.44 0.41 0.35 0.37 1.20 0.98 Dimension: 50 Array 9.64 2.03 0.57 18.74 13.75 4.09 4.29 Matrix 82.98 3.70 1.04 19.87 14.58 6.35 7.91 NumArr 29.72 2.58 0.95 18.40 12.88 8.50 12.90 Numeri 11.97 2.21 0.61 17.71 9.98 1.04 3.22 Matlab 16.98 1.94 1.07 17.86 0.73 1.57 1.77 Dimension: 500 Array 1.22 8.97 2.02 165.03 19.99 3.96 4.25 Matrix 18.13 9.20 2.01 164.90 20.29 4.06 4.35 NumArr 3.16 9.02 2.09 164.83 21.59 4.29 5.72 Numeri 1.46 8.99 2.03 165.01 19.22 3.24 4.50 Matlab 2.09 6.07 2.17 169.45 2.10 2.56 3.06 From Fernando.Perez at colorado.edu Wed Jan 18 09:05:07 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Wed Jan 18 09:05:07 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: <1e2af89e0601180738x3979fa9dyed6616c116333680@mail.gmail.com> References: <1137584647.19613.17.camel@localhost.localdomain> <1e2af89e0601180738x3979fa9dyed6616c116333680@mail.gmail.com> Message-ID: <43CE7531.2080708@colorado.edu> Matthew Brett wrote: > Hi, > > >>Travis asked me to benchmark numpy versus matlab in some basic linear >>algebra operations. Here are the resuts for matrices/vectors of >>dimensions 5, 50 and 500: > > > This is really excellent, thanks. Is there any chance we can make > these and other benchmarks part of the pre-release testing? Apart > from testing for bottlenecks, if we could show that we were in the > ballpark of matlab for speed for each release, this would be very > helpful for those us trying to persuade our matlab colleagues to > switch. +1 It might not be part of test(1), but at test(10) these tests could be automatically run, activating each line as each package is found (or not) in the system (Numeric, numarray, matlab). This way, people who have matlab on their box can even get a real-time check of how their fresh-off-svn numpy fares against matlab that day. cheers, f From pjssilva at ime.usp.br Wed Jan 18 09:08:05 2006 From: pjssilva at ime.usp.br (Paulo J. S. Silva) Date: Wed Jan 18 09:08:05 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: References: <1137584647.19613.17.camel@localhost.localdomain> Message-ID: <1137603902.19613.47.camel@localhost.localdomain> Ops... Forgot to send the benchmark code... (I know it is ugly and very synthetic, I'll improve it with time). -------------- next part -------------- A non-text attachment was scrubbed... Name: benchmark.py Type: text/x-python Size: 4317 bytes Desc: not available URL: From pjssilva at ime.usp.br Wed Jan 18 09:11:06 2006 From: pjssilva at ime.usp.br (Paulo J. S. Silva) Date: Wed Jan 18 09:11:06 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: <1e2af89e0601180738x3979fa9dyed6616c116333680@mail.gmail.com> References: <1137584647.19613.17.camel@localhost.localdomain> <1e2af89e0601180738x3979fa9dyed6616c116333680@mail.gmail.com> Message-ID: <1137604057.19613.51.camel@localhost.localdomain> Em Qua, 2006-01-18 ?s 15:38 +0000, Matthew Brett escreveu: > > This is really excellent, thanks. Is there any chance we can make > these and other benchmarks part of the pre-release testing? Apart > from testing for bottlenecks, if we could show that we were in the > ballpark of matlab for speed for each release, this would be very > helpful for those us trying to persuade our matlab colleagues to > switch. > Sure. As soon as I add some less synthetic benchmark it can be even more interesting. I'll try to make the code easily callable from anyone that has both numpy and matlab on his own machine. As soon as it is polished enough I'll make it available to this comunity. It may take some weeks though. Best, Paulo From oliphant.travis at ieee.org Wed Jan 18 09:51:05 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed Jan 18 09:51:05 2006 Subject: [Numpy-discussion] Re: [SciPy-dev] numpy/scipy some remarks In-Reply-To: <43CE3E27.5080207@gmail.com> References: <43CE3E27.5080207@gmail.com> Message-ID: <43CE7FDF.5030506@ieee.org> Andrew Jaffe wrote: >I have some related numpy/scipy questions: > > - I recently upgraded to the most recent SVN of numpy, without doing >likewise for scipy. I found that the numpy.test() calls failed in a >couple of places -- because *scipy* hadn't been updated with the latest >dtype updates! (I can't reproduce the errors since I've since updated >scipy.) I thought the whole point of the numpy/scipy split was to avoid >'implicit' calls of scipy from numpy, wasn't it? > > Not entirely. The issue was namespace clashes between packages. This issue has come up before and is localized to the numpy.dual module. If somebody would like to re-write the module to contain a register function that scipy hooks into to register new functions for the ones listed there, then that would be a better solution. But, I haven't had time to do it. -Travis From robert.kern at gmail.com Wed Jan 18 10:08:03 2006 From: robert.kern at gmail.com (Robert Kern) Date: Wed Jan 18 10:08:03 2006 Subject: [Numpy-discussion] Re: [SciPy-dev] numpy/scipy some remarks In-Reply-To: <43CE8077.8020706@colorado.edu> References: <43CE3E27.5080207@gmail.com> <43CE8077.8020706@colorado.edu> Message-ID: <43CE83D5.4060105@gmail.com> Fernando Perez wrote: > Anyway, I won't belabor this point any longer. I'd just like to hear from > others their opinion on this matter, and if a decision is made to go ahead > with the overwriting, at least I think the rationale for it should be well > justified (and be more than "it's convenient"). The fact that over the last > few weeks we've had several surprised questions on this is, to me, an > indicator that I'm not the one uncomfortable with this decision. I haven't followed this discussion in great detail, but I believe the current situation is this: 1) If you use numpy.dft and numpy.linalg directly, you will always get the numpy versions no matter what else is installed. 2) If you want to optionally use optimized scipy versions if they are available and regular numpy versions otherwise, then you use the functions exposed in numpy.dual. You do so at your own risk. 3) pkgload() exists to support the loading of subpackages. It does not reach into numpy.dft or numpy.linalg at all. It is not relevant to this issue. 4) There are some places in numpy that use numpy.dual. I think we can address all of your concerns by changing #4. -- Robert Kern robert.kern at gmail.com "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From oliphant.travis at ieee.org Wed Jan 18 10:16:03 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed Jan 18 10:16:03 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: <1137584647.19613.17.camel@localhost.localdomain> References: <1137584647.19613.17.camel@localhost.localdomain> Message-ID: <43CE85AD.1090904@ieee.org> Paulo Jose da Silva e Silva wrote: >Hello, > >Travis asked me to benchmark numpy versus matlab in some basic linear >algebra operations. Here are the resuts for matrices/vectors of >dimensions 5, 50 and 500: > > Hi Paulo, Will you run these again with the latest SVN version of numpy. I couldn't figure out why a copy was being made on transpose (because it shouldn't have been). Then, I dug deep into the PyArray_FromAny code and found bad logic in when a copy was needed that was causing an inappropriate copy. I fixed that and now wonder how things will change. Because presumably, the dotblas function should handle the situation now... -Travis From oliphant.travis at ieee.org Wed Jan 18 10:20:08 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed Jan 18 10:20:08 2006 Subject: [Numpy-discussion] Re: [SciPy-dev] numpy/scipy some remarks In-Reply-To: <43CE83D5.4060105@gmail.com> References: <43CE3E27.5080207@gmail.com> <43CE8077.8020706@colorado.edu> <43CE83D5.4060105@gmail.com> Message-ID: <43CE86C5.1050702@ieee.org> Robert Kern wrote: >Fernando Perez wrote: > > > >>Anyway, I won't belabor this point any longer. I'd just like to hear from >>others their opinion on this matter, and if a decision is made to go ahead >>with the overwriting, at least I think the rationale for it should be well >>justified (and be more than "it's convenient"). The fact that over the last >>few weeks we've had several surprised questions on this is, to me, an >>indicator that I'm not the one uncomfortable with this decision. >> >> > >I haven't followed this discussion in great detail, but I believe the current >situation is this: > >1) If you use numpy.dft and numpy.linalg directly, you will always get the numpy >versions no matter what else is installed. > >2) If you want to optionally use optimized scipy versions if they are available >and regular numpy versions otherwise, then you use the functions exposed in >numpy.dual. You do so at your own risk. > >3) pkgload() exists to support the loading of subpackages. It does not reach >into numpy.dft or numpy.linalg at all. It is not relevant to this issue. > >4) There are some places in numpy that use numpy.dual. > >I think we can address all of your concerns by changing #4. > > > This is an accurate assessment. However, I do not want to eliminate number 4 as I've mentioned before. I think there is a place for having functions that can be over-written with better versions. I agree that it could be implemented better, however, with some kind of register function instead of automatically looking in scipy... -Travis From robert.kern at gmail.com Wed Jan 18 10:32:01 2006 From: robert.kern at gmail.com (Robert Kern) Date: Wed Jan 18 10:32:01 2006 Subject: [SciPy-dev] [Numpy-discussion] Re: numpy/scipy some remarks In-Reply-To: <43CE86C5.1050702@ieee.org> References: <43CE3E27.5080207@gmail.com> <43CE8077.8020706@colorado.edu> <43CE83D5.4060105@gmail.com> <43CE86C5.1050702@ieee.org> Message-ID: <43CE8983.2050804@gmail.com> Travis Oliphant wrote: > Robert Kern wrote: >>4) There are some places in numpy that use numpy.dual. >> >>I think we can address all of your concerns by changing #4. And actually, I think we can eat our cake and have it, too, by providing a way to restrict numpy.dual to only use numpy versions. We won't provide a way to force numpy.dual to only use some_other_version. I think Fernando's examples won't be problematic, then. > This is an accurate assessment. However, I do not want to eliminate > number 4 as I've mentioned before. I think there is a place for having > functions that can be over-written with better versions. I agree that > it could be implemented better, however, with some kind of register > function instead of automatically looking in scipy... Like egg entry_points. Please, let's not reinvent this wheel again. http://peak.telecommunity.com/DevCenter/PkgResources#entry-points -- Robert Kern robert.kern at gmail.com "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From chanley at stsci.edu Wed Jan 18 10:47:06 2006 From: chanley at stsci.edu (Christopher Hanley) Date: Wed Jan 18 10:47:06 2006 Subject: [Numpy-discussion] numpy / numarray chararray incompatibility Message-ID: <43CE8CF1.7000802@stsci.edu> Hi Travis, The following works in numarray but fails in numpy: In [23]: a1 = numarray.strings.array(['abc','def','xx']) In [24]: a1 Out[24]: CharArray(['abc', 'def', 'xx']) In [25]: from numpy.core import char In [26]: a = char.array(['abc', 'def', 'xx']) --------------------------------------------------------------------------- exceptions.TypeError Traceback (most recent call last) /data/sparty1/dev/pyfits-numpy/test/ /data/sparty1/dev/site-packages/lib/python/numpy/core/defchararray.py in array(obj, itemsize, copy, unicode, fortran) 321 dtype += str(itemsize) 322 --> 323 if isinstance(obj, str) or isinstance(obj, unicode): 324 if itemsize is None: 325 itemsize = len(obj) TypeError: isinstance() arg 2 must be a class, type, or tuple of classes and types Chris From schofield at ftw.at Wed Jan 18 11:01:01 2006 From: schofield at ftw.at (Ed Schofield) Date: Wed Jan 18 11:01:01 2006 Subject: [Numpy-discussion] [SciPy-dev] importing madness In-Reply-To: <43CE83D5.4060105@gmail.com> References: <43CE3E27.5080207@gmail.com> <43CE8077.8020706@colorado.edu> <43CE83D5.4060105@gmail.com> Message-ID: <43CE9178.3050602@ftw.at> Robert Kern wrote: >Fernando Perez wrote: > > > >>Anyway, I won't belabor this point any longer. I'd just like to hear from >>others their opinion on this matter, and if a decision is made to go ahead >>with the overwriting, at least I think the rationale for it should be well >>justified (and be more than "it's convenient"). The fact that over the last >>few weeks we've had several surprised questions on this is, to me, an >>indicator that I'm not the one uncomfortable with this decision. >> >> > >I haven't followed this discussion in great detail, but I believe the current >situation is this: > >1) If you use numpy.dft and numpy.linalg directly, you will always get the numpy >versions no matter what else is installed. > >2) If you want to optionally use optimized scipy versions if they are available >and regular numpy versions otherwise, then you use the functions exposed in >numpy.dual. You do so at your own risk. > >3) pkgload() exists to support the loading of subpackages. It does not reach >into numpy.dft or numpy.linalg at all. It is not relevant to this issue. > >4) There are some places in numpy that use numpy.dual. > >I think we can address all of your concerns by changing #4. > > I've been battling for half an hour trying to import scipy.linalg. Is this one of Fernando's scary predictions coming true? I get: >>> from scipy import linalg >>> dir(linalg) ['Heigenvalues', 'Heigenvectors', 'LinAlgError', 'ScipyTest', '__builtins__', '__doc__', '__file__', '__name__', '__path__', 'cholesky', 'cholesky_decomposition', 'det', 'determinant', 'eig', 'eigenvalues', 'eigenvectors', 'eigh', 'eigvals', 'eigvalsh', 'generalized_inverse', 'inv', 'inverse', 'lapack_lite', 'linalg', 'linear_least_squares', 'lstsq', 'pinv', 'singular_value_decomposition', 'solve', 'solve_linear_equations', 'svd', 'test'] >>> linalg.__file__ '/home/schofield/Tools/lib/python2.4/site-packages/numpy/linalg/__init__.pyc' This is the linalg package from numpy, not scipy. It's missing my favourite pinv2 function. What is going on?! I've just cleaned everything out and built from the latest SVN revisions. Using pkgload('linalg') doesn't seem to help: >>> import scipy >>> scipy.pkgload('linalg') >>> linalg.__file__ Traceback (most recent call last): File "", line 1, in ? NameError: name 'linalg' is not defined >>> scipy.linalg.__file__ '/home/schofield/Tools/lib/python2.4/site-packages/numpy/linalg/__init__.pyc' The only thing that helps is calling pkgload() with no arguments: >>> import scipy >>> scipy.pkgload() Overwriting lib= from /home/schofield/Tools/lib/python2.4/site-packages/scipy/lib/__init__.pyc (was from /home/schofield/Tools/lib/python2.4/site-packages/numpy/lib/__init__.pyc) >>> scipy.linalg.__file__ '/home/schofield/Tools/lib/python2.4/site-packages/scipy/linalg/__init__.pyc' Unless I'm doing something very stupid, there seem to be multiple sources of evil here. First, numpy's linalg package is available from the scipy namespace, which seems like a recipe for Ed's madness, since he can't find his pinv2() function. Second, scipy.pkgload('linalg') silently fails to make any visible difference. This is probably a simple bug. Third, ..., there is no third. But bad things usually come in threes. -- Ed From Fernando.Perez at colorado.edu Wed Jan 18 11:03:00 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Wed Jan 18 11:03:00 2006 Subject: [SciPy-dev] [Numpy-discussion] Re: numpy/scipy some remarks In-Reply-To: <43CE86C5.1050702@ieee.org> References: <43CE3E27.5080207@gmail.com> <43CE8077.8020706@colorado.edu> <43CE83D5.4060105@gmail.com> <43CE86C5.1050702@ieee.org> Message-ID: <43CE90AA.2020104@colorado.edu> Travis Oliphant wrote: >>3) pkgload() exists to support the loading of subpackages. It does not reach >>into numpy.dft or numpy.linalg at all. It is not relevant to this issue. >> >>4) There are some places in numpy that use numpy.dual. >> >>I think we can address all of your concerns by changing #4. >> >> >> > > > This is an accurate assessment. However, I do not want to eliminate > number 4 as I've mentioned before. I think there is a place for having > functions that can be over-written with better versions. I agree that > it could be implemented better, however, with some kind of register > function instead of automatically looking in scipy... Mmh, I think I'm confused then: it seemed to me that pkgload() WAS overwriting numpy names, from the messages which the environment variable controls. Is that not true? Here's a recent thread: http://aspn.activestate.com/ASPN/Mail/Message/scipy-dev/2974044 where this was shown: In [3]: import scipy Overwriting fft= from scipy.fftpack.basic (was from numpy.dft.fftpack) Overwriting ifft= from scipy.fftpack.basic (was from numpy.dft.fftpack) I understood this as 'scipy.fftpack.basic.fft overwrote numpy.dft.fftpack.fft'. Does this then not affect the numpy namespace at all? I also would like to propose that, rather than using an environment variable, pkgload() takes a 'verbose=' keyword (or 'quiet='). I think it's much cleaner to say pkgload(quiet=1) or pkgload(verbose=0) than relying on users configuring env. variables for something like this. Cheers, f From pjssilva at ime.usp.br Wed Jan 18 11:03:02 2006 From: pjssilva at ime.usp.br (Paulo J. S. Silva) Date: Wed Jan 18 11:03:02 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: <43CE85AD.1090904@ieee.org> References: <1137584647.19613.17.camel@localhost.localdomain> <43CE85AD.1090904@ieee.org> Message-ID: <1137610937.19613.65.camel@localhost.localdomain> Em Qua, 2006-01-18 ?s 11:15 -0700, Travis Oliphant escreveu: > Will you run these again with the latest SVN version of numpy. I > couldn't figure out why a copy was being made on transpose (because it > shouldn't have been). Then, I dug deep into the PyArray_FromAny code > and found bad logic in when a copy was needed that was causing an > inappropriate copy. > > I fixed that and now wonder how things will change. Because presumably, > the dotblas function should handle the situation now... > Good work Travis :-) Tests x.T*y x*y.T A*x A*B A.T*x half 2in2 Dimension: 5 Array 0.9000 0.2400 0.2000 0.2600 0.7100 0.9400 1.1600 Matrix 4.7800 1.5700 0.6200 0.7600 1.0600 3.0400 4.6500 NumArr 3.2900 0.7400 0.6800 0.7800 8.4800 7.4200 11.6600 Numeri 1.3300 0.3900 0.3100 0.4200 0.7900 0.6800 0.7600 Matlab 1.88 0.44 0.41 0.35 0.37 1.20 0.98 Dimension: 50 Array 9.0000 2.1400 0.5500 18.9500 1.4100 4.2700 4.4500 Matrix 48.7400 3.9200 1.0100 20.2000 1.8000 6.5000 8.1900 NumArr 32.3900 2.6800 1.0000 18.9700 13.0300 8.6300 13.0700 Numeri 13.1000 2.2600 0.6500 18.2700 10.1500 1.0400 3.2600 Matlab 16.98 1.94 1.07 17.86 0.73 1.57 1.77 Dimension: 500 Array 1.1400 9.2300 2.0100 168.2700 2.1800 4.0200 4.2900 Matrix 5.0300 9.3500 2.1500 167.5300 2.1700 4.1100 4.4200 NumArr 3.4400 9.1000 2.1000 168.7100 21.8400 4.3900 5.8900 Numeri 1.5800 9.2700 2.0700 167.5600 20.0500 3.4000 4.6800 Matlab 2.09 6.07 2.17 169.45 2.10 2.56 3.06 Note the 10-fold speed-up for higher dimensions :-) It looks like that now that numpy only looses to matlab in small dimensions. Probably, the problem is the creation of the object to represent the transposed object. Probably Matlab creation of objects is very lightweight (they only have matrices objects to deal with). Probably this phenomenon explains the behavior for the indexing operations too. Paulo From eric at enthought.com Wed Jan 18 11:04:05 2006 From: eric at enthought.com (eric jones) Date: Wed Jan 18 11:04:05 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: <43CE7531.2080708@colorado.edu> References: <1137584647.19613.17.camel@localhost.localdomain> <1e2af89e0601180738x3979fa9dyed6616c116333680@mail.gmail.com> <43CE7531.2080708@colorado.edu> Message-ID: <43CE90E6.2070107@enthought.com> In scipy, we talked about having a benchmark_xyz methods that could be added to the test classes. These weren't run during unit tests (scipy.test()) but would could be run using scipy.benchmark() or something like that. I can't remember if Pearu got the machinery in place, but it seems to me it wouldn't be so hard. You would have to add guards around benchmarks that compare to 3rd party tools, obviously, so that people without them could still run the benchmark suite. Adding a regression process that checks against results from previous builds to flag potential problems when a slow down is noted would be good -- that is more work. Anyway, something flagging these "tests" as benchmarks instead of standard correctness tests seems like a good idea. eric Fernando Perez wrote: > Matthew Brett wrote: >> Hi, >> >> >>> Travis asked me to benchmark numpy versus matlab in some basic linear >>> algebra operations. Here are the resuts for matrices/vectors of >>> dimensions 5, 50 and 500: >> >> >> This is really excellent, thanks. Is there any chance we can make >> these and other benchmarks part of the pre-release testing? Apart >> from testing for bottlenecks, if we could show that we were in the >> ballpark of matlab for speed for each release, this would be very >> helpful for those us trying to persuade our matlab colleagues to >> switch. > > +1 > > It might not be part of test(1), but at test(10) these tests could be > automatically run, activating each line as each package is found (or > not) in the system (Numeric, numarray, matlab). This way, people who > have matlab on their box can even get a real-time check of how their > fresh-off-svn numpy fares against matlab that day. > > cheers, > > f > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log > files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From ndarray at mac.com Wed Jan 18 11:22:00 2006 From: ndarray at mac.com (Sasha) Date: Wed Jan 18 11:22:00 2006 Subject: [Numpy-discussion] Shouldn't bool_ derive from bool? Message-ID: >>> from numpy import * >>> isinstance(bool_(True), bool) False From ndarray at mac.com Wed Jan 18 11:29:05 2006 From: ndarray at mac.com (Sasha) Date: Wed Jan 18 11:29:05 2006 Subject: [Numpy-discussion] Re: Shouldn't bool_ derive from bool? In-Reply-To: References: Message-ID: Oops, bool cannot be subclassed in python, but bool is a subclass of int, so it makes sense to derive from int_, only which int? In python bool inherits all number methods from int and overrides only and, or, and xor. Maybe bool_ can do the same ... -- sasha On 1/18/06, Sasha wrote: > >>> from numpy import * > >>> isinstance(bool_(True), bool) > False > From strawman at astraw.com Wed Jan 18 11:36:05 2006 From: strawman at astraw.com (Andrew Straw) Date: Wed Jan 18 11:36:05 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: <1137610937.19613.65.camel@localhost.localdomain> References: <1137584647.19613.17.camel@localhost.localdomain> <43CE85AD.1090904@ieee.org> <1137610937.19613.65.camel@localhost.localdomain> Message-ID: <43CE985E.9050406@astraw.com> Paulo J. S. Silva wrote: >Em Qua, 2006-01-18 ?s 11:15 -0700, Travis Oliphant escreveu: > > > >>Will you run these again with the latest SVN version of numpy. I >>couldn't figure out why a copy was being made on transpose (because it >>shouldn't have been). Then, I dug deep into the PyArray_FromAny code >>and found bad logic in when a copy was needed that was causing an >>inappropriate copy. >> >>I fixed that and now wonder how things will change. Because presumably, >>the dotblas function should handle the situation now... >> >> >> > >Good work Travis :-) > >Tests x.T*y x*y.T A*x A*B A.T*x half 2in2 > >Dimension: 5 >Array 0.9000 0.2400 0.2000 0.2600 0.7100 0.9400 1.1600 >Matrix 4.7800 1.5700 0.6200 0.7600 1.0600 3.0400 4.6500 >NumArr 3.2900 0.7400 0.6800 0.7800 8.4800 7.4200 11.6600 >Numeri 1.3300 0.3900 0.3100 0.4200 0.7900 0.6800 0.7600 >Matlab 1.88 0.44 0.41 0.35 0.37 1.20 0.98 > >Dimension: 50 >Array 9.0000 2.1400 0.5500 18.9500 1.4100 4.2700 4.4500 >Matrix 48.7400 3.9200 1.0100 20.2000 1.8000 6.5000 8.1900 >NumArr 32.3900 2.6800 1.0000 18.9700 13.0300 8.6300 13.0700 >Numeri 13.1000 2.2600 0.6500 18.2700 10.1500 1.0400 3.2600 >Matlab 16.98 1.94 1.07 17.86 0.73 1.57 1.77 > >Dimension: 500 >Array 1.1400 9.2300 2.0100 168.2700 2.1800 4.0200 4.2900 >Matrix 5.0300 9.3500 2.1500 167.5300 2.1700 4.1100 4.4200 >NumArr 3.4400 9.1000 2.1000 168.7100 21.8400 4.3900 5.8900 >Numeri 1.5800 9.2700 2.0700 167.5600 20.0500 3.4000 4.6800 >Matlab 2.09 6.07 2.17 169.45 2.10 2.56 3.06 > >Note the 10-fold speed-up for higher dimensions :-) > >It looks like that now that numpy only looses to matlab in small >dimensions. Probably, the problem is the creation of the object to >represent the transposed object. Probably Matlab creation of objects is >very lightweight (they only have matrices objects to deal with). >Probably this phenomenon explains the behavior for the indexing >operations too. > >Paulo > > > > Here's an idea Fernando and I have briefly talked about off-list, but which perhaps bears talking about here: Is there speed to be gained by an alternative, very simple, very optimized ndarray constructor? The idea would be a special-case constructor with very limited functionality designed purely for speed. It wouldn't support (m)any of the fantastic things Travis has done, but would be useful only in specialized use cases, such as creating indices. I'm not familiar enough with what the normal constructor does to know if we could implement something, (in C, perhaps) that would do nothing but create a simple, contiguous array significantly faster than what is currently done. Or does the current constructor create a new instance about as fast as possible? I know Travis has optimized it, but it's a general purpose constructor, and I'm thinking these extra features may take some extra CPU cycles. Cheers! Andrew From oliphant at ee.byu.edu Wed Jan 18 11:49:03 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed Jan 18 11:49:03 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: <43CE985E.9050406@astraw.com> References: <1137584647.19613.17.camel@localhost.localdomain> <43CE85AD.1090904@ieee.org> <1137610937.19613.65.camel@localhost.localdomain> <43CE985E.9050406@astraw.com> Message-ID: <43CE9B65.5040803@ee.byu.edu> Andrew Straw wrote: > Here's an idea Fernando and I have briefly talked about off-list, but > which perhaps bears talking about here: Is there speed to be gained by > an alternative, very simple, very optimized ndarray constructor? The > idea would be a special-case constructor with very limited > functionality designed purely for speed. It wouldn't support (m)any of > the fantastic things Travis has done, but would be useful only in > specialized use cases, such as creating indices. The general purpose constructor is PyArray_NewFromDescr(...) I suspect this could be special cased for certain circumstances and the special-case called occasionally. Their are checks on the dimensions that could be avoided in certain circumstances (like when we are getting the dimensions from another arrayobject already...) We could also inline the __array_from_strides code... Besides that, I'm not sure what else to optimize... -Travis From oliphant at ee.byu.edu Wed Jan 18 11:56:07 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed Jan 18 11:56:07 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: <43CE985E.9050406@astraw.com> References: <1137584647.19613.17.camel@localhost.localdomain> <43CE85AD.1090904@ieee.org> <1137610937.19613.65.camel@localhost.localdomain> <43CE985E.9050406@astraw.com> Message-ID: <43CE9D1D.6090509@ee.byu.edu> >> > Here's an idea Fernando and I have briefly talked about off-list, but > which perhaps bears talking about here: Is there speed to be gained by > an alternative, very simple, very optimized ndarray constructor? The > idea would be a special-case constructor with very limited > functionality designed purely for speed. It wouldn't support (m)any of > the fantastic things Travis has done, but would be useful only in > specialized use cases, such as creating indices. > > I'm not familiar enough with what the normal constructor does to know > if we could implement something, (in C, perhaps) that would do nothing > but create a simple, contiguous array significantly faster than what > is currently done. Or does the current constructor create a new > instance about as fast as possible? I know Travis has optimized it, > but it's a general purpose constructor, and I'm thinking these extra > features may take some extra CPU cycles. I think the indexing code will be slower because it is more sophisticated than Numeric's. Basically, it has to check for fancy indexing before defaulting to the old way. I see this as more of a slow-down than array creation. It might be possible to improve it --- more eyeballs are always helpful. But, I'm not sure how at this point. -Travis From robert.kern at gmail.com Wed Jan 18 12:07:02 2006 From: robert.kern at gmail.com (Robert Kern) Date: Wed Jan 18 12:07:02 2006 Subject: [SciPy-dev] [Numpy-discussion] Re: numpy/scipy some remarks In-Reply-To: <43CE90AA.2020104@colorado.edu> References: <43CE3E27.5080207@gmail.com> <43CE8077.8020706@colorado.edu> <43CE83D5.4060105@gmail.com> <43CE86C5.1050702@ieee.org> <43CE90AA.2020104@colorado.edu> Message-ID: <43CE9FA2.3000705@gmail.com> Fernando Perez wrote: > Travis Oliphant wrote: > >>>3) pkgload() exists to support the loading of subpackages. It does not reach >>>into numpy.dft or numpy.linalg at all. It is not relevant to this issue. >>> >>>4) There are some places in numpy that use numpy.dual. >>> >>>I think we can address all of your concerns by changing #4. >> >>This is an accurate assessment. However, I do not want to eliminate >>number 4 as I've mentioned before. I think there is a place for having >>functions that can be over-written with better versions. I agree that >>it could be implemented better, however, with some kind of register >>function instead of automatically looking in scipy... > > Mmh, I think I'm confused then: So am I, now! > it seemed to me that pkgload() WAS overwriting > numpy names, from the messages which the environment variable controls. Is > that not true? Here's a recent thread: > > http://aspn.activestate.com/ASPN/Mail/Message/scipy-dev/2974044 > > where this was shown: > > In [3]: import scipy > Overwriting fft= from > scipy.fftpack.basic (was from > numpy.dft.fftpack) > Overwriting ifft= from > scipy.fftpack.basic (was from > numpy.dft.fftpack) [~]$ python Python 2.4.1 (#2, Mar 31 2005, 00:05:10) [GCC 3.3 20030304 (Apple Computer, Inc. build 1666)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import scipy scipy.>>> scipy.pkgload(verbose=2) Imports to 'scipy' namespace ---------------------------- __all__.append('io') import lib -> success Overwriting lib= from /Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/scipy-0.4.4.1307-py2.4-macosx-10.4-ppc.egg/scipy/lib/__init__.pyc (was from /Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/numpy-0.9.4.1849-py2.4-macosx-10.4-ppc.egg/numpy/lib/__init__.pyc) __all__.append('signal') __all__.append('interpolate') __all__.append('lib.lapack') import cluster -> success __all__.append('montecarlo') __all__.append('fftpack') __all__.append('sparse') __all__.append('integrate') __all__.append('optimize') __all__.append('special') import lib.blas -> success __all__.append('linalg') __all__.append('stats') >>> import numpy >>> numpy.lib >>> scipy.lib [Ignore the SVN version numbers. They are faked. I was using checkouts from last night.] > I understood this as 'scipy.fftpack.basic.fft overwrote > numpy.dft.fftpack.fft'. Does this then not affect the numpy namespace at all? If it does, then I agree with you that this should change. > I also would like to propose that, rather than using an environment variable, > pkgload() takes a 'verbose=' keyword (or 'quiet='). I think it's much cleaner > to say > > pkgload(quiet=1) or pkgload(verbose=0) > > than relying on users configuring env. variables for something like this. It does take a verbose keyword argument. -- Robert Kern robert.kern at gmail.com "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From ndarray at mac.com Wed Jan 18 12:09:07 2006 From: ndarray at mac.com (Sasha) Date: Wed Jan 18 12:09:07 2006 Subject: [Numpy-discussion] Singleton bool_ values Message-ID: As of svn version 1931, numpy bool_ values are singletons: >>> from numpy import * >>> bool_(0) is array([True,False])[1] This change makes bool_ values behavior more similar to python bools and will allow much faster implementation of scalar boolean algebra. In order to allow other modules to take advantage of this property, I would like to propose several additions to python and c interfaces. At the Python level: define True_ and False_ constants. At the C-API level: PyArrayScalar_True PyArrayScalar_False PyArrayScalar_BoolFromLong PyArrayScalar_RETURN_TRUE PyArrayScalar_RETURN_FALSE PyArrayScalar_RETURN_BOOL_FROM_LONG Any objections? -- sasha From robert.kern at gmail.com Wed Jan 18 12:09:09 2006 From: robert.kern at gmail.com (Robert Kern) Date: Wed Jan 18 12:09:09 2006 Subject: [SciPy-dev] [Numpy-discussion] importing madness In-Reply-To: <43CE9178.3050602@ftw.at> References: <43CE3E27.5080207@gmail.com> <43CE8077.8020706@colorado.edu> <43CE83D5.4060105@gmail.com> <43CE9178.3050602@ftw.at> Message-ID: <43CEA042.20108@gmail.com> Ed Schofield wrote: > Unless I'm doing something very stupid, there seem to be multiple > sources of evil here. First, numpy's linalg package is available from > the scipy namespace, which seems like a recipe for Ed's madness, since > he can't find his pinv2() function. Second, scipy.pkgload('linalg') > silently fails to make any visible difference. This is probably a > simple bug. It's certainly not intended behavior. The process by which pkgload() determines where it is being called from is messing up. pkgload lives in numpy._import_tools, but it is also exposed in the scipy namespace, too. Please file a bug report and assign it to Pearu. -- Robert Kern robert.kern at gmail.com "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From Fernando.Perez at colorado.edu Wed Jan 18 14:01:02 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Wed Jan 18 14:01:02 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: <43CE9B65.5040803@ee.byu.edu> References: <1137584647.19613.17.camel@localhost.localdomain> <43CE85AD.1090904@ieee.org> <1137610937.19613.65.camel@localhost.localdomain> <43CE985E.9050406@astraw.com> <43CE9B65.5040803@ee.byu.edu> Message-ID: <43CEBA67.8000104@colorado.edu> Travis Oliphant wrote: > Andrew Straw wrote: > > >>Here's an idea Fernando and I have briefly talked about off-list, but >>which perhaps bears talking about here: Is there speed to be gained by >>an alternative, very simple, very optimized ndarray constructor? The >>idea would be a special-case constructor with very limited >>functionality designed purely for speed. It wouldn't support (m)any of >>the fantastic things Travis has done, but would be useful only in >>specialized use cases, such as creating indices. > > > The general purpose constructor is > > PyArray_NewFromDescr(...) > > I suspect this could be special cased for certain circumstances and the > special-case called occasionally. Their are checks on the dimensions > that could be avoided in certain circumstances (like when we are getting > the dimensions from another arrayobject already...) > > We could also inline the __array_from_strides code... > > Besides that, I'm not sure what else to optimize... Just to give some context: this came to my mind inspired by Blitz++'s TinyVector and TinyMatrix objects. In Blitz, arrays have compile-time rank, but run-time size in all dimensions. Since this introduces some overhead, Blitz offers also the Tiny* classes, which are compile-time fixed _both_ in rank and in size. This allows a number of optimizations to be made on them, at the cost of some flexibility lost. Some info on these guys: http://www.oonumerics.org/blitz/manual/blitz07.html What Andrew and I discussed was the idea of writing some object which would only support the most basic operations: element-wise arithmetic, slicing, linear algebra calls on them (matrix-matrix, matrix-vector and vector operations), and little else. I'd be OK losing fancy indexing, byteswapping, memory-mapping, reshaping, and anything else which costs either: 1. initialization-time CPU cycles 2. memory footprint 3. runtime element access and arithmetic. Such objects could be very useful in many contexts. I'd even like an immutable version, so they could be used as dictionary keys without having to make a tuple out of them. This would allow algorithms which use small arrays as multidimensional indices in sparse tree structures to be used without the hoops one must jump through today, and with higher performance. I wonder if giving up reshaping would allow the indexing code to be faster, as specialized versions could be hard-coded for each rank, with only say ranks 1-4 offered for this kind of object (I know we recently had a discussion about large ranks, but this object would be geared towards pure performance, and certainly working in 32 dimensions is a 'flexibility-driven' case, where the generic objects are called for). Note that I had never mentioned this in public, because I think it may be a slight specialization that isn't needed early on, and currently the library's priority was to get off the ground. But having such objects could be very handy, and now that the C API is starting to stabilize, maybe someone can play with this as a side project. Once they prove their worth, these beasts could be folded as part of the official distribution. I am not really qualified to judge whether there are enough areas for optimization where the sacrifices indicated could really pay off, both in terms of memory and performance. Cheers, f From pearu at scipy.org Wed Jan 18 14:15:01 2006 From: pearu at scipy.org (Pearu Peterson) Date: Wed Jan 18 14:15:01 2006 Subject: [SciPy-dev] [Numpy-discussion] Re: numpy/scipy some remarks In-Reply-To: <43CE90AA.2020104@colorado.edu> References: <43CE3E27.5080207@gmail.com> <43CE8077.8020706@colorado.edu> <43CE83D5.4060105@gmail.com> <43CE86C5.1050702@ieee.org> <43CE90AA.2020104@colorado.edu> Message-ID: On Wed, 18 Jan 2006, Fernando Perez wrote: > Mmh, I think I'm confused then: it seemed to me that pkgload() WAS > overwriting numpy names, from the messages which the environment variable > controls. Is that not true? Here's a recent thread: > > http://aspn.activestate.com/ASPN/Mail/Message/scipy-dev/2974044 pkgload() is overwriting numpy names in *scipy namespace*. To be explicit, the following is going on in scipy/__init__.py when pkgload is called, pseudocode follows: from numpy import * # imports fft old_fft = fft from scipy.fftpack import fft print 'Overwriting',old_fft,'with',fft del old_fft And nothing else! So, scipy.fft, that was numpy.fft, is set to scipy.fftpack.fft. numpy.fft remains the same. ... > I understood this as 'scipy.fftpack.basic.fft overwrote > numpy.dft.fftpack.fft'. Does this then not affect the numpy namespace at > all? No! See above. > I also would like to propose that, rather than using an environment variable, > pkgload() takes a 'verbose=' keyword (or 'quiet='). I think it's much > cleaner to say > > pkgload(quiet=1) or pkgload(verbose=0) > > than relying on users configuring env. variables for something like this. pkgload has already verbose kwd argumend. See ?pkgload for more information. Pearu From Fernando.Perez at colorado.edu Wed Jan 18 14:29:11 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Wed Jan 18 14:29:11 2006 Subject: [SciPy-dev] [Numpy-discussion] Re: numpy/scipy some remarks In-Reply-To: References: <43CE3E27.5080207@gmail.com> <43CE8077.8020706@colorado.edu> <43CE83D5.4060105@gmail.com> <43CE86C5.1050702@ieee.org> <43CE90AA.2020104@colorado.edu> Message-ID: <43CEC128.30601@colorado.edu> Pearu Peterson wrote: >>I understood this as 'scipy.fftpack.basic.fft overwrote >>numpy.dft.fftpack.fft'. Does this then not affect the numpy namespace at >>all? > > > No! See above. Ah, OK. Thanks for the clarification, I misunderstood the message. >>I also would like to propose that, rather than using an environment variable, >>pkgload() takes a 'verbose=' keyword (or 'quiet='). I think it's much >>cleaner to say >> >>pkgload(quiet=1) or pkgload(verbose=0) >> >>than relying on users configuring env. variables for something like this. > > > > pkgload has already verbose kwd argumend. See ?pkgload for more > information. What is this '?foo' syntax you speak of? Certainly not python... ;-) Sorry for not checking, I just don't have new-scipy on my work machine, only at home. Cheers, f From perry at stsci.edu Wed Jan 18 14:38:01 2006 From: perry at stsci.edu (Perry Greenfield) Date: Wed Jan 18 14:38:01 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: <43CEBA67.8000104@colorado.edu> References: <1137584647.19613.17.camel@localhost.localdomain> <43CE85AD.1090904@ieee.org> <1137610937.19613.65.camel@localhost.localdomain> <43CE985E.9050406@astraw.com> <43CE9B65.5040803@ee.byu.edu> <43CEBA67.8000104@colorado.edu> Message-ID: It's not a new idea. I raised it some time ago and I don't think it was new then either. I have to believe that if you allowed only Float64 (and perhaps a complex variant) and used other restrictions then it would be much faster for small arrays. One would think it would be much easier to implement than Numeric/numarray/numpy... I've always thought that those looking for really fast small array performance would be better served by something like this. But you'd really have to fight off feature creep. ("This almost meets my needs. If it could only do xxx") Perry On Jan 18, 2006, at 5:00 PM, Fernando Perez wrote: > Travis Oliphant wrote: >> Andrew Straw wrote: >>> Here's an idea Fernando and I have briefly talked about off-list, >>> but which perhaps bears talking about here: Is there speed to be >>> gained by an alternative, very simple, very optimized ndarray >>> constructor? The idea would be a special-case constructor with very >>> limited functionality designed purely for speed. It wouldn't support >>> (m)any of the fantastic things Travis has done, but would be useful >>> only in specialized use cases, such as creating indices. >> The general purpose constructor is >> PyArray_NewFromDescr(...) >> I suspect this could be special cased for certain circumstances and >> the special-case called occasionally. Their are checks on the >> dimensions that could be avoided in certain circumstances (like when >> we are getting the dimensions from another arrayobject already...) >> We could also inline the __array_from_strides code... >> Besides that, I'm not sure what else to optimize... > > Just to give some context: this came to my mind inspired by Blitz++'s > TinyVector and TinyMatrix objects. In Blitz, arrays have compile-time > rank, but run-time size in all dimensions. Since this introduces some > overhead, Blitz offers also the Tiny* classes, which are compile-time > fixed _both_ in rank and in size. This allows a number of > optimizations to be made on them, at the cost of some flexibility > lost. Some info on these guys: > > http://www.oonumerics.org/blitz/manual/blitz07.html > > What Andrew and I discussed was the idea of writing some object which > would only support the most basic operations: element-wise arithmetic, > slicing, linear algebra calls on them (matrix-matrix, matrix-vector > and vector operations), and little else. I'd be OK losing fancy > indexing, byteswapping, memory-mapping, reshaping, and anything else > which costs either: > > 1. initialization-time CPU cycles > 2. memory footprint > 3. runtime element access and arithmetic. > > Such objects could be very useful in many contexts. I'd even like an > immutable version, so they could be used as dictionary keys without > having to make a tuple out of them. This would allow algorithms which > use small arrays as multidimensional indices in sparse tree structures > to be used without the hoops one must jump through today, and with > higher performance. > > I wonder if giving up reshaping would allow the indexing code to be > faster, as specialized versions could be hard-coded for each rank, > with only say ranks 1-4 offered for this kind of object (I know we > recently had a discussion about large ranks, but this object would be > geared towards pure performance, and certainly working in 32 > dimensions is a 'flexibility-driven' case, where the generic objects > are called for). > > Note that I had never mentioned this in public, because I think it may > be a slight specialization that isn't needed early on, and currently > the library's priority was to get off the ground. But having such > objects could be very handy, and now that the C API is starting to > stabilize, maybe someone can play with this as a side project. Once > they prove their worth, these beasts could be folded as part of the > official distribution. > > I am not really qualified to judge whether there are enough areas for > optimization where the sacrifices indicated could really pay off, both > in terms of memory and performance. > > Cheers, > > f > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log > files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://sel.as-us.falkag.net/sel? > cmd=lnk&kid=103432&bid=230486&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From cookedm at physics.mcmaster.ca Wed Jan 18 14:42:06 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Wed Jan 18 14:42:06 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: <43CEBA67.8000104@colorado.edu> (Fernando Perez's message of "Wed, 18 Jan 2006 15:00:07 -0700") References: <1137584647.19613.17.camel@localhost.localdomain> <43CE85AD.1090904@ieee.org> <1137610937.19613.65.camel@localhost.localdomain> <43CE985E.9050406@astraw.com> <43CE9B65.5040803@ee.byu.edu> <43CEBA67.8000104@colorado.edu> Message-ID: Fernando Perez writes: > Travis Oliphant wrote: >> Andrew Straw wrote: >> >>> Here's an idea Fernando and I have briefly talked about off-list, >>> but which perhaps bears talking about here: Is there speed to be >>> gained by an alternative, very simple, very optimized ndarray >>> constructor? The idea would be a special-case constructor with very >>> limited functionality designed purely for speed. It wouldn't >>> support (m)any of the fantastic things Travis has done, but would >>> be useful only in specialized use cases, such as creating indices. >> The general purpose constructor is >> PyArray_NewFromDescr(...) >> I suspect this could be special cased for certain circumstances and >> the special-case called occasionally. Their are checks on the >> dimensions that could be avoided in certain circumstances (like when >> we are getting the dimensions from another arrayobject already...) >> We could also inline the __array_from_strides code... >> Besides that, I'm not sure what else to optimize... > > Just to give some context: this came to my mind inspired by Blitz++'s > TinyVector and TinyMatrix objects. In Blitz, arrays have compile-time > rank, but run-time size in all dimensions. Since this introduces some > overhead, Blitz offers also the Tiny* classes, which are compile-time > fixed _both_ in rank and in size. This allows a number of > optimizations to be made on them, at the cost of some flexibility > lost. Some info on these guys: > > http://www.oonumerics.org/blitz/manual/blitz07.html > > What Andrew and I discussed was the idea of writing some object which > would only support the most basic operations: element-wise arithmetic, > slicing, linear algebra calls on them (matrix-matrix, matrix-vector > and vector operations), and little else. I'd be OK losing fancy > indexing, byteswapping, memory-mapping, reshaping, and anything else > which costs either: > > 1. initialization-time CPU cycles > 2. memory footprint > 3. runtime element access and arithmetic. > > Such objects could be very useful in many contexts. I'd even like an > immutable version, so they could be used as dictionary keys without > having to make a tuple out of them. This would allow algorithms which > use small arrays as multidimensional indices in sparse tree structures > to be used without the hoops one must jump through today, and with > higher performance. I've done a little bit of work along these lines. I have a module I call vector3 [*] which has 2- and 3-dimensional immutable vectors, using either ints or doubles. It's as fast as I could make it, while keeping it all written in Pyrex. I find it very convenient for anything vector-related. Konrad Hinsen has something similiar in the development version of his ScientificPython package. [*] http://arbutus.mcmaster.ca/dmc/software/vector3.html Also, I've also done some playing around with a n-dimensional vector type (restricted to doubles). My best attempts make it ~4-5x faster than numpy (and 2x faster than Numeric) for vectors of dimension 10 on simple ops like + and *, 2x faster than numpy for dimension 1000, and approaching 1x as you make the vectors larger. Indexing is about 3x faster than numpy, and 1.4x faster than Numeric. So that gives I think some idea of the maximum speed-up possible. I think the speedups mostly come from the utter lack of any polymorphism: it handles vectors of doubles only, and only as contiguous vectors (no strides). -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From Fernando.Perez at colorado.edu Wed Jan 18 14:52:08 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Wed Jan 18 14:52:08 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: References: <1137584647.19613.17.camel@localhost.localdomain> <43CE85AD.1090904@ieee.org> <1137610937.19613.65.camel@localhost.localdomain> <43CE985E.9050406@astraw.com> <43CE9B65.5040803@ee.byu.edu> <43CEBA67.8000104@colorado.edu> Message-ID: <43CEC67B.1090600@colorado.edu> David M. Cooke wrote: > I've done a little bit of work along these lines. I have a module I > call vector3 [*] which has 2- and 3-dimensional immutable vectors, > using either ints or doubles. It's as fast as I could make it, while > keeping it all written in Pyrex. I find it very convenient for > anything vector-related. Konrad Hinsen has something similiar in the > development version of his ScientificPython package. > > [*] http://arbutus.mcmaster.ca/dmc/software/vector3.html > > Also, I've also done some playing around with a n-dimensional vector > type (restricted to doubles). My best attempts make it ~4-5x faster > than numpy (and 2x faster than Numeric) for vectors of dimension 10 > on simple ops like + and *, 2x faster than numpy for dimension 1000, > and approaching 1x as you make the vectors larger. Indexing is about > 3x faster than numpy, and 1.4x faster than Numeric. So that gives I > think some idea of the maximum speed-up possible. > > I think the speedups mostly come from the utter lack of any > polymorphism: it handles vectors of doubles only, and only as > contiguous vectors (no strides). This is excellent, thanks for the pointer. I can see uses for vectors (still 1-d, no strides, etc) with more than 3 elements, and perhaps fixed-size (no reshaping, no striding) 2-d arrays (matrices), but this looks like a good starting point. Sandbox material? Cheers, f From oliphant at ee.byu.edu Wed Jan 18 14:59:03 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed Jan 18 14:59:03 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: <43CEC67B.1090600@colorado.edu> References: <1137584647.19613.17.camel@localhost.localdomain> <43CE85AD.1090904@ieee.org> <1137610937.19613.65.camel@localhost.localdomain> <43CE985E.9050406@astraw.com> <43CE9B65.5040803@ee.byu.edu> <43CEBA67.8000104@colorado.edu> <43CEC67B.1090600@colorado.edu> Message-ID: <43CEC7FC.1040408@ee.byu.edu> Fernando Perez wrote: > David M. Cooke wrote: > >> I've done a little bit of work along these lines. I have a module I >> call vector3 [*] which has 2- and 3-dimensional immutable vectors, >> using either ints or doubles. It's as fast as I could make it, while >> keeping it all written in Pyrex. I find it very convenient for >> anything vector-related. Konrad Hinsen has something similiar in the >> development version of his ScientificPython package. >> >> [*] http://arbutus.mcmaster.ca/dmc/software/vector3.html >> >> Also, I've also done some playing around with a n-dimensional vector >> type (restricted to doubles). My best attempts make it ~4-5x faster >> than numpy (and 2x faster than Numeric) for vectors of dimension 10 >> on simple ops like + and *, 2x faster than numpy for dimension 1000, >> and approaching 1x as you make the vectors larger. Indexing is about >> 3x faster than numpy, and 1.4x faster than Numeric. So that gives I >> think some idea of the maximum speed-up possible. >> >> I think the speedups mostly come from the utter lack of any >> polymorphism: it handles vectors of doubles only, and only as >> contiguous vectors (no strides). > > > This is excellent, thanks for the pointer. I can see uses for vectors > (still 1-d, no strides, etc) with more than 3 elements, and perhaps > fixed-size (no reshaping, no striding) 2-d arrays (matrices), but this > looks like a good starting point. Sandbox material? > With the array interface, these kinds of objects can play very nicely with full ndarray's as well... -Travis From Fernando.Perez at colorado.edu Wed Jan 18 15:22:10 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Wed Jan 18 15:22:10 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: References: <1137584647.19613.17.camel@localhost.localdomain> <43CE85AD.1090904@ieee.org> <1137610937.19613.65.camel@localhost.localdomain> <43CE985E.9050406@astraw.com> <43CE9B65.5040803@ee.byu.edu> <43CEBA67.8000104@colorado.edu> Message-ID: <43CECD88.6050809@colorado.edu> Perry Greenfield wrote: > It's not a new idea. I raised it some time ago and I don't think it was I wasn't claiming authorship, sorry if it sounded like that :) In fact, I remember specifically talking with you about this at scipy'03, in the context of small array performance issues for the at-the-time-nascent numarray, and I'm sure similar things have been done many times before. I've had it floating in my head since I first saw blitz, back in 2001, and blitz probably got it from... There's nothing really new under the sun ;) > new then either. I have to believe that if you allowed only Float64 > (and perhaps a complex variant) and used other restrictions then it > would be much faster for small arrays. One would think it would be much > easier to implement than Numeric/numarray/numpy... I've always thought > that those looking for really fast small array performance would be > better served by something like this. But you'd really have to fight > off feature creep. ("This almost meets my needs. If it could only do > xxx") Couldn't that last issue be well dealt with by the fact that today's numpy is fairly subclassing-friendly? (which, if I remember correctly, wasn't quite the case with at least old Numeric). Cheers, f From rowen at cesmail.net Wed Jan 18 16:02:09 2006 From: rowen at cesmail.net (Russell E. Owen) Date: Wed Jan 18 16:02:09 2006 Subject: [Numpy-discussion] Efficient way to handle None->nan? Message-ID: We're getting numeric data from a (MySQL) database. We'd like to use numarray or NumPy on the resulting data, but some values may be None. Is there a fast, efficient way to replace None with NaN? I'd hate to use a list comprehension on each data tuple before turning it into an array, but I haven't thought of anything else. numarray.array and numarray.where are both intolerant of None in the input data. -- Russell From oliphant.travis at ieee.org Wed Jan 18 16:10:11 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed Jan 18 16:10:11 2006 Subject: [Numpy-discussion] array vs. matrix for matlab-compatible stuff In-Reply-To: <43CE631D.1000006@gmx.net> References: <43CE631D.1000006@gmx.net> Message-ID: <43CE7E68.1010908@ieee.org> Sven Schreiber wrote: > Hi, > I've spent a couple of weeks with scipy/numpy and the old-to-new > transition; now that the transition is over (?) but some confusion is > remaining (on my side) I feel the need to ask a basic question about > matlab compatibility in terms of matrix (linear algebra) programming. > > Take "eye" and "identity" for example; given that "eye" supposedly > exists to facilitate transiton from matlab to numpy/scipy (correct?), > I expected eye to be/return a matrix. Historical is the only reason. Numeric always returned an array for eye not a matrix. We could return a matrix without difficulty especially if we put an eye --> identity transition in convertcode.py -Travis From ndarray at mac.com Wed Jan 18 16:11:13 2006 From: ndarray at mac.com (Sasha) Date: Wed Jan 18 16:11:13 2006 Subject: [Numpy-discussion] Efficient way to handle None->nan? In-Reply-To: References: Message-ID: >>> from numpy.core.ma import masked_values >>> from numpy import nan >>> masked_values([1.0,None,2.0],None).filled(nan).astype(float) array([ 1. , nan, 2. ]) On 1/18/06, Russell E. Owen wrote: > We're getting numeric data from a (MySQL) database. We'd like to use > numarray or NumPy on the resulting data, but some values may be None. Is > there a fast, efficient way to replace None with NaN? I'd hate to use a > list comprehension on each data tuple before turning it into an array, > but I haven't thought of anything else. > > numarray.array and numarray.where are both intolerant of None in the > input data. > > -- Russell > > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From perry at stsci.edu Wed Jan 18 17:00:01 2006 From: perry at stsci.edu (Perry Greenfield) Date: Wed Jan 18 17:00:01 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: <43CECD88.6050809@colorado.edu> References: <1137584647.19613.17.camel@localhost.localdomain> <43CE85AD.1090904@ieee.org> <1137610937.19613.65.camel@localhost.localdomain> <43CE985E.9050406@astraw.com> <43CE9B65.5040803@ee.byu.edu> <43CEBA67.8000104@colorado.edu> <43CECD88.6050809@colorado.edu> Message-ID: <863273026756969dc8715e00bf0776c7@stsci.edu> On Jan 18, 2006, at 6:21 PM, Fernando Perez wrote: > Perry Greenfield wrote: >> It's not a new idea. I raised it some time ago and I don't think it >> was > > I wasn't claiming authorship, sorry if it sounded like that :) In > fact, I remember specifically talking with you about this at scipy'03, > in the context of small array performance issues for the > at-the-time-nascent numarray, and I'm sure similar things have been > done many times before. I've had it floating in my head since I first > saw blitz, back in 2001, and blitz probably got it from... There's > nothing really new under the sun ;) > Really :-). I remember that conversation and wondered if it had something to do with that. (And I remember Paul Dubois talking to me about similar ideas). I think it is worth trying (and has been I see, though I would have expected perhaps even a greater speed improvement; somehow I think it should not take a lot of time if you don't need all the type, shape and striding flexibility). It just needs someone to do it. >> new then either. I have to believe that if you allowed only Float64 >> (and perhaps a complex variant) and used other restrictions then it >> would be much faster for small arrays. One would think it would be >> much easier to implement than Numeric/numarray/numpy... I've always >> thought that those looking for really fast small array performance >> would be better served by something like this. But you'd really have >> to fight off feature creep. ("This almost meets my needs. If it >> could only do xxx") > > Couldn't that last issue be well dealt with by the fact that today's > numpy is fairly subclassing-friendly? (which, if I remember correctly, > wasn't quite the case with at least old Numeric). Does that help? You aren't talking about the fast array subclassing numpy are you? I'm not sure what you mean here. Perry From Fernando.Perez at colorado.edu Wed Jan 18 17:09:02 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Wed Jan 18 17:09:02 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: <863273026756969dc8715e00bf0776c7@stsci.edu> References: <1137584647.19613.17.camel@localhost.localdomain> <43CE85AD.1090904@ieee.org> <1137610937.19613.65.camel@localhost.localdomain> <43CE985E.9050406@astraw.com> <43CE9B65.5040803@ee.byu.edu> <43CEBA67.8000104@colorado.edu> <43CECD88.6050809@colorado.edu> <863273026756969dc8715e00bf0776c7@stsci.edu> Message-ID: <43CEE67D.9000707@colorado.edu> Perry Greenfield wrote: > On Jan 18, 2006, at 6:21 PM, Fernando Perez wrote: > Really :-). I remember that conversation and wondered if it had > something to do with that. (And I remember Paul Dubois talking to me > about similar ideas). I think it is worth trying (and has been I see, > though I would have expected perhaps even a greater speed improvement; > somehow I think it should not take a lot of time if you don't need all > the type, shape and striding flexibility). It just needs someone to do > it. Maybe putting David's code into the sandbox would be a good starting point. >>>new then either. I have to believe that if you allowed only Float64 >>>(and perhaps a complex variant) and used other restrictions then it >>>would be much faster for small arrays. One would think it would be >>>much easier to implement than Numeric/numarray/numpy... I've always >>>thought that those looking for really fast small array performance >>>would be better served by something like this. But you'd really have >>>to fight off feature creep. ("This almost meets my needs. If it >>>could only do xxx") >> >>Couldn't that last issue be well dealt with by the fact that today's >>numpy is fairly subclassing-friendly? (which, if I remember correctly, >>wasn't quite the case with at least old Numeric). > > > Does that help? You aren't talking about the fast array subclassing > numpy are you? I'm not sure what you mean here. What I meant was that by having good subclassing functionality, it's easier to ward off requests for every feature under the sun. It's much easier to say: 'this basic object provides a very small, core set of array features where the focus is on raw speed rather than fancy features; if you need extra features, subclass it and add them yourself' when the subclassing is actually reasonably easy. Note that I haven't actually used array subclassing myself (haven't needed it), so I may be mistaken in my comments here, it's just an intuition. Cheers, f From oliphant at ee.byu.edu Wed Jan 18 17:21:09 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed Jan 18 17:21:09 2006 Subject: [Numpy-discussion] Re: Passing context in __array__ In-Reply-To: References: <43CEB431.3010405@ee.byu.edu> Message-ID: <43CEE946.1050906@ee.byu.edu> Alexander Belopolsky wrote: >On 1/18/06, Travis Oliphant wrote: > > >>This mail got placed in my junk folder so I did not see it until now. >>Sorry about the confusion... >> >> >> >To add to the confusion, I've interpreted your previous e-mail as "go >ahead" and commited the changes. Hopefuly your concerns can be >addressed and I will not have to retract the change. > > > That is actually fine. I don't mind having changes in SVN that we discuss and fix. It's what its for. As long as it builds, then great... I went ahead and altered the C-API based on your desired use and changed numpy to work with it. This means that the SVN version of NumPy requires a recompilation of extension modules.... Some of my previous comments did not understand how you had implemented the routine. Hopefully what's in SVN now works correctly (it passes your tests...) >Yes, I agree and confessed to this problem in the comments to my code. > However, I did not want to touch PyArray_FromAny because it is part >of published C API. The best solution would be to expose parts of >PyArray_FromAny in C API and use those in ufunc code. > > We are pre 1.0 and so I don't have too much problems adding an extra context argument to the PyArray_FromAny call :-) I've also simplified things a bit, too. I moved array_fromobject to PyArray_FromAny and renamed PyArray_FromAny to PyArray_CheckFromAny(...) as it has a couple more checks to ensure DEFAULT_FLAGS are set when ENSURECOPY is set and to check for the NOTSWAPPED requirement... Often flags is 0 or does not have these FLAGS (or we have a native byte-order descriptor anyway and so the NOTSWAPPED requirement is redundant) and so the simpler PyArray_FromAny can be used. Most people are still using PyArray_ContiguousFromObject and friends anyway and so the change doesn't affect them at all. -Travis From perry at stsci.edu Wed Jan 18 17:22:05 2006 From: perry at stsci.edu (Perry Greenfield) Date: Wed Jan 18 17:22:05 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: <43CEE67D.9000707@colorado.edu> References: <1137584647.19613.17.camel@localhost.localdomain> <43CE85AD.1090904@ieee.org> <1137610937.19613.65.camel@localhost.localdomain> <43CE985E.9050406@astraw.com> <43CE9B65.5040803@ee.byu.edu> <43CEBA67.8000104@colorado.edu> <43CECD88.6050809@colorado.edu> <863273026756969dc8715e00bf0776c7@stsci.edu> <43CEE67D.9000707@colorado.edu> Message-ID: <33e3b696b76c303ca00abe37ff0d48f3@stsci.edu> On Jan 18, 2006, at 8:08 PM, Fernando Perez wrote: >>> Couldn't that last issue be well dealt with by the fact that today's >>> numpy is fairly subclassing-friendly? (which, if I remember >>> correctly, wasn't quite the case with at least old Numeric). >> Does that help? You aren't talking about the fast array subclassing >> numpy are you? I'm not sure what you mean here. > > What I meant was that by having good subclassing functionality, it's > easier to ward off requests for every feature under the sun. It's > much easier to say: > > 'this basic object provides a very small, core set of array features > where the focus is on raw speed rather than fancy features; if you > need extra features, subclass it and add them yourself' > > when the subclassing is actually reasonably easy. Note that I haven't > actually used array subclassing myself (haven't needed it), so I may > be mistaken in my comments here, it's just an intuition. > Yeah, that makes sense. I was confused by the reference to numpy. Perry From oliphant.travis at ieee.org Wed Jan 18 18:19:06 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed Jan 18 18:19:06 2006 Subject: [Numpy-discussion] New SVN (>1939) requires a rebuild of extension modules Message-ID: <43CEF708.9070609@ieee.org> Just in case you missed the note in the middle of another message. The SVN version of NumPy has a few new C-API calls and requires a rebuild of extension modules. If you get mysterious segfaults all of a sudden, you haven't recompiled... -Travis From chris at trichech.us Wed Jan 18 19:10:01 2006 From: chris at trichech.us (Christopher Fonnesbeck) Date: Wed Jan 18 19:10:01 2006 Subject: [Numpy-discussion] numpy svn unit test failure Message-ID: Tonight's svn build of numpy on OS X 10.4 fails the unit test: In [2]: numpy.test(1,1) Found 3 tests for numpy.distutils.misc_util Found 10 tests for numpy.core.umath Found 8 tests for numpy.lib.arraysetops Found 42 tests for numpy.lib.type_check Found 9 tests for numpy.lib.twodim_base Found 15 tests for numpy.core.multiarray Found 25 tests for numpy.core.ma Found 6 tests for numpy.core.defmatrix Found 3 tests for numpy.lib.getlimits Found 33 tests for numpy.lib.function_base Found 3 tests for numpy.dft.helper Found 6 tests for numpy.core.records Found 8 tests for numpy.core.numeric Found 4 tests for numpy.lib.index_tricks Found 44 tests for numpy.lib.shape_base Found 0 tests for __main__ ........................................................................ ........................................E...E........................... ........................................................................ ..... ====================================================================== ERROR: check_basic (numpy.core.defmatrix.test_defmatrix.test_algebra) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/numpy-0.9.4.1940-py2.4-macosx-10.4-ppc.egg/ numpy/core/tests/test_defmatrix.py", line 111, in check_basic assert allclose((mA ** -i).A, B) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/numpy-0.9.4.1940-py2.4-macosx-10.4-ppc.egg/ numpy/core/defmatrix.py", line 155, in __pow__ x = self.I File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/numpy-0.9.4.1940-py2.4-macosx-10.4-ppc.egg/ numpy/core/defmatrix.py", line 209, in getI from numpy.dual import inv File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/numpy-0.9.4.1940-py2.4-macosx-10.4-ppc.egg/ numpy/dual.py", line 24, in ? import scipy.linalg as linpkg File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/scipy-0.4.4.1562-py2.4-macosx-10.4-ppc.egg/ scipy/linalg/__init__.py", line 8, in ? from basic import * File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/scipy-0.4.4.1562-py2.4-macosx-10.4-ppc.egg/ scipy/linalg/basic.py", line 17, in ? from flinalg import get_flinalg_funcs File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/scipy-0.4.4.1562-py2.4-macosx-10.4-ppc.egg/ scipy/linalg/flinalg.py", line 15, in ? from numpy.distutils.misc_util import PostponedException ImportError: cannot import name PostponedException ====================================================================== ERROR: check_basic (numpy.core.defmatrix.test_defmatrix.test_properties) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/numpy-0.9.4.1940-py2.4-macosx-10.4-ppc.egg/ numpy/core/tests/test_defmatrix.py", line 34, in check_basic assert allclose(linalg.inv(A), mA.I) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/numpy-0.9.4.1940-py2.4-macosx-10.4-ppc.egg/ numpy/core/defmatrix.py", line 209, in getI from numpy.dual import inv File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/numpy-0.9.4.1940-py2.4-macosx-10.4-ppc.egg/ numpy/dual.py", line 24, in ? import scipy.linalg as linpkg File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/scipy-0.4.4.1562-py2.4-macosx-10.4-ppc.egg/ scipy/linalg/__init__.py", line 8, in ? from basic import * File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/scipy-0.4.4.1562-py2.4-macosx-10.4-ppc.egg/ scipy/linalg/basic.py", line 17, in ? from flinalg import get_flinalg_funcs File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/scipy-0.4.4.1562-py2.4-macosx-10.4-ppc.egg/ scipy/linalg/flinalg.py", line 15, in ? from numpy.distutils.misc_util import PostponedException ImportError: cannot import name PostponedException ---------------------------------------------------------------------- Ran 221 tests in 2.990s FAILED (errors=2) Out[2]: -- Christopher J. Fonnesbeck Population Ecologist, Marine Mammal Section Fish & Wildlife Research Institute (FWC) St. Petersburg, FL Adjunct Assistant Professor Warnell School of Forest Resources University of Georgia Athens, GA T: 727.235.5570 E: chris at trichech.us -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2417 bytes Desc: not available URL: From strawman at astraw.com Wed Jan 18 20:04:03 2006 From: strawman at astraw.com (Andrew Straw) Date: Wed Jan 18 20:04:03 2006 Subject: [Numpy-discussion] New SVN (>1939) requires a rebuild of extension modules In-Reply-To: <43CEF708.9070609@ieee.org> References: <43CEF708.9070609@ieee.org> Message-ID: <43CF0F80.3060301@astraw.com> Travis Oliphant wrote: > > Just in case you missed the note in the middle of another message. > > The SVN version of NumPy has a few new C-API calls and requires a > rebuild of extension modules. If you get mysterious segfaults all of > a sudden, you haven't recompiled... > > -Travis > Hi Travis, I have an idea that might prevent people from accidentally getting mysterious errors due to mixing extensions compiled against different versions of numpy. Basically, the idea is that a version number is changed every time the C API changes. This version number is stored in each extension module at compile time, and at run time, each module init function that calls import_array() will trigger a test against the version number of the numpy core. I've implemented my idea in the included patch, and, as far as I can tell, it actually works. This is based on using the NDARRAY_VERSION define in arrayobject.h, which I'm completely unsure if this is a suitable use for. Furthermore, there's a lot going on that I don't understand in this part of the numpy code. Thus, I consider my patch only a proof of concept and hope you (or someone) will take this idea, perhaps the patch, and run with it. I hope this prevents many headaches in the future... What do you think? -------------- next part -------------- A non-text attachment was scrubbed... Name: API_version.patch Type: text/x-patch Size: 3071 bytes Desc: not available URL: From ndarray at mac.com Wed Jan 18 21:04:01 2006 From: ndarray at mac.com (Sasha) Date: Wed Jan 18 21:04:01 2006 Subject: [Numpy-discussion] New SVN (>1939) requires a rebuild of extension modules In-Reply-To: <43CF0F80.3060301@astraw.com> References: <43CEF708.9070609@ieee.org> <43CF0F80.3060301@astraw.com> Message-ID: +1 I suggest a little more descriptive exception: PyErr_Format(PyExc_ImportError, "extension module compiled against version %X of numpy API" ", not %X", NDARRAY_VERSION, PyArray_GetNDArrayCVersion()) On 1/18/06, Andrew Straw wrote: > Travis Oliphant wrote: > > > > > Just in case you missed the note in the middle of another message. > > > > The SVN version of NumPy has a few new C-API calls and requires a > > rebuild of extension modules. If you get mysterious segfaults all of > > a sudden, you haven't recompiled... > > > > -Travis > > > > Hi Travis, > > I have an idea that might prevent people from accidentally getting > mysterious errors due to mixing extensions compiled against different > versions of numpy. Basically, the idea is that a version number is > changed every time the C API changes. This version number is stored in > each extension module at compile time, and at run time, each module init > function that calls import_array() will trigger a test against the > version number of the numpy core. > > I've implemented my idea in the included patch, and, as far as I can > tell, it actually works. This is based on using the NDARRAY_VERSION > define in arrayobject.h, which I'm completely unsure if this is a > suitable use for. Furthermore, there's a lot going on that I don't > understand in this part of the numpy code. Thus, I consider my patch > only a proof of concept and hope you (or someone) will take this idea, > perhaps the patch, and run with it. I hope this prevents many headaches > in the future... > > What do you think? > > > From faltet at carabos.com Thu Jan 19 01:13:03 2006 From: faltet at carabos.com (Francesc Altet) Date: Thu Jan 19 01:13:03 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: <43CE9D1D.6090509@ee.byu.edu> References: <1137584647.19613.17.camel@localhost.localdomain> <43CE985E.9050406@astraw.com> <43CE9D1D.6090509@ee.byu.edu> Message-ID: <200601191012.14204.faltet@carabos.com> A Dimecres 18 Gener 2006 20:55, Travis Oliphant va escriure: > I think the indexing code will be slower because it is more > sophisticated than Numeric's. Basically, it has to check for fancy > indexing before defaulting to the old way. I see this as more of a > slow-down than array creation. It might be possible to improve it --- > more eyeballs are always helpful. But, I'm not sure how at this point. I'm sure you already know this: http://oprofile.sourceforge.net It is a nice way to do profiling at C level on Linux machines. Running the Paulo benchmarks through oprofile can surely bring some light. Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From arnd.baecker at web.de Thu Jan 19 01:29:03 2006 From: arnd.baecker at web.de (Arnd Baecker) Date: Thu Jan 19 01:29:03 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: <200601191012.14204.faltet@carabos.com> References: <1137584647.19613.17.camel@localhost.localdomain> <43CE985E.9050406@astraw.com> <43CE9D1D.6090509@ee.byu.edu> <200601191012.14204.faltet@carabos.com> Message-ID: On Thu, 19 Jan 2006, Francesc Altet wrote: > A Dimecres 18 Gener 2006 20:55, Travis Oliphant va escriure: > > I think the indexing code will be slower because it is more > > sophisticated than Numeric's. Basically, it has to check for fancy > > indexing before defaulting to the old way. I see this as more of a > > slow-down than array creation. It might be possible to improve it --- > > more eyeballs are always helpful. But, I'm not sure how at this point. > > I'm sure you already know this: > > http://oprofile.sourceforge.net > > It is a nice way to do profiling at C level on Linux machines. Running > the Paulo benchmarks through oprofile can surely bring some light. What might be nice is to use the output of oprofile and stick it into kcachegrind, http://kcachegrind.sourceforge.net/ to get a GUI to the results. I haven't done this, but according to http://linux.com.hk/penguin/man/1/op2calltree.html the tool is `op2calltree` - convert OProfile profiling data to KCachegrind calltree format and further "This utility is part of the KDE Software Development Kit." Best, Arnd From faltet at carabos.com Thu Jan 19 02:01:06 2006 From: faltet at carabos.com (Francesc Altet) Date: Thu Jan 19 02:01:06 2006 Subject: [Numpy-discussion] New SVN (>1939) requires a rebuild of extension modules In-Reply-To: References: <43CEF708.9070609@ieee.org> <43CF0F80.3060301@astraw.com> Message-ID: <200601191100.21715.faltet@carabos.com> A Dijous 19 Gener 2006 06:03, Sasha va escriure: > +1 > > I suggest a little more descriptive exception: > > PyErr_Format(PyExc_ImportError, > "extension module compiled against version %X of numpy > API" ", not %X", NDARRAY_VERSION, PyArray_GetNDArrayCVersion()) I also like this idea. I'd go a step further and add to the above message something like this: "...Nertheless, if you know what are you doing and want to allow this extension to run, set the environment variable FORCE_NUMPY_RUN to 1 (will continue warning you) or 2 (skip all warnings)." Of course, one should provide some code to check the environment variable, but this should be not very difficult. Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From svetosch at gmx.net Thu Jan 19 04:42:09 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Thu Jan 19 04:42:09 2006 Subject: [Numpy-discussion] array vs. matrix for matlab-compatible stuff In-Reply-To: <43CE7E68.1010908@ieee.org> References: <43CE631D.1000006@gmx.net> <43CE7E68.1010908@ieee.org> Message-ID: <43CF8942.3050304@gmx.net> Travis Oliphant schrieb: > Sven Schreiber wrote: > >> Hi, >> I've spent a couple of weeks with scipy/numpy and the old-to-new >> transition; now that the transition is over (?) but some confusion is >> remaining (on my side) I feel the need to ask a basic question about >> matlab compatibility in terms of matrix (linear algebra) programming. >> >> Take "eye" and "identity" for example; given that "eye" supposedly >> exists to facilitate transiton from matlab to numpy/scipy (correct?), >> I expected eye to be/return a matrix. > > > Historical is the only reason. Numeric always returned an array for eye > not a matrix. We could return a matrix without difficulty especially if > we put an eye --> identity transition in convertcode.py > I see, thanks for the quick answer. So wouldn't it be a good idea to have all the specifically matrix-related stuff (afaics, things in numpy/lib/twodim_base.py) return matrices? It seems my question (or misunderstanding) has a broader scope: Coming from matrix languages, I'm glad about short notations like A.I or A*B representing standard matrix operations. Much easier than linalg.inverse(A) or matrixmultiply(A,B). However, many matrix functions (decompositions etc.) seem to return arrays instead of matrices, even if you feed them matrices (is this assumption correct?). So you have to use mat(returned-result) again and again to be able to do return-result.I afterwards, which seems clumsy. So what's the best strategy here? I'm aware that this is very much a basic/newbie question, so a pointer to the answer(s) would also be very welcome. If you think I should purchase the numpy-book to get the authorative answer, feel free to tell me so (but only if it really has the answer ;-) Thanks for your help, Sven From pjssilva at ime.usp.br Thu Jan 19 04:58:02 2006 From: pjssilva at ime.usp.br (Paulo J. S. Silva) Date: Thu Jan 19 04:58:02 2006 Subject: [Numpy-discussion] Updated benchmark Message-ID: <1137675462.2961.7.camel@localhost.localdomain> Hello, I am sending an update benchmark table below. I have noticed that in my benchmarck code for numpy I was using transpose(A) instead of A.transpose() which introduces an extra function call. A very big penalty for tiny matrices. Now the transpose operation doesn't look that bad in numpy when compared to matlab. There is still something on this table that I can't understand. Why are the index operations so expensive in domension 50? If you look closely you'll see that numpy is faster in dimension 5, much slower in dimension 50 (2.6 times slower) and then the gap decreases in dimension 500 (1.6 slower). Best, Paulo Tests x.T*y x*y.T A*x A*B A.T*x half 2in2 Dimension: 5 Array 0.82 0.22 0.18 0.26 0.33 0.90 1.07 Matrix 4.54 1.41 0.56 0.66 0.99 2.87 4.34 NumArr 2.82 0.66 0.61 0.71 7.54 6.61 10.56 Numeri 1.15 0.34 0.28 0.38 0.66 0.60 0.71 Matlab 1.60 0.39 0.31 0.33 0.35 1.13 0.91 Dimension: 50 Array 9.03 1.95 0.48 16.60 0.98 3.89 4.07 Matrix 45.94 3.55 0.90 17.30 1.67 5.93 7.42 NumArr 29.18 2.47 0.90 18.23 11.86 7.87 11.88 Numeri 11.91 2.13 0.59 17.21 9.27 0.93 3.02 Matlab 16.12 1.81 0.99 16.28 0.70 1.46 1.66 Dimension: 500 Array 1.09 8.05 1.77 151.32 1.89 3.63 3.86 Matrix 4.87 8.16 1.82 151.44 1.94 4.22 4.35 NumArr 3.12 8.19 1.80 151.66 19.83 3.94 5.30 Numeri 1.43 8.18 1.81 151.68 17.52 2.94 4.08 Matlab 1.91 5.07 1.89 150.52 1.84 2.27 2.74 From konrad.hinsen at laposte.net Thu Jan 19 06:02:04 2006 From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net) Date: Thu Jan 19 06:02:04 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: <43CD296C.7000700@astraw.com> References: <22F30CE0-7362-44A7-8DA1-79C6FE9E70D0@tamu.edu> <96D6BEBF-8A8D-4B69-9A6C-BC2A86BDA9B0@laposte.net> <89D76E1D-BE44-43FF-AD68-D0A8FACD795D@tamu.edu> <43CD296C.7000700@astraw.com> Message-ID: <79CFDD9A-7239-49EB-9722-4AB5EA36B87D@laposte.net> On 17.01.2006, at 18:29, Andrew Straw wrote: > Also, as for as I know, the only packages that install things into > /usr/include/python2.x/packagename are Numeric and numarray, so I > would > argue it's not "standard", (although it may have lots of history). Two of my own packages, ScientificPython and MMTK, do the same, so that's already four. In fact, I don't know any package that puts its include files elsewhere. Most packages have no include files at all of course. Another argument for this being the "standard" location is that it is in the include file search path when Python modules are built using distutils, and the path where Distutils puts them.. Yet another argument is that it's about the only location where include files can be placed without any necessity for client packages to know the habits of each of its dependencies. On 18.01.2006, at 02:18, Christopher Barker wrote: > As a Linux+OS-X user, I've run into a number of these Apple "Think > Different" issues, but I don't think creating an ugly, non-standard > system to accommodate the fact that Apple like to do things > strangely is the right solution. Apple doesn't maintain their > python distribution well, or really play well with the python > community, so many of us end up installing a new Python anyway. There isn't much difference between MacOS X and Linux: on both systems you need superuser rights to write to /usr/include. On 18.01.2006, at 03:22, Travis Oliphant wrote: > This is the important point in this discussion. There was a > reason it was changed. I remember opposing the change, > originally. However, it solved a real problem and only required > people to give up their arbitrary notions of where include files > *should* be. I don't see this as a real issue since you can get > where the include files are very easily. It's hardly arbitrary - see above. And I don't think anyone cares where include files are, as long as installing them via distutils will put them somewhere where another package installation using distutils will find them. > With that, however, I think it would be helpful for one of those > people who use .eggs to clarify why it solves their problem to have > the include files in their current location. Also what their problem exactly is - I don't remember seeing a description on this list. Perhaps we can come up with a different solution. Konrad. -- ------------------------------------------------------------------------ ------- Konrad Hinsen Laboratoire Leon Brillouin (CEA-CNRS), CEA Saclay, 91191 Gif-sur-Yvette Cedex, France Tel.: +33-1 69 08 79 25 Fax: +33-1 69 08 82 61 E-Mail: khinsen at cea.fr ------------------------------------------------------------------------ ------- From cjw at sympatico.ca Thu Jan 19 06:35:20 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Thu Jan 19 06:35:20 2006 Subject: [Numpy-discussion] An apparent bug in the Windows version of numpy 0.9.2 Message-ID: <43CFA2CB.3000806@sympatico.ca> Example: > PythonWin 2.4.2 (#67, Sep 28 2005, 12:41:11) [MSC v.1310 32 bit (Intel)] > on win32. > Portions Copyright 1994-2004 Mark Hammond (mhammond at skippinet.com.au) - > see 'Help/About PythonWin' for further copyright information. > >>> from numpy.core import umath > THIS CRASHES PYTHONWIN From svetosch at gmx.net Thu Jan 19 07:39:01 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Thu Jan 19 07:39:01 2006 Subject: [Numpy-discussion] An apparent bug in the Windows version of numpy 0.9.2 In-Reply-To: <43CFA2CB.3000806@sympatico.ca> References: <43CFA2CB.3000806@sympatico.ca> Message-ID: <43CFB299.1030505@gmx.net> Colin J. Williams schrieb: > Example: >> PythonWin 2.4.2 (#67, Sep 28 2005, 12:41:11) [MSC v.1310 32 bit (Intel)] >> on win32. >> Portions Copyright 1994-2004 Mark Hammond (mhammond at skippinet.com.au) - >> see 'Help/About PythonWin' for further copyright information. >> >>> from numpy.core import umath >> THIS CRASHES PYTHONWIN > not for me, so there must be some other ingredients as well -sven From strawman at astraw.com Thu Jan 19 09:33:01 2006 From: strawman at astraw.com (Andrew Straw) Date: Thu Jan 19 09:33:01 2006 Subject: [Numpy-discussion] include file location and .eggs Message-ID: <43CFCCF8.8000401@astraw.com> (We've had threads about this before. See http://www.scipy.org/documentation/mailman?fn=scipy-dev/2005-September/003238.html and http://sourceforge.net/mailarchive/forum.php?thread_id=9474853&forum_id=4890) There's been recent discussion regarding the location of the numpy include files. The pre-numpy way (Numeric and numarray) was to use distutils's install_headers, which places them in the long-familiar locations like /usr/local/include/python2.3/ (on *nix). This was a solution we all lived with for many years. Why then, is numpy rocking the boat and putting the include files somewhere else? For me, and I suspect for most others in favor of the new way, the answer boils down to using eggs with out-of-the box numpy. Eggs are the distributable form of Python package distributions made with setuptools (see http://peak.telecommunity.com/DevCenter/setuptools ). This isn't an email to convince everyone to use eggs, but I would like to say what issues eggs solve, what they make possible, and why the current include file location doesn't work with them. I _would_ like to convince people that supporting eggs is a valid thing for numpy to do, even if it means some change in the include file location and therefore some pain. (Although the amount of pain seems minimal to me.) I hope the developers of numpy-based extension modules, even if they never plan on using eggs, are willing to accomodate those of us who do. = Why can't eggs use traditional include locations? = Multiple version support. Using eggs, it is possible to have multiple versions of the same distribution, such as numpy, installed within a single Python installation. (I use "distribution" to mean collection of Python packages and modules created by a call to distutils' setup() and use that word because some distributions, such as matplotlib, include multiple top-level packages.) This means I can have a root-installed, site-wide installation of numpy on my unix machine for all users but I can easily play with a per-user installed numpy installation if I want to. With multiple versions of numpy installed on my system, the include files I need to use to compile an extension module are obviously going to depend on the right version of the headers. This simply isn't possible using the traditional include location, but requires the appropriate headers for the the version of numpy in use. I don't see any easier solution to this issue than what eggs already has implemented -- import the numpy package and ask it where its include files are (which will be in a sub-directory of it's top-level space within site-packages). = What issues do eggs solve? = I already mentioned multiple version support. eggs also offer dependency analysis and resolution. They also allow compatibility with easy_install, meaning automatic downloading and installation of dependencies (see http://peak.telecommunity.com/DevCenter/EasyInstall ). = What else do eggs make possible? = Eggs are distributed as a single ".egg" file, which is really just a specially formatted .zip file. If the package is "zip-safe", eggs are installed in site-packages as a single zip file. Even if they're not zip-safe, eggs are installed in site-packages in their own single directory. This makes package management much easier. Another of my favorite enhancements that setuptools brings to Python is runtime plugin support. If I have package A, which can optionally use plugins from module B, I can install A without installing B. When I install B, A will automatically be able to use B. If I write C to the same plugin interface that B uses, A can also use C, even if A never knew of the existance of C. There is lots more that setuptools enables. This is necessarily only a short list of what eggs can do and are useful for. Please see http://peak.telecommunity.com/DevCenter/setuptools for more information. I hope those with reservations about the change in include file locations now understand the reason for the change and, even more optimistically, I hope those folks are willing to accept the current state of affairs. I will attempt to answer any more questions on the issue. Cheers! Andrew From cjw at sympatico.ca Thu Jan 19 10:26:04 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Thu Jan 19 10:26:04 2006 Subject: [Numpy-discussion] An apparent bug in the Windows version of numpy 0.9.2 In-Reply-To: <43CFB299.1030505@gmx.net> References: <43CFA2CB.3000806@sympatico.ca> <43CFB299.1030505@gmx.net> Message-ID: <43CFD988.5090105@sympatico.ca> Sven Schreiber wrote: >Colin J. Williams schrieb: > > >>Example: >> >> >>>PythonWin 2.4.2 (#67, Sep 28 2005, 12:41:11) [MSC v.1310 32 bit (Intel)] >>>on win32. >>>Portions Copyright 1994-2004 Mark Hammond (mhammond at skippinet.com.au) - >>>see 'Help/About PythonWin' for further copyright information. >>> >>> from numpy.core import umath >>>THIS CRASHES PYTHONWIN >>> >>> > >not for me, so there must be some other ingredients as well >-sven > > > Thanks for this, I'm using Numpy 0.9.2. Does anyone have any suggestion as to what the the other ingredients might be. Similar problems arise with Boa-constructer and Python Scripter. The latter also reports a floating point error. Colin W. From oliphant.travis at ieee.org Thu Jan 19 11:26:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu Jan 19 11:26:02 2006 Subject: [Numpy-discussion] array vs. matrix for matlab-compatible stuff In-Reply-To: <43CF8942.3050304@gmx.net> References: <43CE631D.1000006@gmx.net> <43CE7E68.1010908@ieee.org> <43CF8942.3050304@gmx.net> Message-ID: <43CFE79B.9000509@ieee.org> Sven Schreiber wrote: >I see, thanks for the quick answer. So wouldn't it be a good idea to have all the specifically >matrix-related stuff (afaics, things in numpy/lib/twodim_base.py) return matrices? > >It seems my question (or misunderstanding) has a broader scope: Coming from matrix languages, I'm >glad about short notations like A.I or A*B representing standard matrix operations. Much easier than >linalg.inverse(A) or matrixmultiply(A,B). However, many matrix functions (decompositions etc.) seem >to return arrays instead of matrices, even if you feed them matrices (is this assumption correct?). >So you have to use mat(returned-result) again and again to be able to do return-result.I afterwards, >which seems clumsy. So what's the best strategy here? > > Yes, some of them do still return arrays. Matrices are longer lived in NumPy then the were in Numeric, for sure, but many functions still aren't friendly to matrices and convert all inputs to arrays before operation. Originally, I had the asarray(...) function not convert matrices by default, but this is too surprising because matrices change the '*' and '**' operators which could make your function not work. We should convert all the functions that don't handle matrices so they will. I'd like to see matrices survive longer than they do. There are some functions that try to do that -Travis From svetosch at gmx.net Thu Jan 19 11:47:01 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Thu Jan 19 11:47:01 2006 Subject: [Numpy-discussion] array vs. matrix for matlab-compatible stuff In-Reply-To: <43CF8942.3050304@gmx.net> References: <43CE631D.1000006@gmx.net> <43CE7E68.1010908@ieee.org> <43CF8942.3050304@gmx.net> Message-ID: <43CFECB1.20008@gmx.net> Sven Schreiber schrieb: > So you have to use mat(returned-result) again and again to be able to do return-result.I afterwards, Well I hope the fact that the above line is syntactically scandalous and a good reason to feel ashamed of myself is not the reason that nobody is answering my previous post .... ;-) But seriously, isn't there a preferred way to do matrix computations in numpy? Should I stick with arrays a la "linalg.inv(eye(3))", or does it make more sense to use matrices, as in "mat(eye(3)).I" ? Or, since both notations seem a little clumsy to me, isn't there an even better way that maybe even combines the best of both worlds? I definitely have the feeling that I'm missing something. _Any_ feedback is highly appreciated, thanks, Sven From oliphant.travis at ieee.org Thu Jan 19 12:12:04 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu Jan 19 12:12:04 2006 Subject: [Numpy-discussion] An apparent bug in the Windows version of numpy 0.9.2 In-Reply-To: <43CFD988.5090105@sympatico.ca> References: <43CFA2CB.3000806@sympatico.ca> <43CFB299.1030505@gmx.net> <43CFD988.5090105@sympatico.ca> Message-ID: <43CFE6A1.8060504@ieee.org> Colin J. Williams wrote: >> > Thanks for this, I'm using Numpy 0.9.2. Does anyone have any > suggestion as to what the the other ingredients might be. > Similar problems arise with Boa-constructer and Python Scripter. The > latter also reports a floating point error. I'm pretty sure it has to do with the fact that I'm using the mingw32 compiler to compile NumPy, while Pythonwin was compiled using something else. It might be related to Py_complex numbers which in Python are passed around as structures and my understanding is different compilers treat this differently. But, it might be something else as well. I've been surprised in the past by some of the quirks of compiling on windows. The safest thing to do is compile extensions with the same compiler used to compile Python (but I don't have that compiler, so I can't do it). -Travis From oliphant.travis at ieee.org Thu Jan 19 12:25:03 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu Jan 19 12:25:03 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: <200601191012.14204.faltet@carabos.com> References: <1137584647.19613.17.camel@localhost.localdomain> <43CE985E.9050406@astraw.com> <43CE9D1D.6090509@ee.byu.edu> <200601191012.14204.faltet@carabos.com> Message-ID: <43CFF596.2070007@ieee.org> Francesc Altet wrote: >http://oprofile.sourceforge.net > >It is a nice way to do profiling at C level on Linux machines. Running >the Paulo benchmarks through oprofile can surely bring some light. > > I ran the following code snippet (timed under a Timeit instance) through the oprofile profiler for both NumPy and Numeric, to look at indexing speeds. op = "b = A[::2,::2]; d = A[1:80,:]" This is what I found: overall 61242 53.9606 /usr/bin/python 17647 15.5488 /usr/lib/python2.4/site-packages/numpy/core/multiarray.so 15942 14.0466 /lib/tls/libc-2.3.3.so 7158 6.3069 /no-vmlinux 6995 6.1633 /usr/lib/python2.4/site-packages/Numeric/_numpy.so Showing that more time is spent in NumPy than in Numeric doing indexing... Here's the breakdown for NumPy samples % symbol name 2353 13.3337 PyArray_PyIntAsIntp # This is also slower --- called more often? 2060 11.6734 PyArray_MapIterNew # This calls fancy_indexing_check. 1980 11.2200 slice_GetIndices 1631 9.2424 parse_index 1149 6.5110 arraymapiter_dealloc # Interesting this is taking so long? 1142 6.4714 array_subscript 1121 6.3524 _IsAligned 1069 6.0577 array_dealloc 780 4.4200 fancy_indexing_check 684 3.8760 PyArray_NewFromDescr 627 3.5530 parse_subindex 538 3.0487 PyArray_DescrFromType 534 3.0260 array_subscript_nice 455 2.5783 _IsContiguous 370 2.0967 _IsFortranContiguous 334 1.8927 slice_coerce_index 294 1.6660 PyArray_UpdateFlags 234 1.3260 anonymous symbol from section .plt 161 0.9123 PyArray_Return 120 0.6800 array_alloc 2 0.0113 PyArray_Broadcast 2 0.0113 PyArray_IterNew 1 0.0057 LONG_setitem 1 0.0057 PyArray_EquivTypes 1 0.0057 PyArray_FromAny 1 0.0057 PyArray_FromStructInterface 1 0.0057 PyArray_IntpConverter 1 0.0057 PyArray_SetNumericOps 1 0.0057 initialize_numeric_types Here's the breakdown for Numeric: 1577 22.5447 slice_GetIndices 1155 16.5118 parse_index 912 13.0379 PyArray_FromDimsAndDataAndDescr 792 11.3224 array_subscript 675 9.6497 PyArray_IntegerAsInt 517 7.3910 parse_subindex 401 5.7327 array_dealloc 379 5.4182 slice_coerce_index 339 4.8463 array_subscript_nice 161 2.3016 anonymous symbol from section .plt 82 1.1723 PyArray_Return 5 0.0715 do_sliced_copy Anybody interested in optimization? -Travis From svetosch at gmx.net Thu Jan 19 12:54:21 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Thu Jan 19 12:54:21 2006 Subject: [Numpy-discussion] array vs. matrix for matlab-compatible stuff In-Reply-To: <43CFE79B.9000509@ieee.org> References: <43CE631D.1000006@gmx.net> <43CE7E68.1010908@ieee.org> <43CF8942.3050304@gmx.net> <43CFE79B.9000509@ieee.org> Message-ID: <43CFFC84.7050803@gmx.net> Travis Oliphant schrieb: > Yes, some of them do still return arrays. Matrices are longer lived in > NumPy then the were in Numeric, for sure, but many functions still > aren't friendly to matrices and convert all inputs to arrays before > operation. Originally, I had the asarray(...) function not convert > matrices by default, but this is too surprising because matrices change > the '*' and '**' operators which could make your function not work. > We should convert all the functions that don't handle matrices so they > will. I'd like to see matrices survive longer than they do. There are > some functions that try to do that > Thanks very much for this useful information! I totally agree, long live the matrices... In the ebook you are selling, are things like that (which functions preserve matrix-type) documented? Those kind of things would be a reason for me to buy it. cheers, Sven From ndarray at mac.com Thu Jan 19 12:57:12 2006 From: ndarray at mac.com (Sasha) Date: Thu Jan 19 12:57:12 2006 Subject: [Numpy-discussion] Interesting timing results In-Reply-To: <43CD7E3A.8070404@ieee.org> References: <43CD7E3A.8070404@ieee.org> Message-ID: On 1/17/06, Travis Oliphant wrote: > ... Currently all array scalar math goes through the > entire ufunc machinery. This is clearly sub-optimal. It is the reason > for the scalarmath module that I've started in the src directory. > Eventually, we should be able to have scalar math as fast as Python > scalars. I have implemented "nonzero", &, | and ^ for scalar bools. (See http://projects.scipy.org/scipy/numpy/changeset/1946). Here are the timings: Before: > python -m timeit -s "from numpy import bool_; x = bool_(0)" "x & x" 100000 loops, best of 3: 3.85 usec per loop Now: > python -m timeit -s "from numpy import bool_; x = bool_(0)" "x & x" 10000000 loops, best of 3: 0.174 usec per loop This is close to python bool: > python -m timeit -s "x = bool(0)" "x & x" 10000000 loops, best of 3: 0.142 usec per loop and faster than python int: > python -m timeit -s "from numpy import bool_; x = 0" "x & x" 10000000 loops, best of 3: 0.199 usec per loop But it is in fact all within the timing error now. I did not put it in the scalarmath module for two reasons: first, scalarmath is not hooked to numpy yet and second because C-API does not provide access to scalar bools yet. I have posted a proposal for C-API changes and did not hear any opposition (well, no support either), so I will implement that soon. There are a few issues with the new APIs that I proposed. First is a simple one: I proposed to expose boolean scalars as named constants to Python, the question is how to call them. 1. True_ and False_ 2. true and false The second issue is whether to add new numbers to _ARRAY_API or add a separate _ARRAY_SCALAR_API . I've only optimized scalar-scalar case in binary ops and fall back to old for everything else. I would like to optimize scalar-array and array-scalar cases as well, but I wonder if it is apropriate to make "(bool_(0) | x) is x" true when x is an array. Alternatively (bool_(0) | x) can become a new array that shares data with x. -- sasha From oliphant.travis at ieee.org Thu Jan 19 13:31:05 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu Jan 19 13:31:05 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: <43CFF596.2070007@ieee.org> References: <1137584647.19613.17.camel@localhost.localdomain> <43CE985E.9050406@astraw.com> <43CE9D1D.6090509@ee.byu.edu> <200601191012.14204.faltet@carabos.com> <43CFF596.2070007@ieee.org> Message-ID: <43D004FC.7090108@ieee.org> Travis Oliphant wrote: > > This is what I found: overall > > 61242 53.9606 /usr/bin/python > 17647 15.5488 > /usr/lib/python2.4/site-packages/numpy/core/multiarray.so > 15942 14.0466 /lib/tls/libc-2.3.3.so > 7158 6.3069 /no-vmlinux > 6995 6.1633 /usr/lib/python2.4/site-packages/Numeric/_numpy.so > > Showing that more time is spent in NumPy than in Numeric doing > indexing... > I optimized PyArray_PyIntAsIntp and changed things so that extra allocations are not done if the indexing is not fancy in the new svn of numpy and improved the performance of indexing a bit... The new numbers... samples % symbol name 551 18.4775 slice_GetIndices 482 16.1636 array_subscript 343 11.5023 parse_index 309 10.3622 PyArray_PyIntAsIntp 166 5.5667 fancy_indexing_check 164 5.4997 _IsAligned 140 4.6948 array_dealloc 134 4.4936 parse_subindex 133 4.4601 anonymous symbol from section .plt 127 4.2589 PyArray_NewFromDescr 121 4.0577 slice_coerce_index 79 2.6492 _IsFortranContiguous 67 2.2468 array_subscript_nice 56 1.8779 _IsContiguous 49 1.6432 array_alloc 31 1.0396 PyArray_UpdateFlags 29 0.9725 PyArray_Return 1 0.0335 array_itemsize_get For reference, the Numeric numbers are: samples % symbol name 375 25.9695 slice_GetIndices 255 17.6593 parse_index 198 13.7119 PyArray_IntegerAsInt 135 9.3490 array_subscript 130 9.0028 PyArray_FromDimsAndDataAndDescr 117 8.1025 parse_subindex 81 5.6094 slice_coerce_index 55 3.8089 array_subscript_nice 44 3.0471 array_dealloc 40 2.7701 anonymous symbol from section .plt 13 0.9003 PyArray_Return 1 0.0693 do_sliced_copy These look a bit better and the results show that simple indexing seems to be only slowed down by the fancy indexing check... -Travis From oliphant.travis at ieee.org Thu Jan 19 13:38:11 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu Jan 19 13:38:11 2006 Subject: [Numpy-discussion] Interesting timing results In-Reply-To: References: <43CD7E3A.8070404@ieee.org> Message-ID: <43D00699.10001@ieee.org> Sasha wrote: >On 1/17/06, Travis Oliphant wrote: > > >>... Currently all array scalar math goes through the >>entire ufunc machinery. This is clearly sub-optimal. It is the reason >>for the scalarmath module that I've started in the src directory. >>Eventually, we should be able to have scalar math as fast as Python >>scalars. >> >> > >I have implemented "nonzero", &, | and ^ for scalar bools. (See >http://projects.scipy.org/scipy/numpy/changeset/1946). Here are the >timings: > >Before: > > >>python -m timeit -s "from numpy import bool_; x = bool_(0)" "x & x" >> >> >100000 loops, best of 3: 3.85 usec per loop > >Now: > > >>python -m timeit -s "from numpy import bool_; x = bool_(0)" "x & x" >> >> >10000000 loops, best of 3: 0.174 usec per loop > >This is close to python bool: > > >>python -m timeit -s "x = bool(0)" "x & x" >> >> >10000000 loops, best of 3: 0.142 usec per loop > >and faster than python int: > > >>python -m timeit -s "from numpy import bool_; x = 0" "x & x" >> >> >10000000 loops, best of 3: 0.199 usec per loop > >But it is in fact all within the timing error now. > >I did not put it in the scalarmath module for two reasons: first, >scalarmath is not hooked to numpy yet and second because C-API does >not provide access to scalar bools yet. I have posted a proposal for >C-API changes and did not hear any opposition (well, no support >either), so I will implement that soon. > >There are a few issues with the new APIs that I proposed. First is a >simple one: I proposed to expose boolean scalars as named constants to >Python, the question is >how to call them. > >1. True_ and False_ > > +1 >2. true and false > > -1 >The second issue is whether to add new numbers to _ARRAY_API or add a >separate _ARRAY_SCALAR_API . > > No separate _SCALAR_API.... >I've only optimized scalar-scalar case in binary ops and fall back to >old for everything else. I would like to optimize scalar-array and >array-scalar cases as well, but I wonder if it is apropriate to make >"(bool_(0) | x) is x" true when x is an array. Alternatively >(bool_(0) | x) can become a new array that shares data with x. > > Other operations with scalars return a new array. The fact that this would be different would probably be a bad thing in the end. So, I vote for returning a new copy of the data... -Travis From oliphant.travis at ieee.org Thu Jan 19 15:21:11 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu Jan 19 15:21:11 2006 Subject: [Numpy-discussion] Time for a new release of NumPy Message-ID: <43D01EC7.3040203@ieee.org> I'd like to make a release of NumPy over the weekend. Please submit bugs to the list before Saturday night (GMT -7) NumPy 0.9.2 was based on SVN version 1829 and we are over 100 checkins from that point, including the significant change to the .dtype attribute.... -Travis From info at barclays.co.uk Thu Jan 19 15:30:06 2006 From: info at barclays.co.uk (info at barclays.co.uk) Date: Thu Jan 19 15:30:06 2006 Subject: [Numpy-discussion] Barclays IBank security measure ID 3342025 Message-ID: <20060119222017.15399.qmail@ns32968.ovh.net> An HTML attachment was scrubbed... URL: From ndarray at mac.com Thu Jan 19 16:53:03 2006 From: ndarray at mac.com (Sasha) Date: Thu Jan 19 16:53:03 2006 Subject: [Numpy-discussion] Time for a new release of NumPy In-Reply-To: <43D01EC7.3040203@ieee.org> References: <43D01EC7.3040203@ieee.org> Message-ID: On 1/19/06, Travis Oliphant wrote: > > ... Please submit > bugs to the list before Saturday night (GMT -7) Trac has bug report pages: http://projects.scipy.org/scipy/numpy/report/6 but they don't appear to be in use. Here is my bug report: http://projects.scipy.org/scipy/numpy/ticket/3 -- sasha From mark at mitre.org Thu Jan 19 19:22:05 2006 From: mark at mitre.org (Mark Heslep) Date: Thu Jan 19 19:22:05 2006 Subject: [Numpy-discussion] Speed performance on array constant set Message-ID: <43D0573E.4090301@mitre.org> Im doing some work with the OpenCv* project. Im using swig typemaps to convert the Cv data structures to numarray which works well. Id like to restrict Cv use to what its strengths: complicated vision processing algorithms like optical flow. For the case of simple Cv data manipulations, I'd rather use NumPy functions & methods but was surprised at the performance comparison. - A simple scalar constant fill with cvSet. 'im' here is a wrapped Cv image data structure. > python -m timeit -s "import opencv.cv as cv; im = > cv.cvCreateImage(cv.cvSize(1000,1000), 8, 1)" "cv.cvSet( im, > cv.cvRealScalar( 7 ) )" > 100 loops, best of 3: 2.58 msec per loop - If I try the equivalent with NumPy > python -m timeit -s "import numarray as na; a = na.zeros((1000,1000) > )" "a[:,:] = 7" > 10 loops, best of 3: 45.1 msec per loop A >10x hit. Am I using the preferred / optimal NumPy method here? I scanned the earlier Scalar posts but thought that was boolean type only issue. Mark *OpenCv is an computer vision library, open source, and is sponsored by Intel. It includes many video capable functions for application to motion analysis, tracking and the like. From oliphant.travis at ieee.org Thu Jan 19 19:49:01 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu Jan 19 19:49:01 2006 Subject: [Numpy-discussion] Speed performance on array constant set In-Reply-To: <43D0573E.4090301@mitre.org> References: <43D0573E.4090301@mitre.org> Message-ID: <43D05D99.1060200@ieee.org> Mark Heslep wrote: > Im doing some work with the OpenCv* project. Im using swig typemaps > to convert the Cv data structures to numarray which works well. Id > like to restrict Cv use to what its strengths: complicated vision > processing algorithms like optical flow. For the case of simple Cv > data manipulations, I'd rather use NumPy functions & methods but was > surprised at the performance comparison. > - A simple scalar constant fill with cvSet. 'im' here is a wrapped Cv > image data structure. > >> python -m timeit -s "import opencv.cv as cv; im = >> cv.cvCreateImage(cv.cvSize(1000,1000), 8, 1)" "cv.cvSet( im, >> cv.cvRealScalar( 7 ) )" >> 100 loops, best of 3: 2.58 msec per loop > > > - If I try the equivalent with NumPy > >> python -m timeit -s "import numarray as na; a = na.zeros((1000,1000) >> )" "a[:,:] = 7" > > >> 10 loops, best of 3: 45.1 msec per loop > > This is actually a bit surprising that opencv can create and fill so quickly. Perhaps they are using optimized SSE functions for the Intel platform, or something? -Travis From ndarray at mac.com Thu Jan 19 19:53:02 2006 From: ndarray at mac.com (Sasha) Date: Thu Jan 19 19:53:02 2006 Subject: [Numpy-discussion] Speed performance on array constant set In-Reply-To: References: <43D0573E.4090301@mitre.org> Message-ID: On 1/19/06, Mark Heslep wrote: > A >10x hit. Am I using the preferred / optimal NumPy method here? I > scanned the earlier Scalar posts but thought that was boolean type > only issue. Try a.fill(7). On my system: > python -m timeit -s "import numpy as na; a = na.zeros((1000,1000))" "a[:,:] = 7" 100 loops, best of 3: 17.1 msec per loop > python -m timeit -s "import numpy as na; a = na.zeros((1000,1000))" "a.fill(7)" 100 loops, best of 3: 6.28 msec per loop You can also use empty instead of zeros, but that wont affect the timings that you presented. -- sasha From mark at mitre.org Thu Jan 19 20:24:01 2006 From: mark at mitre.org (Mark Heslep) Date: Thu Jan 19 20:24:01 2006 Subject: [Numpy-discussion] Speed performance on array constant set In-Reply-To: <43D05D99.1060200@ieee.org> References: <43D0573E.4090301@mitre.org> <43D05D99.1060200@ieee.org> Message-ID: <43D065AF.3080307@mitre.org> Travis Oliphant wrote: > This is actually a bit surprising that opencv can create and fill so > quickly. Perhaps they are using optimized SSE functions for the > Intel platform, or something? > -Travis > Ah, sorry, Im an unintentional fraud. Yes I have Intel's optimization library IPP turned on and had forgotten about it. So one more time: With IPP on as before. UseOptimized = # of Cv functions available w/ IPP > python -m timeit -s "import opencv.cv as cv; print > cv.cvUseOptimized(1); im =cv.cvCreateImage(cv.cvSize(1000,1000), 8, > 1)" "cv.cvSet( im, cv.cvRealScalar( 7 ) )" > 305 > 305 > 305 > 305 > 305 > 100 loops, best of 3: 2.24 msec per loop And without: > python -m timeit -s "import opencv.cv as cv; print > cv.cvUseOptimized(0); im =cv.cvCreateImage(cv.cvSize(1000,1000), 8, > 1)" "cv.cvSet( im, cv.cvRealScalar( 7 ) )" > 0 > 0 > 0 > 0 > 0 > 100 loops, best of 3: 6.94 msec per loop So IPP gives me 3X, which leads me to ask about plans for IPP / SSE for NumPy, no offense intended to non Intel users. I believe I recall some post that auto code generation in NumArray was the road block? Mark From mark at mitre.org Thu Jan 19 20:38:00 2006 From: mark at mitre.org (Mark Heslep) Date: Thu Jan 19 20:38:00 2006 Subject: [Numpy-discussion] Speed performance on array constant set In-Reply-To: References: <43D0573E.4090301@mitre.org> Message-ID: <43D06908.9060306@mitre.org> Sasha wrote: >Try a.fill(7). > >On my system: > > >>python -m timeit -s "import numpy as na; a = na.zeros((1000,1000))" "a[:,:] = 7" >> >> >100 loops, best of 3: 17.1 msec per loop > > > >>python -m timeit -s "import numpy as na; a = na.zeros((1000,1000))" "a.fill(7)" >> >> >100 loops, best of 3: 6.28 msec per loop > >You can also use empty instead of zeros, but that wont affect the >timings that you presented. > >-- sasha > > > Thank you. Any equivalent to 'fill' for NumArray or Numeric? Well, I suppose its time to move to Sci/NumPy. Im on Fedora 4, any pointers to a NumPy package? Mean time Ill pull from SVN and try again to build. I have the updated gcc Fortran. Mark From mark at mitre.org Thu Jan 19 20:45:01 2006 From: mark at mitre.org (Mark Heslep) Date: Thu Jan 19 20:45:01 2006 Subject: [Numpy-discussion] Speed performance on array constant set In-Reply-To: <43D06908.9060306@mitre.org> References: <43D0573E.4090301@mitre.org> <43D06908.9060306@mitre.org> Message-ID: <43D06AA2.9080005@mitre.org> Scratch that - got it from SF. > Im on Fedora 4, any pointers to a NumPy package? Mean time Ill pull > from SVN and try again to build. I have the updated gcc Fortran. > > Mark > > From Norbert.Nemec.list at gmx.de Fri Jan 20 00:22:02 2006 From: Norbert.Nemec.list at gmx.de (Norbert Nemec) Date: Fri Jan 20 00:22:02 2006 Subject: [Numpy-discussion] Bug? numpy.matrix.sum() Message-ID: <43D09D6E.5070907@gmx.de> Is it just me who finds this confusing? There is some bug there, but I cannot even say what it is supposed to behave like... --------------------------------- In [1]: import numpy In [2]: numpy.__version__ Out[2]: '0.9.2' In [3]: a = numpy.matrix(numpy.eye(2)) In [4]: a.sum() Out[4]: matrix([[1, 0, 0, 1]]) In [5]: a.A.sum() Out[5]: 2 In [6]: sum(a) Out[6]: matrix([[1, 1]]) In [7]: sum(a.A) Out[7]: array([1, 1]) --------------------------------- From Norbert.Nemec.list at gmx.de Fri Jan 20 00:31:01 2006 From: Norbert.Nemec.list at gmx.de (Norbert Nemec) Date: Fri Jan 20 00:31:01 2006 Subject: [Numpy-discussion] Matrix/Vector norm? Message-ID: <43D09F8F.6080702@gmx.de> Hi there, im missing a norm function in NumPy. Writing one by hand is simple, but doing so both general and efficient is a lot more tricky. Ideas? Norbert From arnd.baecker at web.de Fri Jan 20 00:53:01 2006 From: arnd.baecker at web.de (Arnd Baecker) Date: Fri Jan 20 00:53:01 2006 Subject: [Numpy-discussion] Time for a new release of NumPy In-Reply-To: <43D01EC7.3040203@ieee.org> References: <43D01EC7.3040203@ieee.org> Message-ID: On Thu, 19 Jan 2006, Travis Oliphant wrote: > I'd like to make a release of NumPy over the weekend. Please submit > bugs to the list before Saturday night (GMT -7) > > NumPy 0.9.2 was based on SVN version 1829 and we are over 100 checkins > from that point, including the significant change to the .dtype > attribute.... I would like to get numpy working with intels compiler icc - However, as long as no one can tell me where to get information on my scipy distutils question/problem http://www.scipy.org/mailinglists/mailman?fn=scipy-dev/2006-January/005138.html I can't help further ;-) (Personally I don't need an icc compiled one at the moment, I can always install gcc, but ...) Best, Arnd P.S.: I just saw that on http://www.scipy.org/ there is still reference to new core/ new scipy etc: """ This site does not reflect the rapid growth of scipy that has taken place while a new core scipy array object has been built to replace Numeric. This link provides more information on where to get the new scipy core. A version of scipy that works on newcore is available for anyonymous check out from a subversion repostitory at http://svn.scipy.org/svn/scipy/trunk. File downloads are available on this site or from sourceforge here. """ It would be great if this could be corrected (BTW, what about a new round of name changing - it was so much fun ...;-) From august at attorney.com Fri Jan 20 01:40:01 2006 From: august at attorney.com (=?windows-1251?B?NyAtIDgg9OXi8ODr/yAyMDA2IOPu5OA=?=) Date: Fri Jan 20 01:40:01 2006 Subject: [Numpy-discussion] =?windows-1251?B?z9DAytLI18XRysjJIMzA0MrF0sjNww==?= Message-ID: ?????????? ???-?????????? ? ??????????? ????????????? ?????????? ? ???????, ??????, ??????????????? ???????? ? ?????????????? ?????????? ??????? ??????? ? ????? ?????????? ????????: ???????????? ????????? ????????? ????????? ??????????? ?? ?????????? ? ????? ??????????? ????????? ???????????????? ???? ? ?????????? ???????? 7 - 8 ??????? 2006 ???? ?? ??????? ?? ???????? ??????????? ???????????? ????????? ?????????? ???????? ? ?????? ?????????????? ? ??????? ?? ???????????????, ????? ??????????? ???? ?????????? ??????-????????? ?? ???? ???????? ???????????? ???????????? ? ???????? ?? ??????????. ???? ? ????? ?????? ?????????? ? ??????????????? ????????? ???????? ??????????? ????????? ?????????? ? ????????????????????? ????????. ?????? ? ?????????. ?????????? ????? ??????????. ???????? ???????????????? ?????? ?????????? ? ???????? ? ?? ?????????. ???????? ????? ? ???????????????? ???????? ?? ??? ??????? ????? ??????????. ????????? ?????????????? ? ???????????? ??????? ????????? ??????????. ????? ?????????????? ???????? ? ?????????????. ???? ???????????? ? ?? ????????. ???????? ?????? "??????????" ?????. ?????????? ? ???????? ???????? CRM. ?????????????? ???????? ? ????????? ????????????????????? ???????? ??????????????? ???????? ??????/??????. ????????? ? ?????????? ????? ????? ???????/?????. ????? ?????????? ?1 - ???????????? ???????. ??????????? ??????????? ???????? ??????????????? ? ????????????? ??? ????????????? ????????. ??????????????? ? ??????????? ????? ??????/???????????? ???? ? ???????? ????????/??????. ??????? ?????????? ????, ??????? ??????, ????? ?????? ? ????????????. ??????????? ????????/?????? ?? ????? ? ??????????? ??????????? ??????????????? ? ????????????? ? ?????????? ???????????????????????? ????????. ??????????? ???????????? ?????????????? ?????? ?????????? ? ????????. ????????????? ???????????? ? ?? ???? ? ????????? ????????????????????? ???????? ??????? ??? ?????????? ????????? ????????????????? ???????????? ? ??????/??????. ??? ???????? ??????????? ?????. ??????? ?????? ?????????? ????????? ????????. ????? ?????????? ?2 - ???????????? ???????. ?????????? ????????? ???????????? ???????????? ? ???????? ????????? ?? ???????????. ?????????? ????? ????????? ????????. ?????????????? ?????? ??? ??????????: CD ???? ? ?????? ???????? ??????????????? ?????????? ????? ? ????????????? ?????????, ????? ?????? ???????? "?????????: ????????? ??????-????????". ????? ???????? - ??????????? ???????? ?????????????? ????????. ???????????????? ??????????? ? ????? ????????????? ??????????, ???????????, PR, Direct-marketing, Sale promotions ? ????????????? ???????? ??????-????????? ??? ??????????? ? ???? ????????? ???? ????????????, ????? ???? ????????? ?????????????? ????????????? ? ????????? ????????, ???????? ?? ??????????? ?????????. ????? ???? "???????????? ?????????. ??????? ??????? ????????????", "???????????? ????????? 2. ??????????? ?? ?????????" ? "???????? ? ???????? ??????????", ?????????????? ?????? ? ?????????? ? ??????? ????????????? ???????????????? ????????, ???? ?????????????? ????????? ?????????? ?????????. ????? ??????? ??????? ? ???????? ??? ?????? ???????????, ????????? ?? ?????????: (495) 980-65-36, 980-65-39 ?????? ????????? ?? ?????? ???????? ? ????????????? ????????, ? ?????? ????????? ?????? ??????? ? ?????? ?????????? ??????. -------------- next part -------------- An HTML attachment was scrubbed... URL: From MAILER-DAEMON at ibspublic.ibs.fr Fri Jan 20 03:30:10 2006 From: MAILER-DAEMON at ibspublic.ibs.fr (Mail Delivery Subsystem) Date: Fri Jan 20 03:30:10 2006 Subject: [Numpy-discussion] Returned mail: see transcript for details Message-ID: <200601201129.k0KBTNa7005336@ibspublic.ibs.fr> The original message was received at Fri, 20 Jan 2006 12:29:12 +0100 from adsl196-62-32-217-196.adsl196-10.iam.net.ma [196.217.32.62] ----- The following addresses had permanent fatal errors ----- (reason: 550 5.1.1 ... User unknown) ----- Transcript of session follows ----- ... while talking to ibsprive.ibs.fr.: >>> RCPT To: <<< 550 5.1.1 ... User unknown 550 5.1.1 ... User unknown -------------- next part -------------- An embedded message was scrubbed... From: unknown sender Subject: no subject Date: no date Size: 38 URL: From numpy-discussion at lists.sourceforge.net Fri Jan 20 06:29:09 2006 From: numpy-discussion at lists.sourceforge.net (numpy-discussion at lists.sourceforge.net) Date: Fri, 20 Jan 2006 12:29:09 +0100 Subject: hello Message-ID: <200601201129.k0KBTBa7005322@ibspublic.ibs.fr> An embedded and charset-unspecified text was scrubbed... Name: warning1.txt URL: -------------- next part -------------- The message contains Unicode characters and has been sent as a binary attachment. From aisaac at american.edu Fri Jan 20 07:03:03 2006 From: aisaac at american.edu (Alan G Isaac) Date: Fri Jan 20 07:03:03 2006 Subject: [Numpy-discussion] Bug? numpy.matrix.sum() In-Reply-To: <43D09D6E.5070907@gmx.de> References: <43D09D6E.5070907@gmx.de> Message-ID: On Fri, 20 Jan 2006, Norbert Nemec apparently wrote: > In [1]: import numpy > In [2]: numpy.__version__ > Out[2]: '0.9.2' > In [3]: a = numpy.matrix(numpy.eye(2)) > In [4]: a.sum() > Out[4]: matrix([[1, 0, 0, 1]]) > In [5]: a.A.sum() > Out[5]: 2 > In [6]: sum(a) > Out[6]: matrix([[1, 1]]) > In [7]: sum(a.A) > Out[7]: array([1, 1]) Only [4] looks wrong. However it works "right" if you provide a numerical axis argument. (I put "right" in quotes because it is not clear to me that a row vector is expected as the result when summing columns.) The default is axis=None, which is not working as expected for matrices. Cheers, Alan Isaac From cjw at sympatico.ca Fri Jan 20 07:40:02 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Fri Jan 20 07:40:02 2006 Subject: [Numpy-discussion] Interesting timing results In-Reply-To: <43D00699.10001@ieee.org> References: <43CD7E3A.8070404@ieee.org> <43D00699.10001@ieee.org> Message-ID: <43D10404.5030107@sympatico.ca> Travis Oliphant wrote: > Sasha wrote: > >> On 1/17/06, Travis Oliphant wrote: >> >> >>> ... Currently all array scalar math goes >>> through the >>> entire ufunc machinery. This is clearly sub-optimal. It is the >>> reason >>> for the scalarmath module that I've started in the src directory. >>> Eventually, we should be able to have scalar math as fast as Python >>> scalars. >>> >> >> >> I have implemented "nonzero", &, | and ^ for scalar bools. (See >> http://projects.scipy.org/scipy/numpy/changeset/1946). Here are the >> timings: >> >> Before: >> >> >>> python -m timeit -s "from numpy import bool_; x = bool_(0)" "x & x" >>> >> >> 100000 loops, best of 3: 3.85 usec per loop >> >> Now: >> >> >>> python -m timeit -s "from numpy import bool_; x = bool_(0)" "x & x" >>> >> >> 10000000 loops, best of 3: 0.174 usec per loop >> >> This is close to python bool: >> >> >>> python -m timeit -s "x = bool(0)" "x & x" >>> >> >> 10000000 loops, best of 3: 0.142 usec per loop >> >> and faster than python int: >> >> >>> python -m timeit -s "from numpy import bool_; x = 0" "x & x" >>> >> >> 10000000 loops, best of 3: 0.199 usec per loop >> >> But it is in fact all within the timing error now. >> >> I did not put it in the scalarmath module for two reasons: first, >> scalarmath is not hooked to numpy yet and second because C-API does >> not provide access to scalar bools yet. I have posted a proposal for >> C-API changes and did not hear any opposition (well, no support >> either), so I will implement that soon. >> >> There are a few issues with the new APIs that I proposed. First is a >> simple one: I proposed to expose boolean scalars as named constants to >> Python, the question is >> how to call them. >> >> 1. True_ and False_ >> >> > +1 Why not True and False? >>> type(True) >>> Colin W. > >> 2. true and false >> >> > -1 > >> The second issue is whether to add new numbers to _ARRAY_API or add a >> separate _ARRAY_SCALAR_API . >> >> > No separate _SCALAR_API.... > >> I've only optimized scalar-scalar case in binary ops and fall back to >> old for everything else. I would like to optimize scalar-array and >> array-scalar cases as well, but I wonder if it is apropriate to make >> "(bool_(0) | x) is x" true when x is an array. Alternatively >> (bool_(0) | x) can become a new array that shares data with x. >> >> > Other operations with scalars return a new array. The fact that this > would be different would probably be a bad thing in the end. So, I > vote for returning a new copy of the data... > > -Travis > > > > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log > files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From cjw at sympatico.ca Fri Jan 20 07:43:06 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Fri Jan 20 07:43:06 2006 Subject: [Numpy-discussion] Time for a new release of NumPy In-Reply-To: <43D01EC7.3040203@ieee.org> References: <43D01EC7.3040203@ieee.org> Message-ID: <43D104EE.1070208@sympatico.ca> Travis Oliphant wrote: > > I'd like to make a release of NumPy over the weekend. Please submit > bugs to the list before Saturday night (GMT -7) > > NumPy 0.9.2 was based on SVN version 1829 and we are over 100 checkins > from that point, including the significant change to the .dtype > attribute.... Will there be some note with the release explaining this, or do we have to browse the discussion to get the detail? Colin W. From faltet at carabos.com Fri Jan 20 07:51:07 2006 From: faltet at carabos.com (Francesc Altet) Date: Fri Jan 20 07:51:07 2006 Subject: [Numpy-discussion] Time for a new release of NumPy In-Reply-To: <43D104EE.1070208@sympatico.ca> References: <43D01EC7.3040203@ieee.org> <43D104EE.1070208@sympatico.ca> Message-ID: <200601201650.41924.faltet@carabos.com> A Divendres 20 Gener 2006 16:42, Colin J. Williams va escriure: > Travis Oliphant wrote: > > I'd like to make a release of NumPy over the weekend. Please submit > > bugs to the list before Saturday night (GMT -7) > > > > NumPy 0.9.2 was based on SVN version 1829 and we are over 100 checkins > > from that point, including the significant change to the .dtype > > attribute.... > > Will there be some note with the release explaining this, or do we have > to browse the discussion to get the detail? Here you have the original explanation from Travis posted in this list a couple of days ago: ----------- There was some cruft left over from the change to making data-type descriptors real Python objects. This left lots of .dtype related attributes on the array object --- too many as Francesc Altet graciously pointed out. In the latest SVN, I've cleaned things up (thanks to a nice patch from Francesc to get it started). Basically, there is now only one attribute on the array object dealing with the data-type (arr.dtype). This attribute returns the data-type descriptor object for the array. This object itself has the attributes .char, .str, and .type (among others). I think this will lead to less confusion long term. The cruft was due to the fact that my understanding of the data-type descriptor came in December while seriously looking at records module. This will have some backward-compatibility issues (we are still pre-1.0 and early enough that I hope this is not too difficult to deal with). The compatibility to numpy-0.9.2 issues I can see are: 1) Replacing attributes that are now gone: .dtypechar --> .dtype.char .dtypestr --> .dtype.str .dtypedescr --> .dtype 2) Changing old .dtype -> .dtype.type This is only necessary if you were using a.dtype as a *typeobject* as in issubclass(a.dtype, ) If you were using .dtype as a parameter to dtype= then that usage will still work great (in fact a little faster) because now .dtype returns a "descriptor object" 3) The dtypedescr constructor is now called dtype. This change should have gone into the 0.9.2 release, but things got too hectic with all the name changes. I will quickly release 0.9.4 with these changes unless I hear strong disagreements within the next few days. -Travis P.S. SciPy SVN has been updated and fixed with the changes. Numeric compatibility now implies that .typecode() --> .dtype.char although if .typecode() was used as an argument to a function, then .dtype will very likely work. -Travis -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From oliphant.travis at ieee.org Fri Jan 20 08:42:01 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri Jan 20 08:42:01 2006 Subject: [Numpy-discussion] Speed performance on array constant set In-Reply-To: <43D065AF.3080307@mitre.org> References: <43D0573E.4090301@mitre.org> <43D05D99.1060200@ieee.org> <43D065AF.3080307@mitre.org> Message-ID: <43D06AEA.50707@ieee.org> Mark Heslep wrote: > Travis Oliphant wrote: > >> This is actually a bit surprising that opencv can create and fill so >> quickly. Perhaps they are using optimized SSE functions for the >> Intel platform, or something? >> -Travis >> > Ah, sorry, Im an unintentional fraud. Yes I have Intel's optimization > library IPP turned on and had forgotten about it. So one more time: > > With IPP on as before. UseOptimized = # of Cv functions available w/ > IPP > >> python -m timeit -s "import opencv.cv as cv; print >> cv.cvUseOptimized(1); im =cv.cvCreateImage(cv.cvSize(1000,1000), 8, >> 1)" "cv.cvSet( im, cv.cvRealScalar( 7 ) )" >> 305 >> 305 >> 305 >> 305 >> 305 >> 100 loops, best of 3: 2.24 msec per loop > > > And without: > >> python -m timeit -s "import opencv.cv as cv; print >> cv.cvUseOptimized(0); im =cv.cvCreateImage(cv.cvSize(1000,1000), 8, >> 1)" "cv.cvSet( im, cv.cvRealScalar( 7 ) )" >> 0 >> 0 >> 0 >> 0 >> 0 >> 100 loops, best of 3: 6.94 msec per loop > > > So IPP gives me 3X, which leads me to ask about plans for IPP / SSE > for NumPy, no offense intended to non Intel users. I believe I recall > some post that auto code generation in NumArray was the road block? There was some talk of using liboil for this (similar to what _dotblas does). There could definitely be some gains. I don't see any road block other than time and effort.... With my own tests of a ctypes-wrapped function that just mallocs memory and fills it, I put numpy at about 3x slower than that function. The scalar fill function of numpy could definitely be made faster. Right now, it's still pretty generic. -Travis From oliphant.travis at ieee.org Fri Jan 20 08:42:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri Jan 20 08:42:02 2006 Subject: [Numpy-discussion] Speed performance on array constant set In-Reply-To: <43D0573E.4090301@mitre.org> References: <43D0573E.4090301@mitre.org> Message-ID: <43D05BA0.7000700@ieee.org> Mark Heslep wrote: > Im doing some work with the OpenCv* project. Im using swig typemaps > to convert the Cv data structures to numarray which works well. Id > like to restrict Cv use to what its strengths: complicated vision > processing algorithms like optical flow. For the case of simple Cv > data manipulations, I'd rather use NumPy functions & methods but was > surprised at the performance comparison. > - A simple scalar constant fill with cvSet. 'im' here is a wrapped Cv > image data structure. > >> python -m timeit -s "import opencv.cv as cv; im = >> cv.cvCreateImage(cv.cvSize(1000,1000), 8, 1)" "cv.cvSet( im, >> cv.cvRealScalar( 7 ) )" >> 100 loops, best of 3: 2.58 msec per loop > > > - If I try the equivalent with NumPy > >> python -m timeit -s "import numarray as na; a = na.zeros((1000,1000) >> )" "a[:,:] = 7" > > >> 10 loops, best of 3: 45.1 msec per loop > > > A >10x hit. Am I using the preferred / optimal NumPy method here? I > scanned the earlier Scalar posts but thought that was boolean type > only issue. > First of all, try using NumPy instead of Numarray: import numpy as na Second: use math (i.e. a += 7) Third: There are specialized fill routines .fill() in numpy and a simliar routine in Numarray that can be faster then indexing. Best, -Travis From oliphant.travis at ieee.org Fri Jan 20 08:42:03 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri Jan 20 08:42:03 2006 Subject: [Numpy-discussion] Matrix/Vector norm? In-Reply-To: <43D09F8F.6080702@gmx.de> References: <43D09F8F.6080702@gmx.de> Message-ID: <43D0A290.8080600@ieee.org> Norbert Nemec wrote: >Hi there, > >im missing a norm function in NumPy. Writing one by hand is simple, but >doing so both general and efficient is a lot more tricky. > >Ideas? > > There's one in scipy (i'm not sure if it's the best of breed, but it's a starting point). Look in scipy.linalg.norm -Travis From oliphant.travis at ieee.org Fri Jan 20 08:42:04 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri Jan 20 08:42:04 2006 Subject: [Numpy-discussion] Bug? numpy.matrix.sum() In-Reply-To: <43D09D6E.5070907@gmx.de> References: <43D09D6E.5070907@gmx.de> Message-ID: <43D0A247.7000003@ieee.org> Norbert Nemec wrote: >Is it just me who finds this confusing? There is some bug there, but I >cannot even say what it is supposed to behave like... > >--------------------------------- >In [1]: import numpy > >In [2]: numpy.__version__ >Out[2]: '0.9.2' > >In [3]: a = numpy.matrix(numpy.eye(2)) > >In [4]: a.sum() >Out[4]: matrix([[1, 0, 0, 1]]) > > > Bug... Fixed in SVN. -Travis From Chris.Barker at noaa.gov Fri Jan 20 09:36:09 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri Jan 20 09:36:09 2006 Subject: [Numpy-discussion] Speed performance on array constant set In-Reply-To: <43D065AF.3080307@mitre.org> References: <43D0573E.4090301@mitre.org> <43D05D99.1060200@ieee.org> <43D065AF.3080307@mitre.org> Message-ID: <43D11F90.30602@noaa.gov> Sasha wrote: >>python -m timeit -s "import numpy as na; a = na.zeros((1000,1000))" "a.fill(7)" or use a += 7: $ python2.4 -m timeit -s "import numpy as na; a = na.zeros((1000,1000))" "a.fill(7)" 100 loops, best of 3: 6.95 msec per loop $ python2.4 -m timeit -s "import numpy as na; a = na.zeros((1000,1000))" "a += 7" 100 loops, best of 3: 3.24 msec per loop A factor of 2 speedup for me. I don't know why fill is slower. > So IPP gives me 3X, which leads me to ask about plans for IPP / SSE for > NumPy, no offense intended to non Intel users. I"ve wondered about this as as well, though not necessarily IPP / SEE. It seems that BLAS should provide some optimizations that could be used outside of the strictly linear algebra functions, like element-wise multiplication, array copying, etc. However, I haven't looked into it, and I suppose it would make for a lot of special-case code. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From cookedm at physics.mcmaster.ca Fri Jan 20 09:48:03 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Fri Jan 20 09:48:03 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: <43CEC67B.1090600@colorado.edu> (Fernando Perez's message of "Wed, 18 Jan 2006 15:51:39 -0700") References: <1137584647.19613.17.camel@localhost.localdomain> <43CE85AD.1090904@ieee.org> <1137610937.19613.65.camel@localhost.localdomain> <43CE985E.9050406@astraw.com> <43CE9B65.5040803@ee.byu.edu> <43CEBA67.8000104@colorado.edu> <43CEC67B.1090600@colorado.edu> Message-ID: Fernando Perez writes: > David M. Cooke wrote: > >> I've done a little bit of work along these lines. I have a module I >> call vector3 [*] which has 2- and 3-dimensional immutable vectors, >> using either ints or doubles. It's as fast as I could make it, while >> keeping it all written in Pyrex. I find it very convenient for >> anything vector-related. Konrad Hinsen has something similiar in the >> development version of his ScientificPython package. >> [*] http://arbutus.mcmaster.ca/dmc/software/vector3.html >> Also, I've also done some playing around with a n-dimensional vector >> type (restricted to doubles). My best attempts make it ~4-5x faster >> than numpy (and 2x faster than Numeric) for vectors of dimension 10 >> on simple ops like + and *, 2x faster than numpy for dimension 1000, >> and approaching 1x as you make the vectors larger. Indexing is about >> 3x faster than numpy, and 1.4x faster than Numeric. So that gives I >> think some idea of the maximum speed-up possible. >> I think the speedups mostly come from the utter lack of any >> polymorphism: it handles vectors of doubles only, and only as >> contiguous vectors (no strides). > > This is excellent, thanks for the pointer. I can see uses for vectors > (still 1-d, no strides, etc) with more than 3 elements, and perhaps > fixed-size (no reshaping, no striding) 2-d arrays (matrices), but this > looks like a good starting point. Sandbox material? Well, I'd be pleased to donate vector3 to scipy as a sandbox (although I think it's very useful as a standalone package for others with no need of full scipy. The general vector stuff is still quite rough, and at this point is really only meant as a study on how fast I can make something like that run :-) -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From faltet at carabos.com Fri Jan 20 09:59:04 2006 From: faltet at carabos.com (Francesc Altet) Date: Fri Jan 20 09:59:04 2006 Subject: [Numpy-discussion] Segmentation fault when dealing with nested records Message-ID: <200601201858.34777.faltet@carabos.com> Hi, I'm starting to write a series of torture test for numpy and (nested) heterogeneous arrays. I've found a first problem that causes a segfault: In [7]: dtype=[('x', '0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From Ed.Schofield at ftw.at Fri Jan 20 10:17:02 2006 From: Ed.Schofield at ftw.at (Schofield, Ed) Date: Fri Jan 20 10:17:02 2006 Subject: [Numpy-discussion] Re: array vs. matrix for matlab-compatible stuff Message-ID: (Apologies if this is a duplicate. My mail isn't getting through ...) Travis Oliphant wrote: >> Sven Schreiber wrote: >> >> I see, thanks for the quick answer. So wouldn't it be a good idea to >> have all the specifically >> matrix-related stuff (afaics, things in numpy/lib/twodim_base.py) >> return matrices? >> >> It seems my question (or misunderstanding) has a broader scope: >> Coming from matrix languages, I'm >> glad about short notations like A.I or A*B representing standard >> matrix operations. Much easier than >> linalg.inverse(A) or matrixmultiply(A,B). However, many matrix >> functions (decompositions etc.) seem >> to return arrays instead of matrices, even if you feed them matrices >> (is this assumption correct?). >> So you have to use mat(returned-result) again and again to be able to >> do return-result.I afterwards, >> which seems clumsy. So what's the best strategy here? > > Yes, some of them do still return arrays. Matrices are longer lived > in NumPy then the were in Numeric, for sure, but many functions still > aren't friendly to matrices and convert all inputs to arrays before > operation. Originally, I had the asarray(...) function not convert > matrices by default, but this is too surprising because matrices > change the '*' and '**' operators which could make your function not > work. > We should convert all the functions that don't handle matrices so they > will. I'd like to see matrices survive longer than they do. There > are some functions that try to do that I agree, and I'm willing to help with this. We should also think about handling SciPy's spmatrix objects properly. In this case we don't want NumPy to have any dependency on a particular format for SciPy's spmatrix objects -- but we could design and publish a simple interface for sparse matrix objects (from SciPy or anywhere else) to conform to for basic NumPy compatibility. How should we go about this? Let's take linalg.solve_linear_equations() as an example. We could use isinstance(x, matrix) for dense matrix objects, because they're part of NumPy, then use asarray(x), process them as arrays, and wrap them up with asmatrix(x) at the end. But with sparse matrices this wouldn't work. What if we specify instead that objects passed to functions like solve_linear_equations() need to provide two members: x.toarray(), allowing x to represent itself as an array x.fromarray(a), creating another object of the same type as x out of the given array a We could choose other names, of course, maybe asarray() or whatever else. But this would simplify the code for all these functions while allowing us to support other array-like objects without needing to know any more about them. I'm also +1 on eye() returning a matrix :) -- Ed -------------- next part -------------- An HTML attachment was scrubbed... URL: From dd55 at cornell.edu Fri Jan 20 10:25:05 2006 From: dd55 at cornell.edu (Darren Dale) Date: Fri Jan 20 10:25:05 2006 Subject: [Numpy-discussion] numpy license file Message-ID: <200601201324.34947.dd55@cornell.edu> I also apologize if this is a duplicate, my mail isnt getting through either. I'm looking for the license file in svn numpy, but cant find it. a grep of "license" on the numpy directory turned up several references to terms of the SciPy license, and to the missing LICENSE.txt. Some folks are working on a gentoo ebuild for numpy/scipy in gentoo's bugzilla, and raised questions about numpy's license. Darren From ghall at research.dfci.harvard.edu Fri Jan 20 11:01:08 2006 From: ghall at research.dfci.harvard.edu (Giles Hall) Date: Fri Jan 20 11:01:08 2006 Subject: [Numpy-discussion] Bugfix for arraybase.h Message-ID: <43D13347.3050309@research.dfci.harvard.edu> Hi, This simple python code snippit used to core dump on my modern AMD 64 Linux servers: from numarray import * a = zeros( 10, Float32 ) b = transpose(a) dot( b,a ) This crash was caused by a corrupted "libnumarray_API". The pointer was initially assigned the correct value, but becomes corrupted almost immediately after its initial runtime binding. On my systems, line 725 of _dotblas.c would clobber the "libnumarray_API" pointer while initializing the "dotFunctions" vector table. the problem is "dotFunctions", an array of function pointers, is sized statically by "PyArray_NTYPES", an enumerated constant that reflects the value of "tMaxType" defined in arraybase.h like this: typedef enum { tAny, tBool, tInt8, tUInt8, tInt16, tUInt16, tInt32, tUInt32, tInt64, tUInt64, tFloat32, tFloat64, tComplex32, tComplex64, tObject, /* placeholder... does nothing */ tDefault = tFloat64, #if LP64 tLong = tInt64, #else tLong = tInt32, #endif tMaxType } NumarrayType; The problem is that "tMaxType" is not assigned 17, like we would expect, since it's at the end of the list. On my machine, "tMaxType" is assigned the value 9! The problem is the explicit assignment of "tLong" and "tDefault". Explicitly assigned constants in an enumerated list will modify the running count. Since the macro "LP64" is defined, "tLong" is assigned value of "tInt64", which is 8, and "tMaxType" is assigned the next value, 9. This means the vector table "dotFunctions" is shorter then we intend, and overflowing this table can corrupt important data. The fix is simple, just define tMaxType before the explicitly assigned aliases: typedef enum { tAny, tBool, tInt8, tUInt8, tInt16, tUInt16, tInt32, tUInt32, tInt64, tUInt64, tFloat32, tFloat64, tComplex32, tComplex64, tObject, /* placeholder... does nothing */ tMaxType, tDefault = tFloat64, #if LP64 tLong = tInt64, #else tLong = tInt32, #endif } NumarrayType; From faltet at carabos.com Fri Jan 20 11:06:06 2006 From: faltet at carabos.com (Francesc Altet) Date: Fri Jan 20 11:06:06 2006 Subject: [Numpy-discussion] Time for a new release of NumPy In-Reply-To: <43D104EE.1070208@sympatico.ca> References: <43D01EC7.3040203@ieee.org> <43D104EE.1070208@sympatico.ca> Message-ID: <200601202005.12065.faltet@carabos.com> A Divendres 20 Gener 2006 16:42, Colin J. Williams va escriure: > Travis Oliphant wrote: > > I'd like to make a release of NumPy over the weekend. Please submit > > bugs to the list before Saturday night (GMT -7) > > > > NumPy 0.9.2 was based on SVN version 1829 and we are over 100 checkins > > from that point, including the significant change to the .dtype > > attribute.... > > Will there be some note with the release explaining this, or do we have > to browse the discussion to get the detail? Colin told me that my previous message got truncated. I'm sending now a link to the original message from Travis: http://aspn.activestate.com/ASPN/Mail/Message/numpy-discussion/2979474 Hope this time it would look well. -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From ivazquez at ivazquez.net Fri Jan 20 11:51:05 2006 From: ivazquez at ivazquez.net (Ignacio Vazquez-Abrams) Date: Fri Jan 20 11:51:05 2006 Subject: [Numpy-discussion] numpy license file In-Reply-To: <200601201324.34947.dd55@cornell.edu> References: <200601201324.34947.dd55@cornell.edu> Message-ID: <1137786625.910.3.camel@ignacio.lan> On Fri, 2006-01-20 at 13:24 -0500, Darren Dale wrote: > I also apologize if this is a duplicate, my mail isnt getting through either. > > I'm looking for the license file in svn numpy, but cant find it. a grep of > "license" on the numpy directory turned up several references to terms of the > SciPy license, and to the missing LICENSE.txt. > > Some folks are working on a gentoo ebuild for numpy/scipy in gentoo's > bugzilla, and raised questions about numpy's license. The Numpy code itself is BSD but the included f2py is LGPL, so it seems that the entire package is LGPL. -- Ignacio Vazquez-Abrams http://fedora.ivazquez.net/ gpg --keyserver hkp://subkeys.pgp.net --recv-key 38028b72 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From oliphant at ee.byu.edu Fri Jan 20 14:05:02 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri Jan 20 14:05:02 2006 Subject: [Fwd: Re: [Fwd: Re: [Numpy-discussion] numpy license file]] Message-ID: <43D15D60.3020300@ee.byu.edu> -------------- next part -------------- An embedded message was scrubbed... From: unknown sender Subject: no subject Date: no date Size: 38 URL: From pearu at scipy.org Fri Jan 20 15:47:28 2006 From: pearu at scipy.org (Pearu Peterson) Date: Fri, 20 Jan 2006 14:47:28 -0600 (CST) Subject: [Fwd: Re: [Numpy-discussion] numpy license file] In-Reply-To: <43D1562C.5070103@ee.byu.edu> References: <43D1562C.5070103@ee.byu.edu> Message-ID: On Fri, 20 Jan 2006, Travis Oliphant wrote: > Can we release f2py in numpy under a BSD license so as not to encumber all of > numpy? Yes. Pearu --------------070907070601070307030400-- From dd55 at cornell.edu Fri Jan 20 14:08:30 2006 From: dd55 at cornell.edu (Darren Dale) Date: Fri Jan 20 14:08:30 2006 Subject: [Numpy-discussion] numpy license file In-Reply-To: <1137786625.910.3.camel@ignacio.lan> References: <200601201324.34947.dd55@cornell.edu> <1137786625.910.3.camel@ignacio.lan> Message-ID: <200601201702.28089.dd55@cornell.edu> On Friday 20 January 2006 14:50, Ignacio Vazquez-Abrams wrote: > On Fri, 2006-01-20 at 13:24 -0500, Darren Dale wrote: > > I also apologize if this is a duplicate, my mail isnt getting through > > either. > > > > I'm looking for the license file in svn numpy, but cant find it. a grep > > of "license" on the numpy directory turned up several references to terms > > of the SciPy license, and to the missing LICENSE.txt. > > > > Some folks are working on a gentoo ebuild for numpy/scipy in gentoo's > > bugzilla, and raised questions about numpy's license. > > The Numpy code itself is BSD but the included f2py is LGPL, so it seems > that the entire package is LGPL. I don't want this to seem disrespectful, but could I get a second opinion on this? From oliphant at ee.byu.edu Fri Jan 20 14:17:04 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri Jan 20 14:17:04 2006 Subject: [Numpy-discussion] numpy license file In-Reply-To: <1137786625.910.3.camel@ignacio.lan> References: <200601201324.34947.dd55@cornell.edu> <1137786625.910.3.camel@ignacio.lan> Message-ID: <43D1613F.6050308@ee.byu.edu> Ignacio Vazquez-Abrams wrote: >On Fri, 2006-01-20 at 13:24 -0500, Darren Dale wrote: > > >>I also apologize if this is a duplicate, my mail isnt getting through either. >> >>I'm looking for the license file in svn numpy, but cant find it. a grep of >>"license" on the numpy directory turned up several references to terms of the >>SciPy license, and to the missing LICENSE.txt. >> >>Some folks are working on a gentoo ebuild for numpy/scipy in gentoo's >>bugzilla, and raised questions about numpy's license. >> >> > >The Numpy code itself is BSD but the included f2py is LGPL, so it seems >that the entire package is LGPL. > > Pearu has agreed to release the numpy.f2py under the same license as NumPy. Thus, there should be no problem regardless of the veracity of this implication. -Travis From oliphant at ee.byu.edu Fri Jan 20 14:28:04 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri Jan 20 14:28:04 2006 Subject: [Numpy-discussion] Time for a new release of NumPy In-Reply-To: References: <43D01EC7.3040203@ieee.org> <43D036AC.7030503@ieee.org> <43D04E3C.4020108@ieee.org> Message-ID: <43D163E1.4050509@ee.byu.edu> Sasha wrote: >On 1/19/06, Travis Oliphant wrote: > > >>... >>I don't think this is right. zero-rank float arrays use Python's >>floating-point str or repr function to produce the string, so whatever >>Python is doing will be used. >> >> > >Well, it looks like they use repr for str: > > > >>>>from numpy import * >>>>len(str(1/3.)) >>>> >>>> >14 > > >>>>len(str(array(1/3.))) >>>> >>>> >19 > > >>>>len(repr(1/3.)) >>>> >>>> >19 > > > O.K. That is a bug. And it is now fixed (an oversight...) -Travis From rowen at cesmail.net Fri Jan 20 14:44:05 2006 From: rowen at cesmail.net (Russell E. Owen) Date: Fri Jan 20 14:44:05 2006 Subject: [Numpy-discussion] Re: Efficient way to handle None->nan? References: Message-ID: In article , Sasha wrote: > >>> from numpy.core.ma import masked values > >>> from numpy import nan > >>> masked values([1.0,None,2.0],None).filled(nan).astype(float) > array([ 1. , nan, 2. ]) Neat! Any idea if that's likely to keep working in future versions? It doesn't work in numarray.ma.masked_values and in general it seems like a lot of numpy and numarray raise exception when they get a list that contains None. -- Russell From bsouthey at gmail.com Fri Jan 20 14:48:03 2006 From: bsouthey at gmail.com (Bruce Southey) Date: Fri Jan 20 14:48:03 2006 Subject: [Numpy-discussion] numpy license file In-Reply-To: <200601201702.28089.dd55@cornell.edu> References: <200601201324.34947.dd55@cornell.edu> <1137786625.910.3.camel@ignacio.lan> <200601201702.28089.dd55@cornell.edu> Message-ID: Hi, > The Numpy code itself is BSD but the included f2py is LGPL, so it seems > that the entire package is LGPL. This is not really correct but I am not a lawyer or license expert nor a Numpy developer. Just because f2py is (or rather was) LGPL it does not automatically make all the Numpy code LGPL. If a piece of code is a derivative of f2py then it must be released as LGPL or with a more restrictive license that does not violate the LGPL but you can not release it under a less restrictive license such as BSD. But any piece of code that is not a derivative of f2py (such as being completely independent of f2py) can have it own license (that must not violate other licenses). Exactly what is a derivative is a real mess so seek legal advice. Regards Bruce On 1/20/06, Darren Dale wrote: > > On Friday 20 January 2006 14:50, Ignacio Vazquez-Abrams wrote: > > On Fri, 2006-01-20 at 13:24 -0500, Darren Dale wrote: > > > I also apologize if this is a duplicate, my mail isnt getting through > > > either. > > > > > > I'm looking for the license file in svn numpy, but cant find it. a > grep > > > of "license" on the numpy directory turned up several references to > terms > > > of the SciPy license, and to the missing LICENSE.txt. > > > > > > Some folks are working on a gentoo ebuild for numpy/scipy in gentoo's > > > bugzilla, and raised questions about numpy's license. > > > > The Numpy code itself is BSD but the included f2py is LGPL, so it seems > > that the entire package is LGPL. > > I don't want this to seem disrespectful, but could I get a second opinion > on > this? > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log > files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at ee.byu.edu Fri Jan 20 15:04:02 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri Jan 20 15:04:02 2006 Subject: [Numpy-discussion] Efficient way to handle None->nan? In-Reply-To: References: Message-ID: <43D16C31.3060209@ee.byu.edu> Russell E. Owen wrote: >We're getting numeric data from a (MySQL) database. We'd like to use >numarray or NumPy on the resulting data, but some values may be None. Is >there a fast, efficient way to replace None with NaN? I'd hate to use a >list comprehension on each data tuple before turning it into an array, >but I haven't thought of anything else. > > Current numpy SVN allows array([1.0,None,2.0],dtype=float) array([ 1. , nan, 2. ]) -Travis From dd55 at cornell.edu Fri Jan 20 15:30:03 2006 From: dd55 at cornell.edu (Darren Dale) Date: Fri Jan 20 15:30:03 2006 Subject: [Numpy-discussion] numpy license info in svn? Message-ID: <200601200901.44802.dd55@cornell.edu> I'm looking for the license file in svn numpy, but cant find it. a grep of "license" on the numpy directory turned up several references to terms of the SciPy license, and to the missing LICENSE.txt. Some folks are working on a gentoo ebuild for numpy/scipy in gentoo's bugzilla, and raised questions about numpy's license. Darren From dd55 at cornell.edu Fri Jan 20 18:41:04 2006 From: dd55 at cornell.edu (Darren Dale) Date: Fri Jan 20 18:41:04 2006 Subject: [Numpy-discussion] fromstring feature request Message-ID: <200601202140.26335.dd55@cornell.edu> Would be possible to add a feature to numpy's fromstring function, to make it consistent with fromfile? fromfile will return an array from a binary file or an ascii file, but fromstring will only work with binary data. I would put fromstring to good use if it dealt with ascii data. Darren From apices at etang.com Fri Jan 20 19:21:03 2006 From: apices at etang.com (=?windows-1251?B?MzAg/+3i4PD/IC0gMiD05eLw4Ov/IDIwMDYg4+7k4A==?=) Date: Fri Jan 20 19:21:03 2006 Subject: [Numpy-discussion] =?windows-1251?B?z9DAytLI18XRysjJIMrT0NEgz84gws3T0tDFzc3FzNMgys7N0tDOy94gyCDBztDc?= =?windows-1251?B?wcUg0SDKztDPztDA0sjCzdvMIMzO2MXNzcjXxdHSws7M?= Message-ID: ???????????? ???? ?? ??????????? ???????? ? ?????? ? ????????????? ?????????????? 30 ?????? - 2 ??????? 2006 ???? ???? ????????????? ?????????, ???????? ?????????? ???????????, ???????? ??????????? ? ?????? ??????? ???????????? ????????? ???? ?????????????? ?????????? ? ????????, ????? ?? ???????????? ???????? ???????? ?????????? ????????. ??? ??????? ????? ????????????? ? ?????????? ?????????? ??????????? ?? ???????????? ??????? ????????? ??? ???? ? ????? ? ????????? ???????. ?????? ?? ? ??? ???????? ????? ???????? ?????????? ?????? ??????????? ????????, ??????? ???????? ??????????? ??????? ???????????? ?????, ????????? ??????????????????? ? ???????????????? ?????? ? ?????? ? ????????? ????????????? ????????? ?????????? ???????, ???????? ??? ??????????????, ?????????? ? ????????????????? ?? ??????? ???????????. ??????? ????????? ?? ????????????? ?????????? ??????????? ???????, ?????? ??????? ?????????? ? ??????????? ????????, ? ????? ????????????? ?????????? ?????. ?????????? ??????????, ??????????? ? ???????????? ?????????? ? ????????? ??????????????? ??????????????????? ????????-??????????: ???????????? ???? ?? ??????????? ???????? ? ?????? ? ????????????? ?????????????? 30 ?????? - 2 ??????? 2006 ???? ???? ?????: ??????????? ??????????? ??????? ? ?????????? ??????????? ??????????? ??????????? ? ????????? ? ??????????? ???????? ??????????? ????????, ??????? ???????? ?????????? ????? ?????????????? ??????? ????? ???????? ?????? ??????????????? ???????????? ? ?????????????? ?????????. ????????? ?? ??? ?????????????? ????????????, ????? ???????? ???????? ???????? ???????? ???????? ? ??????? ??????????? ? ????? ??????? ???????? ? ??????????? ????????????? ????? ? ???????? ???????? ?????????? ???????? ?? ???????? ?????????? ? ?????????? ????????. ?? ???????? ????? ???????? ??????-????? "????????????? ??????????? ??????????? ???????? ??? ?????? ?? ????????????? ???????? ?????????????????? ? ?????????????? ???????", ? ????? ??????????? ? ?????? ?????????? ???????? ?? ???????????? ???????? ????????. ???????: (?95) 980-65-l6, 980-65-?9 -------------- next part -------------- An HTML attachment was scrubbed... URL: From borreguero at gmail.com Sat Jan 21 09:16:06 2006 From: borreguero at gmail.com (Jose Borreguero) Date: Sat Jan 21 09:16:06 2006 Subject: [Numpy-discussion] HELP! eig and eigvals hang the python session In-Reply-To: <7cced4ed0601210820q75545136y@mail.gmail.com> References: <7cced4ed0601210820q75545136y@mail.gmail.com> Message-ID: <7cced4ed0601210915v46d38cdev@mail.gmail.com> Hi all, I don't understand what's going on. Here's my python session: $ python >>> from numpy.random import rand >>> a=rand(3,3) >>> from numpy.linalg import det,eig >>> det(a) 0.070796819514446802 >>> eig(a) and the process freezes here (at least 18minutes from now). I checked with 'top' and python is using all CPU. Any ideas, please? jose From aisaac at american.edu Sat Jan 21 09:52:08 2006 From: aisaac at american.edu (Alan G Isaac) Date: Sat Jan 21 09:52:08 2006 Subject: [Numpy-discussion] HELP! eig and eigvals hang the python session In-Reply-To: <7cced4ed0601210915v46d38cdev@mail.gmail.com> References: <7cced4ed0601210820q75545136y@mail.gmail.com><7cced4ed0601210915v46d38cdev@mail.gmail.com> Message-ID: On Sat, 21 Jan 2006, Jose Borreguero apparently wrote: >>>> eig(a) > and the process freezes here (at least 18minutes from now). I checked with > 'top' and python is using all CPU. No problem here. fwiw, Alan Isaac Python 2.4.1 (#65, Mar 30 2005, 09:13:57) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy as N >>> a=N.rand(3,3) >>> from numpy.linalg import det,eig >>> det(a) -0.1313102666029379 >>> eig(a) (array([ 1.65540878, -0.16786818, 0.47252527]), array([[-0.55573211, -0.8712454 8, 0.25278726], [-0.52611295, 0.37246933, -0.5095001 ], [-0.64371343, 0.31968409, 0.82250122]])) >>> N.__version__ '0.9.2' >>> From ndarray at mac.com Sat Jan 21 09:54:02 2006 From: ndarray at mac.com (Sasha) Date: Sat Jan 21 09:54:02 2006 Subject: [Numpy-discussion] HELP! eig and eigvals hang the python session In-Reply-To: <7cced4ed0601210915v46d38cdev@mail.gmail.com> References: <7cced4ed0601210820q75545136y@mail.gmail.com> <7cced4ed0601210915v46d38cdev@mail.gmail.com> Message-ID: You may be hitting a known problem in lapack's _geev functions that rely on computations not being performed with extra precision. See https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=138683 -- sasha On 1/21/06, Jose Borreguero wrote: > Hi all, > I don't understand what's going on. Here's my python session: > $ python > >>> from numpy.random import rand > >>> a=rand(3,3) > >>> from numpy.linalg import det,eig > >>> det(a) > 0.070796819514446802 > >>> eig(a) > and the process freezes here (at least 18minutes from now). I checked with > 'top' and python is using all CPU. > Any ideas, please? > jose > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://sel.as-us.falkag.net/sel?cmdlnk&kid3432&bid#0486&dat1642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From cjw at sympatico.ca Sat Jan 21 16:12:02 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Sat Jan 21 16:12:02 2006 Subject: [Numpy-discussion] Scalar array - a trap for the unwary Message-ID: <43D2CDA8.9030700@sympatico.ca> In the script below, b is presented as an integer, either with str or repr, but it is an array and needs further processing to treat it as in integer. I much prefer the numarray treatment, particularly in a matrix context. I suggest that, if it looks like an integer, b[1] should return a Python scalar. Script: # tScalar.py Scalar vs rank 0 import numpy.core.multiarray as _mu a= _mu.array([1, 2, 4]) print 'a:', a, 'repr(a):', repr(a), 'shape:', a.shape, 'dtype:', a.dtype b= a[1] print 'b:', b, 'repr(b):', repr(b), 'shape:', b.shape, 'dtype:', b.dtype Result: a: array([1, 2, 4], 'l') repr(a): array([1, 2, 4], 'l') shape: (3,) dtype: b: 2 repr(b): 2 shape: () dtype: Colin W. From ndarray at mac.com Sat Jan 21 16:58:00 2006 From: ndarray at mac.com (Sasha) Date: Sat Jan 21 16:58:00 2006 Subject: [Numpy-discussion] Scalar array - a trap for the unwary In-Reply-To: <43D2CDA8.9030700@sympatico.ca> References: <43D2CDA8.9030700@sympatico.ca> Message-ID: On 1/21/06, Colin J. Williams wrote: > ... > I much prefer the numarray treatment, particularly in a matrix context. > I suggest that, if it looks like an integer, b[1] should return a Python > scalar. In NumPy b[1] IS an integer: >>> from numpy import * >>> a = array([1,2,3]) >>> b = a[1] >>> isinstance(b, int) True It IS NOT rank-0 ndarray: >>> isinstance(b, ndarray) False Surely, the whole truth is that numpy scalars are instances of classes derived from Python scalars. What is the problem that numpy is causing you? If b was a python scalar, b.dtype would just raise an AttributeError. -- sasha From matthew.brett at gmail.com Sat Jan 21 17:14:16 2006 From: matthew.brett at gmail.com (Matthew Brett) Date: Sat Jan 21 17:14:16 2006 Subject: [Numpy-discussion] array vs. matrix for matlab-compatible stuff In-Reply-To: <43CFFC84.7050803@gmx.net> References: <43CE631D.1000006@gmx.net> <43CE7E68.1010908@ieee.org> <43CF8942.3050304@gmx.net> <43CFE79B.9000509@ieee.org> <43CFFC84.7050803@gmx.net> Message-ID: <1e2af89e0601211713s7790ac01m100a32e7ccbfcc55@mail.gmail.com> Hi, > Thanks very much for this useful information! I totally agree, long live the matrices... > In the ebook you are selling, are things like that (which functions preserve matrix-type) > documented? Those kind of things would be a reason for me to buy it. A bit off-topic I suppose, but can I second several previous suggestions that we all buy a copy if we can? First, it's very useful, and second to support Travis in his Herculean efforts. Best, Matthew From oliphant.travis at ieee.org Sat Jan 21 23:49:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sat Jan 21 23:49:02 2006 Subject: [Numpy-discussion] numpy-0.9.4 won't build with Python 2.3 Message-ID: <43D338B1.9080802@ieee.org> Due to last-minute use of PyOS_ascii_strtod(), the numpy-0.9.4 release won't build properly with Python 2.3. The fix is easy and I made it to release a windows binary for numpy-0.9.4 on Python 2.3. I'll put a patch up soon. If you need support for Python 2.3, get numpy out of SVN or use the patch. -Travis From NadavH at VisionSense.com Sun Jan 22 03:17:05 2006 From: NadavH at VisionSense.com (Nadav Horesh) Date: Sun Jan 22 03:17:05 2006 Subject: [Numpy-discussion] Running benchmark on AMD64: Strange results Message-ID: <43D36592.8020906@VisionSense.com> I ran benchmark.py from the "Numpy x Matlab: some synthetic benchmarks" thread (the first version), on a amd64 machine (dual-core athlon 4400+, 1GB RAM) running either win64 or gentoo linux (dual boot). And got a strange results in the dimension 500 matrix multiplication benchmark: On windows: Tests x.T*y x*y.T A*x A*B A.T*x half 2in2 Dimension: 5 Array 0.7547 0.2491 0.1939 0.2688 0.8351 0.6387 0.8398 Matrix 6.4227 1.3667 0.7070 0.8192 1.4211 2.2491 3.3690 NumArr 1.7175 3.0416 0.3539 3.0406 5.0468 4.3359 7.0414 Numeri 0.8756 0.2918 0.2639 0.3418 0.6445 0.5888 0.6953 Dimension: 50 Array 7.8431 1.5494 0.7105 12.9425 6.4525 2.1420 2.4210 Matrix 69.4642 2.8870 1.2589 13.6162 7.1357 3.7315 4.9553 NumArr 17.6435 4.6267 0.9372 33.3093 6.6129 4.7710 7.5562 Numeri 9.3534 1.6228 1.0502 12.9967 5.4694 0.9972 2.0816 Dimension: 500 Array 1.1935 6.7593 1.6090 129.6738 12.8805 1.6672 2.0153 <-- Matrix 12.6622 6.8386 1.6488 129.8621 12.6773 1.9591 2.1887 <-- NumArr 2.2546 7.5078 1.0158 545.9313 7.6133 0.6549 1.4395 <-- Numeri 1.3390 6.8766 1.2113 133.2397 12.7436 0.5740 1.7577 <-- On linux: Dimension: 5 Array 0.7500 0.1600 0.1600 0.1800 0.7200 0.5900 0.7900 Matrix 5.9600 1.1800 0.5900 0.6300 1.2700 2.1500 3.3200 NumArr 1.9400 0.4500 0.4300 0.4600 5.4100 4.6500 7.4600 Numeri 0.9100 0.2200 0.2100 0.2400 0.5200 0.4400 0.5600 Dimension: 50 Array 7.8700 1.5900 0.6900 25.9800 7.9200 2.3700 2.6300 Matrix 67.1700 2.8100 1.1500 26.5000 8.4700 3.9500 5.2000 NumArr 20.0700 1.6200 0.9500 10.4400 7.8400 5.3500 8.2600 Numeri 9.4300 1.7700 0.7400 26.1000 9.6500 0.7700 3.1500 Dimension: 500 Array 1.2200 4.6700 1.0700 1188.6400 15.3500 2.5000 2.8100 <-- Matrix 13.1100 4.7800 1.1000 1052.9200 15.3500 2.6000 2.9100 <-- NumArr 2.4700 7.5800 1.1900 76.7100 13.0600 1.8400 3.6600 <-- Numeri 1.3700 7.3900 1.2400 1068.0200 14.6800 1.3600 3.5200 <-- Numeric/numpy matrix multiplication is about 8 fold slower on linux, and about 7 fold fast with numarray. Configuration: On win64 I used the provided binaries for numarray1.5, numpy0.92 and scipy 4.4 (compiled for P4+sse2) On linux I used numarray 1.5.1 (from cvs) numpy0.92, and scipy0.44, all compiled from source for 64 bits, and linked with ATLAS 3.7.11 (linking with ACML provided roughly the same figures). Any idea were this huge performance factor came from? Nadav. From pjssilva at ime.usp.br Sun Jan 22 03:49:03 2006 From: pjssilva at ime.usp.br (Paulo Jose da Silva e Silva) Date: Sun Jan 22 03:49:03 2006 Subject: [Numpy-discussion] Running benchmark on AMD64: Strange results In-Reply-To: <43D36592.8020906@VisionSense.com> References: <43D36592.8020906@VisionSense.com> Message-ID: <1137930520.7397.6.camel@localhost.localdomain> Em Dom, 2006-01-22 ?s 12:59 +0200, Nadav Horesh escreveu: > Dimension: 500 > Array 1.2200 4.6700 1.0700 1188.6400 15.3500 2.5000 2.8100 <-- > Matrix 13.1100 4.7800 1.1000 1052.9200 15.3500 2.6000 2.9100 <-- > NumArr 2.4700 7.5800 1.1900 76.7100 13.0600 1.8400 3.6600 <-- > Numeri 1.3700 7.3900 1.2400 1068.0200 14.6800 1.3600 3.5200 <-- > I bet that Numeric and Numpy are not really accessing the ATLAS libraries or the ATLAS package you are using are using in Linux are not optimized for you architecture. The same test in my system gives: Dimension: 500 Array 1.09 8.05 1.77 151.32 1.89 3.63 3.86 Matrix 4.87 8.16 1.82 151.44 1.94 4.22 4.35 NumArr 3.12 8.19 1.80 151.66 19.83 3.94 5.30 Numeri 1.43 8.18 1.81 151.68 17.52 2.94 4.08 And my system is only a Sempron overclocked to run at 1900Mhz (and a 400Mhz bus) with an SSE optimized ATLAS. Your machine should beat mine by a large margin. Best, Paulo From NadavH at VisionSense.com Sun Jan 22 04:34:01 2006 From: NadavH at VisionSense.com (Nadav Horesh) Date: Sun Jan 22 04:34:01 2006 Subject: [Numpy-discussion] Running benchmark on AMD64: Strange results In-Reply-To: <1137930520.7397.6.camel@localhost.localdomain> References: <43D36592.8020906@VisionSense.com> <1137930520.7397.6.camel@localhost.localdomain> Message-ID: <43D3780C.8000608@VisionSense.com> An HTML attachment was scrubbed... URL: From cjw at sympatico.ca Sun Jan 22 14:25:01 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Sun Jan 22 14:25:01 2006 Subject: [Numpy-discussion] Scalar array - a trap for the unwary In-Reply-To: <43D2E182.2060201@ieee.org> References: <43D2CDA8.9030700@sympatico.ca> <43D2E182.2060201@ieee.org> Message-ID: <43D4061C.5060900@sympatico.ca> Travis Oliphant wrote: > Colin J. Williams wrote: > >> In the script below, b is presented as an integer, either with str or >> repr, but it is an array and needs further processing to treat it as >> in integer. > > > I think there is a misunderstanding here---or, at least you need to > clarify what your concern is. Travis, You are right, it was an incomplete understanding on my part. Sasha helped to explain things. > In the code you provided, b is not an array. It is a scalar that has > the attributes of an array (but it is also subclassed from a Python > integer and acts like one everywhere). Yes, it is dressed like an array but behaves as an integer. >> >> I much prefer the numarray treatment, particularly in a matrix context. > > > What do you mean by "in a matrix context?" numarray returns a simple scalar, without the ArrayType dressing. numpy would appear to be operationally equivalent and thus adequate. In exploring these matters, it seems that numpy does not broadcast scalar operations on arrays, as numarray does. Is this intended? Please see the script below and the result under that. # tArray.py from numpy.core import multiarray as _mu import numarray.numarraycore as _na naa= _na.array([1, 2]) print type(naa[1]), print naa * 3 mua= _mu.array([3, 4]) print type(mua[1]), print mua * 3 # Grief here *** Python 2.4.2 (#67, Sep 28 2005, 12:41:11) [MSC v.1310 32 bit (Intel)] on win32. *** >>> [3 6] Traceback (most recent call last): File "", line 63, in run_nodebug File "C:\Python24\Lib\site-packages\PyMatrix\Misc\Test\tArray.py", line 11, in ? print mua * 3 # Grief here TypeError: unsupported operand type(s) for *: 'numpy.ndarray' and 'int' From ljsnxgtbpd at kaufmanchaiken.com Sun Jan 22 17:19:20 2006 From: ljsnxgtbpd at kaufmanchaiken.com (Castro Bonnie) Date: Sun Jan 22 17:19:20 2006 Subject: [Numpy-discussion] Re[6]: Message-ID: <001201c61fba$eb846ba0$639d48c8@PC5> Witt Alejandro -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: hbadgjzcsd.gif Type: image/gif Size: 6142 bytes Desc: not available URL: From konrad.hinsen at laposte.net Mon Jan 23 03:50:16 2006 From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net) Date: Mon Jan 23 03:50:16 2006 Subject: [Numpy-discussion] include file location and .eggs In-Reply-To: <43CFCCF8.8000401@astraw.com> References: <43CFCCF8.8000401@astraw.com> Message-ID: <7FA168E7-4908-4A1C-937A-DF19764BD4AA@laposte.net> On Jan 19, 2006, at 18:31, Andrew Straw wrote: > For me, and I suspect for most others in favor of the new way, the > answer boils down to using eggs with out-of-the box numpy. Eggs are > the ... Thanks for explaining this issue in some detail! > = Why can't eggs use traditional include locations? = > Multiple version support. Using eggs, it is possible to have multiple > versions of the same distribution, such as numpy, installed within a > single Python installation. (I use "distribution" to mean > collection of How is multiple version support in eggs supposed to work in the case of packages that contain C modules? Let me give a concrete example: Numeric, ScientificPython, and MMTK. Numeric provides a C API through its header files. ScientificPython and MMTK both contain C modules that depend on that C API. ScientificPython also provides a C API for two of its modules, which in turn MMTK depends on. Suppose I would like to have both Numeric 23.5 and Numeric 24.2 on my machine, and also Scientific 2.4.2 and Scientific 2.5.1. I would then need four sets of the binary modules of Scientific: 1) 2.4.2 for Numeric 23.5 2) 2.4.2 for Numeric 24.2 3) 2.5.1 for Numeric 23.5 4) 2.5.1 for Numeric 24.2 I would also need four sets of the binary modules of MMTK, assuming that I am happy with just one version of MMTK. Unless some mechanism (which would have to be part of egg construction or egg installation) makes sure that the correct binaries are used for the particular combination of versions that I wish to use, I will end up having lots of mysterious crashes resulting from incompatible binary versions. If the egg mechanism does take care of such dependencies, then it could easily (and thus should) take care of selecting the right combination of header files. If it doesn't, then it should not be used at all for packages containing C or Pyrex modules, and packages like numpy should better not accomodate to eggs because in the long run that will only cause support headaches. Konrad. -- --------------------------------------------------------------------- Konrad Hinsen Laboratoire L?on Brillouin, CEA Saclay, 91191 Gif-sur-Yvette Cedex, France Tel.: +33-1 69 08 79 25 Fax: +33-1 69 08 82 61 E-Mail: khinsen at cea.fr --------------------------------------------------------------------- From oliphant.travis at ieee.org Mon Jan 23 07:58:30 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Jan 23 07:58:30 2006 Subject: [Numpy-discussion] Scalar array - a trap for the unwary In-Reply-To: <43D4061C.5060900@sympatico.ca> References: <43D2CDA8.9030700@sympatico.ca> <43D2E182.2060201@ieee.org> <43D4061C.5060900@sympatico.ca> Message-ID: <43D4FCEC.5020507@ieee.org> Colin J. Williams wrote: > In exploring these matters, it seems that numpy does not broadcast > scalar operations on arrays, as numarray does. Is this intended? > Please see the script below and the result under that. I don't know what is going on with your installation, because I don't get that error. Your script works fine for me. I wouldn't advise extracting array directly from multiarray as you are doing, because exactly where array lives is an implementation detail that could change in the future. -Travis From mark at mitre.org Mon Jan 23 08:00:02 2006 From: mark at mitre.org (Mark Heslep) Date: Mon Jan 23 08:00:02 2006 Subject: [Numpy-discussion] Speed performance on array constant set In-Reply-To: <43D05BA0.7000700@ieee.org> References: <43D0573E.4090301@mitre.org> <43D05BA0.7000700@ieee.org> Message-ID: <43D14AC0.3060902@mitre.org> Travis Oliphant wrote: > First of all, try using NumPy instead of Numarray: import numpy as na > Ive got some NumArray C API investment in the typemaps. Is the C high level API the same, ie. NA_Input, etc? I havnt found the C API docs on the web site. > Second: use math (i.e. a += 7) > > Third: There are specialized fill routines .fill() in numpy and a > simliar routine in Numarray that can be faster then indexing. Im happy to help out on the optimization of .fill if needed Mark From oliphant.travis at ieee.org Mon Jan 23 08:12:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Jan 23 08:12:02 2006 Subject: [Numpy-discussion] Speed performance on array constant set In-Reply-To: <43D14AC0.3060902@mitre.org> References: <43D0573E.4090301@mitre.org> <43D05BA0.7000700@ieee.org> <43D14AC0.3060902@mitre.org> Message-ID: <43D50020.3030800@ieee.org> Mark Heslep wrote: > Travis Oliphant wrote: > >> First of all, try using NumPy instead of Numarray: import numpy >> as na >> > Ive got some NumArray C API investment in the typemaps. Is the C high > level API the same, ie. NA_Input, etc? I havnt found the C API docs > on the web site. The high-level C-API is the same as the old Numeric C-API. The new numarray C-API is not supported at this point. However, I would like to see a numarray compatibility module written so that it is easy to port numarray-based C-routines (simply by changing the header). I've started such a thing in scipy/Lib/sandbox/nd_image area. Look at these files for guidance: http://svn.scipy.org/svn/scipy/trunk/Lib/sandbox/nd_image/Src/numcompat.h http://svn.scipy.org/svn/scipy/trunk/Lib/sandbox/nd_image/Src/numcompat.c I'm thinking that a module could be written that contains the entire numarray C-API. -Travis From strawman at astraw.com Mon Jan 23 09:09:01 2006 From: strawman at astraw.com (Andrew Straw) Date: Mon Jan 23 09:09:01 2006 Subject: [Numpy-discussion] include file location and .eggs In-Reply-To: <7FA168E7-4908-4A1C-937A-DF19764BD4AA@laposte.net> References: <43CFCCF8.8000401@astraw.com> <7FA168E7-4908-4A1C-937A-DF19764BD4AA@laposte.net> Message-ID: <43D50D56.3000108@astraw.com> konrad.hinsen at laposte.net wrote: > On Jan 19, 2006, at 18:31, Andrew Straw wrote: >> = Why can't eggs use traditional include locations? = >> Multiple version support. Using eggs, it is possible to have multiple >> versions of the same distribution, such as numpy, installed within a >> single Python installation. (I use "distribution" to mean collection of > > > How is multiple version support in eggs supposed to work in the case > of packages that contain C modules? Eggs will allow you to do it, but you'll have to specify the exact dependencies in the setup.py file. > > Let me give a concrete example: Numeric, ScientificPython, and MMTK. > Numeric provides a C API through its header files. ScientificPython > and MMTK both contain C modules that depend on that C API. > ScientificPython also provides a C API for two of its modules, which > in turn MMTK depends on. Let's say you wanted to have Scientific 2.4.2 depend on Numeric 23.5. You'd make sure the call to setup() (which is now imported from setuptools, not distutils.core) included the following: setup( name='Scientific', version='2.4.2', ext_modules=stuff_that_depends_on_Numeric_23_5, install_requires='Numeric==23.5') Clearly, you could do this for all versions of Numeric and Scientific you want to run together. Obviously, it could be done in a more automated way (by querying what version of Numeric is being used to build). To have two versions of Scientific 2.4.2 installed, however, they would have to have different names, so perhaps '2.4.2_Numeric23.5' would work well for the example above. Now, you can just do your imports as normal, which will result in the the highest version number available. To specify versions, use 'import pkg_resources; pkg_resources.require('Numeric==23.5'); import Numeric'. An exception is raised if you do 'import Numeric' and it imports Numeric==24.2 and then you later try and import Scientific with only version 2.4.2_Numeric23.5 available. (At least, if an exception isn't raised, I'm 99% sure this would be considered a bug and would be fixed.) So, the import order is important -- it's probably best to import Scientific first in this example so that the correct version of Numeric can be selected behind the scenes without requiring you to write any pkg_resources.require() calls. N.B. For setuptools to get a package's (e.g. Numeric's) version number, Numeric must be built with setuptools -- not necessarily as an egg, but at least with the egg metadata available. If you don't build as an egg, though, you can't have more than one version installed. FYI, to build a conventional (non-egg) with setuptools so that it includes this metadata from an unmodified package that doesn't itself use setuptools, use the following python -c "import setuptools; execfile('setup.py')" install --single-version-externally-managed > > Suppose I would like to have both Numeric 23.5 and Numeric 24.2 on my > machine, and also Scientific 2.4.2 and Scientific 2.5.1. I would then > need four sets of the binary modules of Scientific: > > 1) 2.4.2 for Numeric 23.5 > 2) 2.4.2 for Numeric 24.2 > 3) 2.5.1 for Numeric 23.5 > 4) 2.5.1 for Numeric 24.2 See above. > > I would also need four sets of the binary modules of MMTK, assuming > that I am happy with just one version of MMTK. The above scales, so I don't think any new info is needed here. > > Unless some mechanism (which would have to be part of egg > construction or egg installation) makes sure that the correct > binaries are used for the particular combination of versions that I > wish to use, I will end up having lots of mysterious crashes > resulting from incompatible binary versions. That's exactly the point -- eggs do ensure the correct binary version is used. To come full circle to the question of header file location, this is exactly why eggs need to have multiple versions of the header files available, each version located within their respective egg. And this is also why keeping the header files in /usr/local/include/python2.4/Numeric/arrayobject.h is incompatible with one of the best features of eggs. > > If the egg mechanism does take care of such dependencies, then it > could easily (and thus should) take care of selecting the right > combination of header files. If it doesn't, then it should not be > used at all for packages containing C or Pyrex modules, and packages > like numpy should better not accomodate to eggs because in the long > run that will only cause support headaches. So how, about the converse? :) If setuptools does take care of dependencies (and it does), should packages like numpy definitely accommodate eggs? I certainly think so! :) Andrew From konrad.hinsen at laposte.net Mon Jan 23 09:28:07 2006 From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net) Date: Mon Jan 23 09:28:07 2006 Subject: [Numpy-discussion] include file location and .eggs In-Reply-To: <43D50D56.3000108@astraw.com> References: <43CFCCF8.8000401@astraw.com> <7FA168E7-4908-4A1C-937A-DF19764BD4AA@laposte.net> <43D50D56.3000108@astraw.com> Message-ID: <7EB1861B-5D0B-4ADE-BFA8-CCDCFF1C30B8@laposte.net> On Jan 23, 2006, at 18:07, Andrew Straw wrote: > Let's say you wanted to have Scientific 2.4.2 depend on Numeric 23.5. > You'd make sure the call to setup() (which is now imported from > setuptools, not distutils.core) included the following: ... OK, that looks sophisticated enough to me :-) Since setuptools keeps track of dependencies, it should be able to provide on demand a list of all include file directories for all packages on which the current build procedure depends, directly or indirectly. To stay within my example, a package that depends on ScientificPython, which in turn depends on Numeric, should be able to get, with a single call, the include directories for *both* ScientificPython and Numeric. Assuming such a mechanism exists, and is also implemented in plain distutils (returning just the standard installation paths), then personally I wouldn't care where header files are, because there is a straightforward way to obtain that information. However, that is not what is currently being discussed for numpy, unless I missed something. What is being discussed is an arrangement that requires the setup.py of every package to reconstruct the dependency tree and figure out where each package in the dependency tree keeps its header files. There is something else that I don't understand. If egg installation uses setup from setuptools instead of distutils, it should be trivial for the setup function from setuptools to implement install_headers in a way that suits its need. So why request that numpy (and other packages) adapt to setuptools, creating compatibility issues with standard distutils, rather than modify setuptools and have it work automatically for all packages? Konrad. -- --------------------------------------------------------------------- Konrad Hinsen Laboratoire L?on Brillouin, CEA Saclay, 91191 Gif-sur-Yvette Cedex, France Tel.: +33-1 69 08 79 25 Fax: +33-1 69 08 82 61 E-Mail: khinsen at cea.fr --------------------------------------------------------------------- From strawman at astraw.com Mon Jan 23 10:04:01 2006 From: strawman at astraw.com (Andrew Straw) Date: Mon Jan 23 10:04:01 2006 Subject: [Numpy-discussion] include file location and .eggs In-Reply-To: <7EB1861B-5D0B-4ADE-BFA8-CCDCFF1C30B8@laposte.net> References: <43CFCCF8.8000401@astraw.com> <7FA168E7-4908-4A1C-937A-DF19764BD4AA@laposte.net> <43D50D56.3000108@astraw.com> <7EB1861B-5D0B-4ADE-BFA8-CCDCFF1C30B8@laposte.net> Message-ID: <43D51A59.9020509@astraw.com> konrad.hinsen at laposte.net wrote: > On Jan 23, 2006, at 18:07, Andrew Straw wrote: > >> Let's say you wanted to have Scientific 2.4.2 depend on Numeric 23.5. >> You'd make sure the call to setup() (which is now imported from >> setuptools, not distutils.core) included the following: > > > ... > > OK, that looks sophisticated enough to me :-) > > Since setuptools keeps track of dependencies, it should be able to > provide on demand a list of all include file directories for all > packages on which the current build procedure depends, directly or > indirectly. To stay within my example, a package that depends on > ScientificPython, which in turn depends on Numeric, should be able to > get, with a single call, the include directories for *both* > ScientificPython and Numeric. > > Assuming such a mechanism exists, and is also implemented in plain > distutils (returning just the standard installation paths), then > personally I wouldn't care where header files are, because there is a > straightforward way to obtain that information. The problem is backwards compatibility. Nothing like this is implemented in plain distutils, which is why the packages will have to support it themselves. This way they can keep working in both old distutils-based environments and new ones, including with setuptools. The current solution _works_ on yesteryear's Python 2.3. In my opinion, relying on distutils to manage header file locations was a mistake, but that's easy to say in hindsight, when it's clear that, as useful as distutils is, it has some deficiencies. Distutils never handled the case when, for example you installed Numeric with '/usr/local/bin/python2.3 setup.py install --prefix=/home/astraw/py23'. In this case, sure, the package got installed where I wanted (in my $PYTHONPATH) in /home/astraw/py23/lib/python2.3/site-packages/Numeric, but the headers were in /home/astraw/py23/include/python2.3/Numeric. Distutils would never find those headers again. This is exactly the same problem, and it's one that distutils never solved, and I don't think we should hold our breath waiting for Python 2.5's distutils to solve it. For one thing, it's already solved at the package level a la numpy. For another, given that it's already solved, who's going to bother attempting to make a patch (which can't be relied on by older pythons) for Python 2.5, which may not even be accepted? > > However, that is not what is currently being discussed for numpy, > unless I missed something. What is being discussed is an arrangement > that requires the setup.py of every package to reconstruct the > dependency tree and figure out where each package in the dependency > tree keeps its header files. Why is "import numpy; include_dirs=numpy.get_numpy_include()" in setup.py unpalatable for you? That's all that is required in setup.py. To me, this is a small price to pay for the benefits gained. If you want to argue this is more pain than you're willing to deal with, I think that, to be fair, you need to give a counter-example whereby you explain how you handle all those versions of Numeric and Scientific installed simultaneously and how you select among them. I'll admit that in the case you gave, of 2 versions of Scientific and 2 versions of Numeric compiled into all 4 possible combinations, you'd also need an "import pkg_resources; pkg_resources.require('Numeric==x.y')" prior to the "import Numeric". I see the case you gave as a rather extreme case -- normally I would like, for example, a single version of (old) scipy compiled against old Numeric and a single new version of scipy compiled against numpy. (If it's not already working, I'm almost willing to backport egg support to old scipy to do just this...) Like I said above, distutils has no such equivalent mechanism, and this _allows_ something that simply wasn't possible before and is arguably broken in distutils. > > > There is something else that I don't understand. If egg installation > uses setup from setuptools instead of distutils, it should be trivial > for the setup function from setuptools to implement install_headers > in a way that suits its need. So why request that numpy (and other > packages) adapt to setuptools, creating compatibility issues with > standard distutils, rather than modify setuptools and have it work > automatically for all packages? That's an interesting idea. The short answer is that setuptools doesn't do it currently. Perhaps it could. I'm not going to implement it, though. It also doesn't solve the problem I gave above with plain distutils, which I think is a real problem that numpy solves in a backwards compatible way. From konrad.hinsen at laposte.net Mon Jan 23 10:40:06 2006 From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net) Date: Mon Jan 23 10:40:06 2006 Subject: [Numpy-discussion] include file location and .eggs In-Reply-To: <43D51A59.9020509@astraw.com> References: <43CFCCF8.8000401@astraw.com> <7FA168E7-4908-4A1C-937A-DF19764BD4AA@laposte.net> <43D50D56.3000108@astraw.com> <7EB1861B-5D0B-4ADE-BFA8-CCDCFF1C30B8@laposte.net> <43D51A59.9020509@astraw.com> Message-ID: <991A3B2E-AE9B-433C-91DB-549C70F47C7C@laposte.net> On Jan 23, 2006, at 19:03, Andrew Straw wrote: > The problem is backwards compatibility. Nothing like this is > implemented > in plain distutils, which is why the packages will have to support it > themselves. This way they can keep working in both old distutils-based > environments and new ones, including with setuptools. The current > solution _works_ on yesteryear's Python 2.3. In my opinion, relying on Well, I'd argue it doesn't work at all, because it doesn't scale to complex dependency trees. > Distutils never handled the case when, for example you installed > Numeric > with '/usr/local/bin/python2.3 setup.py install > --prefix=/home/astraw/py23'. In this case, sure, the package got > installed where I wanted (in my $PYTHONPATH) in > /home/astraw/py23/lib/python2.3/site-packages/Numeric, but the headers > were in /home/astraw/py23/include/python2.3/Numeric. Distutils would True. It was always true that if you install anywhere but in the default location, you are on your own. But if you do keep the default location (as most users do), everything works fine. > never find those headers again. This is exactly the same problem, and > it's one that distutils never solved, and I don't think we should hold > our breath waiting for Python 2.5's distutils to solve it. For one Why not contribute to solving it? > thing, it's already solved at the package level a la numpy. For > another, > given that it's already solved, who's going to bother attempting to > make I don't think it is solved at all. > Why is "import numpy; include_dirs=numpy.get_numpy_include()" in > setup.py unpalatable for you? That's all that is required in setup.py. Let's go back to my initial scenario, and let's even forget about multiple versions. I have ScientificPython, which can be compiled for use with either Numeric or numarray, and some day NumPy will be a third choice. Then I install MMTK, which depends on ScientificPython. How is the MMTK installation script supposed to figure out if ScientificPython was compiled with Numeric, numarray, or NumPy? And how is it supposed to know where each of these packages stores its header files? I don't mind adding some lines to setup.py for each package, that's a small price to pay. But I do mind having to worry about indirect dependencies. > to argue this is more pain than you're willing to deal with, I think > that, to be fair, you need to give a counter-example whereby you > explain > how you handle all those versions of Numeric and Scientific installed > simultaneously and how you select among them. I'll admit that in the I don't, except on my development machine, where I used symbolic links that are changed using scripts. That's a bit of work to set up, but I have only two packages of which I use multiple versions, so doing it once is acceptable for me. In fact, my main worry is not my own machine, nor the machine of any competent Python programmer. My worry is the less competent user who can't get installation of my packages to work and turns to me for help. Tech support takes more of my time than development these days. A fragile dependency system such as the one you propose will only create more support questions on my side - so don't expect me to support it. > Like I said above, distutils has no such equivalent mechanism, and > this > _allows_ something that simply wasn't possible before and is arguably > broken in distutils. I am all for fixing distutils. Konrad. -- --------------------------------------------------------------------- Konrad Hinsen Laboratoire L?on Brillouin, CEA Saclay, 91191 Gif-sur-Yvette Cedex, France Tel.: +33-1 69 08 79 25 Fax: +33-1 69 08 82 61 E-Mail: khinsen at cea.fr --------------------------------------------------------------------- From pearu at scipy.org Mon Jan 23 11:18:02 2006 From: pearu at scipy.org (Pearu Peterson) Date: Mon Jan 23 11:18:02 2006 Subject: [Numpy-discussion] include file location and .eggs In-Reply-To: <991A3B2E-AE9B-433C-91DB-549C70F47C7C@laposte.net> References: <43CFCCF8.8000401@astraw.com> <7FA168E7-4908-4A1C-937A-DF19764BD4AA@laposte.net> <43D50D56.3000108@astraw.com> <7EB1861B-5D0B-4ADE-BFA8-CCDCFF1C30B8@laposte.net> <43D51A59.9020509@astraw.com> <991A3B2E-AE9B-433C-91DB-549C70F47C7C@laposte.net> Message-ID: On Mon, 23 Jan 2006, konrad.hinsen at laposte.net wrote: > Let's go back to my initial scenario, and let's even forget about multiple > versions. I have ScientificPython, which can be compiled for use with either > Numeric or numarray, and some day NumPy will be a third choice. Then I > install MMTK, which depends on ScientificPython. How is the MMTK installation > script supposed to figure out if ScientificPython was compiled with Numeric, > numarray, or NumPy? I don't know about setuptools but distutils cannot solve this problem either without some extra code in MMTK setup.py file and a possibility to determine which choice was used to build ScientificPython. So, it's not really a distutils nor numpy include file location issue. > And how is it supposed to know where each of these packages stores its header files? Here's an answer to this question, namely, when numpy is installed, then one can use the following code in setup.py to support different array backends: from numpy.distutils.system_info import get_info arr_info = get_info('numpy') or get_info('numarray') or get_info('Numeric') # For explanation, arr_info is one of the following dictionaries: # {'define_macros':[('NUMPY',None)], 'include_dirs':['/path/to/numpy/header/dir']} # {'define_macros':[('NUMARRAY',None)], 'include_dirs':['/path/to/numarray/header/dir']} # {'define_macros':[('NUMERIC',None)], 'include_dirs':['/path/to/Numeric/header/dir']} # {} # --- no array backends installed # If one has all three array backend packages installed and wishes to # use, say, numarray, then to disable detecting numpy, one should # define environment variable NUMPY=None. ... setup(..., ext_modules = [Extension(..., **arr_info),..]) Pearu From boisgera at isia.cma.fr Mon Jan 23 11:53:04 2006 From: boisgera at isia.cma.fr (=?ISO-8859-1?Q?S=E9bastien_Boisg=E9rault?=) Date: Mon Jan 23 11:53:04 2006 Subject: [Numpy-discussion] Eigenvalues hangs Message-ID: <43D533EF.2090604@isia.cma.fr> Hi all, Robert Kern suggested to transfer a discussion that originated on comp.lang.python to this mailing-list. My problem is that on my platform the numpy 'eigenvalues' method hangs forever. Some platform info: * numpy 0.9.4 (from the tar.gz) * Linux Mandriva (gcc 4.0.1). * no optimized BLAS version used. Greg Landrum pointed out that it may be a gcc 4.0 related problem and proposed a workaround -- to add the option '-ffloat-store' to CFLAGS. Could anybody tell me how I should modify the numpy sources to test this workaround ? Regards, SB From cookedm at physics.mcmaster.ca Mon Jan 23 12:03:03 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Mon Jan 23 12:03:03 2006 Subject: [Numpy-discussion] Eigenvalues hangs In-Reply-To: <43D533EF.2090604@isia.cma.fr> (=?utf-8?Q?S=C3=A9bastien_Bois?= =?utf-8?Q?g=C3=A9rault's?= message of "Mon, 23 Jan 2006 20:52:15 +0100") References: <43D533EF.2090604@isia.cma.fr> Message-ID: S?bastien Boisg?rault writes: > Hi all, > > Robert Kern suggested to transfer a discussion that originated > on comp.lang.python to this mailing-list. > > My problem is that on my platform the numpy 'eigenvalues' method > hangs forever. > > Some platform info: > * numpy 0.9.4 (from the tar.gz) > * Linux Mandriva (gcc 4.0.1). > * no optimized BLAS version used. > > Greg Landrum pointed out that it may be a gcc 4.0 related problem and > proposed a workaround -- to add the option '-ffloat-store' to CFLAGS. > > Could anybody tell me how I should modify the numpy sources to test > this workaround ? You should just be able to do the setup like this: $ CFLAGS='-ffloat-store' python setup.py build -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From boisgera at isia.cma.fr Mon Jan 23 12:19:03 2006 From: boisgera at isia.cma.fr (=?UTF-8?B?U8OpYmFzdGllbiBCb2lzZ8OpcmF1bHQ=?=) Date: Mon Jan 23 12:19:03 2006 Subject: [Numpy-discussion] Eigenvalues hangs In-Reply-To: References: <43D533EF.2090604@isia.cma.fr> Message-ID: <43D539FF.4070808@isia.cma.fr> David M. Cooke a ?crit : >S?bastien Boisg?rault writes: > > > >>Hi all, >> >>Robert Kern suggested to transfer a discussion that originated >>on comp.lang.python to this mailing-list. >> >>My problem is that on my platform the numpy 'eigenvalues' method >>hangs forever. >> >>Some platform info: >> * numpy 0.9.4 (from the tar.gz) >> * Linux Mandriva (gcc 4.0.1). >> * no optimized BLAS version used. >> >>Greg Landrum pointed out that it may be a gcc 4.0 related problem and >>proposed a workaround -- to add the option '-ffloat-store' to CFLAGS. >> >>Could anybody tell me how I should modify the numpy sources to test >>this workaround ? >> >> > >You should just be able to do the setup like this: > >$ CFLAGS='-ffloat-store' python setup.py build > > > Great ! It solves the problem :) Thanks a lot for the help ! SB -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndarray at mac.com Mon Jan 23 12:26:04 2006 From: ndarray at mac.com (Sasha) Date: Mon Jan 23 12:26:04 2006 Subject: [Numpy-discussion] Eigenvalues hangs In-Reply-To: <43D533EF.2090604@isia.cma.fr> References: <43D533EF.2090604@isia.cma.fr> Message-ID: I think this issue was altimately resolved as a lapack rather than gcc problem and -ffloat-store is not the right work-around. I don't have the right references right now, so unless someone steps in I will follow up when I get home this evening. -- sasha On 1/23/06, S?bastien Boisg?rault wrote: > > Hi all, > > Robert Kern suggested to transfer a discussion that originated > on comp.lang.python to this mailing-list. > > My problem is that on my platform the numpy 'eigenvalues' method > hangs forever. > > Some platform info: > * numpy 0.9.4 (from the tar.gz) > * Linux Mandriva (gcc 4.0.1). > * no optimized BLAS version used. > > Greg Landrum pointed out that it may be a gcc 4.0 related problem and > proposed a workaround -- to add the option '-ffloat-store' to CFLAGS. > > Could anybody tell me how I should modify the numpy sources to test > this workaround ? > > Regards, > > SB > > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From konrad.hinsen at laposte.net Mon Jan 23 12:27:02 2006 From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net) Date: Mon Jan 23 12:27:02 2006 Subject: [Numpy-discussion] include file location and .eggs In-Reply-To: References: <43CFCCF8.8000401@astraw.com> <7FA168E7-4908-4A1C-937A-DF19764BD4AA@laposte.net> <7EB1861B-5D0B-4ADE-BFA8-CCDCFF1C30B8@laposte.net> <991A3B2E-AE9B-433C-91DB-549C70F47C7C@laposte.net> Message-ID: <246A6EC8-8D6B-407A-8224-0276422392B0@laposte.net> On 23.01.2006, at 19:17, Pearu Peterson wrote: > I don't know about setuptools but distutils cannot solve this > problem either without some extra code in MMTK setup.py file and a > possibility to determine which choice was used to build > ScientificPython. So, it's not really a distutils nor numpy include > file location issue. Distutils doesn't provide a perfect solution, but if everyone just lets distutils put header files into the default location, and all packages get installed without an explicit prefix, then it will just work. Considering that I get next to no support questions about such installation issues, I consider the current situation very acceptable. >> And how is it supposed to know where each of these packages stores >> its header files? > > Here's an answer to this question, namely, when numpy is installed, > then one can use the following code in setup.py to support > different array backends: That's for numpy. How about Numeric and numarray? And how about whatever other packages need header files? If everyone makes up his own conventions, we end up with a big mess. Another idea: considering that setup.py needs to be changed anyway to make eggs (because setup must be imported from somewhere else), why not put some flag "make_egg" at the beginning of numpy's setup.py, and depending on the value of the flag either use setuptools.setup and header files in the data section, or distutils.core.setup and header files in the standard location? That should suit everyone, right? Konrad. -- ------------------------------------------------------------------------ ------- Konrad Hinsen Laboratoire Leon Brillouin (CEA-CNRS), CEA Saclay, 91191 Gif-sur-Yvette Cedex, France Tel.: +33-1 69 08 79 25 Fax: +33-1 69 08 82 61 E-Mail: khinsen at cea.fr ------------------------------------------------------------------------ ------- From cjw at sympatico.ca Mon Jan 23 14:32:29 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Mon Jan 23 14:32:29 2006 Subject: [Numpy-discussion] Scalar array - a trap for the unwary In-Reply-To: <43D4FCEC.5020507@ieee.org> References: <43D2CDA8.9030700@sympatico.ca> <43D2E182.2060201@ieee.org> <43D4061C.5060900@sympatico.ca> <43D4FCEC.5020507@ieee.org> Message-ID: <43D55952.8000905@sympatico.ca> Travis Oliphant wrote: > Colin J. Williams wrote: > >> In exploring these matters, it seems that numpy does not broadcast >> scalar operations on arrays, as numarray does. Is this intended? >> Please see the script below and the result under that. > > > I don't know what is going on with your installation, because I don't > get that error. Your script works fine for me. I've just tried this with version 0.9.4 and all works well. >>> a= _mu.array([1, 2], dtype= _nt.uint8) >>> a+254 array([255, 256], dtype=int16) This does bring out a possible problem with the new type. str of the old type returned the input name int8, float64 etc. Is there some method of the new type which returns the name input, 'int8' in the above example? In most cases, the mnemonic is more helpful than the letter coding. > > I wouldn't advise extracting array directly from multiarray as you are > doing, because exactly where array lives is an implementation detail > that could change in the future. I accept this risk for now. Partly, this was done to simplify the namespace and partly to locate a bug concerning the interaction of a Windows debugger and the umath module. > > -Travis > Colin W. From Chris.Barker at noaa.gov Mon Jan 23 14:42:49 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon Jan 23 14:42:49 2006 Subject: [Numpy-discussion] include file location and .eggs In-Reply-To: <246A6EC8-8D6B-407A-8224-0276422392B0@laposte.net> References: <43CFCCF8.8000401@astraw.com> <7FA168E7-4908-4A1C-937A-DF19764BD4AA@laposte.net> <7EB1861B-5D0B-4ADE-BFA8-CCDCFF1C30B8@laposte.net> <991A3B2E-AE9B-433C-91DB-549C70F47C7C@laposte.net> <246A6EC8-8D6B-407A-8224-0276422392B0@laposte.net> Message-ID: <43D55B77.4010105@noaa.gov> konrad.hinsen at laposte.net wrote: > Distutils doesn't provide a perfect solution, but if everyone just lets > distutils put header files into the default location, and all packages > get installed without an explicit prefix, then it will just work. > Considering that I get next to no support questions about such > installation issues, I consider the current situation very acceptable. Except that there is no support whatsoever for multiple versions. It seems your complicated scenario was due to multiple versions. In fact, the one problem I've had with Scientific in the past was trying to use a binary that had been compiled against a different version of Numeric than I was using. If your user is compiling everything, then there is no problem. If your user is using binaries, they they need to be labeled with versions. If they use the wrong one with eggs, then there is at least some hope that they'll get a meaningful error message. If they are compiling MMTK against a Scientific and a Numpy that they didn't compile, then it's going to get ugly -- but I don't think it's any uglier than it is now. As for improving distutils in the next go around -- I'm all for it, but why not have 2.5 include setuptools instead -- it strikes me as a next generation distutils. I do agree that setuptools should handle the header file placement and finding for you, rather than each package having to do it on it's own. and Andrew, thanks for all the info on setuptools and eggs. It's been very informative! -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From mark at mitre.org Mon Jan 23 16:19:06 2006 From: mark at mitre.org (Mark Heslep) Date: Mon Jan 23 16:19:06 2006 Subject: [Numpy-discussion] Speed performance on array constant set In-Reply-To: <43D51669.6080308@noaa.gov> References: <43D0573E.4090301@mitre.org> <43D05BA0.7000700@ieee.org> <43D14AC0.3060902@mitre.org> <43D51669.6080308@noaa.gov> Message-ID: <43D57262.6090004@mitre.org> Christopher Barker wrote: > Mark Heslep wrote: > >> Ive got some NumArray C API investment in the typemaps. > > > Are these SWIG typemaps? If so I think there are a bunch of us that > wold like to see them, and maybe we can all work together to port them > to NumPy. > > -Chris > Yes they are SWIG typemaps for OpenCv* data structures, based on prior Numeric art I'd found and ported to Numarray. Attached. 1. numdata.h example that works well for simple Cv structures containing uniform types: CvPoint2D32F => struct { float x, float y }. Can I use a record array here, or is there some Numpy overhead interlaced with fields? 2. numarray.h Attempt to replace the main Cv Image structures CvMat, IplImage. Needs work. Some success but there's segfault or two in there somewhere. *Details for Cv use: I drop them in the ../interfaces/swig/python path for opencv and them in cv.i Regards, Mark -------------- next part -------------- A non-text attachment was scrubbed... Name: adaptors.h Type: text/x-chdr Size: 2195 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: numarray.h Type: text/x-chdr Size: 2746 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: numdata.h Type: text/x-chdr Size: 1750 bytes Desc: not available URL: From oliphant at ee.byu.edu Mon Jan 23 16:37:05 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Jan 23 16:37:05 2006 Subject: [Numpy-discussion] Speed performance on array constant set In-Reply-To: <43D57262.6090004@mitre.org> References: <43D0573E.4090301@mitre.org> <43D05BA0.7000700@ieee.org> <43D14AC0.3060902@mitre.org> <43D51669.6080308@noaa.gov> <43D57262.6090004@mitre.org> Message-ID: <43D5768C.4090200@ee.byu.edu> Mark Heslep wrote: > Christopher Barker wrote: > >> Mark Heslep wrote: >> >>> Ive got some NumArray C API investment in the typemaps. >> >> >> >> Are these SWIG typemaps? If so I think there are a bunch of us that >> wold like to see them, and maybe we can all work together to port >> them to NumPy. >> >> -Chris >> > Yes they are SWIG typemaps for OpenCv* data structures, based on prior > Numeric art I'd found and ported to Numarray. > Attached. > > 1. numdata.h example that works well for simple Cv structures > containing uniform types: CvPoint2D32F => struct { float x, float y > }. Can I use a record array here, or is there some Numpy overhead > interlaced with fields? You can use a record array, but for uniform types I don't know what the advantage is (except for perhaps named-field slicing---which may actually be faster I'm not sure) over a x2 float array. > 2. numarray.h Attempt to replace the main Cv Image structures CvMat, > IplImage. Needs work. Some success but there's segfault or two in > there somewhere. > These can be ported fairly easily, I think (actually, the old Numeric typemaps would still work --- and work with Numarray), so the basic Numeric C-API still presents itself as the simplest way to support all the array packages. -Travis From ndarray at mac.com Mon Jan 23 19:06:02 2006 From: ndarray at mac.com (Sasha) Date: Mon Jan 23 19:06:02 2006 Subject: [Numpy-discussion] Eigenvalues hangs In-Reply-To: References: <43D533EF.2090604@isia.cma.fr> Message-ID: On 1/23/06, Sasha wrote: > I think this issue was altimately resolved as a lapack rather than gcc > problem and -ffloat-store is not the right work-around. I don't have > the right references right now, so unless someone steps in I will > follow up when I get home this evening. > > -- sasha > > On 1/23/06, S?bastien Boisg?rault wrote: > > > > Hi all, > > > > Robert Kern suggested to transfer a discussion that originated > > on comp.lang.python to this mailing-list. > > > > My problem is that on my platform the numpy 'eigenvalues' method > > hangs forever. > > > > Some platform info: > > * numpy 0.9.4 (from the tar.gz) > > * Linux Mandriva (gcc 4.0.1). > > * no optimized BLAS version used. > > > > Greg Landrum pointed out that it may be a gcc 4.0 related problem and > > proposed a workaround -- to add the option '-ffloat-store' to CFLAGS. > > > > Could anybody tell me how I should modify the numpy sources to test > > this workaround ? > > > > Regards, > > > > SB > > > > > > > > ------------------------------------------------------- > > This SF.net email is sponsored by: Splunk Inc. Do you grep through log files > > for problems? Stop! Download the new AJAX search engine that makes > > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642 > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > From ndarray at mac.com Mon Jan 23 19:15:03 2006 From: ndarray at mac.com (Sasha) Date: Mon Jan 23 19:15:03 2006 Subject: [Numpy-discussion] Eigenvalues hangs In-Reply-To: References: <43D533EF.2090604@isia.cma.fr> Message-ID: Sorry for an empty message. I lost connection and a half a page message with it :-( Here is a brief summary: -ffloat-store may be expensive: """ Reading the gcc documentation, one finds that the -ffloat-store is an expensive hack to attempt to avoid the excess precision in the floating pointer registers on Intel FPUs. Based on recent experience with DONLP2, a noteworthy nonlinear solver, using -ffloat-store everywhere is an unnecessarily costly workaround. """ If it has to be enabled, it should be limited to dlapack_lite . Red Hat patched its own lapack by disabling optimization for a few routines: https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=138683 -- sasha On 1/23/06, Sasha wrote: > On 1/23/06, Sasha wrote: > > I think this issue was altimately resolved as a lapack rather than gcc > > problem and -ffloat-store is not the right work-around. I don't have > > the right references right now, so unless someone steps in I will > > follow up when I get home this evening. > > > > -- sasha > > > > On 1/23/06, S?bastien Boisg?rault wrote: > > > > > > Hi all, > > > > > > Robert Kern suggested to transfer a discussion that originated > > > on comp.lang.python to this mailing-list. > > > > > > My problem is that on my platform the numpy 'eigenvalues' method > > > hangs forever. > > > > > > Some platform info: > > > * numpy 0.9.4 (from the tar.gz) > > > * Linux Mandriva (gcc 4.0.1). > > > * no optimized BLAS version used. > > > > > > Greg Landrum pointed out that it may be a gcc 4.0 related problem and > > > proposed a workaround -- to add the option '-ffloat-store' to CFLAGS. > > > > > > Could anybody tell me how I should modify the numpy sources to test > > > this workaround ? > > > > > > Regards, > > > > > > SB > > > > > > > > > > > > ------------------------------------------------------- > > > This SF.net email is sponsored by: Splunk Inc. Do you grep through log files > > > for problems? Stop! Download the new AJAX search engine that makes > > > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > > > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642 > > > _______________________________________________ > > > Numpy-discussion mailing list > > > Numpy-discussion at lists.sourceforge.net > > > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > > > > From mark at mitre.org Mon Jan 23 19:21:08 2006 From: mark at mitre.org (Mark Heslep) Date: Mon Jan 23 19:21:08 2006 Subject: [Numpy-discussion] Speed performance on array constant set In-Reply-To: <43D5768C.4090200@ee.byu.edu> References: <43D0573E.4090301@mitre.org> <43D05BA0.7000700@ieee.org> <43D14AC0.3060902@mitre.org> <43D51669.6080308@noaa.gov> <43D57262.6090004@mitre.org> <43D5768C.4090200@ee.byu.edu> Message-ID: <43D59D0B.3060907@mitre.org> Travis Oliphant wrote: >> 1. numdata.h example that works well for simple Cv structures >> containing uniform types: CvPoint2D32F => struct { float x, float y >> }. Can I use a record array here, or is there some Numpy overhead >> interlaced with fields? > > > You can use a record array, but for uniform types I don't know what > the advantage is (except for perhaps named-field slicing---which may > actually be faster I'm not sure) over a x2 float array. So that I can step through the array record by record and not field by field. >> 2. numarray.h Attempt to replace the main Cv Image structures CvMat, >> IplImage. Needs work. Some success but there's segfault or two in >> there somewhere. > > These can be ported fairly easily, I think (actually, the old Numeric > typemaps would still work --- and work with Numarray), so the basic > Numeric C-API still presents itself as the simplest way to support all > the array packages. Well good to know. Ive been proceeding directly from the NumArray Users Manual 1.5 by Greenfield/Miller et al that describes the NA High Level API as the fastest of the APIs available to NA. I thought that NA also took steps to insure that unnecessary mem copies were not made, unlike Numeric? Mark From oliphant.travis at ieee.org Mon Jan 23 20:42:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Jan 23 20:42:02 2006 Subject: [Numpy-discussion] Speed performance on array constant set In-Reply-To: <43D59D0B.3060907@mitre.org> References: <43D0573E.4090301@mitre.org> <43D05BA0.7000700@ieee.org> <43D14AC0.3060902@mitre.org> <43D51669.6080308@noaa.gov> <43D57262.6090004@mitre.org> <43D5768C.4090200@ee.byu.edu> <43D59D0B.3060907@mitre.org> Message-ID: <43D5AFEA.9030405@ieee.org> Mark Heslep wrote: > Travis Oliphant wrote: > >>> 1. numdata.h example that works well for simple Cv structures >>> containing uniform types: CvPoint2D32F => struct { float x, float >>> y }. Can I use a record array here, or is there some Numpy overhead >>> interlaced with fields? >> >> >> >> You can use a record array, but for uniform types I don't know what >> the advantage is (except for perhaps named-field slicing---which may >> actually be faster I'm not sure) over a x2 float array. > > > So that I can step through the array record by record and not field by > field. Right, but you could do that using steps of length 2 if you want to manage it yourself (or use a complex data type). Again, I'm not sure of the performance implications at this point as I haven't done enough tests. > >>> 2. numarray.h Attempt to replace the main Cv Image structures >>> CvMat, IplImage. Needs work. Some success but there's segfault or >>> two in there somewhere. >> >> >> These can be ported fairly easily, I think (actually, the old Numeric >> typemaps would still work --- and work with Numarray), so the basic >> Numeric C-API still presents itself as the simplest way to support >> all the array packages. > > > Well good to know. Ive been proceeding directly from the NumArray > Users Manual 1.5 by Greenfield/Miller et al that describes the NA High > Level API as the fastest of the APIs available to NA. I thought that > NA also took steps to insure that unnecessary mem copies were not > made, unlike Numeric? Numeric doesn't make unnecessary copies either. It depends on what you mean by necessary. I think they may be referring to the fact that most people used ContiguousFromObject which always gave you contiguous memory from an array (copying if necessary), but if you learn how to access strided memory then FromObject was just fine (which would never copy). There are still some algorithms that don't deal with strided memory and so will make a copy. Some of these can be changed, others would be more difficult. It really becomes a question of cache-misses versus memory copy time, which I'm not sure how to optimize generally except by per-system experimentation like ATLAS does. -Travis From faltet at carabos.com Tue Jan 24 01:44:04 2006 From: faltet at carabos.com (Francesc Altet) Date: Tue Jan 24 01:44:04 2006 Subject: [Numpy-discussion] Scalar array - a trap for the unwary In-Reply-To: <43D55952.8000905@sympatico.ca> References: <43D2CDA8.9030700@sympatico.ca> <43D4FCEC.5020507@ieee.org> <43D55952.8000905@sympatico.ca> Message-ID: <200601241043.32209.faltet@carabos.com> A Dilluns 23 Gener 2006 23:31, Colin J. Williams va escriure: > >>> a= _mu.array([1, 2], dtype= _nt.uint8) > >>> a+254 > > array([255, 256], dtype=int16) > > This does bring out a possible problem with the new type. str of the > old type returned the input name int8, float64 etc. Is there some > method of the new type which returns the name input, 'int8' in the above > example? In most cases, the mnemonic is more helpful than the letter > coding. Yes. a.dtype.name does the trick. Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From jensj at fysik.dtu.dk Tue Jan 24 04:45:03 2006 From: jensj at fysik.dtu.dk (Jens Jorgen Mortensen) Date: Tue Jan 24 04:45:03 2006 Subject: [Numpy-discussion] Numeric/numpy difference Message-ID: <43D62121.5060504@fysik.dtu.dk> I just discovered (the hard way) that: >>> import numpy >>> a = numpy.empty((2, 3)) >>> a[:] = [1, 2] >>> a array([[1, 2, 1], [2, 1, 2]]) This is with revision 1984. With Numeric I get "ValueError: matrices are not aligned for copy", which I think is the correct answer. Jens J?rgen From schofield at ftw.at Tue Jan 24 06:16:02 2006 From: schofield at ftw.at (Ed Schofield) Date: Tue Jan 24 06:16:02 2006 Subject: [Numpy-discussion] 0-dimensional object arrays Message-ID: <43D6363C.5080509@ftw.at> Hi all, I've been attempting to use an object array to group together a collection of different-length arrays or lists, and I'm getting unexpected results. Here's an example: >>> import numpy >>> a = numpy.array([[1,2],[1,2,3]],object) I expected here that 'a' would be an array of dimension (2,), holding two Python lists of integers. Was I wrong to expect this? It seems instead that 'a' is a 0-dimensional array holding one object: >>> a array([[1, 2], [1, 2, 3]], dtype=object) >>> a.shape () >>> type(a) >>> a[0] Traceback (most recent call last): File "", line 1, in ? IndexError: 0-d arrays can't be indexed. This behaviour seems less useful. Would you agree this is a bug in the array constructor? I also can't explain this behaviour: >>> [] + a array([], dtype=object) >>> a + [] array([], dtype=object) I guess I would have expected these commands to be equivalent array(a.view()+[]). These results seem more sensible: >>> a.transpose() [[1, 2], [1, 2, 3]] >>> a*2 [[1, 2], [1, 2, 3], [1, 2], [1, 2, 3]] although these last two commands return Python lists, without the 0-d array wrapper, which isn't consistent with other 0-d arrays. On the whole, 0-d object arrays seem quite strange beasts. Could someone please enlighten me on why they deserve to exist? ;) They seem inconsistent with the new simplified interface to object array elements. Could we get rid of them entirely?! -- Ed From oliphant.travis at ieee.org Tue Jan 24 07:58:07 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue Jan 24 07:58:07 2006 Subject: [Numpy-discussion] 0-dimensional object arrays In-Reply-To: <43D6363C.5080509@ftw.at> References: <43D6363C.5080509@ftw.at> Message-ID: <43D64E66.9060706@ieee.org> Ed Schofield wrote: >Hi all, > >I've been attempting to use an object array to group together a >collection of different-length arrays or lists, and I'm getting >unexpected results. Here's an example: > > > >>>>import numpy >>>>a = numpy.array([[1,2],[1,2,3]],object) >>>> >>>> > >I expected here that 'a' would be an array of dimension (2,), holding >two Python lists of integers. Was I wrong to expect this? It seems >instead that 'a' is a 0-dimensional array holding one object: > > > Yes, you were wrong to expect this ;-) The array constructor is not really useful for constructing arbitrary object arrays. It is not designed to analyze what you've input and pick the "optimal" assignment. It never has been. So, it's not surprising that it doesn't do what you expect. > >[[1, 2], [1, 2, 3], [1, 2], [1, 2, 3]] > >although these last two commands return Python lists, without the 0-d >array wrapper, which isn't consistent with other 0-d arrays. > > We just changed that remember. When 0-d object arrays result from calculations, the actual objects are returned. >On the whole, 0-d object arrays seem quite strange beasts. Could >someone please enlighten me on why they deserve to exist? ;) They seem >inconsistent with the new simplified interface to object array >elements. Could we get rid of them entirely?! > > No, we can't get rid of them because 0-d arrays exist and OBJECT is a data-type an array can have. Somebody is going to be able to construct one. My advise is to create empty object arrays and fill them appropriately unless something better is written for OBJECT array construction. -Travis From oliphant.travis at ieee.org Tue Jan 24 08:01:08 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue Jan 24 08:01:08 2006 Subject: [Numpy-discussion] Numeric/numpy difference In-Reply-To: <43D62121.5060504@fysik.dtu.dk> References: <43D62121.5060504@fysik.dtu.dk> Message-ID: <43D64F2B.3020506@ieee.org> Jens Jorgen Mortensen wrote: > I just discovered (the hard way) that: > > >>> import numpy > >>> a = numpy.empty((2, 3)) > >>> a[:] = [1, 2] > >>> a > array([[1, 2, 1], > [2, 1, 2]]) > > This is with revision 1984. With Numeric I get "ValueError: matrices > are not aligned for copy", which I think is the correct answer. Well, not necessarily. This is consistent behavior with a[:] = 3 which fills the array with 3's. What's happening is that the array is being filled with [1,2] repeatedly (in C-contiguous order). -Travis From agn at noc.soton.ac.uk Tue Jan 24 08:06:04 2006 From: agn at noc.soton.ac.uk (George Nurser) Date: Tue Jan 24 08:06:04 2006 Subject: [Numpy-discussion] f2py problems Message-ID: I'm not sure whether this is the right list for f2py questions, but anyway. Tried it (v 2_1967) out today. It worked beautifully for a simple function. But for a more complicated (f77) *.F file with preprocessor directives and a lot of data statements I have been having problems. Firstly, it just seemed to ignore all of the lines with preprocessor directives. Secondly, after I used a preprocessor to create the .f file separately, and then ran the .f file through f2py, I got the following error trace. Anybody have any ideas? --George Nurser. f2py -c --verbose -m state2sig state2sig.f running build running config_fc running build_src building extension "state2sig" sources f2py options: [] f2py:> /tmp/tmppzKlnT/src/state2sigmodule.c creating /tmp/tmppzKlnT creating /tmp/tmppzKlnT/src Reading fortran codes... Reading file 'state2sig.f' (format:fix,strict) Post-processing... Block: state2sig Block: state2sig Traceback (most recent call last): File "/data/jrd/mod1/agn/ext/Linux/bin/f2py", line 6, in ? f2py.main() File "/data/jrd/mod1/agn/ext/Linux/lib64/python/numpy/f2py/ f2py2e.py", line 546, in main run_compile() File "/data/jrd/mod1/agn/ext/Linux/lib64/python/numpy/f2py/ f2py2e.py", line 533, in run_compile setup(ext_modules = [ext]) File "/data/jrd/mod1/agn/ext/Linux/lib64/python/numpy/distutils/ core.py", line 93, in setup return old_setup(**new_attr) File "/usr/lib64/python2.3/distutils/core.py", line 149, in setup dist.run_commands() File "/usr/lib64/python2.3/distutils/dist.py", line 907, in run_commands self.run_command(cmd) File "/usr/lib64/python2.3/distutils/dist.py", line 927, in run_command cmd_obj.run() File "/usr/lib64/python2.3/distutils/command/build.py", line 107, in run self.run_command(cmd_name) File "/usr/lib64/python2.3/distutils/cmd.py", line 333, in run_command self.distribution.run_command(command) File "/usr/lib64/python2.3/distutils/dist.py", line 927, in run_command cmd_obj.run() File "/data/jrd/mod1/agn/ext/Linux/lib64/python/numpy/distutils/ command/build_src.py", line 86, in run self.build_sources() File "/data/jrd/mod1/agn/ext/Linux/lib64/python/numpy/distutils/ command/build_src.py", line 99, in build_sources self.build_extension_sources(ext) File "/data/jrd/mod1/agn/ext/Linux/lib64/python/numpy/distutils/ command/build_src.py", line 149, in build_extension_sources sources = self.f2py_sources(sources, ext) File "/data/jrd/mod1/agn/ext/Linux/lib64/python/numpy/distutils/ command/build_src.py", line 355, in f2py_sources f2py2e.run_main(f2py_options + ['--lower', File "/data/jrd/mod1/agn/ext/Linux/lib64/python/numpy/f2py/ f2py2e.py", line 330, in run_main postlist=callcrackfortran(files,options) File "/data/jrd/mod1/agn/ext/Linux/lib64/python/numpy/f2py/ f2py2e.py", line 269, in callcrackfortran postlist=crackfortran.crackfortran(files) File "/data/jrd/mod1/agn/ext/Linux/lib64/python/numpy/f2py/ crackfortran.py", line 2574, in crackfortran postlist=postcrack(grouplist[0]) File "/data/jrd/mod1/agn/ext/Linux/lib64/python/numpy/f2py/ crackfortran.py", line 1488, in postcrack g=postcrack(g,tab=tab+'\t') File "/data/jrd/mod1/agn/ext/Linux/lib64/python/numpy/f2py/ crackfortran.py", line 1506, in postcrack block['body']=analyzebody(block,args,tab=tab) File "/data/jrd/mod1/agn/ext/Linux/lib64/python/numpy/f2py/ crackfortran.py", line 1648, in analyzebody b=postcrack(b,as,tab=tab+'\t') File "/data/jrd/mod1/agn/ext/Linux/lib64/python/numpy/f2py/ crackfortran.py", line 1506, in postcrack block['body']=analyzebody(block,args,tab=tab) File "/data/jrd/mod1/agn/ext/Linux/lib64/python/numpy/f2py/ crackfortran.py", line 1648, in analyzebody b=postcrack(b,as,tab=tab+'\t') File "/data/jrd/mod1/agn/ext/Linux/lib64/python/numpy/f2py/ crackfortran.py", line 1502, in postcrack block['vars']=analyzevars(block) File "/data/jrd/mod1/agn/ext/Linux/lib64/python/numpy/f2py/ crackfortran.py", line 2007, in analyzevars for k in implicitrules[ln0].keys(): KeyError: '(' From robert.kern at gmail.com Tue Jan 24 08:16:09 2006 From: robert.kern at gmail.com (Robert Kern) Date: Tue Jan 24 08:16:09 2006 Subject: [Numpy-discussion] bdist_egg trips up over non-extension extensions Message-ID: <43D6529C.8030105@gmail.com> In particular, numpy can't build into an egg currently because numpy/__config__.py is created using Configuration.add_extension() and a generator function. In order to make extension modules usable from zipped eggs, setuptools generates a pure Python stub loader that will extract the extension module to a .hidden directory and load it from there. In this case, since the Extension isn't actually an extension, the stub loader is broken, because it expects a numpy/__config__.so. Can we find another way to generate Python code instead of abusing Extension this way? -- Robert Kern robert.kern at gmail.com "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From pearu at scipy.org Tue Jan 24 08:18:02 2006 From: pearu at scipy.org (Pearu Peterson) Date: Tue Jan 24 08:18:02 2006 Subject: [Numpy-discussion] f2py problems In-Reply-To: References: Message-ID: On Tue, 24 Jan 2006, George Nurser wrote: > I'm not sure whether this is the right list for f2py questions, but anyway. > > Tried it (v 2_1967) out today. It worked beautifully for a simple function. > > But for a more complicated (f77) *.F file with preprocessor directives and a > lot of data statements I have been having problems. > > Firstly, it just seemed to ignore all of the lines with preprocessor > directives. f2py does not do preprocessing. > Secondly, after I used a preprocessor to create the .f file separately, and > then ran the .f file through f2py, I got the following error trace. > Anybody have any ideas? > line 2007, in analyzevars > for k in implicitrules[ln0].keys(): > KeyError: '(' Could you send me the file that caused this error? The error may be due to a f2py bug or a non-standard Fortran code. Thanks, Pearu From strawman at astraw.com Tue Jan 24 08:43:03 2006 From: strawman at astraw.com (Andrew Straw) Date: Tue Jan 24 08:43:03 2006 Subject: [Numpy-discussion] bdist_egg trips up over non-extension extensions In-Reply-To: <43D6529C.8030105@gmail.com> References: <43D6529C.8030105@gmail.com> Message-ID: <43D658F4.5030602@astraw.com> Just to be clear, you're not having problems with numpy as a non-zipped egg, just a zipped egg, right? I haven't had any problems, but I haven't tried to make zip-safe eggs. (For those who don't know, eggs can be installed either as a zip filed with extension .egg, or as a directory with extension egg.) Robert Kern wrote: >In particular, numpy can't build into an egg currently because >numpy/__config__.py is created using Configuration.add_extension() and a >generator function. In order to make extension modules usable from zipped eggs, >setuptools generates a pure Python stub loader that will extract the extension >module to a .hidden directory and load it from there. In this case, since the >Extension isn't actually an extension, the stub loader is broken, because it >expects a numpy/__config__.so. > >Can we find another way to generate Python code instead of abusing Extension >this way? > > > From pearu at scipy.org Tue Jan 24 08:46:07 2006 From: pearu at scipy.org (Pearu Peterson) Date: Tue Jan 24 08:46:07 2006 Subject: [Numpy-discussion] bdist_egg trips up over non-extension extensions In-Reply-To: <43D6529C.8030105@gmail.com> References: <43D6529C.8030105@gmail.com> Message-ID: On Tue, 24 Jan 2006, Robert Kern wrote: > In particular, numpy can't build into an egg currently because > numpy/__config__.py is created using Configuration.add_extension() and a > generator function. In order to make extension modules usable from zipped eggs, > setuptools generates a pure Python stub loader that will extract the extension > module to a .hidden directory and load it from there. In this case, since the > Extension isn't actually an extension, the stub loader is broken, because it > expects a numpy/__config__.so. > > Can we find another way to generate Python code instead of abusing Extension > this way? Yes, using py_modules instead of ext_modules would be the correct way to go. I'm looking into this issue right now... Pearu From ndarray at mac.com Tue Jan 24 09:22:06 2006 From: ndarray at mac.com (Sasha) Date: Tue Jan 24 09:22:06 2006 Subject: [Numpy-discussion] Replace typechars considered harmful Message-ID: After a day of debugging I found that convertcode replaced '1' with 'b' in the code which was selecting database columns with names ending with '1'. Since it is probably very difficult to automatically differentiate between '1' used as a typecode and other possible uses, I suggest to either make "replace typechars" optional, or make it write verbose message with a few lines of context when it changes the code. -- sasha From pearu at scipy.org Tue Jan 24 10:32:03 2006 From: pearu at scipy.org (Pearu Peterson) Date: Tue Jan 24 10:32:03 2006 Subject: [Numpy-discussion] bdist_egg trips up over non-extension extensions In-Reply-To: <43D6529C.8030105@gmail.com> References: <43D6529C.8030105@gmail.com> Message-ID: On Tue, 24 Jan 2006, Robert Kern wrote: > In particular, numpy can't build into an egg currently because > numpy/__config__.py is created using Configuration.add_extension() and a > generator function. In order to make extension modules usable from zipped eggs, > setuptools generates a pure Python stub loader that will extract the extension > module to a .hidden directory and load it from there. In this case, since the > Extension isn't actually an extension, the stub loader is broken, because it > expects a numpy/__config__.so. Could you try recent version of numpy from SVN? __config__.py is now generated via py_modules list. Pearu From jmiller at stsci.edu Tue Jan 24 10:46:01 2006 From: jmiller at stsci.edu (Todd Miller) Date: Tue Jan 24 10:46:01 2006 Subject: [Numpy-discussion] Speed performance on array constant set In-Reply-To: <43D59D0B.3060907@mitre.org> References: <43D0573E.4090301@mitre.org> <43D05BA0.7000700@ieee.org> <43D14AC0.3060902@mitre.org> <43D51669.6080308@noaa.gov> <43D57262.6090004@mitre.org> <43D5768C.4090200@ee.byu.edu> <43D59D0B.3060907@mitre.org> Message-ID: <43D675A7.4070906@stsci.edu> Mark Heslep wrote: > Travis Oliphant wrote: > >>> 1. numdata.h example that works well for simple Cv structures >>> containing uniform types: CvPoint2D32F => struct { float x, float >>> y }. Can I use a record array here, or is there some Numpy overhead >>> interlaced with fields? >> >> >> >> You can use a record array, but for uniform types I don't know what >> the advantage is (except for perhaps named-field slicing---which may >> actually be faster I'm not sure) over a x2 float array. > > > So that I can step through the array record by record and not field by > field. > >>> 2. numarray.h Attempt to replace the main Cv Image structures >>> CvMat, IplImage. Needs work. Some success but there's segfault or >>> two in there somewhere. >> >> >> These can be ported fairly easily, I think (actually, the old Numeric >> typemaps would still work --- and work with Numarray), so the basic >> Numeric C-API still presents itself as the simplest way to support >> all the array packages. > > > Well good to know. Ive been proceeding directly from the NumArray > Users Manual 1.5 by Greenfield/Miller et al that describes the NA High > Level API as the fastest of the APIs available to NA. The Numeric compatibility layer in numarray is almost identical in performance with the High Level API. I recommend using the Numeric compatibility layer because it makes it easier to port your application back to Numeric or forward to NumPy. > I thought that NA also took steps to insure that unnecessary mem > copies were not made, unlike Numeric? NA does provide flexibility and support for managing mis-behaved (byteswapped, mis-aligned, mis-typed) arrays, but these extra properties weren't a concern for Numeric. Both numarray's High Level API and numarray's Numeric compatibility layer generally make whole copies of arrays as required to present well behaved representations of data to C. Regards, Todd From robert.kern at gmail.com Tue Jan 24 12:24:12 2006 From: robert.kern at gmail.com (Robert Kern) Date: Tue Jan 24 12:24:12 2006 Subject: [Numpy-discussion] bdist_egg trips up over non-extension extensions In-Reply-To: References: <43D6529C.8030105@gmail.com> Message-ID: <43D68CC4.1010004@gmail.com> Pearu Peterson wrote: > Could you try recent version of numpy from SVN? __config__.py is now > generated via py_modules list. It doesn't build an egg, yet, because setuptools expects Distribution.py_modules to contain only strings, not tuples. Modifying setuptools to handle tuples in Distribution.py_modules seems to work. Is there any way you can do it without tuples? -- Robert Kern robert.kern at gmail.com "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From pearu at scipy.org Tue Jan 24 12:37:05 2006 From: pearu at scipy.org (Pearu Peterson) Date: Tue Jan 24 12:37:05 2006 Subject: [Numpy-discussion] bdist_egg trips up over non-extension extensions In-Reply-To: <43D68CC4.1010004@gmail.com> References: <43D6529C.8030105@gmail.com> <43D68CC4.1010004@gmail.com> Message-ID: On Tue, 24 Jan 2006, Robert Kern wrote: > Pearu Peterson wrote: >> Could you try recent version of numpy from SVN? __config__.py is now >> generated via py_modules list. > > It doesn't build an egg, yet, because setuptools expects Distribution.py_modules > to contain only strings, not tuples. Modifying setuptools to handle tuples in > Distribution.py_modules seems to work. Is there any way you can do it without > tuples? Source generators in numpy.distutils are extensions to distutils and if setuptools assumes std distutils then source generators will remain a problem for setuptools. However, if you call build_src then all source generators will be resolved (including 3-tuples in py_modules list) and the file lists in distribution instance should not cause any problems for setuptools. So, would calling `setup.py build_src bdist_egg` be an acceptable solution? __config__.py files could also be handled as data files but that would require hooks in numpy.core.setup function and we want to get rid of numpy.core.setup in future. Pearu From Norbert.Nemec.list at gmx.de Tue Jan 24 13:11:00 2006 From: Norbert.Nemec.list at gmx.de (Norbert Nemec) Date: Tue Jan 24 13:11:00 2006 Subject: [Numpy-discussion] Matrix/Vector norm? In-Reply-To: <43D0A290.8080600@ieee.org> References: <43D09F8F.6080702@gmx.de> <43D0A290.8080600@ieee.org> Message-ID: <43D69794.8030609@gmx.de> Travis Oliphant wrote: > Norbert Nemec wrote: > >> Hi there, >> >> im missing a norm function in NumPy. Writing one by hand is simple, but >> doing so both general and efficient is a lot more tricky. >> >> Ideas? >> >> > There's one in scipy (i'm not sure if it's the best of breed, but it's > a starting point). > > Look in scipy.linalg.norm Is there a reason not to have it in numpy.linalg? That would be the most logical place for it, I would guess. From rowen at cesmail.net Tue Jan 24 14:01:06 2006 From: rowen at cesmail.net (Russell E. Owen) Date: Tue Jan 24 14:01:06 2006 Subject: [Numpy-discussion] Re: Efficient way to handle None->nan? References: <43D16C31.3060209@ee.byu.edu> Message-ID: In article <43D16C31.3060209 at ee.byu.edu>, Travis Oliphant wrote: > Russell E. Owen wrote: > > >We're getting numeric data from a (MySQL) database. We'd like to use > >numarray or NumPy on the resulting data, but some values may be None. Is > >there a fast, efficient way to replace None with NaN? I'd hate to use a > >list comprehension on each data tuple before turning it into an array, > >but I haven't thought of anything else. > > > > > > Current numpy SVN allows > > array([1.0,None,2.0],dtype=float) > array([ 1. , nan, 2. ]) That's great! Thanks!! -- Russell From agn at noc.soton.ac.uk Tue Jan 24 14:33:07 2006 From: agn at noc.soton.ac.uk (agn at noc.soton.ac.uk) Date: Tue Jan 24 14:33:07 2006 Subject: [Numpy-discussion] dtype=float32 not recognised. In-Reply-To: <43D69794.8030609@gmx.de> References: <43D09F8F.6080702@gmx.de> <43D0A290.8080600@ieee.org> <43D69794.8030609@gmx.de> Message-ID: <1138141838.43d6aa8ee811e@webmail.noc.soton.ac.uk> A curious problem (0.9.5.1993): n [12]: theta3 =numpy.zeros(66,dtype=float32) --------------------------------------------------------------------------- exceptions.NameError Traceback (most recent call last) /working/jrd/mod2/agn/OCCAM/AGN_GLOBAL/ NameError: name 'float32' is not defined But n [14]: theta3 =numpy.zeros(66,dtype='f') In [15]: theta3 Out[15]: array([ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.], dtype=float32) George Nuresr ----------------------------------------------------------------------- National Oceanography Centre, Southampton :: http://www.noc.soton.ac.uk From ndarray at mac.com Tue Jan 24 16:57:05 2006 From: ndarray at mac.com (Sasha) Date: Tue Jan 24 16:57:05 2006 Subject: [Numpy-discussion] dtype=float32 not recognised. In-Reply-To: <1138141838.43d6aa8ee811e@webmail.noc.soton.ac.uk> References: <43D09F8F.6080702@gmx.de> <43D0A290.8080600@ieee.org> <43D69794.8030609@gmx.de> <1138141838.43d6aa8ee811e@webmail.noc.soton.ac.uk> Message-ID: You did not import float32. Your options: >>> from numpy import * >>> zeros(5, dtype=float32) array([ 0., 0., 0., 0., 0.], dtype=float32) or >>> import numpy >>> numpy.zeros(5, dtype=numpy.float32) array([ 0., 0., 0., 0., 0.], dtype=float32) or >>> import numpy >>> numpy.zeros(5, dtype=float) array([ 0., 0., 0., 0., 0.]) -- sasha On 1/24/06, agn at noc.soton.ac.uk wrote: > A curious problem (0.9.5.1993): > > n [12]: theta3 =numpy.zeros(66,dtype=float32) > --------------------------------------------------------------------------- > exceptions.NameError Traceback (most recent call > last) > > /working/jrd/mod2/agn/OCCAM/AGN_GLOBAL/ > > NameError: name 'float32' is not defined > > But > n [14]: theta3 =numpy.zeros(66,dtype='f') > > In [15]: theta3 > Out[15]: > array([ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., > 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., > 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., > 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., > 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.], > dtype=float32) > > George Nuresr > > > ----------------------------------------------------------------------- > National Oceanography Centre, Southampton :: http://www.noc.soton.ac.uk > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From mithrandir42 at web.de Tue Jan 24 23:03:01 2006 From: mithrandir42 at web.de (N. Volbers) Date: Tue Jan 24 23:03:01 2006 Subject: [Numpy-discussion] some question on new dtype Message-ID: <43D7225E.20306@web.de> Hello everyone on the list! I have been playing around with the latest and greatest numpy 0.94 and its dtype mechanism. I am especially interested in using the record-array-like facilities, e.g. the following which is based on an example from a mail of Travis to this list: <-- # define array with three "columns". dtype = numpy.dtype( {'names': ['name', 'age', 'weight'], 'formats': ['U30', 'i2', numpy.float32]} ) a = numpy.array( [(u'Bill', 31, 260), (u'Fred', 15, 135)], dtype=dtype ) # specify column by key print a ['name'] print a['age'] print a['weight'] #print a['object'] # specify row by number print a[0] print a[1] # first item of row 1 (Fred's age) print a[1]['age'] # first item of name column (name 'Bill') print a['name'][0] --> I now have a few questions, maybe someone can help me with them: 1) When reading the sample chapter from Travis' documentation, I noticed that there is also a type 'object' with the character 'O'. So I kind of hoped that it would be possible to have arbitrary python objects in an array. However, when I add a fourth "column" of type 'O', then numpy will mem-fault. Is this not allowed or is this some implementation bug? 2) Is it possible to rename the type descriptors? For my application, I need to treat these names as keys of dataset columns, so it should be possible to rename these. More generally speaking: Is it possible to alter parts of the dtype after instantiation? Of course it should be possible to copy the dtype, modify it accordingly and create a new array. However, maybe there is a suggested way to doing this? 3) When I use two identical entries in the names part of the dtype, I get the message 'TypeError: expected a readable buffer object'. It makes sense that it is not allowed to have two identical names, but I think the error message should be worded more descriptive. 4) In the example above, printing any of the strings via 'print' will yield the characters and then the characters up to the string size filled up with \x00, e.g. u'Bill\x00\x00\x00\x00\x00\x00\x00.... (30 characters total)' Why doesn't 'prin't terminate the output when the first \x00 is reached ? Overall I am very much impressed by the new numpy and I thank everyone who contributes to this! Niklas Volbers. From oliphant.travis at ieee.org Tue Jan 24 23:34:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue Jan 24 23:34:02 2006 Subject: [Numpy-discussion] some question on new dtype In-Reply-To: <43D7225E.20306@web.de> References: <43D7225E.20306@web.de> Message-ID: <43D729D9.6020608@ieee.org> N. Volbers wrote: > Hello everyone on the list! > > I have been playing around with the latest and greatest numpy 0.94 and > its dtype mechanism. I am especially interested in using the > record-array-like facilities, e.g. the following which is based on an > example from a mail of Travis to this list: > > <-- > # define array with three "columns". > dtype = numpy.dtype( {'names': ['name', 'age', 'weight'], > 'formats': ['U30', 'i2', numpy.float32]} ) > a = numpy.array( [(u'Bill', 31, 260), (u'Fred', 15, 135)], dtype=dtype ) > > # specify column by key > print a ['name'] > print a['age'] > print a['weight'] > #print a['object'] > > # specify row by number > print a[0] > print a[1] > > # first item of row 1 (Fred's age) > print a[1]['age'] > > # first item of name column (name 'Bill') > print a['name'][0] > --> > > I now have a few questions, maybe someone can help me with them: > > 1) When reading the sample chapter from Travis' documentation, I > noticed that there is also a type 'object' with the character 'O'. So > I kind of hoped that it would be possible to have arbitrary python > objects in an array. However, when I add a fourth "column" of type > 'O', then numpy will mem-fault. Is this not allowed or is this some > implementation bug? It's a bug if it seg-faults, that should be allowed. Please post your code so we can track it down. > > 2) Is it possible to rename the type descriptors? For my application, > I need to treat these names as keys of dataset columns, so it should > be possible to rename these. More generally speaking: Is it possible > to alter parts of the dtype after instantiation? Of course it should > be possible to copy the dtype, modify it accordingly and create a new > array. However, maybe there is a suggested way to doing this? Right now, you would have to construct a new data-type with the new field names, There may be some ways we could make this easier, though. The big thing is that you can't change data-types in-place because they are used by other array objects. Perhaps the easiest way to do this is by getting the arrdescr attribute of the dtype (it's a list of tuples) and then constructing a new list of tuples using it (replaceing the tuples you want to rename) and then using that in the dtype constructor: Something like this: descriptor = a.dtype.arrdescr # this could probably be renamed to descr now that we're # not using that word. # or a.__array_descr__ retrieves the same thing. field = descriptor[field_num] descriptor[field_num] = (newname,)+field[1:] newdtype = numpy.dtype(descriptor) Then you can do: a.dtype = newdtype and have a new field name. Actually, it would probably be faster to do: new = dict(a.dtype.fields) # get a writeable dictionary. new[''] = new[''] del new[''] del new[-1] # get rid of the special ordering entry a.dtype = dtype(new) > > 3) When I use two identical entries in the names part of the dtype, I > get the message 'TypeError: expected a readable buffer object'. It > makes sense that it is not allowed to have two identical names, but I > think the error message should be worded more descriptive. Yeah, we ought to check for this. Could you post your code that raises this error. > > 4) In the example above, printing any of the strings via 'print' will > yield the characters and then the characters up to the string size > filled up with \x00, e.g. > > u'Bill\x00\x00\x00\x00\x00\x00\x00.... (30 characters total)' > > Why doesn't 'prin't terminate the output when the first \x00 is reached ? Because I don't know the Unicode equivalent to PyString_FromString (which is what I'm using to automatically truncate the fixed-field string before printing it). UNICODE_getitem in arraytypes.inc.src is the relevant function. See STRING_getitem for comparison. I think it would be better to truncate it if anybody has a good suggestion... -Travis From oliphant.travis at ieee.org Wed Jan 25 00:24:01 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed Jan 25 00:24:01 2006 Subject: [Numpy-discussion] some question on new dtype In-Reply-To: <43D7225E.20306@web.de> References: <43D7225E.20306@web.de> Message-ID: <43D73570.4070201@ieee.org> N. Volbers wrote: > 4) In the example above, printing any of the strings via 'print' will > yield the characters and then the characters up to the string size > filled up with \x00, e.g. > > u'Bill\x00\x00\x00\x00\x00\x00\x00.... (30 characters total)' > > Why doesn't 'prin't terminate the output when the first \x00 is reached ? O.K. I finally broke down and fixed this in the SVN version. Perhaps people can try it out for bugs. -Travis From faltet at carabos.com Wed Jan 25 00:26:03 2006 From: faltet at carabos.com (Francesc Altet) Date: Wed Jan 25 00:26:03 2006 Subject: [Numpy-discussion] dtype=float32 not recognised. In-Reply-To: References: <43D09F8F.6080702@gmx.de> <1138141838.43d6aa8ee811e@webmail.noc.soton.ac.uk> Message-ID: <200601250925.42792.faltet@carabos.com> A Dimecres 25 Gener 2006 01:56, Sasha va escriure: > You did not import float32. > > Your options: > >>> from numpy import * > >>> zeros(5, dtype=float32) > > array([ 0., 0., 0., 0., 0.], dtype=float32) > > or > > >>> import numpy > >>> numpy.zeros(5, dtype=numpy.float32) > > array([ 0., 0., 0., 0., 0.], dtype=float32) > > or > > >>> import numpy > >>> numpy.zeros(5, dtype=float) > > array([ 0., 0., 0., 0., 0.]) Another one: >>> import numpy >>> numpy.zeros(5, dtype="float32") array([ 0., 0., 0., 0., 0.], dtype=float32) -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From faltet at carabos.com Wed Jan 25 00:41:11 2006 From: faltet at carabos.com (Francesc Altet) Date: Wed Jan 25 00:41:11 2006 Subject: [Numpy-discussion] some question on new dtype In-Reply-To: <43D729D9.6020608@ieee.org> References: <43D7225E.20306@web.de> <43D729D9.6020608@ieee.org> Message-ID: <200601250940.34655.faltet@carabos.com> A Dimecres 25 Gener 2006 08:33, Travis Oliphant va escriure: > Something like this: > > descriptor = a.dtype.arrdescr # this could probably be renamed to > # descr now that we're > # not using that word. +1 for this That way we should have (in case of a int32 type): >>> dtype.num 11 >>> dtype.char 'f' >>> dtype.str '>> dtype.name 'float32' >>> dtype.descr [('', '0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From a.h.jaffe at gmail.com Wed Jan 25 01:57:09 2006 From: a.h.jaffe at gmail.com (Andrew Jaffe) Date: Wed Jan 25 01:57:09 2006 Subject: [Numpy-discussion] Weird (wrong?) real fft 2d behavior Message-ID: <43D74B35.9060204@gmail.com> If I start with what I thought was an appropriate (n, n/2+1) complex matrix which should have a real inverse fft, and then take its real fft, I don't get back the original matrix. Note the presence of very small but nonzero reals in the final matrix, and the fact that the 2d and 4th rows are duplicates. This seems to be a mistake somewhere. Or am I just misunderstanding/misremembering something about 2d real ffts? Andrew In [41]: dims=(4,4) In [42]: delta_k = aniso.dft_realizn(dims, P) #### this just makes an appropriate matrix In [43]: delta_k Out[43]: array([[-0. +0.j , 0.5904+0.0429j, 0.9063+0.j ], [-0.221 +0.j , 0.4169+0.4602j, -0.6909+0.j ], [ 0.9059+0.j , -0.8037+0.2604j, 1.835 +0.j ], [ 1.7436+0.j , -0.2382+0.221j , -0.0607+0.j ]]) #### 16 real numbers In [44]: delta_r = N.dft.inverse_real_fft2d(delta_k) In [45]: delta_r Out[45]: array([[ 0.2718, -0.0956, 0.2805, 0.1505], [ 0.0297, -0.0533, -0.259 , 0.0561], [ 0.1308, -0.2096, 0.2288, -0.3041], [ 0.0895, 0.1105, -0.3188, -0.1076]]) In [46]: delta_kp = N.dft.real_fft2d(delta_r) In [47]: delta_kp Out[47]: array([[ 0. +0.00e+00j, 0.5904 +4.2938e-02j, 0.9063 +0.0000e+00j], [ 0.7613 +5.5511e-17j,0.4169 +4.6020e-01j, -0.3758 +5.5511e-17j], [ 0.9059 +0.0000e+00j,-0.8037+2.6038e-01j, 1.835 +0.0000e+00j], [ 0.7613 -5.5511e-17j,-0.2382+2.2099e-01j,-0.3758-5.5511e-17j]]) In [48]: N.__version__ Out[48]: '0.9.5.1982' ------------------------ From a.h.jaffe at gmail.com Wed Jan 25 02:05:03 2006 From: a.h.jaffe at gmail.com (Andrew Jaffe) Date: Wed Jan 25 02:05:03 2006 Subject: [Numpy-discussion] Re: Weird (wrong?) real fft 2d behavior In-Reply-To: <43D74B35.9060204@gmail.com> References: <43D74B35.9060204@gmail.com> Message-ID: <43D74D17.7010103@gmail.com> Andrew Jaffe wrote: > If I start with what I thought was an appropriate (n, n/2+1) complex > matrix which should have a real inverse fft, and then take its real fft, > I don't get back the original matrix. > > Note the presence of very small but nonzero reals in the final matrix, > and the fact that the 2d and 4th rows are duplicates. This seems to be a > mistake somewhere. > > Or am I just misunderstanding/misremembering something about 2d real ffts? and I should point out that delta_rp = N.dft.real_fft2d(delta_kp) is 'allclose' to the original delta_r (which leads me to believe that I may indeed be misunderstanding something). Andrew > > Andrew > > > In [41]: dims=(4,4) > > In [42]: delta_k = aniso.dft_realizn(dims, P) > #### this just makes an appropriate matrix > > In [43]: delta_k > Out[43]: > array([[-0. +0.j , 0.5904+0.0429j, 0.9063+0.j ], > [-0.221 +0.j , 0.4169+0.4602j, -0.6909+0.j ], > [ 0.9059+0.j , -0.8037+0.2604j, 1.835 +0.j ], > [ 1.7436+0.j , -0.2382+0.221j , -0.0607+0.j ]]) > #### 16 real numbers > > In [44]: delta_r = N.dft.inverse_real_fft2d(delta_k) > > In [45]: delta_r > Out[45]: > array([[ 0.2718, -0.0956, 0.2805, 0.1505], > [ 0.0297, -0.0533, -0.259 , 0.0561], > [ 0.1308, -0.2096, 0.2288, -0.3041], > [ 0.0895, 0.1105, -0.3188, -0.1076]]) > > In [46]: delta_kp = N.dft.real_fft2d(delta_r) > > In [47]: delta_kp > Out[47]: > array([[ 0. +0.00e+00j, 0.5904 +4.2938e-02j, 0.9063 +0.0000e+00j], > [ 0.7613 +5.5511e-17j,0.4169 +4.6020e-01j, -0.3758 +5.5511e-17j], > [ 0.9059 +0.0000e+00j,-0.8037+2.6038e-01j, 1.835 +0.0000e+00j], > [ 0.7613 -5.5511e-17j,-0.2382+2.2099e-01j,-0.3758-5.5511e-17j]]) > > In [48]: N.__version__ > Out[48]: '0.9.5.1982' > ------------------------ From faltet at carabos.com Wed Jan 25 04:16:02 2006 From: faltet at carabos.com (Francesc Altet) Date: Wed Jan 25 04:16:02 2006 Subject: [Numpy-discussion] Creating multirow numpy rec.arrays Message-ID: <200601251314.54933.faltet@carabos.com> Hi, This exposes a weird behaviour when building heterogeneous arrays in numpy: In [85]: xcol = numpy.ones((3,2), 'int32') In [86]: numpy.array([(xcol[0],)], dtype=[('x','i4', (2,))]) Out[86]: array([(array([1, 1]),)], dtype=(void,8)) So far so good. Now: In [89]: numpy.array([xcol], dtype=[('x','i4', (2,))]) array([[[(array([-1208633192, -1208633192]),), (array([15, 15]),)], [(array([ 0, 185]),), (array([ 18, -1208633016]),)], [(array([480, 21]),), (array([21, 21]),)]]], dtype=(void,8)) While I'd expect: In [89]: numpy.array([xcol], dtype=[('x','i4', (2,))]) Out[89]: array([(array([1, 1]), (array([1, 1]), (array([1, 1]))], dtype=(void,8)) Also, for the whishlist: It would be nice if one can specify the shape of the array to be built in the factory itself. I mean, allowing something like: In [90]: numpy.array([xcol], dtype=[('x','i4', (2,))], shape=2) Out[90]: array([(array([1, 1]), (array([1, 1]))], dtype=(void,8)) Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From borreguero at gmail.com Wed Jan 25 04:53:05 2006 From: borreguero at gmail.com (Jose Borreguero) Date: Wed Jan 25 04:53:05 2006 Subject: [Numpy-discussion] installation under other than .../site-packages Message-ID: <7cced4ed0601250452x197f0d08i@mail.gmail.com> My system administrator has told me numpy has to be installed under /local , instead of under /usr/lib/pythonx.x/site-packages. Can anybody point to me what modifications do I have to make for this. ? Also, will other libraries that call numpy work after I install ? jose -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.marti at upf.edu Wed Jan 25 08:35:01 2006 From: daniel.marti at upf.edu (Dani Marti) Date: Wed Jan 25 08:35:01 2006 Subject: [Numpy-discussion] [Newbie] Matching a subarray Message-ID: <20060125162603.GA1765@upf.edu> Hi all, I have a long 1D boolean array. I'd like to know where a certain 'sub'array (like, e.g., ([1,1,1,1,1,1])) occurs in the array. That is, I want the index of the first occurrence of a subarray in an array, avoiding the use of explicit loop structures to carry out the search. I am sorry if this happens to be a trivial question, but I was not able to find the answer from the documentation. Thanks a lot, dani From chris at trichech.us Wed Jan 25 09:23:10 2006 From: chris at trichech.us (Christopher Fonnesbeck) Date: Wed Jan 25 09:23:10 2006 Subject: [Numpy-discussion] error in beta random variate generation Message-ID: There appears to be a bug in the beta random number generator in numpy. A 2-parameter beta distribution has an expected value of a/(a+b). Hence, a beta(1,1) should have an expected value of 0.5 (since it is the equivalent of a uniform random variable on [0,1]). However, numpy gives the following: In [6]: from numpy.random import beta In [7]: from numpy import * In [8]: mean(beta(1,1,size=10000)) Out[8]: 0.33397400928538834 -- Christopher J. Fonnesbeck Population Ecologist, Marine Mammal Section Fish & Wildlife Research Institute (FWC) St. Petersburg, FL Adjunct Assistant Professor Warnell School of Forest Resources University of Georgia Athens, GA T: 727.235.5570 E: chris at trichech.us -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2417 bytes Desc: not available URL: From oliphant.travis at ieee.org Wed Jan 25 10:47:08 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed Jan 25 10:47:08 2006 Subject: [Numpy-discussion] installation under other than .../site-packages In-Reply-To: <7cced4ed0601250452x197f0d08i@mail.gmail.com> References: <7cced4ed0601250452x197f0d08i@mail.gmail.com> Message-ID: <43D7C791.7030106@ieee.org> Jose Borreguero wrote: > My system administrator has told me numpy has to be installed under > /local , instead of under /usr/lib/pythonx.x/site-packages. Can > anybody point to me what modifications do I have to make for this. ? > Also, will other libraries that call numpy work after I install ? > jose > As long as you change the sys.path list and insert the location where numpy is installed, it should work and other packages should find it. I think modification of the PYTHONPATH environment variable will do this for all scripts as well. -Travis From cookedm at physics.mcmaster.ca Wed Jan 25 11:05:00 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Wed Jan 25 11:05:00 2006 Subject: [Numpy-discussion] installation under other than .../site-packages In-Reply-To: <7cced4ed0601250452x197f0d08i@mail.gmail.com> References: <7cced4ed0601250452x197f0d08i@mail.gmail.com> Message-ID: On Jan 25, 2006, at 07:52 , Jose Borreguero wrote: > My system administrator has told me numpy has to be installed > under /local , instead of under /usr/lib/pythonx.x/site-packages. > Can anybody point to me what modifications do I have to make for > this. ? Also, will other libraries that call numpy work after I > install ? > jose First, to install it under /usr/local, run setup.py like this: $ python setup.py --prefix=/usr/local This will put numpy into /usr/local/lib/pythonx.x/site-packages. Next, check if you can import it. If you can't, it's likely that that path is not in your sys.path. Probably the easiest way to make it available for _all_ users is to ask your sysadmin to put a file called 'usrlocal.pth' in /usr/lib/pythonx.x/site-packages with these contents import site; site.addsitedir('/usr/local/lib/pythonx.x') (hopefully he'd be amiable to adding only one file :) This adds /usr/ local/lib/pythonx.x to the sys.path *and* processes any .pth in it. You can get fancier and add import os, site; site.addsitedir(os.path.expanduser('~/lib/pythonx.x')) which would also look in user directories. Failing that, you'd set the environment variable PYTHONPATH to include the right directory. This way, .pth files won't be processed. You could add a sitecustomize.py with the appropriate 'import site; site.addsitedir(...)'. The addsitedir is what processes .pth files. These are all generic instructions, not specific to numpy. numpy itself doesn't need .pth files, but there are other packages that do (Numeric, PIL, pygtk, amongst others), so it's handy to have. -- |>|\/|< /------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From oliphant at ee.byu.edu Wed Jan 25 11:41:08 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed Jan 25 11:41:08 2006 Subject: [Numpy-discussion] [Newbie] Matching a subarray In-Reply-To: <20060125162603.GA1765@upf.edu> References: <20060125162603.GA1765@upf.edu> Message-ID: <43D7D418.9090802@ee.byu.edu> Dani Marti wrote: >Hi all, > >I have a long 1D boolean array. I'd like to know where a certain >'sub'array (like, e.g., ([1,1,1,1,1,1])) occurs in the array. That is, I >want the index of the first occurrence of a subarray in an array, >avoiding the use of explicit loop structures to carry out the search. > > Actually, I don't think that's a trivial question. You could look at the data as a string and use the find method of the string. Or you could use (int8) correlation. The largest value of the correlation should be the index where it matches. -Travis From mithrandir42 at web.de Wed Jan 25 12:08:02 2006 From: mithrandir42 at web.de (N. Volbers) Date: Wed Jan 25 12:08:02 2006 Subject: [Numpy-discussion] some question on new dtype In-Reply-To: <43D729D9.6020608@ieee.org> References: <43D7225E.20306@web.de> <43D729D9.6020608@ieee.org> Message-ID: <43D7DA54.10307@web.de> Thanks for your quick answer! >> 1) When reading the sample chapter from Travis' documentation, I >> noticed that there is also a type 'object' with the character 'O'. So >> I kind of hoped that it would be possible to have arbitrary python >> objects in an array. However, when I add a fourth "column" of type >> 'O', then numpy will mem-fault. Is this not allowed or is this some >> implementation bug? > > > It's a bug if it seg-faults, that should be allowed. Please post your > code so we can track it down. Attachment 1 contains a four-liner that reproduces the segfault. But maybe there is something wrong on my system, because I also get the following message when importing numpy: "import linalg -> failed: /usr/lib/python2.4/site-packages/numpy/linalg/lapack_lite.so: undefined symbol: s_wsfe" I removed numpy, checked out from SVN, re-installed and the error is still there. Regarding my question about changing dtypes: Your sample code... > new = dict(a.dtype.fields) # get a writeable dictionary. > new[''] = new[''] > del new[''] > del new[-1] # get rid of the special ordering entry > a.dtype = dtype(new) > ... is short and exactly what I asked for. >> >> 3) When I use two identical entries in the names part of the dtype, I >> get the message 'TypeError: expected a readable buffer object'. It >> makes sense that it is not allowed to have two identical names, but I >> think the error message should be worded more descriptive. > > > Yeah, we ought to check for this. Could you post your code that > raises this error. It seems you have already fixed the error message for the case in which you have two identical names for dtype entries. Man, you are fast! But... ;-) you forgot the situation described in my second code attachment, which is the following situation: >>> dtype = numpy.dtype( [('name', 'S30'), ('name', 'i2')] ) >>> a = numpy.zeros( (4,), dtype=dtype) This works, but leads to silly results. The last instance of 'name' determines the field type, i.e. a[0] = ('Niklas', 1) is invalid, while a[0] = (1,1) is valid! >> 4) In the example above, printing any of the strings via 'print' will >> yield the characters and then the characters up to the string size >> filled up with \x00, e.g. >> >> u'Bill\x00\x00\x00\x00\x00\x00\x00.... (30 characters total)' >> >> Why doesn't 'prin't terminate the output when the first \x00 is reached ? > Thanks for your bug-fix. Unicode display works now. Best regards, Niklas Volbers. -------------- next part -------------- A non-text attachment was scrubbed... Name: dtype_test_1.py Type: text/x-python Size: 98 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: dtype_test_2.py Type: text/x-python Size: 484 bytes Desc: not available URL: From mithrandir42 at web.de Wed Jan 25 12:37:04 2006 From: mithrandir42 at web.de (N. Volbers) Date: Wed Jan 25 12:37:04 2006 Subject: [Numpy-discussion] iterating over array items Message-ID: <43D7E149.9040909@web.de> Hello everyone, the attached source code contains iterations on elements in a numpy array: The first one (for row in a) works, the second one (for item in row) does not. The non-numpy equivalent with lists and tuples works however. Best regards, Niklas Volbers. -------------- next part -------------- A non-text attachment was scrubbed... Name: iterating.py Type: text/x-python Size: 586 bytes Desc: not available URL: From oliphant at ee.byu.edu Wed Jan 25 13:37:03 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed Jan 25 13:37:03 2006 Subject: [Numpy-discussion] some question on new dtype In-Reply-To: <43D7DA54.10307@web.de> References: <43D7225E.20306@web.de> <43D729D9.6020608@ieee.org> <43D7DA54.10307@web.de> Message-ID: <43D7EF5B.8050808@ee.byu.edu> N. Volbers wrote: > Thanks for your quick answer! > >>> 1) When reading the sample chapter from Travis' documentation, I >>> noticed that there is also a type 'object' with the character 'O'. >>> So I kind of hoped that it would be possible to have arbitrary >>> python objects in an array. However, when I add a fourth "column" of >>> type 'O', then numpy will mem-fault. Is this not allowed or is this >>> some implementation bug? >> >> >> >> It's a bug if it seg-faults, that should be allowed. Please post >> your code so we can track it down. > > > Attachment 1 contains a four-liner that reproduces the segfault. But > maybe there is something wrong on my system, because I also get the > following message when importing numpy: "import linalg -> failed: > /usr/lib/python2.4/site-packages/numpy/linalg/lapack_lite.so: > undefined symbol: s_wsfe" > > I removed numpy, checked out from SVN, re-installed and the error is > still there. > > Regarding my question about changing dtypes: Your sample code... > >> new = dict(a.dtype.fields) # get a writeable dictionary. >> new[''] = new[''] >> del new[''] >> del new[-1] # get rid of the special ordering entry >> a.dtype = dtype(new) >> > ... is short and exactly what I asked for. > >>> >>> 3) When I use two identical entries in the names part of the dtype, >>> I get the message 'TypeError: expected a readable buffer object'. It >>> makes sense that it is not allowed to have two identical names, but >>> I think the error message should be worded more descriptive. >> >> >> >> Yeah, we ought to check for this. Could you post your code that >> raises this error. > > > It seems you have already fixed the error message for the case in > which you have two identical names for dtype entries. Man, you are > fast! But... ;-) you forgot the situation described in my second code > attachment, which is the following situation: > > >>> dtype = numpy.dtype( [('name', 'S30'), ('name', 'i2')] ) > >>> a = numpy.zeros( (4,), dtype=dtype) > > This works, but leads to silly results. The last instance of 'name' > determines the field type, i.e. a[0] = ('Niklas', 1) is invalid, while > a[0] = (1,1) is valid! > >>> 4) In the example above, printing any of the strings via 'print' >>> will yield the characters and then the characters up to the string >>> size filled up with \x00, e.g. >>> >>> u'Bill\x00\x00\x00\x00\x00\x00\x00.... (30 characters total)' >>> >>> Why doesn't 'prin't terminate the output when the first \x00 is >>> reached ? >> >> > > Thanks for your bug-fix. Unicode display works now. > > > Best regards, > > Niklas Volbers. > >------------------------------------------------------------------------ > >import numpy >dtype = numpy.dtype( [('object', 'O')] ) >a = numpy.zeros( (4,), dtype=dtype) >print a > > > > I know what the problem is. Too many places, I did not account for the fact that a VOID array could in-fact have OBJECT sub-fields. Thus, any special processing on OBJECT subfields to handle reference counts must also be done on possible sub-fields. This may take a while to fix properly.... If anyone has any ideas, let me know. -Travis From oliphant at ee.byu.edu Wed Jan 25 15:03:06 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed Jan 25 15:03:06 2006 Subject: [Numpy-discussion] some question on new dtype In-Reply-To: <43D7EF5B.8050808@ee.byu.edu> References: <43D7225E.20306@web.de> <43D729D9.6020608@ieee.org> <43D7DA54.10307@web.de> <43D7EF5B.8050808@ee.byu.edu> Message-ID: <43D80371.70103@ee.byu.edu> Travis Oliphant wrote: > I know what the problem is. Too many places, I did not account for > the fact that a VOID array could in-fact have OBJECT sub-fields. > Thus, any special processing on OBJECT subfields to handle reference > counts must also be done on possible sub-fields. > This may take a while to fix properly.... If anyone has any ideas, > let me know. > -Travis For now, I've added a hasobject member to the dtype object structure and filled it in properly on construction. Then, I disable construction of such arrays so that we don't get the segfault. I think this is too harsh. I think at some point we can offer some support for them (there may be a few methods and functions that don't work and we can give an error on those particular ones rather than not allow their construction). -Travis From MTHursts at cisp.cc Wed Jan 25 16:11:19 2006 From: MTHursts at cisp.cc (Lane Margarita) Date: Wed Jan 25 16:11:19 2006 Subject: [Numpy-discussion] Re: penicillin Message-ID: <001401c6220c$f4d94730$6acd1153@m4> fill Very little, sir, I am afraid; I answered, speaking to him as rigid. Mr. Spenlow shut the door, motioned me to a chair, and obligatory Debates, to cool. It was one of the irons I began to heat Very little, sir, I am afraid; I answered, speaking to him as -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: lvwbleeding.gif Type: image/gif Size: 14848 bytes Desc: not available URL: From rudolphv at gmail.com Thu Jan 26 00:45:01 2006 From: rudolphv at gmail.com (Rudolph van der Merwe) Date: Thu Jan 26 00:45:01 2006 Subject: [Numpy-discussion] Possible typo in numpy.dft.fft docstring Message-ID: <97670e910601260044v171cbd38i8b42b15fbeef44ee@mail.gmail.com> Part of the fft function's docstring currently reads: "... The packing of the result is "standard": If A = fft(a, n), then A[0] contains the zero-frequency term, A[1:n/2+1] contains the positive-frequency terms, and A[n/2+1:] contains the negative-frequency terms, in order of decreasingly negative frequency. So for an 8-point transform, the frequencies of the result are [ 0, 1, 2, 3, 4, -3, -2, -1]. ..." Shouldn't the frequency arrangement in the last sentence be [0, 1, 2, 3, -4, -3, -2, -1] and not [ 0, 1, 2, 3, 4, -3, -2, -1] ? -- Rudolph van der Merwe From oliphant.travis at ieee.org Thu Jan 26 06:40:08 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu Jan 26 06:40:08 2006 Subject: [Numpy-discussion] Possible typo in numpy.dft.fft docstring In-Reply-To: <97670e910601260044v171cbd38i8b42b15fbeef44ee@mail.gmail.com> References: <97670e910601260044v171cbd38i8b42b15fbeef44ee@mail.gmail.com> Message-ID: <43D8DF01.9020607@ieee.org> Rudolph van der Merwe wrote: >Part of the fft function's docstring currently reads: > >"... > >The packing of the result is "standard": If A = fft(a, n), then A[0] > contains the zero-frequency term, A[1:n/2+1] contains the > positive-frequency terms, and A[n/2+1:] contains the negative-frequency > terms, in order of decreasingly negative frequency. So for an 8-point > transform, the frequencies of the result are [ 0, 1, 2, 3, 4, -3, -2, -1]. > >..." > >Shouldn't the frequency arrangement in the last sentence be > >[0, 1, 2, 3, -4, -3, -2, -1] > > > >and not > >[ 0, 1, 2, 3, 4, -3, -2, -1] > > Of course these are completely equivalent because of the periodicity of the DFT, but the statement A[1:n/2+1] contains the positive-frequency terms is consistent with what's given. It think the issue is that this statement is true for both even add odd values of n, while writing it the way you propose would require different statements for odd and even terms. I do typically interpret the middle frequency as "negative" though (as does fftshift). -Travis From focke at slac.stanford.edu Thu Jan 26 18:36:10 2006 From: focke at slac.stanford.edu (Warren Focke) Date: Thu Jan 26 18:36:10 2006 Subject: [Numpy-discussion] Re: Weird (wrong?) real fft 2d behavior In-Reply-To: <43D74D17.7010103@gmail.com> References: <43D74B35.9060204@gmail.com> <43D74D17.7010103@gmail.com> Message-ID: On Wed, 25 Jan 2006, Andrew Jaffe wrote: > Andrew Jaffe wrote: > > If I start with what I thought was an appropriate (n, n/2+1) complex > > matrix which should have a real inverse fft, and then take its real fft, > > I don't get back the original matrix. > > > > Note the presence of very small but nonzero reals in the final matrix, That's just roundoff. > > and the fact that the 2d and 4th rows are duplicates. This seems to be a > > mistake somewhere. > > > > Or am I just misunderstanding/misremembering something about 2d real ffts? It looks wrong to me, and I think I wrote those functions. I get the same results in Numeric. I'll try to look into the problem. > and I should point out that > delta_rp = N.dft.real_fft2d(delta_kp) > is 'allclose' to the original delta_r (which leads me to believe that I > may indeed be misunderstanding something). "Stable" does not neccessarily imply "correct". w > > Andrew > > > > > > Andrew > > > > > > In [41]: dims=(4,4) > > > > In [42]: delta_k = aniso.dft_realizn(dims, P) > > #### this just makes an appropriate matrix > > > > In [43]: delta_k > > Out[43]: > > array([[-0. +0.j , 0.5904+0.0429j, 0.9063+0.j ], > > [-0.221 +0.j , 0.4169+0.4602j, -0.6909+0.j ], > > [ 0.9059+0.j , -0.8037+0.2604j, 1.835 +0.j ], > > [ 1.7436+0.j , -0.2382+0.221j , -0.0607+0.j ]]) > > #### 16 real numbers > > > > In [44]: delta_r = N.dft.inverse_real_fft2d(delta_k) > > > > In [45]: delta_r > > Out[45]: > > array([[ 0.2718, -0.0956, 0.2805, 0.1505], > > [ 0.0297, -0.0533, -0.259 , 0.0561], > > [ 0.1308, -0.2096, 0.2288, -0.3041], > > [ 0.0895, 0.1105, -0.3188, -0.1076]]) > > > > In [46]: delta_kp = N.dft.real_fft2d(delta_r) > > > > In [47]: delta_kp > > Out[47]: > > array([[ 0. +0.00e+00j, 0.5904 +4.2938e-02j, 0.9063 +0.0000e+00j], > > [ 0.7613 +5.5511e-17j,0.4169 +4.6020e-01j, -0.3758 +5.5511e-17j], > > [ 0.9059 +0.0000e+00j,-0.8037+2.6038e-01j, 1.835 +0.0000e+00j], > > [ 0.7613 -5.5511e-17j,-0.2382+2.2099e-01j,-0.3758-5.5511e-17j]]) > > > > In [48]: N.__version__ > > Out[48]: '0.9.5.1982' > > ------------------------ > > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From humufr at yahoo.fr Thu Jan 26 20:05:09 2006 From: humufr at yahoo.fr (Humufr) Date: Thu Jan 26 20:05:09 2006 Subject: [Numpy-discussion] installation under other than .../site-packages In-Reply-To: References: <7cced4ed0601250452x197f0d08i@mail.gmail.com> Message-ID: <43D7DB16.5080104@yahoo.fr> This solution is the one I'm using. Unfortunatly it's not working good with scipy. I tried to install scipy in my own directory and everythong seems to work like a charm but the lapack link is bad. I tried everything write in the wiki to install your own BLAS/LAPACK library but scipy want tu use the one of the system who have a problem. I'm trying to find a solution but until I found it numpy/scipy is not an option. N. David M. Cooke wrote: > On Jan 25, 2006, at 07:52 , Jose Borreguero wrote: > >> My system administrator has told me numpy has to be installed under >> /local , instead of under /usr/lib/pythonx.x/site-packages. Can >> anybody point to me what modifications do I have to make for this. ? >> Also, will other libraries that call numpy work after I install ? >> jose > > First, to install it under /usr/local, run setup.py like this: > > $ python setup.py --prefix=/usr/local > > This will put numpy into /usr/local/lib/pythonx.x/site-packages. > > Next, check if you can import it. If you can't, it's likely that that > path is not in your sys.path. Probably the easiest way to make it > available for _all_ users is to ask your sysadmin to put a file > called 'usrlocal.pth' in /usr/lib/pythonx.x/site-packages with these > contents > > import site; site.addsitedir('/usr/local/lib/pythonx.x') > > (hopefully he'd be amiable to adding only one file :) This adds /usr/ > local/lib/pythonx.x to the sys.path *and* processes any .pth in it. > You can get fancier and add > > import os, site; site.addsitedir(os.path.expanduser('~/lib/pythonx.x')) > > which would also look in user directories. > > Failing that, you'd set the environment variable PYTHONPATH to > include the right directory. This way, .pth files won't be processed. > You could add a sitecustomize.py with the appropriate 'import site; > site.addsitedir(...)'. The addsitedir is what processes .pth files. > > These are all generic instructions, not specific to numpy. numpy > itself doesn't need .pth files, but there are other packages that do > (Numeric, PIL, pygtk, amongst others), so it's handy to have. > From ashen at la.com Thu Jan 26 20:48:01 2006 From: ashen at la.com (=?windows-1251?B?MzEg/+3i4PD/?=) Date: Thu Jan 26 20:48:01 2006 Subject: [Numpy-discussion] =?windows-1251?B?0fLw7ujy5ev87fvlIO7w4+Dt6Ofg9ujoLiDB8/Xj4Ovy5fDx6ujpIPP35fIg6CDt?= =?windows-1251?B?4Ovu4+7u4evu5uXt6OU=?= Message-ID: <075301c62225$0a430df9$43599232@la.com> 31 ?????? ???????????? ???????????.????????????? ???? ? ??????????????? ???????????????? ?????????????: ????????? ?.?. - ???????????? ??????????? ?????? ?? ??????????? ?????????????? ????? ???????????? ???????? ??? ??? ??????, ??????????? ???????? ??? ?????-??????, ????????? ???????: 1 ???? (8 ?????), 10.00-17.00 ?????????? ?????????: ?????????? ? ????????? ???????????? ?????????????? ???????. ????????????? ???????????????? ??????????? ???????? ?????????? ??? ?? 10 ?????.??? ?????? ?? ???????? ? ????????? ?????????????? ?????????? ??????????? ?? ?????????: (095) 1?6-48-65, 1?4-97-?9 ? ?????????: ????? ??????? ???????????? ????????????: ???????? ???????????? ???????????? ? ?????????? ??????????????? ????? ????. ???????? ?????????? ????????? ????? ?????????? ???????????? ????????????. ????????????? ??????????? ??????????? ????????, ??????? ?????????????. ??????? ??????????? ????????????? ?????????????????? ?????????? ? ??????????? ?????????????. ?????????? ????????? ????????? ? ??????????? ???? ???????? ???????. ?????????? ????? ????? ???????????? ?????????? ????????? ???????????? ????????????. ??????????????? ????? ????????? ??????? ?????????? ? ????????????? ? ??????? ?? ??????????. ????????????? ???? ? ??????????????? ???????????? ? ?????????? ???????? ?????????????: ????????????? ???? ???????????? ??? ????????????? ??????????? ??????????? ????????, ??????? ????? ???. ??????? ?????????????? ????? ? ??????????????? ???????????? ??? ??????? ?????????????. ??????????? ?????? ?? ???????? ?????????????. ????????????? ???? ? ??????????????? ?????????? ???????? ????????????? - ??????????? ???. ????????????? ???? ? ??????????????? ?????????????????? ??????????: ???? ? ??????????????? ?????????? ? ??????????? ?? ???????? ???????????? ?? ????????????. ???? ???????, ?????????? ?? ????????????. ?????????????? ??????????. ??????? ????????? ??????? ? ???????? ? ????????????? ????? ? ??? ????? ???????????????. ??? ??? ????????????? ??????? ?????????. ????????????? ???? ? ??????????????? ???????????: ??????? ????????? ??????? ? ????????????? ????? ? ??? ????? ??????????????? ? ???????????? ? ???????????? ???????????? ???????????. ?????? ????????? ?????????? ??? ????? ?????????? ???. ??????? ????????? ???????? ? ????????????? ? ????????? ?????. ???? ??????????. ???? ?????????? ?????????. ????????????? ?????????. ?????????? ?????????. ????????? ?????? ? ??????????. ????????? ????? ?? ???. ?????? ??????????. ?????? ?? ???? ???????? ? ????????????. ??????? ?????????? ????????? ??????????????? ??????????. * * *???????: (O95) 106-?865, 1??-9709 -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.h.jaffe at gmail.com Fri Jan 27 00:08:04 2006 From: a.h.jaffe at gmail.com (Andrew Jaffe) Date: Fri Jan 27 00:08:04 2006 Subject: [Numpy-discussion] Re: Weird (wrong?) real fft 2d behavior In-Reply-To: References: <43D74B35.9060204@gmail.com> <43D74D17.7010103@gmail.com> Message-ID: <43D9D342.3080204@gmail.com> Dear Warren, > > On Wed, 25 Jan 2006, Andrew Jaffe wrote: > >> Andrew Jaffe wrote: >>> If I start with what I thought was an appropriate (n, n/2+1) complex >>> matrix which should have a real inverse fft, and then take its real fft, >>> I don't get back the original matrix. >>> >>> Note the presence of very small but nonzero reals in the final matrix, > > That's just roundoff. > >>> and the fact that the 2d and 4th rows are duplicates. This seems to be a >>> mistake somewhere. >>> >>> Or am I just misunderstanding/misremembering something about 2d real ffts? > > It looks wrong to me, and I think I wrote those functions. I get the same > results in Numeric. I'll try to look into the problem. > >> and I should point out that >> delta_rp = N.dft.real_fft2d(delta_kp) >> is 'allclose' to the original delta_r (which leads me to believe that I >> may indeed be misunderstanding something). > > "Stable" does not neccessarily imply "correct". Indeed! And more to the point, it's actually the case that "delta_kp" doesn't actually have the requisite 16 (non-small) real degrees of freedom -- so it can't really be right. Andrew From a.h.jaffe at gmail.com Fri Jan 27 00:11:02 2006 From: a.h.jaffe at gmail.com (Andrew Jaffe) Date: Fri Jan 27 00:11:02 2006 Subject: [Numpy-discussion] Re: Weird (wrong?) real fft 2d behavior In-Reply-To: References: <43D74B35.9060204@gmail.com> <43D74D17.7010103@gmail.com> Message-ID: <43D9D39F.30307@gmail.com> > On Wed, 25 Jan 2006, Andrew Jaffe wrote: > >> Andrew Jaffe wrote: >>> If I start with what I thought was an appropriate (n, n/2+1) complex >>> matrix which should have a real inverse fft, and then take its real fft, >>> I don't get back the original matrix. >>> >>> Note the presence of very small but nonzero reals in the final matrix, > > That's just roundoff. > >>> and the fact that the 2d and 4th rows are duplicates. This seems to be a >>> mistake somewhere. >>> >>> Or am I just misunderstanding/misremembering something about 2d real ffts? > > It looks wrong to me, and I think I wrote those functions. I get the same > results in Numeric. I'll try to look into the problem. > >> and I should point out that >> delta_rp = N.dft.real_fft2d(delta_kp) >> is 'allclose' to the original delta_r (which leads me to believe that I >> may indeed be misunderstanding something). > > "Stable" does not neccessarily imply "correct". Indeed! And more to the point, it's actually the case that "delta_kp" doesn't actually have the requisite 16 (non-small) real degrees of freedom -- so it can't really be right. Andrew From agn at noc.soton.ac.uk Fri Jan 27 01:12:07 2006 From: agn at noc.soton.ac.uk (George Nurser) Date: Fri Jan 27 01:12:07 2006 Subject: [Numpy-discussion] installation under other than .../site-packages Message-ID: <2BA043A4-A260-4EC0-AFCF-A22CEAE74DC2@noc.soton.ac.uk> On 25 Jan 2006, at 20:09, Humufr wrote: > This solution is the one I'm using. Unfortunatly it's not working > good with scipy. I tried to install scipy in my own directory and > everythong seems to work like a charm but the lapack link is bad. I > tried everything write in the wiki to install your own BLAS/LAPACK > library but scipy want tu use the one of the system who have a > problem. I'm trying to find a solution but until I found it numpy/ > scipy is not an option. > This wasn't obvious to me either. To use your own blas/lapck in scipy, you need to put a site.cfg file in the distutils directory of your numpy installation. So if you installed numpy with --home=$HOME it should produce a distutils directory at $HOME/lib/python/numpy/ distutils or $HOME/lib64/python/numpy/distutils If you want to build scipy with the same site.cfg as you used to build numpy, then, assuming the root source numpy directory is $PYSRC/ numpy cp $PYSRC/numpy/numpy/distutils/site.cfg $HOME/lib[64]/python/numpy/ distutils Hope this helps. George. From alteration at onet.pl Fri Jan 27 03:31:00 2006 From: alteration at onet.pl (=?windows-1251?B?MDIg9OXi8ODr/w==?=) Date: Fri Jan 27 03:31:00 2006 Subject: [Numpy-discussion] =?windows-1251?B?z/Do4uvl9+Xt6OUg6CDo8e/u6/zn7uLg7ejlIOIg0NQg6O3u8fLw4O3t7ukg8ODh?= =?windows-1251?B?7vfl6SDx6Ov7Lg==?= Message-ID: <84b101c62333$56628296$0572b9c0@onet.pl> ?????????? ??? ? ????? ?????? ??????? ??????? ? ???????????? ?????????: 02 ??????? - ??????????? ? ????????????? ? ?? ??????????? ??????? ????. 06 ??????? - ??????????????? ????????? ?????: ??????????, ??????? ?????????? ????????, ???????? ?????? ?????????????. ???????? ?????. ?????????? ? ?????????????? ?? ??????????? ??????? ????02 ??????? 2006 ?. 1. ????? ???????????????? ? ????? ??????????? ??????????? ??????? ????. 2. ??????????????? ???????????? ?? ????????? ???????????? ????????????????. ???????????????? ??????????????? ??????????? ???????????????? ??????????????? ??????????? ??? ???????????????? ??????????????? ?????????? ??? ?????? ??????????? ????????? 3. ???????? ????????? ??????????? ?????????? ? ?? ??????? ??????-?????? ??????????? ???? ? ?????????? ??????????????? ??????????? ????????? ???????????????? ?? ??????????? (????? ???????? ???????, ?????? ?????, ????????????, ?????? ?????, ?????????? ???????????, ???????????????) ??????????? ??????????? ?? ?????? ??????? ??? ????? ?? ?????? ??????? ?????? - ?? ????????? ????????????? ??????????-???????? ????????? ? ??????????? ?????? ? ?????. 4. ??????????? ?????????????, ???????????? ???? ??????????? ????????? ?????????? ?? ??????????? ??????????? ??????? ???? ?????????? ????????????? ?? ????? ?????? ?????????, ??????? ?? ????????? ?????????? ?? ??????????, ?? ????????????? ???????? ???????????? ????????????? ???????????????? ?.?????? - ???????? ????????? 5. ?????????????? ?????? ? ?? ??????????? ??????????????? ????? ??????????? ???????????? ?????? ??? ?.?????? ???? ?.?????? ?????????? ?? ????? ???????? ??????????????? ????????? ????? ? ?.?????? 6. ?????????????? ???????????? ????????? ???????: 5800 ???. + ??? ?????????????????? ? ???????? ?????????????? ?????????? ?? ??????:?? ???. (495) 2?2-2l?3 ??????????????? ????????? ?????: ??????????, ??????? ?????????? ????????, ???????? ?????? ?????????????. ???????? ?????.06 ??????? 2006 ???? ????????? ??????? ???????? ????????? ?? ??????? ??????????? ???????? ????????????? ????????, ?? ???????? ? ????? ??????? ?????????? ?????????????. ?????? ???????? ????? ??????? ?? ?????? ???????? ???????, ??????? ????????? ????????????, ?? ? ??? ????????, ??????? ?????????? ?? ?????? ?????????. ????? ????? ????? ???? ?????????? ???????????? ?? ?????? ???? ???????????? ??? ??????-??? ??????????????? ?????????? ????? ? ??????????? ?????????? ??????. 1. ??????????????? ????????????? ? ??????????? ???, ?? ????????? ????????? ????????????????: ????????????????, ?????????. ??? ????? ???????? ?? ?????????? ????????? 2. ??????????? ??????: ??????????????? ????????? ?????: ???????? ??????????? ? ??????????? ?? ?????? ?????, ???????????, ???????? ? ??. 3. ?????????? ? ??????? ??????????????? ????????? ????? (???). 4. ?????????? ? ???????? ??????????, ??????? ????? ????????? ???. 5. ????? ? ??????? ?????????? ????????, ??????????? ???????? ? ????????? ???????. 6. ???????????? ?????? ??? ???????????? ?????????? ?? ??????????? ????????????? ????????. ??? ???????? ????? ???????????? 7. ????? ???????? ? ????? ??????? ????????? ???????????????? ? ?? ???????????: 7.1. ???????? ?????? ??? ?????? ?? ??????. ?????????? ?????????? ??????? ???????? ?????????. ?????????? ?????????? ????????? ???????. 7.2. ????????? ???????????????? ?? ?????? ?????. ?????????? ?????????? ??????? ????. ?????????? ??????????? ??????????? ? ?.?. 7.3. ????????? ?? ?????? ?????. ????????? ? ????????? ????? ?????????. ????????? ?????????? ????? ??? ???? ? ?????. ????????? ??? ????????????. 7.4. ?????????, ????????? ? ??????????????? ?????????? ???????. ??????? ????????????? ?? ????????. 7.5. ????????? ? ??????? ?????? ? ?????????? ???????? ??????. ????????? ? ??????? ???????????? ? ??????????? ?????? ? ????????? ????????. 8. ????????? ??????????? ?????????? ??????. ????????? 5500 ???. + ???. ? ????????? ???????? ????????: ??????????? ?????????, ????, ????-?????. ?????????????????? ? ???????? ?????????????? ?????????? ?? ??????:?? ???. (495) 2?2-2l?3 -------------- next part -------------- An HTML attachment was scrubbed... URL: From faltet at carabos.com Fri Jan 27 03:31:05 2006 From: faltet at carabos.com (Francesc Altet) Date: Fri Jan 27 03:31:05 2006 Subject: [Numpy-discussion] Upcasting rules in numpy Message-ID: <200601271230.23005.faltet@carabos.com> Hi, I don't know if the next behaviour in NumPy is intended or not: In [22]: (numpy.zeros((2,1), dtype="int8")*3.).dtype.type Out[22]: while numarray and Numeric gives: In [36]: (numarray.zeros((2,1), type="Int8")*3.).type() Out[36]: Float64 In [37]: (Numeric.zeros((2,1), typecode="b")*3.).typecode() Out[37]: 'd' Would not be float64 a more sensible default for numpy as well? Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From ndarray at mac.com Fri Jan 27 08:50:12 2006 From: ndarray at mac.com (Sasha) Date: Fri Jan 27 08:50:12 2006 Subject: [Numpy-discussion] Numpy does not compile with PY_ARRAY_TYPES_PREFIX Message-ID: I have attempted to compile RPy (see http://rpy.sourceforge.net/) with numpy and found that both R and numpy headers define Int32: .../numpy/core/include/numpy/arrayobject.h:304: conflicting types for `Int32' .../lib/R/include/R_ext/Random.h:57: previous declaration of `Int32' I have tried to fix the problem by setting PY_ARRAY_TYPES_PREFIX as follows: $ CFLAGS=-DPY_ARRAY_TYPES_PREFIX=PyArray_NS_ python setup.py build This resulted in compile error in almost every Numpy source file. Am I using PY_ARRAY_TYPES_PREFIX correctly? If not, what is the recommended way to solve name conflicts involving Numpy typoes? Thanks. From strawman at astraw.com Fri Jan 27 09:14:07 2006 From: strawman at astraw.com (Andrew Straw) Date: Fri Jan 27 09:14:07 2006 Subject: [Numpy-discussion] Numpy does not compile with PY_ARRAY_TYPES_PREFIX In-Reply-To: References: Message-ID: <43DA54AE.4090704@astraw.com> Sasha wrote: >I have attempted to compile RPy (see http://rpy.sourceforge.net/) with >numpy and found that both R and numpy headers define Int32: > >.../numpy/core/include/numpy/arrayobject.h:304: conflicting types for `Int32' >.../lib/R/include/R_ext/Random.h:57: previous declaration of `Int32' > >I have tried to fix the problem by setting PY_ARRAY_TYPES_PREFIX as follows: > >$ CFLAGS=-DPY_ARRAY_TYPES_PREFIX=PyArray_NS_ python setup.py build > >This resulted in compile error in almost every Numpy source file. > >Am I using PY_ARRAY_TYPES_PREFIX correctly? If not, what is the >recommended way to solve name conflicts involving Numpy typoes? > > David Cooke informed me some time ago that the #include "numpy/arrayobject.h" line should go _after_ the other #includes. It's worked for me well enough ever since, although it seems rather fragile. Otherwise I'm not having the same issue with current svn. There was an issue with release 0.9.4 for C++ code, anyway. Cheers! Andrew From alexander.belopolsky at gmail.com Fri Jan 27 10:14:01 2006 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Fri Jan 27 10:14:01 2006 Subject: Fwd: [Numpy-discussion] Numpy does not compile with PY_ARRAY_TYPES_PREFIX In-Reply-To: References: <43DA54AE.4090704@astraw.com> Message-ID: Numpy-discussion list rejected an attachment. Here are a few relevant lines from the log: In file included from numpy/core/src/multiarraymodule.c:65: numpy/core/src/arrayobject.c: In function `PyArray_FromDimsAndDataAndDescr': numpy/core/src/arrayobject.c:766: `intp' undeclared (first use in this function) numpy/core/src/arrayobject.c:766: parse error before ')' token In file included from numpy/core/src/multiarraymodule.c:65: numpy/core/src/arrayobject.c: At top level: numpy/core/src/arrayobject.c:822: parse error before "intp" numpy/core/src/arrayobject.c:822: warning: function declaration isn't a prototype numpy/core/src/arrayobject.c: In function `PyArray_Scalar': numpy/core/src/arrayobject.c:841: `Bool' undeclared (first use in this function) numpy/core/src/arrayobject.c:841: parse error before ')' token numpy/core/src/arrayobject.c:841: parse error before ')' token numpy/core/src/arrayobject.c:841: warning: left-hand operand of comma expression has no effect ---------- Forwarded message ---------- From: Sasha Date: Jan 27, 2006 12:33 PM Subject: Re: [Numpy-discussion] Numpy does not compile with PY_ARRAY_TYPES_PREFIX To: Andrew Straw , numpy-discussion I've just updated to svn r2006. Regular build works fine, but $ CFLAGS=-DPY_ARRAY_TYPES_PREFIX=PyArray_NS_ python setup.py build produced the attached log. Note that I am not compiling any extention, just regular numpy. On 1/27/06, Andrew Straw wrote: > Sasha wrote: > > >I have attempted to compile RPy (see http://rpy.sourceforge.net/) with > >numpy and found that both R and numpy headers define Int32: > > > >.../numpy/core/include/numpy/arrayobject.h:304: conflicting types for `Int32' > >.../lib/R/include/R_ext/Random.h:57: previous declaration of `Int32' > > > >I have tried to fix the problem by setting PY_ARRAY_TYPES_PREFIX as follows: > > > >$ CFLAGS=-DPY_ARRAY_TYPES_PREFIX=PyArray_NS_ python setup.py build > > > >This resulted in compile error in almost every Numpy source file. > > > >Am I using PY_ARRAY_TYPES_PREFIX correctly? If not, what is the > >recommended way to solve name conflicts involving Numpy typoes? > > > > > > David Cooke informed me some time ago that the #include > "numpy/arrayobject.h" line should go _after_ the other #includes. It's > worked for me well enough ever since, although it seems rather fragile. > Otherwise I'm not having the same issue with current svn. There was an > issue with release 0.9.4 for C++ code, anyway. > > Cheers! > Andrew > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From ndarray at mac.com Fri Jan 27 10:15:03 2006 From: ndarray at mac.com (Sasha) Date: Fri Jan 27 10:15:03 2006 Subject: Fwd: [Numpy-discussion] Numpy does not compile with PY_ARRAY_TYPES_PREFIX In-Reply-To: References: <43DA54AE.4090704@astraw.com> Message-ID: Numpy-discussion list rejected an attachment. Here are a few relevant lines from the log: In file included from numpy/core/src/multiarraymodule.c:65: numpy/core/src/arrayobject.c: In function `PyArray_FromDimsAndDataAndDescr': numpy/core/src/arrayobject.c:766: `intp' undeclared (first use in this function) numpy/core/src/arrayobject.c:766: parse error before ')' token In file included from numpy/core/src/multiarraymodule.c:65: numpy/core/src/arrayobject.c: At top level: numpy/core/src/arrayobject.c:822: parse error before "intp" numpy/core/src/arrayobject.c:822: warning: function declaration isn't a prototype numpy/core/src/arrayobject.c: In function `PyArray_Scalar': numpy/core/src/arrayobject.c:841: `Bool' undeclared (first use in this function) numpy/core/src/arrayobject.c:841: parse error before ')' token numpy/core/src/arrayobject.c:841: parse error before ')' token numpy/core/src/arrayobject.c:841: warning: left-hand operand of comma expression has no effect ---------- Forwarded message ---------- From: Sasha Date: Jan 27, 2006 12:33 PM Subject: Re: [Numpy-discussion] Numpy does not compile with PY_ARRAY_TYPES_PREFIX To: Andrew Straw , numpy-discussion I've just updated to svn r2006. Regular build works fine, but $ CFLAGS=-DPY_ARRAY_TYPES_PREFIX=PyArray_NS_ python setup.py build produced the attached log. Note that I am not compiling any extention, just regular numpy. On 1/27/06, Andrew Straw wrote: > Sasha wrote: > > >I have attempted to compile RPy (see http://rpy.sourceforge.net/) with > >numpy and found that both R and numpy headers define Int32: > > > >.../numpy/core/include/numpy/arrayobject.h:304: conflicting types for `Int32' > >.../lib/R/include/R_ext/Random.h:57: previous declaration of `Int32' > > > >I have tried to fix the problem by setting PY_ARRAY_TYPES_PREFIX as follows: > > > >$ CFLAGS=-DPY_ARRAY_TYPES_PREFIX=PyArray_NS_ python setup.py build > > > >This resulted in compile error in almost every Numpy source file. > > > >Am I using PY_ARRAY_TYPES_PREFIX correctly? If not, what is the > >recommended way to solve name conflicts involving Numpy typoes? > > > > > > David Cooke informed me some time ago that the #include > "numpy/arrayobject.h" line should go _after_ the other #includes. It's > worked for me well enough ever since, although it seems rather fragile. > Otherwise I'm not having the same issue with current svn. There was an > issue with release 0.9.4 for C++ code, anyway. > > Cheers! > Andrew > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From ndarray at mac.com Fri Jan 27 11:23:04 2006 From: ndarray at mac.com (Sasha) Date: Fri Jan 27 11:23:04 2006 Subject: [Numpy-discussion] UnsignedInt32 Message-ID: Numeric has both UInt32 and UnsignedInt32 defined, but Numpy only has UInt32. Is it intentional or an oversight? Thanks. From oliphant at ee.byu.edu Fri Jan 27 11:28:01 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri Jan 27 11:28:01 2006 Subject: [Numpy-discussion] UnsignedInt32 In-Reply-To: References: Message-ID: <43DA740F.4090707@ee.byu.edu> Sasha wrote: > Numeric has both UInt32 and UnsignedInt32 defined, but Numpy only has >UInt32. Is it intentional or an oversight? > > > an oversight. -Travis From oliphant at ee.byu.edu Fri Jan 27 11:32:09 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri Jan 27 11:32:09 2006 Subject: [Numpy-discussion] UnsignedInt32 In-Reply-To: References: Message-ID: <43DA7453.5030906@ee.byu.edu> Sasha wrote: > Numeric has both UInt32 and UnsignedInt32 defined, but Numpy only has >UInt32. Is it intentional or an oversight? > > > It's actually there, it's just not in the __all__ list of oldnumeric Try: from numpy.core.oldnumeric import UnsignedInt32 -Travis From ndarray at mac.com Fri Jan 27 13:32:32 2006 From: ndarray at mac.com (Sasha) Date: Fri Jan 27 13:32:32 2006 Subject: Fwd: [Numpy-discussion] Numpy does not compile with PY_ARRAY_TYPES_PREFIX In-Reply-To: <43DA7625.6010804@ee.byu.edu> References: <43DA54AE.4090704@astraw.com> <43DA7220.8070605@ee.byu.edu> <43DA7625.6010804@ee.byu.edu> Message-ID: On 1/27/06, Travis Oliphant wrote: > Show an example of using it in a file before you include the numpy > header. Because that *should* work. I know, I just tested it a few > days ago... I've tried to put just two lines in "test.c": #define PY_ARRAY_TYPES_PREFIX XYZ_ #include and compile it with $ gcc -I$NUMPYINCLUDE -c test.c Where NUMPYINCLUDE points to the location of numpy/arrayobject.h I get the following errors: In file included from test.c:2: numpy-0.9.4/lib/python2.3/site-packages/numpy/core/include/numpy/arrayobject.h:699: error: syntax error before " XYZ_intp" numpy-0.9.4/lib/python2.3/site-packages/numpy/core/include/numpy/arrayobject.h:699: warning: data definition has no type or storage class numpy-0.9.4/lib/python2.3/site-packages/numpy/core/include/numpy/arrayobject.h:700: error: syntax error before " XYZ_uintp" ... ... From cookedm at physics.mcmaster.ca Fri Jan 27 13:38:06 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Fri Jan 27 13:38:06 2006 Subject: Fwd: [Numpy-discussion] Numpy does not compile with PY_ARRAY_TYPES_PREFIX In-Reply-To: (ndarray@mac.com's message of "Fri, 27 Jan 2006 16:31:42 -0500") References: <43DA54AE.4090704@astraw.com> <43DA7220.8070605@ee.byu.edu> <43DA7625.6010804@ee.byu.edu> Message-ID: Sasha writes: > On 1/27/06, Travis Oliphant wrote: > >> Show an example of using it in a file before you include the numpy >> header. Because that *should* work. I know, I just tested it a few >> days ago... > > I've tried to put just two lines in "test.c": > > #define PY_ARRAY_TYPES_PREFIX XYZ_ > #include > > and compile it with > > $ gcc -I$NUMPYINCLUDE -c test.c > > Where NUMPYINCLUDE points to the location of numpy/arrayobject.h You need to add #include "Python.h" as the first include (and add a -I to your gcc line, of course). -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From ndarray at mac.com Fri Jan 27 14:51:03 2006 From: ndarray at mac.com (Sasha) Date: Fri Jan 27 14:51:03 2006 Subject: Fwd: [Numpy-discussion] Numpy does not compile with PY_ARRAY_TYPES_PREFIX In-Reply-To: References: <43DA54AE.4090704@astraw.com> <43DA7220.8070605@ee.byu.edu> <43DA7625.6010804@ee.byu.edu> Message-ID: On 1/27/06, David M. Cooke wrote: > You need to add #include "Python.h" as the first include (and add a > -I to your gcc line, of course). Still does not work > cat test.c #define PY_ARRAY_TYPES_PREFIX XYZ_ #include #include > gcc -I Python-2.4.1/include/python2.4 -I lib/python2.4/site-packages/numpy/core/include -c test.c In file included from lib/python2.4/site-packages/numpy/core/include/numpy/arrayobject.h:1346, from test.c:3: lib/python2.4/site-packages/numpy/core/include/numpy/__multiarray_api.h: In function `import_array': lib/python2.4/site-packages/numpy/core/include/numpy/__multiarray_api.h:765: parse error before ')' token lib/python2.4/site-packages/numpy/core/include/numpy/__multiarray_api.h:765: parse error before ')' token lib/python2.4/site-packages/numpy/core/include/numpy/__multiarray_api.h: At top level: lib/python2.4/site-packages/numpy/core/include/numpy/__multiarray_api.h:771: parse error before "return" If I remove #define PY_ARRAY_TYPES_PREFIX or define it as nothing, compilation works. The RPy.h header I was trying to compile in the first plase includes numpy/arrayobject.h after Python.h . From oliphant at ee.byu.edu Fri Jan 27 16:27:04 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri Jan 27 16:27:04 2006 Subject: Fwd: [Numpy-discussion] Numpy does not compile with PY_ARRAY_TYPES_PREFIX In-Reply-To: References: <43DA54AE.4090704@astraw.com> <43DA7220.8070605@ee.byu.edu> <43DA7625.6010804@ee.byu.edu> Message-ID: <43DABA30.6020002@ee.byu.edu> Sasha wrote: >On 1/27/06, David M. Cooke wrote: > > >>You need to add #include "Python.h" as the first include (and add a >>-I to your gcc line, of course). >> >> > >Still does not work > > It works for me. Try the attached file. gcc -I /usr/include/python2.4 -I /usr/lib/python2.4/site-packages/numpy/core/include -c test.c Worked fine. Maybe Python.h has to come *before* the special #define. -Travis -------------- next part -------------- A non-text attachment was scrubbed... Name: test.c Type: text/x-csrc Size: 85 bytes Desc: not available URL: From oliphant at ee.byu.edu Fri Jan 27 16:34:04 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri Jan 27 16:34:04 2006 Subject: Fwd: [Numpy-discussion] Numpy does not compile with PY_ARRAY_TYPES_PREFIX In-Reply-To: <43DABA30.6020002@ee.byu.edu> References: <43DA54AE.4090704@astraw.com> <43DA7220.8070605@ee.byu.edu> <43DA7625.6010804@ee.byu.edu> <43DABA30.6020002@ee.byu.edu> Message-ID: <43DABBBC.3040505@ee.byu.edu> Travis Oliphant wrote: > Sasha wrote: > >> On 1/27/06, David M. Cooke wrote: >> >> >>> You need to add #include "Python.h" as the first include (and add a >>> -I to your gcc line, of course). >>> >> >> >> Still does not work >> >> > It works for me. Try the attached file. > > gcc -I /usr/include/python2.4 -I > /usr/lib/python2.4/site-packages/numpy/core/include -c test.c > > Worked fine. > > Maybe Python.h has to come *before* the special #define. On my system, it didn't seem to matter. All incarnations worked. Perhaps there is something wrong with your installation. Could you show the very first errors you are getting. Is the wrong header being picked up from somewhere. Also perhaps you could look at the output of the compilation using the -E (pre-processor only) flag. This might help nail down the problem. -Travis From cookedm at physics.mcmaster.ca Fri Jan 27 16:43:03 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Fri Jan 27 16:43:03 2006 Subject: Fwd: [Numpy-discussion] Numpy does not compile with PY_ARRAY_TYPES_PREFIX In-Reply-To: (ndarray@mac.com's message of "Fri, 27 Jan 2006 17:50:27 -0500") References: <43DA54AE.4090704@astraw.com> <43DA7220.8070605@ee.byu.edu> <43DA7625.6010804@ee.byu.edu> Message-ID: Sasha writes: > On 1/27/06, David M. Cooke wrote: >> You need to add #include "Python.h" as the first include (and add a >> -I to your gcc line, of course). > > Still does not work > >> cat test.c > #define PY_ARRAY_TYPES_PREFIX XYZ_ > #include > #include > >> gcc -I Python-2.4.1/include/python2.4 -I lib/python2.4/site-packages/numpy/core/include -c test.c > In file included from > lib/python2.4/site-packages/numpy/core/include/numpy/arrayobject.h:1346, > from test.c:3: > lib/python2.4/site-packages/numpy/core/include/numpy/__multiarray_api.h: > In function `import_array': > lib/python2.4/site-packages/numpy/core/include/numpy/__multiarray_api.h:765: > parse error before ')' token > lib/python2.4/site-packages/numpy/core/include/numpy/__multiarray_api.h:765: > parse error before ')' token > lib/python2.4/site-packages/numpy/core/include/numpy/__multiarray_api.h: > At top level: > lib/python2.4/site-packages/numpy/core/include/numpy/__multiarray_api.h:771: > parse error before "return" Hmm, the location of that error would seem to imply it's something to do with NDARRAY_VERSION not being defined. Maybe for some reason you've got an old arrayobject.h, but a newer __multiarray_api.h? I'd try blowing away your lib/python2.4/site-packages/numpy directory, and installing again. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From ndarray at mac.com Fri Jan 27 18:13:05 2006 From: ndarray at mac.com (Sasha) Date: Fri Jan 27 18:13:05 2006 Subject: Fwd: [Numpy-discussion] Numpy does not compile with PY_ARRAY_TYPES_PREFIX In-Reply-To: <43DABBBC.3040505@ee.byu.edu> References: <43DA7220.8070605@ee.byu.edu> <43DA7625.6010804@ee.byu.edu> <43DABA30.6020002@ee.byu.edu> <43DABBBC.3040505@ee.byu.edu> Message-ID: Thanks for your advise. With the help of -E option, I've found the problem. I was using the version of the header that used uint in the C-API. the problem was fixed in svn r1983: http://projects.scipy.org/scipy/numpy/changeset/1983 . -- sasha On 1/27/06, Travis Oliphant wrote: > Travis Oliphant wrote: > > > Sasha wrote: > > > >> On 1/27/06, David M. Cooke wrote: > >> > >> > >>> You need to add #include "Python.h" as the first include (and add a > >>> -I to your gcc line, of course). > >>> > >> > >> > >> Still does not work > >> > >> > > It works for me. Try the attached file. > > > > gcc -I /usr/include/python2.4 -I > > /usr/lib/python2.4/site-packages/numpy/core/include -c test.c > > > > Worked fine. > > > > Maybe Python.h has to come *before* the special #define. > > On my system, it didn't seem to matter. All incarnations worked. > Perhaps there is something wrong with your installation. Could you > show the very first errors you are getting. Is the wrong header being > picked up from somewhere. > > Also perhaps you could look at the output of the compilation using the > -E (pre-processor only) flag. This might help nail down the problem. > > -Travis > > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From strawman at astraw.com Fri Jan 27 20:28:13 2006 From: strawman at astraw.com (Andrew Straw) Date: Fri Jan 27 20:28:13 2006 Subject: Fwd: [Numpy-discussion] Numpy does not compile with PY_ARRAY_TYPES_PREFIX In-Reply-To: References: <43DA7220.8070605@ee.byu.edu> <43DA7625.6010804@ee.byu.edu> <43DABA30.6020002@ee.byu.edu> <43DABBBC.3040505@ee.byu.edu> Message-ID: <43DAF2BA.5090307@astraw.com> Sasha wrote: >Thanks for your advise. With the help of -E option, I've found the >problem. I was using the version of the header that used uint in the >C-API. the problem was fixed in svn r1983: >http://projects.scipy.org/scipy/numpy/changeset/1983 . > >-- sasha > > That's strange. In your 2nd email, you said you were using revision 2006, which made me think this couldn't be the issue... From ndarray at mac.com Fri Jan 27 20:41:13 2006 From: ndarray at mac.com (Sasha) Date: Fri Jan 27 20:41:13 2006 Subject: Fwd: [Numpy-discussion] Numpy does not compile with PY_ARRAY_TYPES_PREFIX In-Reply-To: <43DAF2BA.5090307@astraw.com> References: <43DA7625.6010804@ee.byu.edu> <43DABA30.6020002@ee.byu.edu> <43DABBBC.3040505@ee.byu.edu> <43DAF2BA.5090307@astraw.com> Message-ID: Sorry, I did not clean up an old build properly. It's hard to keep up with numpy's pace these days! -- sasha On 1/27/06, Andrew Straw wrote: > That's strange. In your 2nd email, you said you were using revision > 2006, which made me think this couldn't be the issue... From ndarray at mac.com Fri Jan 27 22:42:05 2006 From: ndarray at mac.com (Sasha) Date: Fri Jan 27 22:42:05 2006 Subject: [Numpy-discussion] RPy with NumPy Message-ID: I have succesfully (according to testall.py :-) converted RPy to use numpy instead of Numeric. Attached patch was tested on Linux only. I did not attempt to create a package that would support both numpy and Numeric, but it is easy to do. Please let me know if you are interested. For numpy-discussion readers: "RPy is a very simple, yet robust, Python interface to the R Programming Language." For rpy-list readers: "Development for Numeric has ceased, and users should transisition to NumPy as quickly as possible." For readers of both lists: Sorry for cross-posting. -- sasha -------------- next part -------------- diff -Naur rpy-0.4.6/rpy.py rpy-0.4.6-numpy/rpy.py --- rpy-0.4.6/rpy.py 2005-07-27 13:56:54.000000000 -0400 +++ rpy-0.4.6-numpy/rpy.py 2006-01-27 12:06:13.628796000 -0500 @@ -30,7 +30,7 @@ # installation time and RPy should have been compiled properly. So, # don't complain. try: - from Numeric import * + from numpy import * HAS_NUMERIC = 1 except ImportError: HAS_NUMERIC = 0 diff -Naur rpy-0.4.6/setup.py rpy-0.4.6-numpy/setup.py --- rpy-0.4.6/setup.py 2005-07-28 00:52:29.000000000 -0400 +++ rpy-0.4.6-numpy/setup.py 2006-01-28 01:18:46.148833000 -0500 @@ -34,7 +34,6 @@ from distutils.errors import * import rpy_tools - try: if sys.platform=="win32": RHOMES = os.environ['RHOMES'].split(';') @@ -72,7 +71,7 @@ if sys.platform=="win32": DEFINE.append(('R_HOME', '\\"%s\\"' % RHOME.replace("\\","/") )) else: - DEFINE.append(('R_HOME', '"%s"' % RHOME)) + DEFINE.append(('R_HOME', r'\"%s\"' % RHOME)) r_libs = [ os.path.join(RHOME, 'bin'), @@ -132,8 +131,10 @@ # check whether Numeric is present try: - import Numeric + import numpy DEFINE.append(('WITH_NUMERIC', None)) + DEFINE.append(('PY_ARRAY_TYPES_PREFIX', 'PyArray_')) + include_dirs.append(numpy.get_numpy_include()) except ImportError: UNDEF.append('WITH_NUMERIC') @@ -148,12 +149,11 @@ # use R version specific shared library name shlib_name = "_rpy%s" % RVER if sys.platform=="win32": - DEFINE.append( ('RPY_SHNAME', '\\"_rpy%s\\"' % RVER ) ) - else: DEFINE.append( ('RPY_SHNAME', '"_rpy%s"' % RVER ) ) + else: + DEFINE.append( ('RPY_SHNAME', r'\"_rpy%s\"' % RVER ) ) DEFINE.append( ('INIT_RPY', 'init_rpy%s' % RVER ) ) - modules.append( Extension( shlib_name, source_files, diff -Naur rpy-0.4.6/src/RPy.h rpy-0.4.6-numpy/src/RPy.h --- rpy-0.4.6/src/RPy.h 2005-07-26 23:05:29.000000000 -0400 +++ rpy-0.4.6-numpy/src/RPy.h 2006-01-26 17:18:37.750157000 -0500 @@ -68,12 +68,13 @@ #include +#ifdef WITH_NUMERIC +#include "numpy/arrayobject.h" +#endif + #include "robjobject.h" #include "setenv.h" -#ifdef WITH_NUMERIC -#include "Numeric/arrayobject.h" -#endif #define MAXIDSIZE 256 diff -Naur rpy-0.4.6/src/rpymodule.c rpy-0.4.6-numpy/src/rpymodule.c --- rpy-0.4.6/src/rpymodule.c 2005-07-28 10:23:33.000000000 -0400 +++ rpy-0.4.6-numpy/src/rpymodule.c 2006-01-28 01:19:29.281899000 -0500 @@ -1517,7 +1517,7 @@ if(use_numeric) { import_array(); - multiarray = PyImport_ImportModule("multiarray"); + multiarray = PyImport_ImportModule("numpy"); if (multiarray) { dict = PyModule_GetDict(multiarray); if (dict) diff -Naur rpy-0.4.6/tests/test_array.py rpy-0.4.6-numpy/tests/test_array.py --- rpy-0.4.6/tests/test_array.py 2005-07-18 14:54:54.000000000 -0400 +++ rpy-0.4.6-numpy/tests/test_array.py 2006-01-28 00:58:41.100628000 -0500 @@ -28,7 +28,7 @@ r.array.autoconvert(BASIC_CONVERSION) def testConversionToPy(self): - self.failUnless(self.py == self.ra.as_py(), + self.failUnless((self.py == self.ra.as_py()).all(), 'wrong conversion to Python') def testConversionToR(self): diff -Naur rpy-0.4.6/tests/test_numeric.py rpy-0.4.6-numpy/tests/test_numeric.py --- rpy-0.4.6/tests/test_numeric.py 2005-07-26 23:08:49.000000000 -0400 +++ rpy-0.4.6-numpy/tests/test_numeric.py 2006-01-27 12:07:38.309828000 -0500 @@ -2,7 +2,7 @@ import unittest try: - from Numeric import * + from numpy import * except ImportError: print 'Numeric not available. Skipping.\n' import sys diff -Naur rpy-0.4.6/tests/test_topy.py rpy-0.4.6-numpy/tests/test_topy.py --- rpy-0.4.6/tests/test_topy.py 2005-07-19 16:20:08.000000000 -0400 +++ rpy-0.4.6-numpy/tests/test_topy.py 2006-01-28 01:05:57.007083000 -0500 @@ -27,7 +27,7 @@ assert( r(-2147483648) == r.NA ) def testNAreal(self): - assert(r.NAN == r("as.numeric(NA)") ) + assert(repr(r.NAN) == repr(r("as.numeric(NA)")) ) def testNAstring(self): assert(r("as.character(NA)") == 'NA') @@ -48,14 +48,14 @@ def testNAList(self): xi = [1,2,r.NA,r.NAN,4] assert(r.as_character(xi) == ['1', '2', 'NA', 'NA', '4']) - assert(r.as_numeric(xi) == [1.0, 2.0, r.NAN, r.NAN, 4.0]) + assert(repr(r.as_numeric(xi)) == repr([1.0, 2.0, r.NAN, r.NAN, 4.0])) assert(r.as_integer(xi) == [1, 2, r.NA, r.NA, 4]) assert(r.as_factor(xi) == ['1', '2', 'NA', 'NA', '4']) assert(r.is_na(xi) == [False, False, True, True, False] ) xd = [1.01,2.02,r.NA,r.NAN,4.04] assert(r.as_character(xd) == ['1.01', '2.02', 'NA', 'NA', '4.04']) - assert(r.as_numeric(xd) == [1.01, 2.02, r.NAN, r.NAN, 4.04]) + assert(repr(r.as_numeric(xd)) == repr([1.01, 2.02, r.NAN, r.NAN, 4.04])) assert(r.as_integer(xd) == [1, 2, r.NA, r.NA, 4]) assert(r.as_factor(xd) == ['1.01', '2.02', 'NA', 'NA', '4.04']) assert(r.is_na(xd) == [False, False, True, True, False] ) From ndarray at mac.com Fri Jan 27 22:45:04 2006 From: ndarray at mac.com (Sasha) Date: Fri Jan 27 22:45:04 2006 Subject: [Numpy-discussion] [Rpy] RPy with NumPy Message-ID: I have succesfully (according to testall.py :-) converted RPy to use numpy instead of Numeric. Attached patch was tested on Linux only. I did not attempt to create a package that would support both numpy and Numeric, but it is easy to do. Please let me know if you are interested. For numpy-discussion readers: "RPy is a very simple, yet robust, Python interface to the R Programming Language." For rpy-list readers: "Development for Numeric has ceased, and users should transisition to NumPy as quickly as possible." For readers of both lists: Sorry for cross-posting. -- sasha -------------- next part -------------- diff -Naur rpy-0.4.6/rpy.py rpy-0.4.6-numpy/rpy.py --- rpy-0.4.6/rpy.py 2005-07-27 13:56:54.000000000 -0400 +++ rpy-0.4.6-numpy/rpy.py 2006-01-27 12:06:13.628796000 -0500 @@ -30,7 +30,7 @@ # installation time and RPy should have been compiled properly. So, # don't complain. try: - from Numeric import * + from numpy import * HAS_NUMERIC = 1 except ImportError: HAS_NUMERIC = 0 diff -Naur rpy-0.4.6/setup.py rpy-0.4.6-numpy/setup.py --- rpy-0.4.6/setup.py 2005-07-28 00:52:29.000000000 -0400 +++ rpy-0.4.6-numpy/setup.py 2006-01-28 01:18:46.148833000 -0500 @@ -34,7 +34,6 @@ from distutils.errors import * import rpy_tools - try: if sys.platform=="win32": RHOMES = os.environ['RHOMES'].split(';') @@ -72,7 +71,7 @@ if sys.platform=="win32": DEFINE.append(('R_HOME', '\\"%s\\"' % RHOME.replace("\\","/") )) else: - DEFINE.append(('R_HOME', '"%s"' % RHOME)) + DEFINE.append(('R_HOME', r'\"%s\"' % RHOME)) r_libs = [ os.path.join(RHOME, 'bin'), @@ -132,8 +131,10 @@ # check whether Numeric is present try: - import Numeric + import numpy DEFINE.append(('WITH_NUMERIC', None)) + DEFINE.append(('PY_ARRAY_TYPES_PREFIX', 'PyArray_')) + include_dirs.append(numpy.get_numpy_include()) except ImportError: UNDEF.append('WITH_NUMERIC') @@ -148,12 +149,11 @@ # use R version specific shared library name shlib_name = "_rpy%s" % RVER if sys.platform=="win32": - DEFINE.append( ('RPY_SHNAME', '\\"_rpy%s\\"' % RVER ) ) - else: DEFINE.append( ('RPY_SHNAME', '"_rpy%s"' % RVER ) ) + else: + DEFINE.append( ('RPY_SHNAME', r'\"_rpy%s\"' % RVER ) ) DEFINE.append( ('INIT_RPY', 'init_rpy%s' % RVER ) ) - modules.append( Extension( shlib_name, source_files, diff -Naur rpy-0.4.6/src/RPy.h rpy-0.4.6-numpy/src/RPy.h --- rpy-0.4.6/src/RPy.h 2005-07-26 23:05:29.000000000 -0400 +++ rpy-0.4.6-numpy/src/RPy.h 2006-01-26 17:18:37.750157000 -0500 @@ -68,12 +68,13 @@ #include +#ifdef WITH_NUMERIC +#include "numpy/arrayobject.h" +#endif + #include "robjobject.h" #include "setenv.h" -#ifdef WITH_NUMERIC -#include "Numeric/arrayobject.h" -#endif #define MAXIDSIZE 256 diff -Naur rpy-0.4.6/src/rpymodule.c rpy-0.4.6-numpy/src/rpymodule.c --- rpy-0.4.6/src/rpymodule.c 2005-07-28 10:23:33.000000000 -0400 +++ rpy-0.4.6-numpy/src/rpymodule.c 2006-01-28 01:19:29.281899000 -0500 @@ -1517,7 +1517,7 @@ if(use_numeric) { import_array(); - multiarray = PyImport_ImportModule("multiarray"); + multiarray = PyImport_ImportModule("numpy"); if (multiarray) { dict = PyModule_GetDict(multiarray); if (dict) diff -Naur rpy-0.4.6/tests/test_array.py rpy-0.4.6-numpy/tests/test_array.py --- rpy-0.4.6/tests/test_array.py 2005-07-18 14:54:54.000000000 -0400 +++ rpy-0.4.6-numpy/tests/test_array.py 2006-01-28 00:58:41.100628000 -0500 @@ -28,7 +28,7 @@ r.array.autoconvert(BASIC_CONVERSION) def testConversionToPy(self): - self.failUnless(self.py == self.ra.as_py(), + self.failUnless((self.py == self.ra.as_py()).all(), 'wrong conversion to Python') def testConversionToR(self): diff -Naur rpy-0.4.6/tests/test_numeric.py rpy-0.4.6-numpy/tests/test_numeric.py --- rpy-0.4.6/tests/test_numeric.py 2005-07-26 23:08:49.000000000 -0400 +++ rpy-0.4.6-numpy/tests/test_numeric.py 2006-01-27 12:07:38.309828000 -0500 @@ -2,7 +2,7 @@ import unittest try: - from Numeric import * + from numpy import * except ImportError: print 'Numeric not available. Skipping.\n' import sys diff -Naur rpy-0.4.6/tests/test_topy.py rpy-0.4.6-numpy/tests/test_topy.py --- rpy-0.4.6/tests/test_topy.py 2005-07-19 16:20:08.000000000 -0400 +++ rpy-0.4.6-numpy/tests/test_topy.py 2006-01-28 01:05:57.007083000 -0500 @@ -27,7 +27,7 @@ assert( r(-2147483648) == r.NA ) def testNAreal(self): - assert(r.NAN == r("as.numeric(NA)") ) + assert(repr(r.NAN) == repr(r("as.numeric(NA)")) ) def testNAstring(self): assert(r("as.character(NA)") == 'NA') @@ -48,14 +48,14 @@ def testNAList(self): xi = [1,2,r.NA,r.NAN,4] assert(r.as_character(xi) == ['1', '2', 'NA', 'NA', '4']) - assert(r.as_numeric(xi) == [1.0, 2.0, r.NAN, r.NAN, 4.0]) + assert(repr(r.as_numeric(xi)) == repr([1.0, 2.0, r.NAN, r.NAN, 4.0])) assert(r.as_integer(xi) == [1, 2, r.NA, r.NA, 4]) assert(r.as_factor(xi) == ['1', '2', 'NA', 'NA', '4']) assert(r.is_na(xi) == [False, False, True, True, False] ) xd = [1.01,2.02,r.NA,r.NAN,4.04] assert(r.as_character(xd) == ['1.01', '2.02', 'NA', 'NA', '4.04']) - assert(r.as_numeric(xd) == [1.01, 2.02, r.NAN, r.NAN, 4.04]) + assert(repr(r.as_numeric(xd)) == repr([1.01, 2.02, r.NAN, r.NAN, 4.04])) assert(r.as_integer(xd) == [1, 2, r.NA, r.NA, 4]) assert(r.as_factor(xd) == ['1.01', '2.02', 'NA', 'NA', '4.04']) assert(r.is_na(xd) == [False, False, True, True, False] ) From strawman at astraw.com Sat Jan 28 13:19:02 2006 From: strawman at astraw.com (Andrew Straw) Date: Sat Jan 28 13:19:02 2006 Subject: [Numpy-discussion] a virtual work party: help migrate scipy.org to new wiki Message-ID: <43DBDFA5.7010404@astraw.com> As many of you know, a number of us have been busy migrating the scipy.org website to a new, more user-friendly wiki system. The new wiki is up-and-running, is more-up-to-date, and prettier-to-look at than the old site. We are working to make scipy.org a portal website for all scientific software written in Python, and not just for the packages numpy and scipy, which happen to have their natural homes there. This wiki structure allows peer-review-by-wiki, so please do not feel bashful about updating any aspect of the wiki to suit your opinions. The current address of the new site is http://new.scipy.org/Wiki Once we go "live", we'll point http://scipy.org to this wiki. Before we make the switch, however, there are a few things that need to be done and a few things that should be done. This is a request for two things: 1) Review the page http://new.scipy.org/Wiki/MigratingFromPlone and make sure you can live with what it says. If you can't edit it. 2) Please undertake any of the tasks listed at that page. Update the page that you're working on the task, and again when you're done. 3) (Optional) Bask in the glory of our shiny new website which will allow NumPy/ SciPy/ Matplotlib/ IPython/ whateverelse to rule the scientific computing world! I'd like to ask that we try and complete these tasks as quickly as possible. I know I'm asking a busy group of people to pitch in, but I hope you'll agree the results will be worth it. Thank you, Andrew Straw PS Sorry for the cross postings. I thought it would be important to get as many hands involved as possible. From byrnes at bu.edu Sat Jan 28 14:20:04 2006 From: byrnes at bu.edu (John Byrnes) Date: Sat Jan 28 14:20:04 2006 Subject: [Numpy-discussion] Linux Installation Instructions Message-ID: <20060128221945.GB6662@localhost.localdomain> Is anyone else unable to access the Linux install instructions at http://pong.tamu.edu/tiki/tiki-view_blog_post.php?blogId=6&postId=97 I get a 403 - Forbidden Page. Regards, John -- Well done is better than well said. -- Benjamin Franklin From chris at trichech.us Sat Jan 28 14:31:01 2006 From: chris at trichech.us (Christopher Fonnesbeck) Date: Sat Jan 28 14:31:01 2006 Subject: [Numpy-discussion] array dtype=object problem Message-ID: <47C8EC1D-2FE0-48CD-B9CF-FF88CFBAD209@trichech.us> I am having trouble with numpy arrays that have (for some reason) an "object" dtype. All that happens in my code is that float values are appended to a multidimensional array, this array is transposed, and the transposed array is sent to matplotlib. Unfortunately, the array ends up being of type "object", which is not recognized by matplotlib: (Pdb) y[:5] array([575.060695363, 618.395159954, 817.767177867, 601.659726372, 978.607427424], dtype=object) (Pdb) scatter(x,y) *** TypeError: function not supported for these types, and can't coerce safely to supported types Moreover, this array, which clearly contains nothing but floats cannot be cast to a float array: (Pdb) array(y,float) *** TypeError: array cannot be safely cast to required type Can anyone shed some light on this? I am utterly confused. Thanks, C. -- Christopher J. Fonnesbeck Population Ecologist, Marine Mammal Section Fish & Wildlife Research Institute (FWC) St. Petersburg, FL Adjunct Assistant Professor Warnell School of Forest Resources University of Georgia Athens, GA T: 727.235.5570 E: chris at trichech.us -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2417 bytes Desc: not available URL: From ryanlists at gmail.com Sat Jan 28 14:55:02 2006 From: ryanlists at gmail.com (Ryan Krauss) Date: Sat Jan 28 14:55:02 2006 Subject: [Numpy-discussion] Re: [SciPy-user] Linux Installation Instructions In-Reply-To: <20060128221945.GB6662@localhost.localdomain> References: <20060128221945.GB6662@localhost.localdomain> Message-ID: I can't access it either right now. On 1/28/06, John Byrnes wrote: > Is anyone else unable to access the Linux install instructions at > http://pong.tamu.edu/tiki/tiki-view_blog_post.php?blogId=6&postId=97 > > I get a 403 - Forbidden Page. > > Regards, > John > > -- > Well done is better than well said. > -- Benjamin Franklin > > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.net > http://www.scipy.net/mailman/listinfo/scipy-user > From oliphant.travis at ieee.org Sat Jan 28 15:05:04 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sat Jan 28 15:05:04 2006 Subject: [Numpy-discussion] array dtype=object problem In-Reply-To: <47C8EC1D-2FE0-48CD-B9CF-FF88CFBAD209@trichech.us> References: <47C8EC1D-2FE0-48CD-B9CF-FF88CFBAD209@trichech.us> Message-ID: <43DBF896.6070600@ieee.org> Christopher Fonnesbeck wrote: > I am having trouble with numpy arrays that have (for some reason) an > "object" dtype. All that happens in my code is that float values are > appended to a multidimensional array, this array is transposed, and > the transposed array is sent to matplotlib. Unfortunately, the array > ends up being of type "object", which is not recognized by matplotlib: I'm not sure why that is, but I seem to recall from code that you've sent me before that you intentionally use object arrays from time to time. Object arrays stay object arrays unless you request differently. But, object arrays will usually run more slowly than fixed-precision arrays. > > (Pdb) scatter(x,y) > *** TypeError: function not supported for these types, and can't > coerce safely to supported types > > Moreover, this array, which clearly contains nothing but floats > cannot be cast to a float array: > > (Pdb) array(y,float) > *** TypeError: array cannot be safely cast to required type > > Can anyone shed some light on this? I am utterly confused. > When using the array constructor to cast to another type, there is a check that is equivalent to the numpy command can_cast(from_dtype, to_dtype) that is run to see if the conversion is "safe". Of course casting from an object (which could have infinite precision) to a fixed-precision floating-point data-type is not safe and you get this error. The .astype(to_dtype) method of numpy arrays will always convert (force-cast) to the request type. So, y.astype(float) will work as you want. -Travis From oliphant.travis at ieee.org Sat Jan 28 15:16:01 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sat Jan 28 15:16:01 2006 Subject: [Numpy-discussion] a virtual work party: help migrate scipy.org to new wiki In-Reply-To: <43DBDFA5.7010404@astraw.com> References: <43DBDFA5.7010404@astraw.com> Message-ID: <43DBFAF0.1090308@ieee.org> Andrew Straw wrote: >As many of you know, a number of us have been busy migrating the >scipy.org website to a new, more user-friendly wiki system. The new wiki >is up-and-running, is more-up-to-date, and prettier-to-look at than the >old site. > > Andrew has done an amazing job on this. I think the time has come to move forward with the new site. The most important pages have been moved over in my mind. There is *a lot* of outdated material on the old SciPy site. I think the best strategy is to just start using the new site with a link to the old stuff should someone realize that something is missing. My vote is to go "live" now. -Travis From strawman at astraw.com Sat Jan 28 15:20:01 2006 From: strawman at astraw.com (Andrew Straw) Date: Sat Jan 28 15:20:01 2006 Subject: [Numpy-discussion] a virtual work party: help migrate scipy.org to new wiki In-Reply-To: <43DBFAF0.1090308@ieee.org> References: <43DBDFA5.7010404@astraw.com> <43DBFAF0.1090308@ieee.org> Message-ID: <43DBFBE4.7010604@astraw.com> I'm happy with going live now, too. So, Enthought, I humbly ask you unleash the new site! :) Cheers! Andrew Travis Oliphant wrote: > Andrew Straw wrote: > >> As many of you know, a number of us have been busy migrating the >> scipy.org website to a new, more user-friendly wiki system. The new wiki >> is up-and-running, is more-up-to-date, and prettier-to-look at than the >> old site. >> >> > Andrew has done an amazing job on this. I think the time has come to > move forward with the new site. The most important pages have been > moved over in my mind. There is *a lot* of outdated material on the > old SciPy site. I think the best strategy is to just start using the > new site with a link to the old stuff should someone realize that > something is missing. > My vote is to go "live" now. > -Travis > > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log > files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From agn at noc.soton.ac.uk Sat Jan 28 16:37:03 2006 From: agn at noc.soton.ac.uk (George Nurser) Date: Sat Jan 28 16:37:03 2006 Subject: [Numpy-discussion] numpy can't use ACML In-Reply-To: References: <43DBE594.50301@gmail.com> Message-ID: <4A52806B-348E-4138-92B8-1D3F50E1D39B@noc.soton.ac.uk> I'm sure this is well known, but I just realized that numpy cannot use the _dotblas.so that it makes when it is compiled with ACML. This is because ACML only has the fortran blas libraries, not cblas. numpy will find the acml libraries and use them to make a _dotblas.so without complaining, if you have [blas] blas_libs = acml language = f77 in your site.cfg. But attempting to import this _dotblas.so gives errors .... so numpy never actually uses it. >>> import _dotblas.so Traceback (most recent call last): File "", line 1, in ? ImportError: ./_dotblas.so: undefined symbol: cblas_zaxpy nm _dotblas.so gives a whole stream of undefined cblas_xxxx symbols. So what are the options? Forget about ACML? Find an optimized cblas for numpy _dotblas but use the ACML flapack for scipy? Persuade somebody to write a scipy-style ?f2py interface to generate _dotblas.so using the fblas? George Nurser. From luszczek at cs.utk.edu Sat Jan 28 18:04:04 2006 From: luszczek at cs.utk.edu (Piotr Luszczek) Date: Sat Jan 28 18:04:04 2006 Subject: [Numpy-discussion] numpy can't use ACML In-Reply-To: <4A52806B-348E-4138-92B8-1D3F50E1D39B@noc.soton.ac.uk> References: <4A52806B-348E-4138-92B8-1D3F50E1D39B@noc.soton.ac.uk> Message-ID: <200601282102.53281.luszczek@cs.utk.edu> On Saturday 28 January 2006 19:36, George Nurser wrote: > I'm sure this is well known, but I just realized that numpy cannot > use the _dotblas.so that it makes when it is compiled with ACML. > This is because ACML only has the fortran blas libraries, not cblas. > > numpy will find the acml libraries and use them to make a _dotblas.so > without complaining, if you have > > [blas] > blas_libs = acml > language = f77 > in your site.cfg. > > But attempting to import this _dotblas.so gives errors .... so numpy > never actually uses it. > > >>> import _dotblas.so > > Traceback (most recent call last): > File "", line 1, in ? > ImportError: ./_dotblas.so: undefined symbol: cblas_zaxpy > > nm _dotblas.so gives a whole stream of undefined cblas_xxxx symbols. > > So what are the options? Forget about ACML? Find an optimized cblas > for numpy _dotblas but use the ACML flapack for scipy? Persuade > somebody to write a scipy-style ?f2py interface to generate > _dotblas.so using the fblas? > > George Nurser. There is code for that on netlib: http://www.netlib.org/blas/blast-forum/cblas.tgz I used it myself for my C code before and it worked just fine. Piotr From brendansimons at yahoo.ca Sat Jan 28 18:28:02 2006 From: brendansimons at yahoo.ca (Brendan Simons) Date: Sat Jan 28 18:28:02 2006 Subject: [Numpy-discussion] numpy.dot behavior with column vector and length 1 vector In-Reply-To: <20060128230601.C843888D45@sc8-sf-spam1.sourceforge.net> References: <20060128230601.C843888D45@sc8-sf-spam1.sourceforge.net> Message-ID: <015961AF-70D4-43F2-81D8-9FCC5E34E796@yahoo.ca> Is this behavior expected? If so, why am I wrong in assuming these statements should all return the same result? PyCrust 0.9.5 - The Flakiest Python Shell Python 2.4.1 (#2, Mar 31 2005, 00:05:10) [GCC 3.3 20030304 (Apple Computer, Inc. build 1666)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> colVect = numpy.ones((3, 1)) >>> colVect * 5.3 array([[ 5.3], [ 5.3], [ 5.3]]) >>> numpy.dot(colVect, 5.3) array([[ 5.3], [ 5.3], [ 5.3]]) >>> colVect * [5.3] array([[ 5.3], [ 5.3], [ 5.3]]) >>> numpy.dot(colVect, [5.3]) array([ 5.3, 0. , 0. ]) Brendan -- Brendan Simons __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From oliphant.travis at ieee.org Sat Jan 28 18:56:01 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sat Jan 28 18:56:01 2006 Subject: [Numpy-discussion] numpy.dot behavior with column vector and length 1 vector In-Reply-To: <015961AF-70D4-43F2-81D8-9FCC5E34E796@yahoo.ca> References: <20060128230601.C843888D45@sc8-sf-spam1.sourceforge.net> <015961AF-70D4-43F2-81D8-9FCC5E34E796@yahoo.ca> Message-ID: <43DC2E7D.9050609@ieee.org> Brendan Simons wrote: > Is this behavior expected? If so, why am I wrong in assuming these > statements should all return the same result? > > PyCrust 0.9.5 - The Flakiest Python Shell > Python 2.4.1 (#2, Mar 31 2005, 00:05:10) > [GCC 3.3 20030304 (Apple Computer, Inc. build 1666)] on darwin > Type "help", "copyright", "credits" or "license" for more information. > >>> import numpy > >>> colVect = numpy.ones((3, 1)) > >>> colVect * 5.3 > array([[ 5.3], > [ 5.3], > [ 5.3]]) > >>> numpy.dot(colVect, 5.3) > array([[ 5.3], > [ 5.3], > [ 5.3]]) > >>> colVect * [5.3] > array([[ 5.3], > [ 5.3], > [ 5.3]]) > >>> numpy.dot(colVect, [5.3]) > array([ 5.3, 0. , 0. ]) This is a dotblas bug. You can always tell by testing out the original dot function: from numpy.core.multiarray import dot as dot_ dot_(colVect, [5.3]) does not give this result. -Travis From oliphant.travis at ieee.org Sat Jan 28 19:53:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sat Jan 28 19:53:02 2006 Subject: [Numpy-discussion] numpy.dot behavior with column vector and length 1 vector In-Reply-To: <015961AF-70D4-43F2-81D8-9FCC5E34E796@yahoo.ca> References: <20060128230601.C843888D45@sc8-sf-spam1.sourceforge.net> <015961AF-70D4-43F2-81D8-9FCC5E34E796@yahoo.ca> Message-ID: <43DC3BE4.8080906@ieee.org> Brendan Simons wrote: > Is this behavior expected? If so, why am I wrong in assuming these > statements should all return the same result? This is now fixed in SVN. -Travis From brendansimons at yahoo.ca Sat Jan 28 20:47:04 2006 From: brendansimons at yahoo.ca (Brendan Simons) Date: Sat Jan 28 20:47:04 2006 Subject: [Numpy-discussion] numpy.dot behavior with column vector and length 1 vector In-Reply-To: <20060129041416.B07AA125B4@sc8-sf-spam2.sourceforge.net> References: <20060129041416.B07AA125B4@sc8-sf-spam2.sourceforge.net> Message-ID: <85D4958E-9208-4B4F-A662-77C1DB50BF73@yahoo.ca> > > --__--__-- > > Message: 7 > Date: Sat, 28 Jan 2006 20:52:04 -0700 > From: Travis Oliphant > To: Brendan Simons , > numpy-discussion > Subject: Re: [Numpy-discussion] numpy.dot behavior with column > vector and > length 1 vector > > Brendan Simons wrote: > >> Is this behavior expected? If so, why am I wrong in assuming these >> statements should all return the same result? > > > This is now fixed in SVN. > > -Travis Works for me now. Thanks! Brendan -- Brendan Simons __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From gerard.vermeulen at grenoble.cnrs.fr Sun Jan 29 00:10:01 2006 From: gerard.vermeulen at grenoble.cnrs.fr (Gerard Vermeulen) Date: Sun Jan 29 00:10:01 2006 Subject: [Numpy-discussion] bug triggered by array.astype() Message-ID: <20060129090918.005c440c.gerard.vermeulen@grenoble.cnrs.fr> >>> from numpy import * >>> core.__version__ '0.9.5.2019' >>> v = linspace(0, 2*pi) >>> c = (255<<24)*cos(v) + (255<<16)*cos(v+2*pi/3) + (255<<8)*cos(v+4*pi/3) + 255 >>> c.astype(UInt32) Traceback (most recent call last): File "", line 1, in ? SystemError: Objects/longobject.c:257: bad argument to internal function >>> Gerard From oliphant.travis at ieee.org Sun Jan 29 00:25:04 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sun Jan 29 00:25:04 2006 Subject: [Numpy-discussion] bug triggered by array.astype() In-Reply-To: <20060129090918.005c440c.gerard.vermeulen@grenoble.cnrs.fr> References: <20060129090918.005c440c.gerard.vermeulen@grenoble.cnrs.fr> Message-ID: <43DC7BC1.6050408@ieee.org> Gerard Vermeulen wrote: >>>>from numpy import * >>>>core.__version__ >>>> >>>> >'0.9.5.2019' > > >>>>v = linspace(0, 2*pi) >>>>c = (255<<24)*cos(v) + (255<<16)*cos(v+2*pi/3) + (255<<8)*cos(v+4*pi/3) + 255 >>>>c.astype(UInt32) >>>> >>>> >Traceback (most recent call last): > File "", line 1, in ? >SystemError: Objects/longobject.c:257: bad argument to internal function > > > > > Thanks for this test. The problem stems from the (255<<24) which (on 32-bit platform) generates a long-integer object. Currently, long-integers are converted to object arrays, thus this operation causes your entire calculation to be done using Python objects. Then, you try to convert the whole thing to UInt32 which is giving the error. I'll look into the error. In the meantime, you can avoid going through object arrays using an int64 scalar: int64(255<24)*cos(v) + ... -Travis From oliphant.travis at ieee.org Sun Jan 29 00:52:01 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sun Jan 29 00:52:01 2006 Subject: [Numpy-discussion] bug triggered by array.astype() In-Reply-To: <20060129090918.005c440c.gerard.vermeulen@grenoble.cnrs.fr> References: <20060129090918.005c440c.gerard.vermeulen@grenoble.cnrs.fr> Message-ID: <43DC81F1.3020309@ieee.org> Gerard Vermeulen wrote: >>>>from numpy import * >>>>core.__version__ >>>> >>>> >'0.9.5.2019' > > >>>>v = linspace(0, 2*pi) >>>>c = (255<<24)*cos(v) + (255<<16)*cos(v+2*pi/3) + (255<<8)*cos(v+4*pi/3) + 255 >>>>c.astype(UInt32) >>>> >>>> >Traceback (most recent call last): > File "", line 1, in ? >SystemError: Objects/longobject.c:257: bad argument to internal function > > > > > In SVN, now a Overflow error is raised instead in this example because negative long integer objects are trying to be converted to unsigned integers (this is a Python error being raised). We could check for this and instead do what the c-compiler would do in converting from signed to unsigned objects. Is that preferrable? -Travis From gerard.vermeulen at grenoble.cnrs.fr Sun Jan 29 01:19:02 2006 From: gerard.vermeulen at grenoble.cnrs.fr (Gerard Vermeulen) Date: Sun Jan 29 01:19:02 2006 Subject: [Numpy-discussion] bug triggered by array.astype() In-Reply-To: <43DC81F1.3020309@ieee.org> References: <20060129090918.005c440c.gerard.vermeulen@grenoble.cnrs.fr> <43DC81F1.3020309@ieee.org> Message-ID: <20060129101822.609fb955.gerard.vermeulen@grenoble.cnrs.fr> On Sun, 29 Jan 2006 01:50:57 -0700 Travis Oliphant wrote: [ .. ] > In SVN, now a Overflow error is raised instead in this example because > negative long integer objects are trying to be converted to unsigned > integers (this is a Python error being raised). We could check for this > and instead do what the c-compiler would do in converting from signed to > unsigned objects. > > Is that preferrable? > Well, the current SVN does now almost what Numeric does: >>> from Numeric import * >>> v = arange(0, 10, 1, Float32) >>> c = (255<<24)*cos(v)+ (255<<16)*cos(v+2*pi/3) + (255<<8)*cos(v+4*pi/3) + 255 >>> c.astype(UInt32) Traceback (most recent call last): File "", line 1, in ? OverflowError: long int too large to convert to int >>> However, numarray does: >>> from numarray import * >>> v = arange(0, 10, 1, Float32) >>> c = (255<<24)*cos(v)+ (255<<16)*cos(v+2*pi/3) + (255<<8)*cos(v+4*pi/3) + 255 >>> c.astype(UInt32) array([4269801984, 2294853120, 2504994432, 65861376, 1514949120, 1225005568, 4103764992, 3209541888, 3659448448, 398681088], type=UInt32) >>> This is more user friendly, but may cost more CPU cycles. Gerard From oliphant.travis at ieee.org Sun Jan 29 02:26:01 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sun Jan 29 02:26:01 2006 Subject: [Numpy-discussion] bug triggered by array.astype() In-Reply-To: <20060129101822.609fb955.gerard.vermeulen@grenoble.cnrs.fr> References: <20060129090918.005c440c.gerard.vermeulen@grenoble.cnrs.fr> <43DC81F1.3020309@ieee.org> <20060129101822.609fb955.gerard.vermeulen@grenoble.cnrs.fr> Message-ID: <43DC9806.8030904@ieee.org> Gerard Vermeulen wrote: >On Sun, 29 Jan 2006 01:50:57 -0700 >Travis Oliphant wrote: > >[ .. ] > > > >>In SVN, now a Overflow error is raised instead in this example because >>negative long integer objects are trying to be converted to unsigned >>integers (this is a Python error being raised). We could check for this >>and instead do what the c-compiler would do in converting from signed to >>unsigned objects. >> >>Is that preferrable? >> >> >> > >Well, the current SVN does now almost what Numeric does: > > > >>>>from Numeric import * >>>>v = arange(0, 10, 1, Float32) >>>>c = (255<<24)*cos(v)+ (255<<16)*cos(v+2*pi/3) + (255<<8)*cos(v+4*pi/3) + 255 >>>>c.astype(UInt32) >>>> >>>> >Traceback (most recent call last): > File "", line 1, in ? >OverflowError: long int too large to convert to int > > > >However, numarray does: > > > >>>>from numarray import * >>>>v = arange(0, 10, 1, Float32) >>>>c = (255<<24)*cos(v)+ (255<<16)*cos(v+2*pi/3) + (255<<8)*cos(v+4*pi/3) + 255 >>>>c.astype(UInt32) >>>> >>>> >array([4269801984, 2294853120, 2504994432, 65861376, 1514949120, > 1225005568, 4103764992, 3209541888, 3659448448, 398681088], type=UInt32) > > > >This is more user friendly, but may cost more CPU cycles. > > The issue here is really that in Numeric (and numpy) c is an object array while in numarray c is a float array. Because Numeric and numpy support object arrays, long integers have been converted to objects. I think what should be changed is that long integers should only be converted to objects if they are bigger than the maximum integer-type on the platform. That is what numarray allows and makes sense. In other words, the root of this difference is what array(255<<24) returns. In Numeric and numpy it returns an object array. In numarray it returns a UInt64 array. -Travis From oliphant.travis at ieee.org Sun Jan 29 02:40:08 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sun Jan 29 02:40:08 2006 Subject: [Numpy-discussion] bug triggered by array.astype() In-Reply-To: <20060129090918.005c440c.gerard.vermeulen@grenoble.cnrs.fr> References: <20060129090918.005c440c.gerard.vermeulen@grenoble.cnrs.fr> Message-ID: <43DC9B78.9010705@ieee.org> Gerard Vermeulen wrote: >>>>from numpy import * >>>>core.__version__ >>>> >>>> >'0.9.5.2019' > > >>>>v = linspace(0, 2*pi) >>>>c = (255<<24)*cos(v) + (255<<16)*cos(v+2*pi/3) + (255<<8)*cos(v+4*pi/3) + 255 >>>>c.astype(UInt32) >>>> >>>> >Traceback (most recent call last): > File "", line 1, in ? >SystemError: Objects/longobject.c:257: bad argument to internal function > > This behaves like numarray now in numpy SVN where-in (255<<24) is converted to an int64 array during the multiplication step so that object arrays are not constructed. -Travis From gerard.vermeulen at grenoble.cnrs.fr Sun Jan 29 05:00:01 2006 From: gerard.vermeulen at grenoble.cnrs.fr (Gerard Vermeulen) Date: Sun Jan 29 05:00:01 2006 Subject: [Numpy-discussion] performance problem Message-ID: <20060129135902.4b1291c5.gerard.vermeulen@grenoble.cnrs.fr> Hi Travis, thanks for your incredible quick fixes. numpy is about 5 times slower than numarray (on my numarray-friendly bi-pentium): >>> from timeit import Timer >>> import numarray; print numarray.__version__ 1.5.0 >>> import numpy; print numpy.__version__ 0.9.5.2021 >>> t1 = Timer('a <<= 8', 'import numarray as NX; a = NX.ones(10**6, NX.UInt32)') >>> t2 = Timer('a <<= 8', 'import numpy as NX; a = NX.ones(10**6, NX.UInt32)') >>> t1.timeit(100) 0.21813011169433594 >>> t2.timeit(100) 1.1523458957672119 >>> Numeric-23.1 is about as fast as numarray for inplace left shifts. Gerard From schofield at ftw.at Sun Jan 29 08:34:06 2006 From: schofield at ftw.at (Ed Schofield) Date: Sun Jan 29 08:34:06 2006 Subject: [Numpy-discussion] Segfault with dodgy type descriptors Message-ID: <43DCEE45.2020608@ftw.at> Hi Travis, Here's a segfault from the crazy input department: >>> cs = [('A', 1), ('B', 2)] >>> a = numpy.array(cs, dtype=(object,5)) Using the descriptor dtype=(object,x) for other values of x gives varying results, but all of them should probably raise a ValueError. On a related note, it seems that it's now possible again to use an array as a dtype. You changed this behaviour in November after I suggested that this should be illegal. (See http://www.scipy.net/pipermail/scipy-dev/2005-November/004126.html). My argument then was that calling array(a, b.dtype) is clearer and safer than using array(a, b), which can be confused with array((a,b)). Is it an oversight that arrays can again be used as dtypes? -- Ed From gerard.vermeulen at grenoble.cnrs.fr Sun Jan 29 10:14:04 2006 From: gerard.vermeulen at grenoble.cnrs.fr (Gerard Vermeulen) Date: Sun Jan 29 10:14:04 2006 Subject: [Numpy-discussion] another performance problem Message-ID: <20060129191327.49acbb26.gerard.vermeulen@grenoble.cnrs.fr> Hi Travis, max and min are really slow in numpy: >>> from timeit import Timer >>> import Numeric; Numeric.__version__ '23.1' >>> import numarray; numarray.__version__ '1.2.3' >>> import numpy; numpy.__version__ '0.9.5.2021' >>> t1 = Timer('a = max(b)', 'import Numeric as N; b=N.cos(N.arange(0, 10**5, 1, N.Float))') >>> t2 = Timer('a = max(b)', 'import numarray as N; b=N.cos(N.arange(0, 10**5, 1, N.Float))') >>> t3 = Timer('a = max(b)', 'import numpy as N; b=N.cos(N.arange(0, 10**5, 1, N.Float))') >>> t1.timeit(10) 0.13748002052307129 >>> t2.timeit(10) 0.64549708366394043 >>> t3.timeit(10) 4.5433549880981445 >>> Gerard From oliphant.travis at ieee.org Sun Jan 29 17:17:07 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sun Jan 29 17:17:07 2006 Subject: [Numpy-discussion] another performance problem In-Reply-To: <20060129191327.49acbb26.gerard.vermeulen@grenoble.cnrs.fr> References: <20060129191327.49acbb26.gerard.vermeulen@grenoble.cnrs.fr> Message-ID: <43DD68E1.30102@ieee.org> Gerard Vermeulen wrote: >Hi Travis, > >max and min are really slow in numpy: > > > >>>>from timeit import Timer >>>>import Numeric; Numeric.__version__ >>>> >>>> >'23.1' > > >>>>import numarray; numarray.__version__ >>>> >>>> >'1.2.3' > > >>>>import numpy; numpy.__version__ >>>> >>>> >'0.9.5.2021' > > >>>>t1 = Timer('a = max(b)', 'import Numeric as N; b=N.cos(N.arange(0, 10**5, 1, N.Float))') >>>>t2 = Timer('a = max(b)', 'import numarray as N; b=N.cos(N.arange(0, 10**5, 1, N.Float))') >>>>t3 = Timer('a = max(b)', 'import numpy as N; b=N.cos(N.arange(0, 10**5, 1, N.Float))') >>>>t1.timeit(10) >>>> >>>> >0.13748002052307129 > > >>>>t2.timeit(10) >>>> >>>> >0.64549708366394043 > > >>>>t3.timeit(10) >>>> >>>> >4.5433549880981445 > > You shouldn't be using max(b)!!! You should use b.max() because max is a Python function that is using the generic sequence interface to do the comparisons. Very likely it is slower because in numpy you get an array scalar (which don't yet have their own math defined and so use the full ufunc machinery do to computations). At some point the array scalars will have their own math defined and should be as fast as the Python scalars, so this particular "slowness" will go away. But, you use maximum.reduce() instead anyway. -Travis From faltet at carabos.com Sun Jan 29 22:38:01 2006 From: faltet at carabos.com (Francesc Altet) Date: Sun Jan 29 22:38:01 2006 Subject: [Numpy-discussion] Segfaults when trying to build an array using the description in array protocol Message-ID: <1138603012.7530.14.camel@localhost.localdomain> Hi, Apparently something wrong went in latest version of numpy SVN: In [4]:numpy.__version__ Out[4]:'0.9.5.2021' In [5]:numpy.empty((1,), dtype='i8') Out[5]:array([-5193749191705487200], dtype=int64) In [6]:numpy.empty((1,), dtype=[('p', 'i8')]) Segmentation fault This usually worked well with older versions: In [7]:numpy.__version__ Out[7]:'0.9.5.1980' In [8]:numpy.empty((1,), dtype=[('p', 'i8')]) Out[8]:array([(-5192306666809247592L,)], dtype=(void,8)) BTW, which is the recommended way to report that sort of things, the list or the Trac interface for SVN numpy? I'd like to start using the second, but I haven't manage to create a user for reporting issues. Thanks, -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From faltet at carabos.com Sun Jan 29 23:19:01 2006 From: faltet at carabos.com (Francesc Altet) Date: Sun Jan 29 23:19:01 2006 Subject: [Numpy-discussion] Segfaults when trying to build an array using the description in array protocol In-Reply-To: <43DDB96A.4030307@ieee.org> References: <1138603012.7530.14.camel@localhost.localdomain> <43DDB96A.4030307@ieee.org> Message-ID: <1138605500.7530.21.camel@localhost.localdomain> El dg 29 de 01 del 2006 a les 23:59 -0700, en/na Travis Oliphant va escriure: > I think you can report anonymously. I've just done it. However, I'd like to be informed about the follow-ups of the ticket by e-mail. I've put my e-mail address in the ticket, although I'm not sure if I'll get e-mail on this. Cheers, -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From oliphant.travis at ieee.org Mon Jan 30 00:24:01 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Jan 30 00:24:01 2006 Subject: [Numpy-discussion] performance problem In-Reply-To: <20060129135902.4b1291c5.gerard.vermeulen@grenoble.cnrs.fr> References: <20060129135902.4b1291c5.gerard.vermeulen@grenoble.cnrs.fr> Message-ID: <43DDCCE4.8090705@ieee.org> Gerard Vermeulen wrote: >Hi Travis, thanks for your incredible quick fixes. > >numpy is about 5 times slower than numarray (on my numarray-friendly bi-pentium): > > > This is a coercion issue. NumPy is exercising the most complicated section of the ufunc code requiring buffered casting because you are asking it to combine a uint32 array with a signed scalar (so the only commensurable type is a signed scalar of type int64. This is then coerced back into the unsigned array). Look at the data-type of b = a << 8. (a<<8).dtype.name 'int64' Try a<<=val where val = uint32(8) is in the header You should see more commensurate times. We can of course discuss the appropriateness of the coercion rules. Basically scalars don't cause coercion unless they are of a different kind but as of now signed and unsigned integers are considered to be of different kinds. I think that there is a valid point to be made that all scalar integers should be treated the same since Python only has one way to enter an integer. Right now, this is not what's done, but it could be changed rather easily. -Travis From matthew.brett at gmail.com Mon Jan 30 00:32:04 2006 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon Jan 30 00:32:04 2006 Subject: [Numpy-discussion] Verbose output from setup.py --help intentional? Message-ID: <1e2af89e0601300031k4790d3f5m96ddda611f4421e5@mail.gmail.com> Hi, Sorry if this is low priority, but I noticed that the output from setup.py --help is more than 400 lines. Also the output of (e.g) setup.py --help install and setup.py install --help differ (also more than 400 lines). Best, Matthew From oliphant.travis at ieee.org Mon Jan 30 00:53:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Jan 30 00:53:02 2006 Subject: [Numpy-discussion] performance problem In-Reply-To: <20060129135902.4b1291c5.gerard.vermeulen@grenoble.cnrs.fr> References: <20060129135902.4b1291c5.gerard.vermeulen@grenoble.cnrs.fr> Message-ID: <43DDD3E5.80902@ieee.org> Gerard Vermeulen wrote: >coercion issue snipped > > In current SVN, I think improved on the current state by only calling a scalar a signed integer if it is actually negative (previously only it's data-type was checked and all Python integers get converted to PyArray_LONG data-types which are signed integers). This fixes the immediate problem, I think. What are opinions on this scalar-coercion rule? -Travis From faltet at carabos.com Mon Jan 30 00:57:02 2006 From: faltet at carabos.com (Francesc Altet) Date: Mon Jan 30 00:57:02 2006 Subject: [Numpy-discussion] Problems with creating arrays from array descriptors and lists Message-ID: <1138611386.7530.28.camel@localhost.localdomain> Hi, I think this worked some time ago, but I cannot tell for sure: In [19]:numpy.__version__ Out[19]:'0.9.5.1980' In [20]:numpy.array([1,1], dtype=[('f1', 'i8'),('f2','i4')]) --------------------------------------------------------------------------- exceptions.TypeError Traceback (most recent call last) /home/faltet/ TypeError: expected a readable buffer object Sorry, I cannot check with latest SVN because it segfaults. -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From oliphant.travis at ieee.org Mon Jan 30 01:04:03 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Jan 30 01:04:03 2006 Subject: [Numpy-discussion] Segfault with dodgy type descriptors In-Reply-To: <43DCEE45.2020608@ftw.at> References: <43DCEE45.2020608@ftw.at> Message-ID: <43DDD65F.8090209@ieee.org> Ed Schofield wrote: >Hi Travis, > >Here's a segfault from the crazy input department: > > > >>>>cs = [('A', 1), ('B', 2)] >>>>a = numpy.array(cs, dtype=(object,5)) >>>> >>>> > >Using the descriptor dtype=(object,x) for other values of x gives >varying results, but all of them should probably raise a ValueError. > > These data-types are technically o.k. because (object, 5) means a data-type which is an array of 5 objects. The problem was that your constructor was not creating 5 objects and so an error should have been raised. It is now. >On a related note, it seems that it's now possible again to use an array >as a dtype. You changed this behaviour in November after I suggested >that this should be illegal. (See >http://www.scipy.net/pipermail/scipy-dev/2005-November/004126.html). My >argument then was that calling array(a, b.dtype) is clearer and safer >than using array(a, b), which can be confused with array((a,b)). Is it >an oversight that arrays can again be used as dtypes? > > I think this is an oversight, but I'm not exactly sure how to fix it. Basically during design of the new data-types, I allowed the possibility that any object with a dtype attribute could be considered a data-type. I wasn't thinking about arrays (which of course have that attribute as well), I was thinking about record objects with nested records and making it relatively easy to specify record data-types using objects with a dtype attribute. But, I could be persuaded (again), that it is easy enough to request the data-type attribute should you every actually need it, then have it specially-coded in the descriptor converter. In fact, I don't see any real benefit to it at all. My thinking about record objects changed during development and this could be just a holdover. -Travis From hurak at control.felk.cvut.cz Mon Jan 30 01:31:06 2006 From: hurak at control.felk.cvut.cz (=?ISO-8859-2?Q?Zden=ECk_Hur=E1k?=) Date: Mon Jan 30 01:31:06 2006 Subject: [Numpy-discussion] Numerical Mathematics Consortium vs. scipy and numpy Message-ID: Hello all, the other day I ran across the following link ----> http://www.nmconsortium.org/. Are you, Python numerics guys, aware of this? I admit I have absolutely no idea about the actual usefullness and relevancy of this consortium and their drafts. But it might be useful just to be aware... Important players in scientific computation are involved. Best regards, Zdenek Hurak From matthew.brett at gmail.com Mon Jan 30 04:20:06 2006 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon Jan 30 04:20:06 2006 Subject: [Numpy-discussion] ATLAS binaries - where are we? Message-ID: <1e2af89e0601300412o7e98a9bdy2b85c288e81d7d33@mail.gmail.com> Hi, I wonder if the forthcoming release of numpy is a good stimulus to revisit the issue of ATLAS binaries, as for: http://www.scipy.org/download/atlasbinaries/ and http://www.kevinsheppard.com/research/matlabatlas/matlab_atlas.aspx It seems to me that it will be very important to have ATLAS binaries, at very least for windows, which install seamlessly with NumPy. I am very happy to contribute builds, and can probably make a little windows compile farm at my lab (they _love_ windows where I work). Would that be useful? How would that best be integrated with the distribution? Best, Matthew From oliphant.travis at ieee.org Mon Jan 30 09:26:07 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Jan 30 09:26:07 2006 Subject: [Numpy-discussion] Problems with creating arrays from array descriptors and lists In-Reply-To: <1138614033.7530.57.camel@localhost.localdomain> References: <1138611386.7530.28.camel@localhost.localdomain> <43DDDB1B.8070007@ieee.org> <1138614033.7530.57.camel@localhost.localdomain> Message-ID: <43DE4BD5.3010709@ieee.org> Francesc Altet wrote: >El dl 30 de 01 del 2006 a les 02:23 -0700, en/na Travis Oliphant va >escriure: > > >>>In [20]:numpy.array([1,1], dtype=[('f1', 'i8'),('f2','i4')]) >>>--------------------------------------------------------------------------- >>>exceptions.TypeError Traceback (most >>>recent call last) >>> >>>/home/faltet/ >>> >>>TypeError: expected a readable buffer object >>> >>>Sorry, I cannot check with latest SVN because it segfaults. >>> >>> >>> >>> >>It shouldn't. Latest SVN works fine. >> >> > >True. > > > >>But, your example does not work (Records are only settable with tuples). >> >>numpy.array((1,1), ....) would work (returning an 0-d array) >> >> > >Why is this? I think that allowing lists as well would be highly useful. >In fact, I'm sure that many users are more used to provide lists instead >of tuples. Moreover, allowing tuples and not lists can be a bit >misleading, IMO. > > You can provide lists. It's just that fields must be entered in the array constructor as tuples to fill them in. This was the easiest to implement (it fit in naturally with the code already written from Numeric, and does not seem to me to be a real limitation and in fact can make things more readable. Thus, you can do numpy.array([[obj1, obj2, obj3]], dtype=) as long as the objects are tuples. -Travis From oliphant.travis at ieee.org Mon Jan 30 09:28:35 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Jan 30 09:28:35 2006 Subject: [Numpy-discussion] ATLAS binaries - where are we? In-Reply-To: <1e2af89e0601300412o7e98a9bdy2b85c288e81d7d33@mail.gmail.com> References: <1e2af89e0601300412o7e98a9bdy2b85c288e81d7d33@mail.gmail.com> Message-ID: <43DE4C8B.7020203@ieee.org> Matthew Brett wrote: >Hi, > >I wonder if the forthcoming release of numpy is a good stimulus to >revisit the issue of ATLAS binaries, as for: > >http://www.scipy.org/download/atlasbinaries/ > >and > >http://www.kevinsheppard.com/research/matlabatlas/matlab_atlas.aspx > >It seems to me that it will be very important to have ATLAS binaries, >at very least for windows, which install seamlessly with NumPy. > >I am very happy to contribute builds, and can probably make a little >windows compile farm at my lab (they _love_ windows where I work). >Would that be useful? How would that best be integrated with the >distribution? > > Absolutely useful... I suggest making a page on the new.scipy.org Wiki and uploading them there (to upload to the Wiki I think you create an attachment using attachment: Then, when the page is rendered you can click on the resulting link to upload the file). We could easily place links to this site for downloading. The old SciPy site already had such a page of pre-compiled ATLAS builds. Thanks for offering. -Travis From anthem at taipeilink.net Mon Jan 30 09:28:42 2006 From: anthem at taipeilink.net (=?windows-1251?B?MTQgLSAxNSD05eLw4Ov/IDIwMDYg4+7k4A==?=) Date: Mon Jan 30 09:28:42 2006 Subject: [Numpy-discussion] =?windows-1251?B?zc7C28kgzMXQ18DNxMDJx8jNwyAtIMXZxSDBzsvc2MUgz9DIwdvLyA==?= Message-ID: ?????????? ??????????? ?????????? ???????? ? ????????? ????, ?????????? ?? ???????, ?????????? ????????? ???????? ???????, ??????? ???????? ?????? ? ????????? "?????? ???????? 2005 ????"! ????? ????????????? - ??? ?????? ??????? 14 - 15 ??????? 2006 ???? ???????????? ??????? ?? ??????????????, ?????????? ???????, ??????? ?? ?????? ?????, ??? ??? ??????, ?? ? ??? ???????? ??????? ?????????????? ? ????????? ?????? ? ??????? ???????? ?????????? ??????????. ???? ????????: ???????????? ?????????? ???????? ?????????? ??????????? ?????? ?? ????? ???????, ???????????: ??????? ??????????? ??????? ???????????????? ??????????; ??????? ???????? ??????? "?????????" ? ???????????; ??????? ? ???????? ?????????, "?????????????" ?????????? ? ???????; ??????? ???????? ???????? ??????, ?????????? ? ?????? ??????????. ? ????????? ????????: ?????????? ??? ????? ?????????????? ??????? ??????????? ? ????????????? ??????????????. 1. ?????????? ???????????? ? ??????? ?????????????? ????????? ?????????? ??????????????? ???????? ? ??????????. ????? ?????????????? ? ??????????. ???? ????????????: ?????? ????? ?? ??????. 2. ??????? ?????????? ???????? ??????? ?????????? ???????? ??????? ????????? ? ????????? ? ???????? ???????. ????? ????????: ????? ?? ??????? ????. ???? ? ???????: ??????? ?????????? ?? ?????. 3. ????????? ???????? ?????????? ??????????, ???????? ?? ????????? ????????. ????????? ????????? ??? ??????? ???????. ?????????? ?????????. ????. ??????. ??????. 4. ?????????? ????????? ???????????? ??????? ???????????? ????????. ??????? ????????????? ??? . ??????????? ? ????????? ??????????. 5. ???????? ?????? ??????? ???????? ?????????????. ??????? ?????? ? ????? ????. ??????? "????????????? ???" ?? ??????". ??????? ????????????? ??????. ??????? ????????? ??????. ??????? ??????????? ??????. ??????? ??????? ?????? ?? ?????. ??????? "????? ? ??????????". ????????? ?????? ???????? "?????? ? ?????". 6. ???????????????? ?????????? ?????????????? ??????????. ??????? ??????? ????????? ?????????. ??????? ?????????? ????????. ???????? ???????, ?????????? ?????. ??????? ?????????????? ??????????. 7. ????????? ???????? ??? ????? ????????? ?????????????? ??????? ?????? ??????????. ??????????? ??????????. ???????? ???????????? ????????. ?????????? ?????? ?????? ? ???????????. ???????? ???????? ?????????? ??????????? ?? ????? ??????. ??????????? ?????????? ???????????. ?????? ???????? ?????? ?????????. ????? ???????? - ??????????? ??????? ? ??????? ??????????? ?????????????? ?????, ????? ??????? MBA. ??????? ?????????? ? ??????? ?????????? ?????? ?????????????? ? ????????? ????????. ????????? ????????????????? ???? ????????? "????????????? ???", ? ????? ?????? ????? ?? ??????? ??????? ? ??????????? ???????: DIXIS, LEFUTUR ? ??. ????? ????????????? ???????? - ?????????? ?????? ?????????????? ? ???????? ????????? ????????? ?? ??????? ????????? ???????, ??????, ???????????? ?????????? ? ??. ? ????????? ????? ?????????? ???????????? ??????? ?????????????? ? ???? ????????? ?????? "?????????? ????". ????? ??????????? ? ??????? ????? "????????????? - ??????? ??? ??????" ? ???????? ?????????? ?????? ? ?????????????????? ???????? ?? ???????? ??????????? ????????? ??????. ?????? ????????? ?? ?????? ???????? ? ????????????? ????????, ? ?????? ????????? ?????? ??????? ? ?????? ?????????? ??????. ???????: (495) 98?-65-39, 98?-65-36 -------------- next part -------------- An HTML attachment was scrubbed... URL: From service at paypal.com Mon Jan 30 09:52:09 2006 From: service at paypal.com (Paypal) Date: Mon Jan 30 09:52:09 2006 Subject: [Numpy-discussion] Your Paypal Account. Message-ID: <1130384585.13653@paypal.com> An HTML attachment was scrubbed... URL: From svetosch at gmx.net Mon Jan 30 10:19:02 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Mon Jan 30 10:19:02 2006 Subject: [Numpy-discussion] (some) generalized eigenvalues in numpy Message-ID: <43DE5841.6070006@gmx.net> Hi, in econometrics/statistics it's relatively common to solve a special generalized eigenvalue problem, where the matrices are real, symmetric, and positive definite. It bothered me to depend on scipy just because of that. Also, scipy.linalg.eig is much more general and returns complex types which is inconvenient for sorting etc. So I've written the workaround function below, but it's probably not very efficient (at least I hope the algebra is right). I'd be happy to hear your comments, suggestions; first with respect to the function itself, but also whether you agree that the implemented functionality would be useful enough to include in official numpy (in a more professional way than I have done). cheers, Sven def geneigsympos(A, B): """ Solves symmetric-positive-definite generalized eigenvalue problem Az=lBz. Takes two real-valued symmetric matrices A and B (B must also be positive-definite) and returns the corresponding generalized (but also real-valued) eigenvalues and eigenvectors. Return format: as in scipy.linalg.eig, tuple (l, Z); l is directly taken from eigh output (a 1-dim array of length A.shape[0] ?), and Z is an array or matrix (depending on type of input A) with the corresponding eigenvectors in columns (hopefully). Eigenvalues are ordered ascending (thanks to eigh). """ from numpy import asmatrix, asarray, linalg #fixme: input checks on the matrices LI = asmatrix(linalg.cholesky(B)).I C = LI * asmatrix(A) * LI.T evals, evecs = linalg.eigh(C) if type(A) == type(asarray(A)): output = "array" # A was passed as numpy-array else: output = "matrix" #but the evecs need to be transformed back: evecs = LI.T * asmatrix(evecs) if output == "array": return evals, asarray(evecs) else: return asmatrix(evals), evecs From Chris.Barker at noaa.gov Mon Jan 30 10:25:07 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon Jan 30 10:25:07 2006 Subject: [Numpy-discussion] Numerical Mathematics Consortium vs. scipy and numpy In-Reply-To: References: Message-ID: <43DE59CB.6070601@noaa.gov> Zdene(k Hur?k wrote: > the other day I ran across the following link ----> > http://www.nmconsortium.org/. Thanks for the pointer, it looks like a great project, let's hope it gets some traction. A few thoughts that are NumPy related: -- If nothing else, it's a pretty good list of stuff that "should" be included in Stock NumPy (and/or the core of SciPy). -- I notice they have: Matrix Operations, like: Elementwise Addition, etc. I don't, however, see any N-d array stuff: Too bad. (they do have "Multi-Dimensional Arrays Of Matrices" listed under present additional topics. Why array of matrices, rather than arrays of scalars? -- I'm personally not thrilled with the linear-algebra focus: for instance the matlab style for the operators: * matrix multiply .* elementwise addition. I'd rather see it the other way around. I wonder if we could get a Pythonista to join? In general, unfortunately, it looks to be quite commercially-focused: how does one get the open source community to be represented in this kind of thing? http://www.nmconsortium.org/client-area/registration/ -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From aisaac at american.edu Mon Jan 30 11:30:12 2006 From: aisaac at american.edu (Alan G Isaac) Date: Mon Jan 30 11:30:12 2006 Subject: [Numpy-discussion] Numerical Mathematics Consortium vs. scipy and numpy In-Reply-To: <43DE59CB.6070601@noaa.gov> References: <43DE59CB.6070601@noaa.gov> Message-ID: On Mon, 30 Jan 2006, Christopher Barker apparently wrote: > how does one get the open source community to be > represented in this kind of thing? http://www.nmconsortium.org/FldRte/?id=70&page=Downloads hth, Alan Isaac From pearu at scipy.org Mon Jan 30 13:22:01 2006 From: pearu at scipy.org (Pearu Peterson) Date: Mon Jan 30 13:22:01 2006 Subject: [Numpy-discussion] Verbose output from setup.py --help intentional? In-Reply-To: <1e2af89e0601300031k4790d3f5m96ddda611f4421e5@mail.gmail.com> References: <1e2af89e0601300031k4790d3f5m96ddda611f4421e5@mail.gmail.com> Message-ID: On Mon, 30 Jan 2006, Matthew Brett wrote: > Hi, > > Sorry if this is low priority, but I noticed that the output from > setup.py --help is more than 400 lines. Also the output of (e.g) > setup.py --help install and setup.py install --help differ (also more > than 400 lines). This issue is fixed in numpy SVN. Pearu From oliphant at ee.byu.edu Mon Jan 30 13:34:07 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Jan 30 13:34:07 2006 Subject: [Numpy-discussion] performance problem In-Reply-To: <20060129135902.4b1291c5.gerard.vermeulen@grenoble.cnrs.fr> References: <20060129135902.4b1291c5.gerard.vermeulen@grenoble.cnrs.fr> Message-ID: <43DE8635.2040901@ee.byu.edu> Gerard Vermeulen wrote: >0.9.5.2021 > > >>>>t1 = Timer('a <<= 8', 'import numarray as NX; a = NX.ones(10**6, NX.UInt32)') >>>>t2 = Timer('a <<= 8', 'import numpy as NX; a = NX.ones(10**6, NX.UInt32)') >>>>t1.timeit(100) >>>> >>>> >0.21813011169433594 > > >>>>t2.timeit(100) >>>> >>>> >1.1523458957672119 > > While ultimately, this slow-down was related to a coercion issue, I did still wonder about the extra dereference in the 1-d loop when one of the inputs is a scalar. So, I added a patch that checks for that case and defines a different loop. It seemed to give a small performance boost on my system. I'm wondering if such special-case coding is wise in general. Are there other ways to get C-compilers to produce faster code on modern machines? -Travis From Chris.Barker at noaa.gov Mon Jan 30 13:39:10 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon Jan 30 13:39:10 2006 Subject: [Numpy-discussion] Numerical Mathematics Consortium vs. scipy and numpy In-Reply-To: References: <43DE59CB.6070601@noaa.gov> Message-ID: <43DE8747.40705@noaa.gov> Alan G Isaac wrote: > On Mon, 30 Jan 2006, Christopher Barker apparently wrote: >> how does one get the open source community to be >> represented in this kind of thing? > > http://www.nmconsortium.org/FldRte/?id=70&page=Downloads Were you pointing us to any particular document there? Full membership (that is, with voting rights) requires a $2500.00 a year membership fee. I think it'll be a bit tricky for an open source project to raise that kind of cash. The result is that you have to buy your way into the group, which favors commercial entities. Granted, they have operating expenses, and $2500 isn't all that much, but it's probably too much for open-source groups to get representation. An associate membership is $250./year, which is far more manageable, but then you don't get voting rights: """ The Associate Membership with no voting rights will have full access to the consortium?s output and be engaged in some of the consortium?s activities and is typically for the user community, both academia and industry ($200 annual fee). """ The SciPy community is not a "user" group, it's a developer group. there are other open source projects worth of representation, such as the GNU Scientific Library. Oh well. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From gerard.vermeulen at grenoble.cnrs.fr Mon Jan 30 13:52:07 2006 From: gerard.vermeulen at grenoble.cnrs.fr (Gerard Vermeulen) Date: Mon Jan 30 13:52:07 2006 Subject: [Numpy-discussion] performance problem In-Reply-To: <43DDD3E5.80902@ieee.org> References: <20060129135902.4b1291c5.gerard.vermeulen@grenoble.cnrs.fr> <43DDD3E5.80902@ieee.org> Message-ID: <20060130225144.201784de.gerard.vermeulen@grenoble.cnrs.fr> On Mon, 30 Jan 2006 01:52:53 -0700 Travis Oliphant wrote: > Gerard Vermeulen wrote: > > >coercion issue snipped > > > > > > In current SVN, I think improved on the current state by only calling a > scalar a signed integer if it is actually negative (previously only it's > data-type was checked and all Python integers get converted to > PyArray_LONG data-types which are signed integers). > > This fixes the immediate problem, I think. > > What are opinions on this scalar-coercion rule? > Hmm, this is a consequence of your rule: >>> from numpy import *; core.__version__ '0.9.5.2024' >>> a = arange(3, dtype=uint32) >>> a-3 array([4294967293, 4294967294, 4294967295], dtype=uint32) >>> a+-3 array([-3, -2, -1], dtype=int64) >>> (a-3) == (a+-3) array([False, False, False], dtype=bool) Do you think that the speed-up justifies this? I don't think so. All my performance issues are discovered while writing demo examples which transfer data between a Python wrapped C++-library and Numeric, numarray, or numpy. In that state of mind, it surprises me when an uint32 ndarray gets coerced to an int64 ndarray. I rather prefer numarray's approach which raises an overflow error for the >>> a+-3 above. Agreed that sometimes a silent coercion is a good thing, but somebody who has started with an ubyte ndarray is likely to want an ubyte array in the end. I don't want to start a flame war, happy to write a - uint32(3) for numpy specific code. Gerard From oliphant.travis at ieee.org Mon Jan 30 13:53:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Jan 30 13:53:02 2006 Subject: [Numpy-discussion] Re: Status of 1.0 release and website? In-Reply-To: <463e11f90601231251n14d49c8dq4b919fb45fcf4150@mail.gmail.com> References: <463e11f90601231251n14d49c8dq4b919fb45fcf4150@mail.gmail.com> Message-ID: <43DE8A9B.2000101@ieee.org> Jonathan Taylor wrote: > I am wondering what needs to happen before a 1.0 release of both numpy > and scipy? For a 1.0 release of numpy: In terms of features. The only thing I see as necessary is that array scalars have their own math defined so that they don't go through the full ufunc machinery. This is probably about 20-40 hours of work. I would also like to see a numcompat module that allows numarray users to compile their C-extensions with a simple include file change. Also, a convert_numarraycode.py module would be useful for converting Python code. Mostly, though, I would like to see more people using NumPy in different circumstances so that we get the bugs worked out and needed C-API changes before going to 1.0. This is the process that may take several months as I see it. -Travis From Fernando.Perez at colorado.edu Mon Jan 30 13:57:03 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Mon Jan 30 13:57:03 2006 Subject: [Numpy-discussion] Re: Status of 1.0 release and website? In-Reply-To: <43DE8A9B.2000101@ieee.org> References: <463e11f90601231251n14d49c8dq4b919fb45fcf4150@mail.gmail.com> <43DE8A9B.2000101@ieee.org> Message-ID: <43DE8B71.8040809@colorado.edu> Travis Oliphant wrote: > Jonathan Taylor wrote: > >>I am wondering what needs to happen before a 1.0 release of both numpy >>and scipy? > > > For a 1.0 release of numpy: Small suggestion: I think this info would make an excellent 'timetable' page on our new wiki. It will give people a sense of where things stand, and what needs done before we go 1.0. I know people can always search the list, but something like this I think deserves more prominent mention. Cheers, f From oliphant at ee.byu.edu Mon Jan 30 13:59:02 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Jan 30 13:59:02 2006 Subject: [Numpy-discussion] performance problem In-Reply-To: <20060130225144.201784de.gerard.vermeulen@grenoble.cnrs.fr> References: <20060129135902.4b1291c5.gerard.vermeulen@grenoble.cnrs.fr> <43DDD3E5.80902@ieee.org> <20060130225144.201784de.gerard.vermeulen@grenoble.cnrs.fr> Message-ID: <43DE8C02.2040000@ee.byu.edu> Gerard Vermeulen wrote: >>In current SVN, I think improved on the current state by only calling a >>scalar a signed integer if it is actually negative (previously only it's >>data-type was checked and all Python integers get converted to >>PyArray_LONG data-types which are signed integers). >> >>This fixes the immediate problem, I think. >> >>What are opinions on this scalar-coercion rule? >> >> >> > >Hmm, this is a consequence of your rule: > > > >>>>from numpy import *; core.__version__ >>>> >>>> >'0.9.5.2024' > > >>>>a = arange(3, dtype=uint32) >>>>a-3 >>>> >>>> >array([4294967293, 4294967294, 4294967295], dtype=uint32) > > >>>>a+-3 >>>> >>>> >array([-3, -2, -1], dtype=int64) > > >>>>(a-3) == (a+-3) >>>> >>>> >array([False, False, False], dtype=bool) > >Do you think that the speed-up justifies this? I don't think so. > > > It's still hard to say if it justifies it or not. One way of writing a-3 causes automatic upcasting while the other way doesn't. This might be a good thing, depending on your point of view. I could see people needing both situations. These things are never as clear as we'd like them to be. But, I could also accept a rule that treated *all* integers as the same kind in which case a-3 and a+(-3) would always return the same thing. I'm fine with it either way. So what are other opinions? -Travis From ndarray at mac.com Mon Jan 30 14:39:02 2006 From: ndarray at mac.com (Sasha) Date: Mon Jan 30 14:39:02 2006 Subject: [Numpy-discussion] performance problem In-Reply-To: <43DE8635.2040901@ee.byu.edu> References: <20060129135902.4b1291c5.gerard.vermeulen@grenoble.cnrs.fr> <43DE8635.2040901@ee.byu.edu> Message-ID: On 1/30/06, Travis Oliphant wrote: > Are there other ways to > get C-compilers to produce faster code on modern machines? I would recommend to take a look at if you have not seen it before. Although wtitten by AMD, many recommendations apply to most modern CPUs. I've found Chapter 3 particulary informative. In fact I've changed my coding habits after reading some of their recommendations (for example "Use Array-Style Instead of Pointer-Style Code"). -- sasha From jh at oobleck.astro.cornell.edu Mon Jan 30 14:59:03 2006 From: jh at oobleck.astro.cornell.edu (Joe Harrington) Date: Mon Jan 30 14:59:03 2006 Subject: [Numpy-discussion] Numerical Mathematics Consortium vs. scipy and numpy In-Reply-To: <20060130220002.44B8B1368F@sc8-sf-spam2.sourceforge.net> (numpy-discussion-request@lists.sourceforge.net) References: <20060130220002.44B8B1368F@sc8-sf-spam2.sourceforge.net> Message-ID: <200601302258.k0UMw3PW006700@oobleck.astro.cornell.edu> Currently the NMC membership is tiny. Likely the linalg focus is due to the biases of that group. Getting an open-source project involved in something like this isn't so terrible, if the vendors allow it (i.e., if it isn't a sham for one set of vendors to gang up on another, as the Open Software Foundation was about 15 years ago). I notice that Research Systems (IDL) conspicuously isn't on their list (they'd definitely be more array-oriented). Also conspicuous by their absences are prominent groups like NAG and netlib. As for coming up with the fee, I think that it wouldn't be a huge investment for STScI and Enthought to split. If successful, NMC will impact both organizations quite a bit, so it should be worth it to them. Of course, I don't work for either organization so that's easy for me to say. Note that these people expect their members to do a lot of work, so the greater commitment might be time. However, the group is conveniently located in Austin... Alternatively, we could petition for a waiver based on our lack of profit. That's been done successfully in the past, too, for example by the Gutenprint folks (who then proceeded to teach the developers at Lexmark et al. a thing or two about dithering). XFree86.org had to beat on the X Consortium (or whatever they were called then) a bit, but eventually got in. I'm not sure what the deal was with fees for them. --jh-- From aisaac at american.edu Mon Jan 30 15:36:07 2006 From: aisaac at american.edu (Alan G Isaac) Date: Mon Jan 30 15:36:07 2006 Subject: [Numpy-discussion] Numerical Mathematics Consortium vs. scipy and numpy In-Reply-To: <43DE8747.40705@noaa.gov> References: <43DE59CB.6070601@noaa.gov><43DE8747.40705@noaa.gov> Message-ID: On Mon, 30 Jan 2006, Christopher Barker apparently wrote: > The SciPy community is not a "user" group, it's > a developer group. there are other open source projects > worth of representation, such as the GNU Scientific > Library. It would not hurt to ask for a waiver for open source developers. Whichever way it goes might contain a lot of information. Cheers, Alan From Chris.Barker at noaa.gov Mon Jan 30 16:48:02 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon Jan 30 16:48:02 2006 Subject: [Numpy-discussion] performance problem In-Reply-To: References: <20060129135902.4b1291c5.gerard.vermeulen@grenoble.cnrs.fr> <43DE8635.2040901@ee.byu.edu> Message-ID: <43DEB3A3.2090608@noaa.gov> Sasha wrote: > I would recommend to take a look at > Nice reference, thanks. From that: """ Copy Frequently Dereferenced Pointer Arguments to Local Variables: Avoid frequently dereferencing pointer arguments inside a function. Since the compiler has no knowledge of whether aliasing exists between the pointers, such dereferencing cannot be optimized away by the compiler. This prevents data from being kept in registers and significantly increases memory traffic. Note that many compilers have an ?assume no aliasing? optimization switch. This allows the compiler to assume that two different pointers always have disjoint contents and does not require copying of pointer arguments to local variables. Otherwise, copy the data pointed to by the pointer arguments to local variables at the start of the function and if necessary copy them back at the end of the function. """ Which perhaps helps answer Travis' original question. Did it make much difference in this case, Travis? -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From oliphant.travis at ieee.org Mon Jan 30 19:29:04 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Jan 30 19:29:04 2006 Subject: [Numpy-discussion] performance problem In-Reply-To: <43DEB3A3.2090608@noaa.gov> References: <20060129135902.4b1291c5.gerard.vermeulen@grenoble.cnrs.fr> <43DE8635.2040901@ee.byu.edu> <43DEB3A3.2090608@noaa.gov> Message-ID: <43DED96D.5080509@ieee.org> Christopher Barker wrote: > > Which perhaps helps answer Travis' original question. > > Did it make much difference in this case, Travis? Some difference in that case. For 10**6 elements, the relevant loops went from about 34 msec/loop (using the timeit module) to about 31 msec/loop. For a savings of about 3 msec/loop on my AMD platform. -Travis From vel.accel at gmail.com Tue Jan 31 01:47:07 2006 From: vel.accel at gmail.com (dHering) Date: Tue Jan 31 01:47:07 2006 Subject: [Numpy-discussion] Casting RecArray fields? Message-ID: <1e52e0880601310146t60454e29ue70971137b4daf1e@mail.gmail.com> Hi all, I need to cast fields of type 'Bool' to type 'Int32'. I'm not sure how to do this. Thank you, Dieter import numarray.records as rec b = rec.array([[23, 0],[45, 1]], names=' integer, boolean',formats='Int32, Bool') print b.field(1).type() # following line is not permissable # b.field(1) = b.field(1).astype('Int32') b.field(1)[:] = b.field(1).astype('Int32') print b.field(1).type() ########################## ########################## /usr/bin/python -u Bool Bool -------------- next part -------------- An HTML attachment was scrubbed... URL: From lito.kusnadi at gmail.com Tue Jan 31 02:33:02 2006 From: lito.kusnadi at gmail.com (Lito Kusnadi) Date: Tue Jan 31 02:33:02 2006 Subject: [Numpy-discussion] NumPy installed but aborted when do import Message-ID: Hi, I'm running RHEL 4.0 2.6.9-5.EL with Python 2.3.4. I managed to compile NumPy from src.rpm and make an i386 rpm from it. Version of NumPy is numpy-0.9.4-1. When I run Python interactive line and try to import NumPy, it aborts and kicks me to the command prompt. The following error is generated: Python 2.3.4 (#1, Nov 4 2004, 14:06:56) [GCC 3.4.2 20041017 (Red Hat 3.4.2-6.fc3)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from numpy import * import core -> failed: /usr/lib/python2.3/site-packages/numpy/core/multiarray.so: undefined symbol: PyOS_ascii_strtod import random -> failed: 'module' object has no attribute 'dtype' import lib -> failed: /usr/lib/python2.3/site-packages/numpy/core/multiarray.so: undefined symbol: PyOS_ascii_strtod Fatal Python error: can't initialize module lapack_lite Aborted Anyone knows what package I am missing? Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From faltet at carabos.com Tue Jan 31 02:46:01 2006 From: faltet at carabos.com (Francesc Altet) Date: Tue Jan 31 02:46:01 2006 Subject: [Numpy-discussion] Casting RecArray fields? In-Reply-To: <1e52e0880601310146t60454e29ue70971137b4daf1e@mail.gmail.com> References: <1e52e0880601310146t60454e29ue70971137b4daf1e@mail.gmail.com> Message-ID: <200601311144.45324.faltet@carabos.com> A Dimarts 31 Gener 2006 10:46, dHering va escriure: > Hi all, > > I need to cast fields of type 'Bool' to type 'Int32'. > I'm not sure how to do this. > > Thank you, > Dieter > > > import numarray.records as rec > > b = rec.array([[23, 0],[45, 1]], names=' integer, boolean',formats='Int32, > Bool') > > print b.field(1).type() > # following line is not permissable > # b.field(1) = b.field(1).astype('Int32') > b.field(1)[:] = b.field(1).astype('Int32') > print b.field(1).type() You can't do this because this implies a change on the column type definition (and also in the type size) of the recarray. One way that works for me, but that implies a complete data copy, is: In [15]: c = rec.array([b.field(0),b.field(1)], formats='Int32,Int32') In [16]: c.field(1) Out[16]: array([0, 1]) HTH, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From gerard.vermeulen at grenoble.cnrs.fr Tue Jan 31 02:53:04 2006 From: gerard.vermeulen at grenoble.cnrs.fr (Gerard Vermeulen) Date: Tue Jan 31 02:53:04 2006 Subject: [Numpy-discussion] NumPy installed but aborted when do import In-Reply-To: References: Message-ID: <20060131115245.3d538224.gerard.vermeulen@grenoble.cnrs.fr> On Tue, 31 Jan 2006 21:32:14 +1100 Lito Kusnadi wrote: > Hi, I'm running RHEL 4.0 2.6.9-5.EL with Python 2.3.4. > I managed to compile NumPy from src.rpm and make an i386 rpm from it. > Version of NumPy is numpy-0.9.4-1. > When I run Python interactive line and try to import NumPy, it aborts and > kicks me to the command prompt. > The following error is generated: > > Python 2.3.4 (#1, Nov 4 2004, 14:06:56) > [GCC 3.4.2 20041017 (Red Hat 3.4.2-6.fc3)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from numpy import * > import core -> failed: > /usr/lib/python2.3/site-packages/numpy/core/multiarray.so: undefined symbol: > PyOS_ascii_strtod > import random -> failed: 'module' object has no attribute 'dtype' > import lib -> failed: > /usr/lib/python2.3/site-packages/numpy/core/multiarray.so: undefined symbol: > PyOS_ascii_strtod > Fatal Python error: can't initialize module lapack_lite > Aborted > > Anyone knows what package I am missing? > There is a patch for Python-2.3 (PyOS_ascii_strtod does not exist in Python-2.3): http://sourceforge.net/project/showfiles.php?group_id=1369&package_id=175103 Look for: patch-for-py2.3 Gerard From vel.accel at gmail.com Tue Jan 31 02:56:00 2006 From: vel.accel at gmail.com (dHering) Date: Tue Jan 31 02:56:00 2006 Subject: [Numpy-discussion] Casting RecArray fields? In-Reply-To: <200601311144.45324.faltet@carabos.com> References: <1e52e0880601310146t60454e29ue70971137b4daf1e@mail.gmail.com> <200601311144.45324.faltet@carabos.com> Message-ID: <1e52e0880601310255g6a3eba66tedc303e5fefb4619@mail.gmail.com> On 1/31/06, Francesc Altet wrote: > > A Dimarts 31 Gener 2006 10:46, dHering va escriure: > > Hi all, > > > > I need to cast fields of type 'Bool' to type 'Int32'. > > I'm not sure how to do this. > > > > Thank you, > > Dieter > > > > > > import numarray.records as rec > > > > b = rec.array([[23, 0],[45, 1]], names=' integer, > boolean',formats='Int32, > > Bool') > > > > print b.field(1).type() > > # following line is not permissable > > # b.field(1) = b.field(1).astype('Int32') > > b.field(1)[:] = b.field(1).astype('Int32') > > print b.field(1).type() > > You can't do this because this implies a change on the column type > definition (and also in the type size) of the recarray. > > One way that works for me, but that implies a complete data copy, is: > > In [15]: c = rec.array([b.field(0),b.field(1)], formats='Int32,Int32') > > In [16]: c.field(1) > Out[16]: array([0, 1]) > > HTH, > > -- > >0,0< Francesc Altet http://www.carabos.com/ > V V C?rabos Coop. V. Enjoy Data > "-" Yes it helps very much Francesc. Thank you. Dieter -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris at trichech.us Tue Jan 31 06:48:57 2006 From: chris at trichech.us (Christopher Fonnesbeck) Date: Tue Jan 31 06:48:57 2006 Subject: [Numpy-discussion] numpy 0.9.4 for OSX Message-ID: <04A22C43-8960-4A1F-8E2C-97E2E3A1C2FD@trichech.us> I have added a OS X 10.4 build of numpy 0.9.4 to the sourceforge site. For (approximately) bi-weekly builds of numpy and scipy for OS X, go to http://trichech.us. Chris -- Christopher J. Fonnesbeck Population Ecologist, Marine Mammal Section Fish & Wildlife Research Institute (FWC) St. Petersburg, FL Adjunct Assistant Professor Warnell School of Forest Resources University of Georgia Athens, GA T: 727.235.5570 E: chris at trichech.us -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2417 bytes Desc: not available URL: From oliphant.travis at ieee.org Tue Jan 31 06:56:10 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue Jan 31 06:56:10 2006 Subject: [Numpy-discussion] Re: [SciPy-user] Strange behaviour of dot function In-Reply-To: <19508.1138639848@www036.gmx.net> References: <19508.1138639848@www036.gmx.net> Message-ID: <43DF7A55.3040602@ieee.org> Johannes L?hnert wrote: >Hi, > >I just found out that the dot function which multiplies matrices gives >strange results for a 3-dimensional array. Consider the following example: > > You just found two bugs in numpy.dot one of which is also there in Numeric. I just committed a fix to both bugs by using the ever-useful N-d array iterator (it sure makes it easier to write algorithms for strided arrays...). All three of your tests now produce the same answer. Thank you for finding this problem. -Travis From matthew.brett at gmail.com Tue Jan 31 09:01:02 2006 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue Jan 31 09:01:02 2006 Subject: [Numpy-discussion] gfortran, numpy build Message-ID: <1e2af89e0601310859y5b291420i6767e4efbb161a46@mail.gmail.com> Hi, I have the latest version of Mandrake, which has no g77, only gfortran - and needed the following patch to compile (with some added g77, gfortran outputs): Index: numpy/distutils/fcompiler/gnu.py =================================================================== --- numpy/distutils/fcompiler/gnu.py (revision 2033) +++ numpy/distutils/fcompiler/gnu.py (working copy) @@ -20,8 +20,13 @@ # GNU Fortran (GCC) 3.3.3 (Debian 20040401) # GNU Fortran 0.5.25 20010319 (prerelease) # Redhat: GNU Fortran (GCC 3.2.2 20030222 (Red Hat Linux 3.2.2-5)) 3.2.2 20030222 (Red Hat Linux 3.2.2-5) + # Mandrake: GNU Fortran (GCC) 3.3.6 (Mandrakelinux release 3.3.6-2mdk) + # + # 'gfortran --version' results + # Redhat: GNU Fortran 95 (GCC 4.0.2 20051125 (Red Hat 4.0.2-8)) + # Mandrake: GNU Fortran 95 (GCC 4.0.1 (4.0.1-5mdk for Mandriva Linux release 2006.0)) - for fc_exe in map(find_executable,['g77','f77']): + for fc_exe in map(find_executable,['g77','f77','gfortran']): if os.path.isfile(fc_exe): break executables = { @@ -41,7 +46,10 @@ if os.name != 'nt' and sys.platform!='cygwin': pic_flags = ['-fPIC'] - g2c = 'g2c' + if os.path.basename(fc_exe) == 'gfortran': + g2c = 'gfortran' + else: + g2c = 'g2c' #def get_linker_so(self): # # win32 linking should be handled by standard linker Can I suggest this a patch to svn numpy? Thanks, Matthew -------------- next part -------------- A non-text attachment was scrubbed... Name: gfortran.patch Type: text/x-diff Size: 1259 bytes Desc: not available URL: From faltet at carabos.com Tue Jan 31 09:17:07 2006 From: faltet at carabos.com (Francesc Altet) Date: Tue Jan 31 09:17:07 2006 Subject: [Numpy-discussion] A couple of errors in numpy/records.py Message-ID: <200601311816.25946.faltet@carabos.com> Hi, I've detected a couple of issues in records.py. For the first one, the next should fix it: Index: numpy/core/records.py =================================================================== --- numpy/core/records.py (revision 2033) +++ numpy/core/records.py (working copy) @@ -410,7 +410,8 @@ offset=offset) elif isinstance(obj, sb.ndarray): res = obj.view(recarray) - if issubclass(res.dtype, nt.void): + if issubclass(res.dtype.type, nt.void): res.dtype = sb.dtype((record, res.dtype)) + return res else: raise ValueError("Unknown input type") Now, another curious thing (sorry, no patch this time): In [6]: for i in numpy.rec.array([(1,2)], formats='u4,u4'): print i ...: (1L, 2L) [No problem with this] In [7]: for i in numpy.rec.array([(1,2)], formats='u4,u4')[0]: print i ...: --------------------------------------------------------------------------- exceptions.KeyError Traceback (most recent call last) /home/faltet/computacio.nobackup/hdf5-1.7.51/test/ /usr/lib/python2.3/site-packages/numpy/core/records.py in __getitem__(self, obj) 126 127 def __getitem__(self, obj): --> 128 return self.getfield(*(self.dtype.fields[obj][:2])) 129 130 def __setitem__(self, obj, val): KeyError: 0 I'd expect something like 1L 2L Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From oliphant.travis at ieee.org Tue Jan 31 11:36:23 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue Jan 31 11:36:23 2006 Subject: [Numpy-discussion] A couple of errors in numpy/records.py In-Reply-To: <200601311816.25946.faltet@carabos.com> References: <200601311816.25946.faltet@carabos.com> Message-ID: <43DFBC0B.7070007@ieee.org> Francesc Altet wrote: >Hi, > >I've detected a couple of issues in records.py. For the first one, the >next should fix it: > > Thanks for the patch. Also thank you for catching that we weren't allowing field access on the record objects. This is now fixed (by simply getting rid of the specialized getitem call and leeting the base-class take care of it). -Travis From bsouthey at gmail.com Tue Jan 31 12:05:05 2006 From: bsouthey at gmail.com (Bruce Southey) Date: Tue Jan 31 12:05:05 2006 Subject: [Numpy-discussion] gfortran, numpy build In-Reply-To: <1e2af89e0601310859y5b291420i6767e4efbb161a46@mail.gmail.com> References: <1e2af89e0601310859y5b291420i6767e4efbb161a46@mail.gmail.com> Message-ID: Hi, Perhaps it would also make sense to add g95 (http://g95.sourceforge.net/ or g95.org) which is different than gfortran ( http://gcc.gnu.org/wiki/TheOtherGCCBasedFortranCompiler). Regards Bruce On 1/31/06, Matthew Brett wrote: > > Hi, > > I have the latest version of Mandrake, which has no g77, only gfortran > - and needed the following patch to compile (with some added g77, > gfortran outputs): > > Index: numpy/distutils/fcompiler/gnu.py > =================================================================== > --- numpy/distutils/fcompiler/gnu.py (revision 2033) > +++ numpy/distutils/fcompiler/gnu.py (working copy) > @@ -20,8 +20,13 @@ > # GNU Fortran (GCC) 3.3.3 (Debian 20040401) > # GNU Fortran 0.5.25 20010319 (prerelease) > # Redhat: GNU Fortran (GCC 3.2.2 20030222 (Red Hat Linux > 3.2.2-5)) 3.2.2 20030222 (Red Hat Linux 3.2.2-5) > + # Mandrake: GNU Fortran (GCC) 3.3.6 (Mandrakelinux release 3.3.6-2mdk > ) > + # > + # 'gfortran --version' results > + # Redhat: GNU Fortran 95 (GCC 4.0.2 20051125 (Red Hat 4.0.2-8)) > + # Mandrake: GNU Fortran 95 (GCC 4.0.1 (4.0.1-5mdk for Mandriva > Linux release 2006.0)) > > - for fc_exe in map(find_executable,['g77','f77']): > + for fc_exe in map(find_executable,['g77','f77','gfortran']): > if os.path.isfile(fc_exe): > break > executables = { > @@ -41,7 +46,10 @@ > if os.name != 'nt' and sys.platform!='cygwin': > pic_flags = ['-fPIC'] > > - g2c = 'g2c' > + if os.path.basename(fc_exe) == 'gfortran': > + g2c = 'gfortran' > + else: > + g2c = 'g2c' > > #def get_linker_so(self): > # # win32 linking should be handled by standard linker > > Can I suggest this a patch to svn numpy? > > Thanks, > > Matthew > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pearu at scipy.org Tue Jan 31 12:21:02 2006 From: pearu at scipy.org (Pearu Peterson) Date: Tue Jan 31 12:21:02 2006 Subject: [Numpy-discussion] gfortran, numpy build In-Reply-To: References: <1e2af89e0601310859y5b291420i6767e4efbb161a46@mail.gmail.com> Message-ID: Hi, numpy.distutils has already support for both g95 and gfortran compilers. Run python setup.py config_fc --help-fcompiler to see a list of supported Fortran compilers. Pearu On Tue, 31 Jan 2006, Bruce Southey wrote: > Hi, > Perhaps it would also make sense to add g95 (http://g95.sourceforge.net/ or > g95.org) which is different than gfortran ( > http://gcc.gnu.org/wiki/TheOtherGCCBasedFortranCompiler). > > Regards > Bruce > > On 1/31/06, Matthew Brett wrote: >> >> Hi, >> >> I have the latest version of Mandrake, which has no g77, only gfortran >> - and needed the following patch to compile (with some added g77, >> gfortran outputs): >> >> Index: numpy/distutils/fcompiler/gnu.py >> =================================================================== >> --- numpy/distutils/fcompiler/gnu.py (revision 2033) >> +++ numpy/distutils/fcompiler/gnu.py (working copy) >> @@ -20,8 +20,13 @@ >> # GNU Fortran (GCC) 3.3.3 (Debian 20040401) >> # GNU Fortran 0.5.25 20010319 (prerelease) >> # Redhat: GNU Fortran (GCC 3.2.2 20030222 (Red Hat Linux >> 3.2.2-5)) 3.2.2 20030222 (Red Hat Linux 3.2.2-5) >> + # Mandrake: GNU Fortran (GCC) 3.3.6 (Mandrakelinux release 3.3.6-2mdk >> ) >> + # >> + # 'gfortran --version' results >> + # Redhat: GNU Fortran 95 (GCC 4.0.2 20051125 (Red Hat 4.0.2-8)) >> + # Mandrake: GNU Fortran 95 (GCC 4.0.1 (4.0.1-5mdk for Mandriva >> Linux release 2006.0)) >> >> - for fc_exe in map(find_executable,['g77','f77']): >> + for fc_exe in map(find_executable,['g77','f77','gfortran']): >> if os.path.isfile(fc_exe): >> break >> executables = { >> @@ -41,7 +46,10 @@ >> if os.name != 'nt' and sys.platform!='cygwin': >> pic_flags = ['-fPIC'] >> >> - g2c = 'g2c' >> + if os.path.basename(fc_exe) == 'gfortran': >> + g2c = 'gfortran' >> + else: >> + g2c = 'g2c' >> >> #def get_linker_so(self): >> # # win32 linking should be handled by standard linker >> >> Can I suggest this a patch to svn numpy? >> >> Thanks, >> >> Matthew >> >> >> > From oliphant.travis at ieee.org Tue Jan 31 12:34:06 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue Jan 31 12:34:06 2006 Subject: [Numpy-discussion] Re: [Numpy-user] possible error with isarrtype In-Reply-To: References: Message-ID: <43DFC973.70902@ieee.org> O'Keefe, Michael wrote: >I was just updating my code to work with python 2.4.2 and numpy 0.9.4 and observed the following behavior with the function isarrtype. Just thought I would report it (see below). > > > Thanks for this report. I've fixed the problem, but you should probably not be using these functions anymore. These were put in place before the data-type objects materialized. In the past, what is now the typeobject of the array scalars was thought to be the basic way to determine the data-type of the array. Now, the data-type object itself is the preferred way. I need to figure out what to do with these functions now and see if they even have any use anymore. What use were you making of them? Best, -Travis From Michael_OKeefe at nrel.gov Tue Jan 31 13:07:14 2006 From: Michael_OKeefe at nrel.gov (O'Keefe, Michael) Date: Tue Jan 31 13:07:14 2006 Subject: [Numpy-discussion] RE: [Numpy-user] possible error with isarrtype Message-ID: > Now, the data-type object itself is the preferred way. I need to figure > out what to do with these functions now and see if they even have any > use anymore. Thanks Travis. With regard to what to do with the functions, I only ended up trying them because they were there. From my point of view, I'd love to learn how to code the "right" way. To that end, any means of informing the user of "deprecated" functions (or just taking them out) would be fine with me. However, I realize some may have legacy code and so I would defer to their opinions. > What use were you making of them? I was just migrating some working scipy/numeric code I'd been using to the newer numpy (0.9.4), python (2.4.2), and scipy (0.4.4). When I ran my unit-tests, one line that gave me problems was: import scipy as sp ... assert( type(deltaTime_sec)==float ) This line of code was there to inform me if I was accidentally passing in an array type instead of a scalar. This no longer worked in the new environment. The type of deltaTime_sec came out as float64_arrtype which did not equal type float causing the assertion to trip which I believe to be caused by the following: >>> a=sp.array([1.],float) >>> type(a) >>> type(a[0]) # somewhere I assign deltaTime_sec = someArray[0] I ended up fixing it with the following code which seems to do the trick (though I'm always keen to learn better ways of doing things): assert( sp.size(deltaTime_sec)==1 ) Thanks! Michael From oliphant.travis at ieee.org Tue Jan 31 13:23:03 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue Jan 31 13:23:03 2006 Subject: [Numpy-discussion] Re: [Numpy-user] possible error with isarrtype In-Reply-To: References: Message-ID: <43DFD50D.1000200@ieee.org> O'Keefe, Michael wrote: >I was just migrating some working scipy/numeric code I'd been using to the newer numpy (0.9.4), python (2.4.2), and scipy (0.4.4). > >When I ran my unit-tests, one line that gave me problems was: > >import scipy as sp >... >assert( type(deltaTime_sec)==float ) > >This line of code was there to inform me if I was accidentally passing in an array type instead of a scalar. This no longer worked in the new environment. The type of deltaTime_sec came out as float64_arrtype which did not equal type float causing the assertion to trip which I believe to be caused by the following: > > Ah... This is something that needs to be added to the FAQ, because it is common. Basically, you should almost never test for type equality like this since with >Python2.2 the Python float can be inherited from (that's what a float64scalar does --- used to be called float64_arrtype). So, instead you should use assert (isinstance(deltaTime_sec, float)) which would still work with the new float scalar objects. The big advantage of the new scalar types and objects is that there is a 1-1 relationship between the scalar types and the basic array data-types and the new scalars have all the methods and attributes of arrays. So, in fact you could write: assert(deltaTime_sec.size == 1) and this would work whether deltaTime is a scalar or an array with only one element. -Travis From Fernando.Perez at colorado.edu Tue Jan 31 13:25:15 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Tue Jan 31 13:25:15 2006 Subject: [Numpy-discussion] RE: [Numpy-user] possible error with isarrtype In-Reply-To: References: Message-ID: <43DFD598.5000503@colorado.edu> O'Keefe, Michael wrote: >> Now, the data-type object itself is the preferred way. I need to figure >> out what to do with these functions now and see if they even have any >> use anymore. > > > Thanks Travis. > > With regard to what to do with the functions, I only ended up trying them > because they were there. From my point of view, I'd love to learn how to > code the "right" way. To that end, any means of informing the user of > "deprecated" functions (or just taking them out) would be fine with me. > However, I realize some may have legacy code and so I would defer to their > opinions. Python has a proper framework for deprecation warnings. I'd say we use that, and Travis can ruthlessly deprecate anything that's just compatibility noise. Perhaps 1.0 can keep old but deprecated things around (numpy will see a LOT of use once 1.0 comes out, far more than the current svn testing crowd). Then, as of 1.1 or 1.2 or whatever, those can be removed. Cheers, f From cookedm at physics.mcmaster.ca Tue Jan 31 15:42:01 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Tue Jan 31 15:42:01 2006 Subject: [Numpy-discussion] bdist_egg trips up over non-extension extensions In-Reply-To: References: <43D6529C.8030105@gmail.com> <43D68CC4.1010004@gmail.com> Message-ID: <90D73325-334E-4A59-8FB8-9C20904BA12C@physics.mcmaster.ca> On Jan 24, 2006, at 14:36 , Pearu Peterson wrote: > On Tue, 24 Jan 2006, Robert Kern wrote: > >> Pearu Peterson wrote: >>> Could you try recent version of numpy from SVN? __config__.py is now >>> generated via py_modules list. >> >> It doesn't build an egg, yet, because setuptools expects >> Distribution.py_modules >> to contain only strings, not tuples. Modifying setuptools to >> handle tuples in >> Distribution.py_modules seems to work. Is there any way you can do >> it without >> tuples? > > Source generators in numpy.distutils are extensions to distutils > and if setuptools assumes std distutils then source generators will > remain a problem for setuptools. However, if you call build_src > then all source generators will be resolved (including 3-tuples in > py_modules list) and the file lists in distribution instance should > not cause any problems for setuptools. > > So, would calling `setup.py build_src bdist_egg` be an acceptable > solution? Fixed in svn. I added a wrapper around the setuptools.command.egg_info.egg_info.run method that runs the build_src command first. -- |>|\/|< /------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From matthew.brett at gmail.com Tue Jan 31 16:11:13 2006 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue Jan 31 16:11:13 2006 Subject: [Numpy-discussion] gfortran, numpy build In-Reply-To: References: <1e2af89e0601310859y5b291420i6767e4efbb161a46@mail.gmail.com> Message-ID: <1e2af89e0601311602l4c1f363fpb11d1d405ababf7e@mail.gmail.com> > numpy.distutils has already support for both g95 and gfortran compilers. > Run > python setup.py config_fc --help-fcompiler > to see a list of supported Fortran compilers. Oops - sorry, I completely missed that - and thanks a lot for the pointer. For those as slow as I am, the build command seemed to be: python setup.py config --fcompiler=gnu95 build I am afraid this is a sign that I am lost in the build system. Is there a good place to start to get to grips with options like the above? And, is there a simple way to pass my favorite compiler optimization flags into the build? Thanks a lot, Matthew From cjw at sympatico.ca Tue Jan 31 19:53:02 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Tue Jan 31 19:53:02 2006 Subject: [Numpy-discussion] RE: [Numpy-user] possible error with isarrtype In-Reply-To: <43DFD598.5000503@colorado.edu> References: <43DFD598.5000503@colorado.edu> Message-ID: <43E0309D.5050700@sympatico.ca> Fernando Perez wrote: > O'Keefe, Michael wrote: >>> Now, the data-type object itself is the preferred way. I need to >>> figure >>> out what to do with these functions now and see if they even have >>> any use anymore. >> >> >> Thanks Travis. >> >> With regard to what to do with the functions, I only ended up trying >> them >> because they were there. From my point of view, I'd love to learn how to >> code the "right" way. To that end, any means of informing the user of >> "deprecated" functions (or just taking them out) would be fine with me. >> However, I realize some may have legacy code and so I would defer to >> their >> opinions. > > > Python has a proper framework for deprecation warnings. I'd say we > use that, and Travis can ruthlessly deprecate anything that's just > compatibility noise. > > Perhaps 1.0 can keep old but deprecated things around (numpy will see > a LOT of use once 1.0 comes out, far more than the current svn testing > crowd). Then, as of 1.1 or 1.2 or whatever, those can be removed. > > Cheers, > > f > One of the deprecated names is ArrayType. This seems to be closer to the Python style than ndarray. Colin W. From oliphant.travis at ieee.org Tue Jan 31 20:05:18 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue Jan 31 20:05:18 2006 Subject: [Numpy-discussion] RE: [Numpy-user] possible error with isarrtype In-Reply-To: <43E0309D.5050700@sympatico.ca> References: <43DFD598.5000503@colorado.edu> <43E0309D.5050700@sympatico.ca> Message-ID: <43E03349.10208@ieee.org> Colin J. Williams wrote: > One of the deprecated names is ArrayType. This seems to be closer to > the Python style than ndarray. Not really. Rather than test: type(var) == types.IntType you should be testing isinstance(var, int) just like rather than testing type(somearray) == ArrayType you should be testing isinstance(somearray, ndarray) Python style has changed a bit since 2.2 allowed sub-typing builtings -Travis From accession at mail.ru Tue Jan 31 21:45:01 2006 From: accession at mail.ru (=?windows-1251?B?wvvz9+jy/CDg7ePr6Onx6ujpIC0g7/Du8fLu?=) Date: Tue Jan 31 21:45:01 2006 Subject: [Numpy-discussion] =?windows-1251?B?wO3j6+jp8ero6SDt4CDz8O7i7eUg7e7x6PLl6/8g/+f76uAh?= Message-ID: <4adf01c626f1$41036721$d6d74eb6@mail.ru> ?? ?????? ??? ?? ???????????????? ??????????? ????? ??????????? ?? ???? ????????? ??????????? ??? ???????????? ? ??????? ??????????? ??????????? ????? ?? 3 ?????? ????????? ?????????? ??????????? ?????! ? ?????? ?????????? ????????? ????????? ???????? ????, ??????, ???????, ??: ? ?????????, ???? ???? ?? ??????, ???? ?? ???????????????? ?? ???????? ??????. ???? ????? ???????? ????? ? ?????????? ????????????? ?????, ???? ????????? ????????? ??????, ???????? ??????? ? ??????????? ??????????? ?? ???? ?????????? ?????. ?????????? ??? ????? ?? ?????? ?????? ???????? ??????, ??????? ?? ???????? ??????????? ????, ??? ???? ? ????, - ???????? ???????? ?? ????????? ?????????? ? ?????????? ?? ?????? ???????. ????? ????, ?? ????? ??????? ??????? ???? ????????? ????? ?? ?????? ???????? ?????. ???????????? ???????? ? ?? ???????????????? ???????? ?????????? ?????. ???????? ???????? ? ???? ?????????? ????????? ??????????? ????????????, ????????? ??????? ?????? ???????????? ????????? ? ??? ????? 3 ?????? ???????? ? ???????? ???????? ? ??????. ? ???????? ???????? ?? ?????????: ???????????? ???? ?? ???? ??????? ?????????? ?????? ??????, ? ??? ????? ????? ? ????????? ??????, ? ????? ???????????? ???????? ????????? ????? ???????????? ?????? ???????????? ???????????? ???????? ???? ?????? ??????? ??????????? ??????? ?????????? ?????????? ?????? ? ????????????????? ??????? ?????? ? ?????????? ???? ??????? ???????????? ?????? ???????????? ???????? ?????? ????????? ???????? ?????? ????????? ???????? ??????????? ?????????? ? ???????? ?????????? ????? ???? ???????? ???????? ???????? ????? ???????? - ??? ???????????????, ??????? ? ???, ?????? ????????? ? ??? ? ?? ?????? ????????? - ?????????????? ???????????? ???? ????????????? ??????? ???????? ??????????, ????????-????????, c??????? ??? ????? ? ???? ?????? ??????????? ???, ??????????? ? ????????????? ???????????? ?? ??????? ??????? ???????????????? ???????????? ? ????????? ???????????? ????????. ?????????? ?? ???????????????? ??????????? ?????? ????? ????? ? ??? ??????? ???????? ???????? ?? ????????? ??????? ????? ???????? ????????, ????????????? ???????????? Who's Who of Professionals, ?????????? ????? ????????????? ?????????? ???????? ???? ? ??. ???????? ???????. ? 9 ?? 23 ??????? 2005 ?. ????? ?????? ????? ????-????? ??????????? ????? ? ???????????????? ?????????? ??? ??????? ??????????? (?????? ??????????) ??????????? ??????????? ?????? - ??? ?????. ??????? ????? - ?? ?????? ?????????? ??????? ???? ???????? ??????????? ????? ?? CD ? DVD. ??? ????? ????? ?????????? ????? ?? 1-2 ???? ?????? ???? ? ??????? 2-3 ???????. ????-??????? ????? - 16 ??? 40 ????? ???????? ??????????? ??????????????? ??????? ???????? ? 4 ???? ???????????????? ?????????? + ?????? ??????? ???????? ??????? ???? ??? ??????????????? ???????. ??????? ?????? ? ??????? ???? ???????? ????? ??????? ???????? ?????????? ?????? ? ??????????. ????????????? ???????? - ????? ???????? ????? ???????? ? ?????? ??????? ??????? ?? ???????????? ?????????. ? ???? ????? ????????????? ??????? ????????? ??????? ?? ????? ???????. ??????? ? ??????????? ?????????? ????? ??????????? ??????? ? ?????? ?????? ????????????. ????? ????, ?? ???? ???? ????????? ? ?????????, ????????? ???????? ?? ??????????? ?????, ?? ??????? ????. ???? ?? ?????????????? ? ???????? ?????? ?????????, ?, ????????, ???-???????????, ??????????? ? ??? ??? ????????, ???????? ??????????? ???????? ????????? (???????? + ????? ?? ???????). ????????? ???: ??? ???? ??????: ???. +7 (926) 197-99-05 ??????? ?????: ???. +7(495) 291-72-64 (????? 18.00), +7 (916) 855-17-34 ????????: englishbest at netscape.net -------------- next part -------------- An HTML attachment was scrubbed... URL: From edcjones at comcast.net Mon Jan 2 09:54:11 2006 From: edcjones at comcast.net (Edward C. Jones) Date: Mon Jan 2 09:54:11 2006 Subject: [Numpy-discussion] Clipping, casting, and maximum values Message-ID: <43B968AE.8090406@comcast.net> #! /usr/bin/env python import numarray from numarray.numerictypes import * # Is this program too complicated? # On my machine (AMD64 chip, Debian unstable i386, numarray 1.5.0): # float(2**63-1) == float(2**63) def clipcast(farr): if farr.type() != Float64: raise TypeError('input must be a Float64 array') farr = numarray.clip(farr, 0.0, float(2**63)) print 'L13 %30.4f' % farr[0] top = (farr == float(2**63)) numarray.Error.pushMode(invalid='ignore') arr = farr.astype(Int64) numarray.Error.popMode() return numarray.where(top, 2**63-1, arr) farr = numarray.array([1.0e300], Float64) print farr.type(), farr[0] arr = clipcast(farr) print arr.type(), arr[0] From jmiller at stsci.edu Tue Jan 3 04:18:12 2006 From: jmiller at stsci.edu (Todd Miller) Date: Tue Jan 3 04:18:12 2006 Subject: [Numpy-discussion] Clipping, casting, and maximum values In-Reply-To: <43B968AE.8090406@comcast.net> References: <43B968AE.8090406@comcast.net> Message-ID: <43BA6B5F.1030604@stsci.edu> On my machine running FC4 x86_64, I get: Float64 1e+300 L13 9223372036854775808.0000 Int64 9223372036854775807 What are you getting? What did you expect to get? At 19 digits, the numbers we're talking about are outside the precision of Float64 which is ~16 digits. I did see one (for me) surprise: >>> farr = numarray.clip(farr, 0.0, float(2**63)) >>> farr array([ 9.22337204e+18]) >>> numarray.Error.pushMode(invalid='ignore') >>> arr = farr.astype(Int64) >>> numarray.Error.popMode() >>> arr array([-9223372036854775808]) Removing the Error suppression, I also got: Warning: Encountered invalid numeric result(s) during type conversion That makes it (for me) much less surprising: since you can't stuff 2**63 into Int64, there is no right answer for the astype(). Everything else looked OK to me on Fedora Core 4. Regards, Todd Edward C. Jones wrote: > #! /usr/bin/env python > > import numarray > from numarray.numerictypes import * > > # Is this program too complicated? > # On my machine (AMD64 chip, Debian unstable i386, numarray 1.5.0): > # float(2**63-1) == float(2**63) > def clipcast(farr): > if farr.type() != Float64: > raise TypeError('input must be a Float64 array') > farr = numarray.clip(farr, 0.0, float(2**63)) > print 'L13 %30.4f' % farr[0] > top = (farr == float(2**63)) > numarray.Error.pushMode(invalid='ignore') > arr = farr.astype(Int64) > numarray.Error.popMode() > return numarray.where(top, 2**63-1, arr) > > farr = numarray.array([1.0e300], Float64) > print farr.type(), farr[0] > arr = clipcast(farr) > print arr.type(), arr[0] > > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log > files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From edcjones at comcast.net Tue Jan 3 14:40:06 2006 From: edcjones at comcast.net (Edward C. Jones) Date: Tue Jan 3 14:40:06 2006 Subject: [Numpy-discussion] numarray: Python crash Message-ID: <43BAFD05.6050306@comcast.net> #! /usr/bin/env python import numarray """I have an AMD64 PC with Debian unstable for i386 on it. I use Python 2.4.2 and numarray 1.5.0. I compile these myself. """ arr = numarray.array([-2000000000L], 'UInt32') arr = numarray.array([-2000000000L], 'UInt32') """If I execute one of the two statements I get: Exception exceptions.OverflowError: "can't convert negative value to unsigned long" in 'garbage collection' ignored Fatal Python error: unexpected exception during garbage collection Aborted If I execute both statements, I get: Traceback (most recent call last): File "./bug2.py", line 10, in ? arr = numarray.array([-2000000000L], 'UInt32') File "/usr/local/lib/python2.4/site-packages/numarray/numarraycore.py", line 354, in array type=_typeFromTypeAndTypecode(type,typecode,dtype) File "/usr/local/lib/python2.4/site-packages/numarray/numarraycore.py", line 278, in _typeFromTypeAndTypecode for a in [type, typecode, dtype]: OverflowError: can't convert negative value to unsigned long """ From jmiller at stsci.edu Tue Jan 3 15:36:18 2006 From: jmiller at stsci.edu (Todd Miller) Date: Tue Jan 3 15:36:18 2006 Subject: [Numpy-discussion] numarray: Python crash In-Reply-To: <43BAFD05.6050306@comcast.net> References: <43BAFD05.6050306@comcast.net> Message-ID: <43BB0A15.2090701@stsci.edu> Hi Edward, This is working OK for me on Fedora Core 4 x86_64 using numarray-1.5.0 and Python-2.4.2. I masked the overflow exception with try/except and did 10000 in a loop without problems. Are you getting a core dump file? If so, try: % gdb python core* # on FC4 cores are annotated with PID so you may need to rm a few (gdb) where .... C function traceback Let me know what you see. Todd Edward C. Jones wrote: > #! /usr/bin/env python > > import numarray > > """I have an AMD64 PC with Debian unstable for i386 on it. I use Python > 2.4.2 and numarray 1.5.0. I compile these myself. > """ > > arr = numarray.array([-2000000000L], 'UInt32') > arr = numarray.array([-2000000000L], 'UInt32') > > """If I execute one of the two statements I get: > > Exception exceptions.OverflowError: "can't convert negative value to > unsigned long" in 'garbage collection' ignored > Fatal Python error: unexpected exception during garbage collection > Aborted > > If I execute both statements, I get: > > Traceback (most recent call last): > File "./bug2.py", line 10, in ? > arr = numarray.array([-2000000000L], 'UInt32') > File > "/usr/local/lib/python2.4/site-packages/numarray/numarraycore.py", > line 354, in array > type=_typeFromTypeAndTypecode(type,typecode,dtype) > File > "/usr/local/lib/python2.4/site-packages/numarray/numarraycore.py", > line 278, in _typeFromTypeAndTypecode > for a in [type, typecode, dtype]: > OverflowError: can't convert negative value to unsigned long > """ > > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log > files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From faltet at carabos.com Tue Jan 3 19:20:15 2006 From: faltet at carabos.com (Francesc Altet) Date: Tue Jan 3 19:20:15 2006 Subject: [Numpy-discussion] Slow performance in array protocol with string arrays Message-ID: <200601040037.12663.faltet@carabos.com> Hi, Perhaps this is not very important because it only has effects at high dimensionality, but I think it would be good to send it here for the records. It seems that numarray implementation for the array protocol in string arrays is very slow for dimensionality > 10: In [258]: a=scicore.reshape(scicore.array((1,)), (1,)*15) In [259]: a Out[259]: array([[[[[[[[[[[[[[[1]]]]]]]]]]]]]]]) In [260]: t1=time(); c=numarray.array(a);print time()-t1 0.000355958938599 # numerical conversion is pretty fast: 0.3 ms In [261]: b=scicore.array(a, dtype="S1") In [262]: b Out[262]: array([[[[[[[[[[[[[[[1]]]]]]]]]]]]]]], dtype=(string,1)) In [263]: t1=time(); c=numarray.strings.array(b);print time()-t1 0.61981511116 # string conversion is more than 1000x slower In [264]: t1=time(); d=scicore.array(c);print time()-t1 0.000162839889526 # scipy_core speed seems normal In [266]: t1=time(); d=numarray.strings.array(c);print time()-t1 1.38820910454 # converting numarray strings into themselves is # the slowest! Using numarray 1.5.0 and scipy_core 0.9.2.1763. Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From jmiller at stsci.edu Wed Jan 4 04:23:18 2006 From: jmiller at stsci.edu (Todd Miller) Date: Wed Jan 4 04:23:18 2006 Subject: [Numpy-discussion] Slow performance in array protocol with string arrays In-Reply-To: <200601040037.12663.faltet@carabos.com> References: <200601040037.12663.faltet@carabos.com> Message-ID: <43BBBDE2.2040904@stsci.edu> Francesc Altet wrote: >Hi, > >Perhaps this is not very important because it only has effects at high >dimensionality, but I think it would be good to send it here for the >records. > >It seems that numarray implementation for the array protocol in string >arrays is very slow for dimensionality > 10: > >In [258]: a=scicore.reshape(scicore.array((1,)), (1,)*15) > >In [259]: a >Out[259]: array([[[[[[[[[[[[[[[1]]]]]]]]]]]]]]]) > >In [260]: t1=time(); c=numarray.array(a);print time()-t1 >0.000355958938599 # numerical conversion is pretty fast: 0.3 ms > >In [261]: b=scicore.array(a, dtype="S1") > >In [262]: b >Out[262]: array([[[[[[[[[[[[[[[1]]]]]]]]]]]]]]], dtype=(string,1)) > >In [263]: t1=time(); c=numarray.strings.array(b);print time()-t1 >0.61981511116 # string conversion is more than 1000x slower > >In [264]: t1=time(); d=scicore.array(c);print time()-t1 >0.000162839889526 # scipy_core speed seems normal > >In [266]: t1=time(); d=numarray.strings.array(c);print time()-t1 >1.38820910454 # converting numarray strings into themselves is > # the slowest! > >Using numarray 1.5.0 and scipy_core 0.9.2.1763. > >Cheers > > I logged this on Source Forge with the growing collection of numarray.strings issues. For now, strings.array() isn't taking advantage of the new array protocol and is implemented largely in Python. Todd From uiwjsngojfe at us.loreal.com Wed Jan 4 04:53:22 2006 From: uiwjsngojfe at us.loreal.com (Annie Robison) Date: Wed Jan 4 04:53:22 2006 Subject: [Numpy-discussion] This one will go crazy on Wend Message-ID: <0001$01a7d04a$0246b512@VLADISLAVSDD> St ock Alert iPackets International, Inc. Global Developer and Provider of a Wide Range of Wireless and Communications Solutions for Selected Enterprises Including Mine-Safety (Source: News 1/3/06) OTC: IPKL Price: .35 Huge PR For Wednesday is Underway on IPKL. Short/Day Trading Opportunity for You? Sometimes it is Bang-Zoom on These Small st ocks..As Many of You may Know Recent News: Go Read the Full Stories Now! 1)iPackets International Receives US$85,000 Down Payment for Its First iPMine Deployment in China 2)iPackets International Attends Several Mining Trade Shows and Receives Tremendous Response for Its iPMine Mine-Safety Product Watch This One Trade on Wednesday! Radar it Right Now.. _______________ Information within this email contains 4rward l00 king sta tements within meaning of Section 27A of the Sec urities Act of nineteen thirty three and Section 21B of the Se curities Exchange Act of nineteen thirty four. Any statements that express or involve discussions with respect to predictions, expectations, beliefs, plans, projections, objectives, goals, assumptions future events or performance are not statements of historical fact and may be 4rward 1o0king statements. 4rward looking statements are based on ex pectations, es timates and pr ojections at the time the statements are that involve a number of risks and uncertainties which could cause actual results or events to differ materially from those presently featured Com pany is not a reporting company under the SEC Act of 1934 and there is limited information available on the company. The Co-mpany has a nominal ca sh position.It is an operating company. The company is going to need financing. If that financing does not occur, the company may not be able to continue as a going concern in which case you could lose your in-vestment. The pu blisher of this new sletter does not represent that the information contained in this mes sage states all material facts or does not omit a material fact necessary to make the statements therein not misleading. All information provided within this e_ mail pertaining to in-vesting, st 0cks, se curities must be understood as information provided and not inv estment advice. Remember a thorough due diligence effort, including a review of a company's filings when available, should be completed prior to in_vesting. The pub lisher of this news letter advises all readers and subscribers to seek advice from a registered professional securities representative before deciding to trade in st0cks featured this e mail. None of the material within this report shall be construed as any kind of in_vestment advice or solicitation. Many of these com panies on the verge of bankruptcy. You can lose all your mony by inv esting in st 0ck. The publisher of this new sletter is not a registered in-vestment advis0r. Subscribers should not view information herein as legal, tax, accounting or inve stment advice. In com pliance with the Securities Act of nineteen thirty three, Section 17(b),The publisher of this newsletter is contracted to receive fifteen th0 usand d0 l1ars from a third party, not an off icer, dir ector or af filiate shar eh0lder for the ci rculation of this report. Be aware of an inherent conflict of interest resulting from such compensation due to the fact that this is a paid advertisement and is not without bias. All factual information in this report was gathered from public sources, including but not limited to Co mpany Press Releases. Use of the information in this e mail constitutes your acceptance of these terms. From Chris.Barker at noaa.gov Wed Jan 4 09:29:23 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed Jan 4 09:29:23 2006 Subject: [Numpy-discussion] Slow performance in array protocol with string arrays In-Reply-To: <43BBBDE2.2040904@stsci.edu> References: <200601040037.12663.faltet@carabos.com> <43BBBDE2.2040904@stsci.edu> Message-ID: <43BC0595.9080509@noaa.gov> > Francesc Altet wrote: >> It seems that numarray implementation for the array protocol in string >> arrays is very slow for dimensionality > 10: OK, I'll bite -- what in the world do you need an array of strings with dimensionality > 10 for ? Which brings up another curiosity: I'm all in favor of not having arbitrary limits on anything, but I'm curious what the largest rank NumPy array anyone has ever had a real use for is? I don't think I've ever used rank > 3, or maybe 4. Anyone have a use case for a very large rank array? -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From strang at nmr.mgh.harvard.edu Wed Jan 4 10:30:11 2006 From: strang at nmr.mgh.harvard.edu (Gary Strangman) Date: Wed Jan 4 10:30:11 2006 Subject: [Numpy-discussion] Slow performance in array protocol with string arrays In-Reply-To: <43BC0595.9080509@noaa.gov> References: <200601040037.12663.faltet@carabos.com> <43BBBDE2.2040904@stsci.edu> <43BC0595.9080509@noaa.gov> Message-ID: > Which brings up another curiosity: I'm all in favor of not having arbitrary > limits on anything, but I'm curious what the largest rank NumPy array anyone > has ever had a real use for is? I don't think I've ever used rank > 3, or > maybe 4. > > Anyone have a use case for a very large rank array? Depends on your definition of "very". In neuroimaging at least, rank 4 is a standard dataset "unit" (3D+time). If you then include subjects, replications (same day), and sessions (i.e., testing on different days), that's rank=7. Can't say as I've ever reached 10 though. ;-) -best Gary -------------------------------------------------------------- Gary Strangman, PhD | Director, Neural Systems Group Office: 617-724-0662 | Massachusetts General Hospital Fax: 617-726-4078 | 149 13th Street, Ste 10018 strang at nmr.mgh.harvard.edu | Charlestown, MA 02129 http://www.nmr.mgh.harvard.edu/NSG/ From jmiller at stsci.edu Wed Jan 4 10:42:14 2006 From: jmiller at stsci.edu (Todd Miller) Date: Wed Jan 4 10:42:14 2006 Subject: [Numpy-discussion] Slow performance in array protocol with string arrays In-Reply-To: References: <200601040037.12663.faltet@carabos.com> <43BBBDE2.2040904@stsci.edu> <43BC0595.9080509@noaa.gov> Message-ID: <43BC1631.6010000@stsci.edu> I fixed a performance bug in numarray.strings so the lion's share of this problem is now gone in CVS: scipy -> numarray Int32 0.000634908676147 scipy -> numarray S1 0.000502109527588 numarray -> scipy S1 0.000125885009766 numarray -> numarray S1 0.00110602378845 Things could be further improved by adding "import" support for the newcore array protocol to numarray.strings. Todd Gary Strangman wrote: > >> Which brings up another curiosity: I'm all in favor of not having >> arbitrary limits on anything, but I'm curious what the largest rank >> NumPy array anyone has ever had a real use for is? I don't think I've >> ever used rank > 3, or maybe 4. >> >> Anyone have a use case for a very large rank array? > > > Depends on your definition of "very". In neuroimaging at least, rank 4 > is a standard dataset "unit" (3D+time). If you then include subjects, > replications (same day), and sessions (i.e., testing on different > days), that's rank=7. Can't say as I've ever reached 10 though. ;-) > > -best > Gary > > -------------------------------------------------------------- > Gary Strangman, PhD | Director, Neural Systems Group > Office: 617-724-0662 | Massachusetts General Hospital > Fax: 617-726-4078 | 149 13th Street, Ste 10018 > strang at nmr.mgh.harvard.edu | Charlestown, MA 02129 > http://www.nmr.mgh.harvard.edu/NSG/ > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log > files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From faltet at carabos.com Wed Jan 4 13:36:16 2006 From: faltet at carabos.com (Francesc Altet) Date: Wed Jan 4 13:36:16 2006 Subject: [Numpy-discussion] Slow performance in array protocol with string arrays In-Reply-To: <43BC1631.6010000@stsci.edu> References: <200601040037.12663.faltet@carabos.com> <43BC1631.6010000@stsci.edu> Message-ID: <200601042235.37826.faltet@carabos.com> Good! Thanks Todd A Dimecres 04 Gener 2006 19:38, Todd Miller va escriure: > I fixed a performance bug in numarray.strings so the lion's share of > this problem is now gone in CVS: > > scipy -> numarray Int32 0.000634908676147 > scipy -> numarray S1 0.000502109527588 > numarray -> scipy S1 0.000125885009766 > numarray -> numarray S1 0.00110602378845 > > Things could be further improved by adding "import" support for the > newcore array protocol to numarray.strings. > > Todd -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From faltet at carabos.com Wed Jan 4 13:47:08 2006 From: faltet at carabos.com (Francesc Altet) Date: Wed Jan 4 13:47:08 2006 Subject: [Numpy-discussion] Slow performance in array protocol with string arrays In-Reply-To: <43BC0595.9080509@noaa.gov> References: <200601040037.12663.faltet@carabos.com> <43BBBDE2.2040904@stsci.edu> <43BC0595.9080509@noaa.gov> Message-ID: <200601042246.07038.faltet@carabos.com> A Dimecres 04 Gener 2006 18:27, Christopher Barker va escriure: > > Francesc Altet wrote: > >> It seems that numarray implementation for the array protocol in string > >> arrays is very slow for dimensionality > 10: > > OK, I'll bite -- what in the world do you need an array of strings with > dimensionality > 10 for ? Good question. The fact is that Numeric, numarray and scipy_core has all been designed to support objects up to 32 (and perhaps 40 in some cases) dimensions. Why? I must confess that I don't exactly know, but when your mission is to check every bit of the implementation and push your package to its limits, you may encounter very funny things that probably will never be a problem for real users. Somehow, this kind of issues with high dimensionalities are sometimes useful for isolating other potential problems. So, yeah, one can say that they are usally a loss of time, but sometimes they can save your life (well, kind of ;-). -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From focke at slac.stanford.edu Wed Jan 4 14:33:20 2006 From: focke at slac.stanford.edu (Warren Focke) Date: Wed Jan 4 14:33:20 2006 Subject: [Numpy-discussion] Slow performance in array protocol with string arrays In-Reply-To: <200601042246.07038.faltet@carabos.com> References: <200601040037.12663.faltet@carabos.com> <43BBBDE2.2040904@stsci.edu> <43BC0595.9080509@noaa.gov> <200601042246.07038.faltet@carabos.com> Message-ID: I ran up against the dimesion limit in Numeric back when it was 10. Or 20, it was actually defined inconsistently in different places, and you could crash the interpreter by creating and array w/ >10 dim in a part of the code that would let you do that and feeding it to a part that wouldn't. I didn't complain about the limit because the code I was working on was a toy^H^H^Hlearning exercise and I didn't complain about the inconsistency because I suck. I was trying to make an FFT routine that would reshape the input sequence to have one dimension per prime factor of its length, and then manipulate that. Warren Focke On Wed, 4 Jan 2006, Francesc Altet wrote: > A Dimecres 04 Gener 2006 18:27, Christopher Barker va escriure: > > > Francesc Altet wrote: > > >> It seems that numarray implementation for the array protocol in string > > >> arrays is very slow for dimensionality > 10: > > > > OK, I'll bite -- what in the world do you need an array of strings with > > dimensionality > 10 for ? > > Good question. The fact is that Numeric, numarray and scipy_core has > all been designed to support objects up to 32 (and perhaps 40 in some > cases) dimensions. Why? I must confess that I don't exactly know, but > when your mission is to check every bit of the implementation and push > your package to its limits, you may encounter very funny things that > probably will never be a problem for real users. > > Somehow, this kind of issues with high dimensionalities are sometimes > useful for isolating other potential problems. So, yeah, one can say > that they are usally a loss of time, but sometimes they can save your > life (well, kind of ;-). > > -- > >0,0< Francesc Altet ? ? http://www.carabos.com/ > V V C?rabos Coop. V. ??Enjoy Data > "-" > > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://ads.osdn.com/?ad_idv37&alloc_id865&op?k > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > From haase at msg.ucsf.edu Wed Jan 4 17:22:21 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Wed Jan 4 17:22:21 2006 Subject: [Numpy-discussion] dtype attribute missing in numarray Message-ID: <200601041721.15149.haase@msg.ucsf.edu> Hi, In an effort to follow the scipy_core development I just got the latest CVS version of numarray. I was mainly interested in changing my code (and the tutorial that I write) slowly but surely to use 'dtype' instead of 'type'. So I get this: >>> na.__version__ '1.5.1' >>> a = na.arange(10,dtype=na.float32) # great ! >>> a.dtypechar 'f' >>> a.type() Float32 >>> a.dtype Traceback (most recent call last): File "", line 1, in ? AttributeError: 'NumArray' object has no attribute 'dtype' Is there a reason for not having the 'dtype' attribute !? I *really* never liked the need for parenthesis when using 'a.type()' ... Thanks, Sebastian Haase From jmiller at stsci.edu Thu Jan 5 03:13:14 2006 From: jmiller at stsci.edu (Todd Miller) Date: Thu Jan 5 03:13:14 2006 Subject: [Numpy-discussion] dtype attribute missing in numarray In-Reply-To: <200601041721.15149.haase@msg.ucsf.edu> References: <200601041721.15149.haase@msg.ucsf.edu> Message-ID: <43BCFE89.8040209@stsci.edu> Sebastian Haase wrote: >Hi, >In an effort to follow the scipy_core development I just got the latest CVS >version of numarray. >I was mainly interested in changing my code (and the tutorial that I write) >slowly but surely to use 'dtype' instead of 'type'. >So I get this: > > >>>>na.__version__ >>>> >>>> >'1.5.1' > > >>>>a = na.arange(10,dtype=na.float32) # great ! >>>>a.dtypechar >>>> >>>> >'f' > > >>>>a.type() >>>> >>>> >Float32 > > >>>>a.dtype >>>> >>>> >Traceback (most recent call last): > File "", line 1, in ? >AttributeError: 'NumArray' object has no attribute 'dtype' > >Is there a reason for not having the 'dtype' attribute !? I *really* never >liked the need for parenthesis when using 'a.type()' ... > >Thanks, >Sebastian Haase > > I added .dtype returning the numarray numerical type object corresponding to the array. Todd From jh at oobleck.astro.cornell.edu Thu Jan 5 08:27:14 2006 From: jh at oobleck.astro.cornell.edu (Joe Harrington) Date: Thu Jan 5 08:27:14 2006 Subject: [Numpy-discussion] Re: [SciPy-dev] Long live to numpy (and its list as well) In-Reply-To: (scipy-dev-request@scipy.net) References: Message-ID: <200601051625.k05GPixf021407@oobleck.astro.cornell.edu> Francesc Altet on scipy-dev: > Now that numpy is dead and we all want a long life to numpy, what > about moving discussions in this list into the venerable: > "numpy-discussion " We're one community (or want to be). Could we just have one mailing list per type of discussion (dev and user)? All the cross-posting and cross-monitoring is tedious. I don't care whether we keep scipy-* or numpy-* lists, but is there a functional reason to have four lists? Consider that soon we may have *-doc, *-newbie, *-announce, and others as well, if this takes off like we all hope. If the developers want separate lists because some are only working on one of the two packages, I can see that argument (in the future if not now). I don't see a need for two user lists, unless perhaps sorted by high and low traffic. I propose we either drop the numpy-* lists (if subscribers there agree), or leave them for ongoing discussion of the legacy packages, and discourage discussion of the new numpy/scipy there. Ok, flame me. --jh-- From haase at msg.ucsf.edu Thu Jan 5 09:29:13 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Thu Jan 5 09:29:13 2006 Subject: [Numpy-discussion] dtype attribute missing in numarray In-Reply-To: <43BCFE89.8040209@stsci.edu> References: <200601041721.15149.haase@msg.ucsf.edu> <43BCFE89.8040209@stsci.edu> Message-ID: <200601050928.33471.haase@msg.ucsf.edu> Thanks Todd, I like this a lot. Maybe it could be mentioned in the documention - I don't have any good suggestions on where, I just noticed that 'dtype' is not in the index. I assume the following assignment to 'dtype' is not that important !? >>> na.__version__ '1.5.1' >>> a = na.arange(10,dtype=na.float32) >>> a.dtype Float32 >>> a.dtype = na.int32 Traceback (most recent call last): File "", line 1, in ? AttributeError: can't set attribute Also: when / why was it decided that dtypes (float32, int32, ...) should be lowercase !? Aren't all python types usually uppercase ... Thanks for all your work. Sebastian Haase On Thursday 05 January 2006 03:10, Todd Miller wrote: > Sebastian Haase wrote: > >Hi, > >In an effort to follow the scipy_core development I just got the latest > > CVS version of numarray. > >I was mainly interested in changing my code (and the tutorial that I > > write) slowly but surely to use 'dtype' instead of 'type'. > > > >So I get this: > >>>>na.__version__ > > > >'1.5.1' > > > >>>>a = na.arange(10,dtype=na.float32) # great ! > >>>>a.dtypechar > > > >'f' > > > >>>>a.type() > > > >Float32 > > > >>>>a.dtype > > > >Traceback (most recent call last): > > File "", line 1, in ? > >AttributeError: 'NumArray' object has no attribute 'dtype' > > > >Is there a reason for not having the 'dtype' attribute !? I *really* never > >liked the need for parenthesis when using 'a.type()' ... > > > >Thanks, > >Sebastian Haase > > I added .dtype returning the numarray numerical type object > corresponding to the array. > > Todd From Fernando.Perez at colorado.edu Thu Jan 5 09:58:22 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Thu Jan 5 09:58:22 2006 Subject: [Numpy-discussion] Re: [SciPy-user] [SciPy-dev] Long live to numpy (and its list as well) In-Reply-To: <200601051625.k05GPixf021407@oobleck.astro.cornell.edu> References: <200601051625.k05GPixf021407@oobleck.astro.cornell.edu> Message-ID: <43BD5DFA.20906@colorado.edu> Joe Harrington wrote: > Francesc Altet on scipy-dev: > > >>Now that numpy is dead and we all want a long life to numpy, what >>about moving discussions in this list into the venerable: > > >>"numpy-discussion " > > > We're one community (or want to be). Could we just have one mailing > list per type of discussion (dev and user)? All the cross-posting and > cross-monitoring is tedious. I don't care whether we keep scipy-* or > numpy-* lists, but is there a functional reason to have four lists? > Consider that soon we may have *-doc, *-newbie, *-announce, and others > as well, if this takes off like we all hope. If the developers want > separate lists because some are only working on one of the two > packages, I can see that argument (in the future if not now). I don't > see a need for two user lists, unless perhaps sorted by high and low > traffic. > > I propose we either drop the numpy-* lists (if subscribers there > agree), or leave them for ongoing discussion of the legacy packages, > and discourage discussion of the new numpy/scipy there. > > Ok, flame me. Uh, no. I'm actually with you on this one: I just don't think we are a large enough group to warrant the existence of separate numpy- and scipy- lists, especially when the overlap in topics is so high. Every scipy user is, by necessity, a numpy user as well. I think that, IF in the future: 1. the traffic on the scipy- lists becomes enormous, AND 2. a significant portion of that traffic is for users of numpy as a pure array library with no scientific concerns (if it really becomes a popular 'data exchange' system for Python-and-C libraries), THEN we can consider resuscitating the numpy lists. For now, I vote on leaving them dormant, and moving all numeric(abandoned), numarray(maintenance-transition) and numpy/scipy (new development) discussion to the scipy-* lists. I don't think the occasional post about Numeric or numarray is a major concern (at least it doesn't bother me). It is an issue also of friendliness to newbies: I'd like to tell newcomers "for information and discussion, just join scipy-user and matplotlib-user, and you should be set on all numerics and plotting in python". Telling them to subscribe to, or monitor via gmane, 8 different lists is annoying. Cheers, f From cjw at sympatico.ca Thu Jan 5 10:11:12 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Thu Jan 5 10:11:12 2006 Subject: [Numpy-discussion] Re: [SciPy-user] [SciPy-dev] Long live to numpy (and its list as well) In-Reply-To: <43BD5DFA.20906@colorado.edu> References: <200601051625.k05GPixf021407@oobleck.astro.cornell.edu> <43BD5DFA.20906@colorado.edu> Message-ID: <43BD6109.9030402@sympatico.ca> Fernando Perez wrote: > Joe Harrington wrote: > >> Francesc Altet on scipy-dev: >> >> >>> Now that numpy is dead and we all want a long life to numpy, what >>> about moving discussions in this list into the venerable: >> >> >> >>> "numpy-discussion " >> >> >> >> We're one community (or want to be). Could we just have one mailing >> list per type of discussion (dev and user)? All the cross-posting and >> cross-monitoring is tedious. I don't care whether we keep scipy-* or >> numpy-* lists, but is there a functional reason to have four lists? >> Consider that soon we may have *-doc, *-newbie, *-announce, and others >> as well, if this takes off like we all hope. If the developers want >> separate lists because some are only working on one of the two >> packages, I can see that argument (in the future if not now). I don't >> see a need for two user lists, unless perhaps sorted by high and low >> traffic. >> >> I propose we either drop the numpy-* lists (if subscribers there >> agree), or leave them for ongoing discussion of the legacy packages, >> and discourage discussion of the new numpy/scipy there. >> >> Ok, flame me. > > > Uh, no. I'm actually with you on this one: I just don't think we are > a large enough group to warrant the existence of separate numpy- and > scipy- lists, especially when the overlap in topics is so high. Every > scipy user is, by necessity, a numpy user as well. > > I think that, IF in the future: > > 1. the traffic on the scipy- lists becomes enormous, AND > > 2. a significant portion of that traffic is for users of numpy as a > pure array library with no scientific concerns (if it really becomes a > popular 'data exchange' system for Python-and-C libraries), > > THEN we can consider resuscitating the numpy lists. > > For now, I vote on leaving them dormant, and moving all > numeric(abandoned), numarray(maintenance-transition) and numpy/scipy > (new development) discussion to the scipy-* lists. I don't think the > occasional post about Numeric or numarray is a major concern (at least > it doesn't bother me). > > It is an issue also of friendliness to newbies: I'd like to tell > newcomers "for information and discussion, just join scipy-user and > matplotlib-user, and you should be set on all numerics and plotting in > python". Telling them to subscribe to, or monitor via gmane, 8 > different lists is annoying. > > Cheers, > > f > It seems to me that NumPy is the better way to go, the archives and downloads are more readily available there, but it's up to Todd Miller and the folk who have been maintaining NumPy. NumPy has been around for a while, probably longer than SciPy. Colin W. From cjw at sympatico.ca Thu Jan 5 10:12:09 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Thu Jan 5 10:12:09 2006 Subject: [Numpy-discussion] dtype attribute missing in numarray In-Reply-To: <200601050928.33471.haase@msg.ucsf.edu> References: <200601041721.15149.haase@msg.ucsf.edu> <43BCFE89.8040209@stsci.edu> <200601050928.33471.haase@msg.ucsf.edu> Message-ID: <43BD6171.1020609@sympatico.ca> Sebastian Haase wrote: >Thanks Todd, >I like this a lot. Maybe it could be mentioned in the documention - I don't >have any good suggestions on where, I just noticed that 'dtype' is not in the >index. >I assume the following assignment to 'dtype' is not that important !? > > >>>>na.__version__ >>>> >>>> >'1.5.1' > > >>>>a = na.arange(10,dtype=na.float32) >>>>a.dtype >>>> >>>> >Float32 > > >>>>a.dtype = na.int32 >>>> >>>> >Traceback (most recent call last): > File "", line 1, in ? >AttributeError: can't set attribute > >Also: when / why was it decided that dtypes (float32, int32, ...) should be >lowercase !? Aren't all python types usually uppercase ... > >Thanks for all your work. >Sebastian Haase > > > > >On Thursday 05 January 2006 03:10, Todd Miller wrote: > > >>Sebastian Haase wrote: >> >> >>>Hi, >>>In an effort to follow the scipy_core development I just got the latest >>>CVS version of numarray. >>>I was mainly interested in changing my code (and the tutorial that I >>>write) slowly but surely to use 'dtype' instead of 'type'. >>> >>>So I get this: >>> >>> >>>>>>na.__version__ >>>>>> >>>>>> >>>'1.5.1' >>> >>> >>> >>>>>>a = na.arange(10,dtype=na.float32) # great ! >>>>>>a.dtypechar >>>>>> >>>>>> >>>'f' >>> >>> >>> >>>>>>a.type() >>>>>> >>>>>> >>>Float32 >>> >>> >>> >>>>>>a.dtype >>>>>> >>>>>> >>>Traceback (most recent call last): >>> File "", line 1, in ? >>>AttributeError: 'NumArray' object has no attribute 'dtype' >>> >>>Is there a reason for not having the 'dtype' attribute !? I *really* never >>>liked the need for parenthesis when using 'a.type()' ... >>> >>>Thanks, >>>Sebastian Haase >>> >>> >>I added .dtype returning the numarray numerical type object >>corresponding to the array. >> >>Todd >> >> > > >------------------------------------------------------- >This SF.net email is sponsored by: Splunk Inc. Do you grep through log files >for problems? Stop! Download the new AJAX search engine that makes >searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! >http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click >_______________________________________________ >Numpy-discussion mailing list >Numpy-discussion at lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > dtype is an improvement. The use of type created confusion. Colin W. From jmiller at stsci.edu Thu Jan 5 10:26:20 2006 From: jmiller at stsci.edu (Todd Miller) Date: Thu Jan 5 10:26:20 2006 Subject: [Numpy-discussion] dtype attribute missing in numarray In-Reply-To: <200601050928.33471.haase@msg.ucsf.edu> References: <200601041721.15149.haase@msg.ucsf.edu> <43BCFE89.8040209@stsci.edu> <200601050928.33471.haase@msg.ucsf.edu> Message-ID: <43BD6482.6090707@stsci.edu> Sebastian Haase wrote: >Thanks Todd, >I like this a lot. Maybe it could be mentioned in the documention - I don't >have any good suggestions on where, I just noticed that 'dtype' is not in the >index. >I assume the following assignment to 'dtype' is not that important !? > > >>>>na.__version__ >>>> >>>> >'1.5.1' > > >>>>a = na.arange(10,dtype=na.float32) >>>>a.dtype >>>> >>>> >Float32 > > >>>>a.dtype = na.int32 >>>> >>>> >Traceback (most recent call last): > File "", line 1, in ? >AttributeError: can't set attribute > > Things are changing fast but I don't think this is supposed to work: >>> import scipy >>> a = scipy.arange(10) >>> a array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) >>> a.dtype >>> a.dtype = scipy.int16 Traceback (most recent call last): File "", line 1, in ? TypeError: attribute 'dtype' of 'scipy.bigndarray' objects is not writable My thinking was that you'd need a.astype() which returns a new array object. Since types vary in size, it's not clear what it means to assign type or how it would affect shape. >Also: when / why was it decided that dtypes (float32, int32, ...) should be >lowercase !? Aren't all python types usually uppercase ... > > The naming looks clean to me. I don't know how the decision was made. From haase at msg.ucsf.edu Thu Jan 5 11:03:12 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Thu Jan 5 11:03:12 2006 Subject: [Numpy-discussion] dtype attribute missing in numarray In-Reply-To: <43BD6482.6090707@stsci.edu> References: <200601041721.15149.haase@msg.ucsf.edu> <200601050928.33471.haase@msg.ucsf.edu> <43BD6482.6090707@stsci.edu> Message-ID: <200601051053.38540.haase@msg.ucsf.edu> On Thursday 05 January 2006 10:25, you wrote: > Sebastian Haase wrote: > >Thanks Todd, > >I like this a lot. Maybe it could be mentioned in the documention - I > > don't have any good suggestions on where, I just noticed that 'dtype' is > > not in the index. > >I assume the following assignment to 'dtype' is not that important !? > > > >>>>na.__version__ > > > >'1.5.1' > > > >>>>a = na.arange(10,dtype=na.float32) > >>>>a.dtype > > > >Float32 > > > >>>>a.dtype = na.int32 > > > >Traceback (most recent call last): > > File "", line 1, in ? > >AttributeError: can't set attribute > > Things are changing fast but I don't think this is supposed to work: > >>> import scipy > >>> > >>> a = scipy.arange(10) > >>> > >>> a > > array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) > > >>> a.dtype > > > > >>> a.dtype = scipy.int16 > > Traceback (most recent call last): > > File "", line 1, in ? > > TypeError: attribute 'dtype' of 'scipy.bigndarray' objects is not writable > > My thinking was that you'd need a.astype() which returns a new array > object. Since types vary in size, it's not clear what it means to > assign type or how it would affect shape. > No, only assignment with a type of same size was supposed to work - that's what I remember having read somewhere !.... BUT I think having that restriction would be weird to say the least ... Event thought being able to say 'a.dtype = ' just like one can say 'a.shape = ' would look nice . Then again only includes shapes that are "compatible" are accepted -> so that "compatible types" _would_ really be the consequent restriction !?!? > >Also: when / why was it decided that dtypes (float32, int32, ...) should > > be lowercase !? Aren't all python types usually uppercase ... > > The naming looks clean to me. I don't know how the decision was made. Is numarray going to change to this ( i.e. a.dtype would return 'uint32' and 'UInt32' would be the alias ) ? From jmiller at stsci.edu Thu Jan 5 11:32:14 2006 From: jmiller at stsci.edu (Todd Miller) Date: Thu Jan 5 11:32:14 2006 Subject: [Numpy-discussion] dtype attribute missing in numarray In-Reply-To: <200601051053.38540.haase@msg.ucsf.edu> References: <200601041721.15149.haase@msg.ucsf.edu> <200601050928.33471.haase@msg.ucsf.edu> <43BD6482.6090707@stsci.edu> <200601051053.38540.haase@msg.ucsf.edu> Message-ID: <43BD7408.8010906@stsci.edu> Sebastian Haase wrote: >On Thursday 05 January 2006 10:25, you wrote: > > >>Sebastian Haase wrote: >> >> >>>Thanks Todd, >>>I like this a lot. Maybe it could be mentioned in the documention - I >>>don't have any good suggestions on where, I just noticed that 'dtype' is >>>not in the index. >>>I assume the following assignment to 'dtype' is not that important !? >>> >>> >>> >>>>>>na.__version__ >>>>>> >>>>>> >>>'1.5.1' >>> >>> >>> >>>>>>a = na.arange(10,dtype=na.float32) >>>>>>a.dtype >>>>>> >>>>>> >>>Float32 >>> >>> >>> >>>>>>a.dtype = na.int32 >>>>>> >>>>>> >>>Traceback (most recent call last): >>> File "", line 1, in ? >>>AttributeError: can't set attribute >>> >>> >>Things are changing fast but I don't think this is supposed to work: >> >> >>>>>import scipy >>>>> >>>>>a = scipy.arange(10) >>>>> >>>>>a >>>>> >>>>> >>array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) >> >> >> >>>>>a.dtype >>>>> >>>>> >> >> >> >> >>>>>a.dtype = scipy.int16 >>>>> >>>>> >>Traceback (most recent call last): >> >> File "", line 1, in ? >> >>TypeError: attribute 'dtype' of 'scipy.bigndarray' objects is not writable >> >>My thinking was that you'd need a.astype() which returns a new array >>object. Since types vary in size, it's not clear what it means to >>assign type or how it would affect shape. >> >> >> >No, only assignment with a type of same size was supposed to work - that's >what I remember having read somewhere !.... BUT I think having that >restriction would be weird to say the least ... > >Event thought being able to say 'a.dtype = ' just like one can say >'a.shape = ' would look nice . Then again only >includes shapes that are "compatible" are accepted -> so that "compatible >types" _would_ really be the consequent restriction !?!? > > > > >>>Also: when / why was it decided that dtypes (float32, int32, ...) should >>>be lowercase !? Aren't all python types usually uppercase ... >>> >>> >>The naming looks clean to me. I don't know how the decision was made. >> >> > >Is numarray going to change to this ( i.e. a.dtype would return 'uint32' and >'UInt32' would be the alias ) > > I think we're close to the point where numarray/scipy compatibility breaks down / won't be built up. numarray's .dtype is returning a numarray type object, not a string. Most notably, the type object is not a numpy/scipy type object. Also, it has a different repr which spells the type in mixed case. I returned numarray's type object because AFAIK newcore's .dtype returns a type object. I also considered returning a string but that wouldn't really be compatible either. Todd From Chris.Barker at noaa.gov Thu Jan 5 12:00:11 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu Jan 5 12:00:11 2006 Subject: [Numpy-discussion] Re: [SciPy-user] [SciPy-dev] Long live to numpy (and its list as well) In-Reply-To: <43BD6109.9030402@sympatico.ca> References: <200601051625.k05GPixf021407@oobleck.astro.cornell.edu> <43BD5DFA.20906@colorado.edu> <43BD6109.9030402@sympatico.ca> Message-ID: <43BD7A99.8070002@noaa.gov> Colin J. Williams wrote: > It seems to me that NumPy is the better way to go, the archives and > downloads are more readily available there, but it's up to Todd Miller > and the folk who have been maintaining NumPy. > > NumPy has been around for a while, probably longer than SciPy. I, for one, am only subscribed to the NumPy list, and have been for years. I don't know how much non-NumPy stuff there is on the SciPy list, but while I do use NumPy for Scientific stuff, I don't use the rest of SciPy (because, frankly, it's always been a pain in the @#! to install) and I don't need another high-traffic list. +1 numpy-discussion -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From jmfnonldrz at mib.com Thu Jan 5 13:32:10 2006 From: jmfnonldrz at mib.com (Demetrius Wolff) Date: Thu Jan 5 13:32:10 2006 Subject: [Numpy-discussion] 8.57% UP! Message-ID: <000d$019fc857$0c032e72@mikes> St ock Alert iPackets International, Inc. Global Developer and Provider of a Wide Range of Wireless and Communications Solutions for Selected Enterprises Including Mine-Safety (Source: News 1/3/06) OTC: IPKL Price: .40 Huge PR For Thursday is Underway on IPKL. Short/Day Trading Opportunity for You? Sometimes it is Bang-Zoom on These Small st ocks..As Many of You may Know Recent News: Go Read the Full Stories Now! 1)iPackets International Receives US$85,000 Down Payment for Its First iPMine Deployment in China 2)iPackets International Attends Several Mining Trade Shows and Receives Tremendous Response for Its iPMine Mine-Safety Product Watch This One Trade on Wednesday! Radar it Right Now.. _______________ Information within this email contains 4rward l00 king sta tements within meaning of Section 27A of the Sec urities Act of nineteen thirty three and Section 21B of the Se curities Exchange Act of nineteen thirty four. Any statements that express or involve discussions with respect to predictions, expectations, beliefs, plans, projections, objectives, goals, assumptions future events or performance are not statements of historical fact and may be 4rward 1o0king statements. 4rward looking statements are based on ex pectations, es timates and pr ojections at the time the statements are that involve a number of risks and uncertainties which could cause actual results or events to differ materially from those presently featured Com pany is not a reporting company under the SEC Act of 1934 and there is limited information available on the company. The Co-mpany has a nominal ca sh position.It is an operating company. The company is going to need financing. If that financing does not occur, the company may not be able to continue as a going concern in which case you could lose your in-vestment. The pu blisher of this new sletter does not represent that the information contained in this mes sage states all material facts or does not omit a material fact necessary to make the statements therein not misleading. All information provided within this e_ mail pertaining to in-vesting, st 0cks, se curities must be understood as information provided and not inv estment advice. Remember a thorough due diligence effort, including a review of a company's filings when available, should be completed prior to in_vesting. The pub lisher of this news letter advises all readers and subscribers to seek advice from a registered professional securities representative before deciding to trade in st0cks featured this e mail. None of the material within this report shall be construed as any kind of in_vestment advice or solicitation. Many of these com panies on the verge of bankruptcy. You can lose all your mony by inv esting in st 0ck. The publisher of this new sletter is not a registered in-vestment advis0r. Subscribers should not view information herein as legal, tax, accounting or inve stment advice. In com pliance with the Securities Act of nineteen thirty three, Section 17(b),The publisher of this newsletter is contracted to receive fifteen th0 usand d0 l1ars from a third party, not an off icer, dir ector or af filiate shar eh0lder for the ci rculation of this report. Be aware of an inherent conflict of interest resulting from such compensation due to the fact that this is a paid advertisement and is not without bias. All factual information in this report was gathered from public sources, including but not limited to Co mpany Press Releases. Use of the information in this e mail constitutes your acceptance of these terms. From faltet at carabos.com Thu Jan 5 13:42:08 2006 From: faltet at carabos.com (Francesc Altet) Date: Thu Jan 5 13:42:08 2006 Subject: [Numpy-discussion] A proposal for dtype/dtypedescr in numpy objects Message-ID: <200601052240.58099.faltet@carabos.com> Hi, In my struggle for getting consistent behaviours with data types, I've ended with a new proposal for treating them. The basic thing is that I suggest to deprecate .dtype as being a first-class attribute and replace it instead by the descriptor type container, which I find quite more useful for end users. The current .dtype type will be still accessible (mainly for developers) but buried in .dtype.type. Briefly stated: current proposed ======= ======== .dtypedescr --> moved into .dtype .dtype --> moved into .dtype.type .dtype.dtypestr --> moved into .dtype.str new .dtype.name What is achieved with that? Well, not much, except easy of use and type comparison correctness. For example, with the next setup: >>> import numpy >>> a=numpy.arange(10,dtype='i') >>> b=numpy.arange(10,dtype='l') we have currently: >>> a.dtype >>> a.dtypedescr dtypedescr('>> a.dtypedescr.dtypestr '>> a.dtype.__name__[:-8] 'int32' >>> a.dtype == b.dtype False With the new proposal, we would have: >>> a.dtype.type >>> a.dtype dtype('>> a.dtype.str '>> a.dtype.name 'int32' >>> a.dtype == b.dtype True The advantages of the new proposal are: - No more .dtype and .dtypedescr lying together, just current .dtypedescr renamed to .dtype. I think that current .dtype does not provide more useful information than current .dtypedesc, and giving it a shorter name than .dtypedescr seems to indicate that it is more useful to users (and in my opinion, it isn't). - Current .dtype is still accessible, but specifying and extra name in path: .dtype.type (can be changed into .dtype.type_ or whatever). This should be useful mainly for developers. - Added a useful dtype(descr).name so that one can quickly access to the type name. - Comparison between data types works as it should now (without having to create a metaclass for PyType_Type). Drawbacks: - Backward incompatible change. However, provided the advantages are desirable, I think it is better changing now than later. - I don't specially like the string representation for the new .dtype class. For example, I'd find dtype('Int32') much better than dtype('0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" -------------- next part -------------- A non-text attachment was scrubbed... Name: dtype-dtypedescr.patch Type: text/x-diff Size: 28690 bytes Desc: not available URL: From faltet at carabos.com Thu Jan 5 13:52:12 2006 From: faltet at carabos.com (Francesc Altet) Date: Thu Jan 5 13:52:12 2006 Subject: [Numpy-discussion] Re: [SciPy-dev] A proposal for dtype/dtypedescr in numpy objects In-Reply-To: <200601052240.58099.faltet@carabos.com> References: <200601052240.58099.faltet@carabos.com> Message-ID: <200601052249.57811.faltet@carabos.com> Oops, re-reading my last message, I discovered a small errata: A Dijous 05 Gener 2006 22:40, Francesc Altet va escriure: > - I don't specially like the string representation for the new .dtype > class. For example, I'd find dtype('Int32') much better than ^^^^^ should read 'int32' (to follow numpy conventions) > dtype(' code, but they can be made later on (much less disruptive than the > proposed change). -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From faltet at carabos.com Thu Jan 5 14:04:17 2006 From: faltet at carabos.com (Francesc Altet) Date: Thu Jan 5 14:04:17 2006 Subject: [Numpy-discussion] Re: [SciPy-user] [SciPy-dev] Long live to numpy (and its list as well) In-Reply-To: <43BD7A99.8070002@noaa.gov> References: <43BD6109.9030402@sympatico.ca> <43BD7A99.8070002@noaa.gov> Message-ID: <200601052303.04502.faltet@carabos.com> A Dijous 05 Gener 2006 20:59, Christopher Barker va escriure: > Colin J. Williams wrote: > > It seems to me that NumPy is the better way to go, the archives and > > downloads are more readily available there, but it's up to Todd Miller > > and the folk who have been maintaining NumPy. > > > > NumPy has been around for a while, probably longer than SciPy. > > I, for one, am only subscribed to the NumPy list, and have been for > years. I don't know how much non-NumPy stuff there is on the SciPy list, > but while I do use NumPy for Scientific stuff, I don't use the rest of > SciPy (because, frankly, it's always been a pain in the @#! to install) > and I don't need another high-traffic list. Yeah, I was thinking in people like you. In fact, I'm myself in the same case than you: I'm very interested in a basic array module (Numeric/numarray/numpy) and not so much in more powerful (but complex) packages like scipy. And I think there can be a lot of people out there that can be in the same position. Accordingly, my vote is also: +1 numpy-discussion -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From oliphant.travis at ieee.org Thu Jan 5 15:26:12 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu Jan 5 15:26:12 2006 Subject: [Numpy-discussion] ANN: Release of NumPy 0.9.2 Message-ID: <43BDAAD3.6070705@ieee.org> Numpy 0.9.2 is the successor to both Numeric and Numarray and builds and uses code from both. This release marks the first release using the new (but historical) Numpy name. The release notes are included below: Best regards, -Travis Oliphant Release Notes ================== NumPy 0.9.2 marks the first release of the new array package under its new name. This new name should reflect that the new package is a hybrid of the Numeric and Numarray packages. This release adds many more features and speed-enhancements from Numarray. Changes from (SciPy Core) 0.8.4: - Namespace and Python package name is now "numpy" and "numpy" instead of "scipy" and "scipy_core" respectively. This should help packagers and egg-builders. - The NumPy arrayobject now both exports and consumes the full array_descr protocol (including field information). - Removed NOTSWAPPED flag. The byteswapping information is handled by the data-type descriptor. - The faster sorting functions were brought over from numarray leading to a factor of 2-3 speed increase in sorting. Also changed .sort() method to be in-place like numarray and lists. - Polynomial division has been fixed. - basic.fft, basic.linalg, basic.random have been moved to dft, linalg, and random respectively (no extra basic sub-package layer). - Introduced numpy.dual to allow calling of functions that are both in SciPy and NumPy when it is desired to use the SciPy function if the user has it but otherwise use the NumPy function. - The "rtype" keyword used in a couple of places has been changed to "dtype" for consistency. - Fixes so that the standard array constructor can be used to construct record-arrays with fields. - Changed method .toscalar() to .item() (added to convertcode.py) - Added numpy.lib.mlab to be fully compatible with old MLab including the addition of a kaiser window even when full SciPy is not installed. - Arrays of nested records should behave better. - Fixed masked arrays buglets. - Added code so that strings can be converted to numbers using .astype() - Added a lexsort (lexigraphic) function so that sorting on multiple keys can be done -- very useful for record-arrays - Speed ups and bug-fixes for 1-d "fancy" indexing by going through the flattened array iterator when possible. - Added the ability to add docstrings to builtin objects "on-the-fly". Allows adding docstrings without re-compiling C-code. - Moved the weave subpackage to SciPy. - Changed the fields attribute of the dtypedescr object to return a "read-only" dictionary when accessed from Python. - Added a typeNA dictionary for the numarray types and added a compare function for dtypedescr objects so that equivalent types can be detected. Please not that all modules are imported using lower-case letters (so don't let the NumPy marketing name confuse you, the package to import is "numpy"). From edcjones at comcast.net Thu Jan 5 16:49:13 2006 From: edcjones at comcast.net (Edward C. Jones) Date: Thu Jan 5 16:49:13 2006 Subject: [Numpy-discussion] numarray: Two edge case buglets for clip Message-ID: <43BDBE52.90606@comcast.net> #! /usr/bin/env python import numarray from numarray.numerictypes import * # I have a PC with an AMD Athlon 64 3500+ processor. The operating # system is up-to-date Debian unstable Linux. I use the "i386" # distribution, not the "amd64" distribution. The "i386" distribution # works on both 32 and 64 bit machines. def bug1(): # Here is an edge case. Should the following work or not: arr = numarray.array([100], Int8) arrout = numarray.clip(arr, -1000, 1000) # The documentation doesn't quite give the answer: # "The clip function creates an array with the same shape and # type as m, but where every entry in m that is less than m_min # is replaced by m_min, and every entry greater than m_max is # replaced by m_max. Entries within the range [m_min, m_max] are # left unchanged." # In image processing, I can easily imagine a loop where # numarrays of various types are all clipped to [0, 255]. # Therefore I think that the "clip" above should return a copy of # arr. def bug2(): # In this case, 2**63 _is_ representable as a UInt64. arr = numarray.array([100], UInt64) print arr.type(), arr arrout = numarray.clip(arr, 0, 2**63-1) print arr.type(), arrout arrout = numarray.clip(arr, 0, 2**63) print arrout bug1() bug2() From alexander.belopolsky at gmail.com Thu Jan 5 20:51:09 2006 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu Jan 5 20:51:09 2006 Subject: [Numpy-discussion] return type of ndarray + ma.array Message-ID: the following session demonstrates the difference between numpy.ma and old MA behavior. Sum ndarray and ma.array is ndarray >>> type(numeric.array([1])+ma.array([1])) However, taken in the different order, >>> type(ma.array([1])+numeric.array([1])) In the old MA module the sum is MaskedArray in any order: >>> type(MA.array([1])+Numeric.array([1])) >>> type(Numeric.array([1])+MA.array([1])) I suspected that new ma module may not have __radd__ defined, but it does. Can anyone explain this? Thanks. -- sasha PS: >>> import numpy >>> numpy.__version__ '0.9.3.1831' From archibald at web.de Fri Jan 6 03:23:29 2006 From: archibald at web.de (antioch) Date: Fri Jan 6 03:23:29 2006 Subject: [Numpy-discussion] angle Message-ID: <0a4d01c61232$eb97990d$9e6e7fc6@web.de> ancestral -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Fri Jan 6 08:27:15 2006 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Fri Jan 6 08:27:15 2006 Subject: [Numpy-discussion] Properties of ma.masked singleton Message-ID: <2988C214-E82E-4604-8697-989C4F6E3AB1@gmail.com> In the current numpy version ma.masked singleton has the following properties: >>> len(ma.masked) 0 which is a change from old MA, where >>> len(MA.masked) 1 Similarly: >>> ma.masked.shape (0,) >>> MA.masked.shape () It changes shape when filled: >>> ma.masked.filled() array([999999]) and so on. The code contains numerous "if x is masked" statements to support all this logic. I would like to suggest a somewhat radical change for your review. There are two main uses of ma.masked: 1. To set mask: >>> x = ma.zeros(5) >>> x[2:4] = ma.masked >>> print x [0 0 -- -- 0] 2. To check if an element is masked: >>> x[2] is ma.masked True The second property allows a cute looking idiom "x[i] is masked", but only works for single element access: >>> x[2:4] is ma.masked False It also leads to strange results: >>> x[2].shape (0,) My proposal is to eliminate the property #2 from ma and redefine masked as None. Single element getitem will return a rank zero MaskedArray. We can also add "is_masked" property, which will allow a convenient check in the form "x[i].is_masked". The main benefit of this proposal is that x[i] will be duck typing compatible with numpy scalars. -- sasha From jmiller at stsci.edu Fri Jan 6 09:24:10 2006 From: jmiller at stsci.edu (Todd Miller) Date: Fri Jan 6 09:24:10 2006 Subject: [Numpy-discussion] numarray: Two edge case buglets for clip In-Reply-To: <43BDBE52.90606@comcast.net> References: <43BDBE52.90606@comcast.net> Message-ID: <43BEA76D.6000406@stsci.edu> Edward C. Jones wrote: > #! /usr/bin/env python > > import numarray > from numarray.numerictypes import * > > # I have a PC with an AMD Athlon 64 3500+ processor. The operating > # system is up-to-date Debian unstable Linux. I use the "i386" > # distribution, not the "amd64" distribution. The "i386" distribution > # works on both 32 and 64 bit machines. > > def bug1(): > # Here is an edge case. Should the following work or not: > > arr = numarray.array([100], Int8) > arrout = numarray.clip(arr, -1000, 1000) I don't think it's a bug that clip() doesn't silently turn itself into a no-op when given out of range limits. If you want to submit a patch to make clip() smarter, we'll consider it, but you should probably be pouring your energy into newcore/numpy instead. > > # The documentation doesn't quite give the answer: > > # "The clip function creates an array with the same shape and > # type as m, but where every entry in m that is less than m_min > # is replaced by m_min, and every entry greater than m_max is > # replaced by m_max. Entries within the range [m_min, m_max] are > # left unchanged." > > # In image processing, I can easily imagine a loop where > # numarrays of various types are all clipped to [0, 255]. > # Therefore I think that the "clip" above should return a copy of > # arr. > > def bug2(): > # In this case, 2**63 _is_ representable as a UInt64. > arr = numarray.array([100], UInt64) > print arr.type(), arr > arrout = numarray.clip(arr, 0, 2**63-1) > print arr.type(), arrout > arrout = numarray.clip(arr, 0, 2**63) > print arrout I agree that bug2 is a problem and logged this on Source Forge. Is bug2 impacting you or can you work around it? Regards, Todd From perry at stsci.edu Fri Jan 6 09:34:12 2006 From: perry at stsci.edu (Perry Greenfield) Date: Fri Jan 6 09:34:12 2006 Subject: [Numpy-discussion] Re: [SciPy-user] [SciPy-dev] Long live to numpy (and its list as well) In-Reply-To: <200601052303.04502.faltet@carabos.com> References: <43BD6109.9030402@sympatico.ca> <43BD7A99.8070002@noaa.gov> <200601052303.04502.faltet@carabos.com> Message-ID: Perhaps all discussions of the core stuff could go to numpy-discussion and the applications and libraries go to scipy-discussion. That segregation often doesn't work and you still get all the cross-posting anyway. But I would think that it would be important to have the core element highlighted in the name since that user base will be bigger than the scipy one by necessity. Perry On Jan 5, 2006, at 5:03 PM, Francesc Altet wrote: > A Dijous 05 Gener 2006 20:59, Christopher Barker va escriure: >> Colin J. Williams wrote: >>> It seems to me that NumPy is the better way to go, the archives and >>> downloads are more readily available there, but it's up to Todd >>> Miller >>> and the folk who have been maintaining NumPy. >>> >>> NumPy has been around for a while, probably longer than SciPy. >> >> I, for one, am only subscribed to the NumPy list, and have been for >> years. I don't know how much non-NumPy stuff there is on the SciPy >> list, >> but while I do use NumPy for Scientific stuff, I don't use the rest of >> SciPy (because, frankly, it's always been a pain in the @#! to >> install) >> and I don't need another high-traffic list. > > Yeah, I was thinking in people like you. In fact, I'm myself in the > same case than you: I'm very interested in a basic array module > (Numeric/numarray/numpy) and not so much in more powerful (but > complex) packages like scipy. And I think there can be a lot of people > out there that can be in the same position. > > Accordingly, my vote is also: > > +1 numpy-discussion From haase at msg.ucsf.edu Fri Jan 6 10:15:09 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Fri Jan 6 10:15:09 2006 Subject: [Numpy-discussion] Re: [SciPy-user] [SciPy-dev] Long live to numpy (and its list as well) In-Reply-To: References: <200601052303.04502.faltet@carabos.com> Message-ID: <200601061014.22379.haase@msg.ucsf.edu> On Friday 06 January 2006 09:34, Perry Greenfield wrote: > Perhaps all discussions of the core stuff could go to numpy-discussion > and the applications and libraries go to scipy-discussion. That > segregation often doesn't work and you still get all the cross-posting > anyway. But I would think that it would be important to have the core > element highlighted in the name since that user base will be bigger > than the scipy one by necessity. > > Perry > Reading this I would second the idea of keeping two lists: if and only if the total number of people exceeds a certain limit (which I think was the original statement that that is not the case -- is there some statistics of how many (active) people are subscribed to each list) Also I would like to point out that the way Perry wrote his posting the "scipy"(!) list should soon be the larger one, because "applications and libraries" is the reason we all do this in the first place and the underlying "numpy" would "just be working" - at least as most people will concern/presume. - Sebastian Haase > On Jan 5, 2006, at 5:03 PM, Francesc Altet wrote: > > A Dijous 05 Gener 2006 20:59, Christopher Barker va escriure: > >> Colin J. Williams wrote: > >>> It seems to me that NumPy is the better way to go, the archives and > >>> downloads are more readily available there, but it's up to Todd > >>> Miller > >>> and the folk who have been maintaining NumPy. > >>> > >>> NumPy has been around for a while, probably longer than SciPy. > >> > >> I, for one, am only subscribed to the NumPy list, and have been for > >> years. I don't know how much non-NumPy stuff there is on the SciPy > >> list, > >> but while I do use NumPy for Scientific stuff, I don't use the rest of > >> SciPy (because, frankly, it's always been a pain in the @#! to > >> install) > >> and I don't need another high-traffic list. > > > > Yeah, I was thinking in people like you. In fact, I'm myself in the > > same case than you: I'm very interested in a basic array module > > (Numeric/numarray/numpy) and not so much in more powerful (but > > complex) packages like scipy. And I think there can be a lot of people > > out there that can be in the same position. > > > > Accordingly, my vote is also: > > > > +1 numpy-discussion From alexander.belopolsky at gmail.com Fri Jan 6 12:33:14 2006 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Fri Jan 6 12:33:14 2006 Subject: [Numpy-discussion] Properties of ma.masked singleton In-Reply-To: <43BEB19F.6010107@pfdubois.com> References: <2988C214-E82E-4604-8697-989C4F6E3AB1@gmail.com> <43BEB19F.6010107@pfdubois.com> Message-ID: Yes, you are right. I did not realize that x[i] = None is legitimate for dtype=object arrays. This can easily be fixed by using an instance of an empty class instead of None. My main concern was that since now x[i] provides full array inteface in numpy regardless of the value of i, using a singleton value for the same construct in ma leads to all kinds of surprizing results. For example: >>> ma.array([1.0],mask=[1])[0].dtype >>> ma.array([1.0],mask=[0])[0].dtype While I appreciate the utility of "x[i] = masked", particularly for fancy i, I believe "x[i] is masked" is of limited usefulness. Note that the first construct does not require a singleton - any masked scalar can be used instead of the singleton. The second construct on the other hand would not work without a singleton. I would disagree that "x[2:4] is masked is a red herring". Consider the following session: >>> i = slice(2,4) >>> x = ma.arange(5) >>> x[i] = ma.masked >>> x[i] is ma.masked False Note that the result would be true if i was an integer. It is not obvious that "x[2:4] is masked" should be false is all elements in 2:4 range are masked. Numpy deprecated conversion of size > 1 arrays to bool in favor of explicit any or all: >>> bool(array([1,1])) Traceback (most recent call last): File "", line 1, in ? ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() In my view "x[i] is masked" is ambiguous as well and x.mask.any() or x.mask.all() seems to be a better choice (this would require use of bool_(0) instead of None to indicate no mask). How much code out there relies on "x[i] is masked" construct? I've used MA for many years (thanks for an excellent product, Paul!) but never used this construct in my code. Thanks. -- sasha On 1/6/06, Paul F. Dubois wrote: > I'll look into your complaints. Your proposed solution does not work: > if x[2] is masked: > and > if x[2] is None: > become indistinguishable. Likewise for assignment. > > Your example if x[2:4] is masked is a red herring. Obviously this > statement is false no matter how you define 'masked'. > > I redid ma in a bit of a hurry and didn't pay any attention to the > issues you raise as to len(masked), which I agree should be 1. > > Alexander Belopolsky wrote: > > In the current numpy version ma.masked singleton has the following > > properties: > > > > >>> len(ma.masked) > > 0 > > > > which is a change from old MA, where > > >>> len(MA.masked) > > 1 > > > > Similarly: > > > > >>> ma.masked.shape > > (0,) > > >>> MA.masked.shape > > () > > > > It changes shape when filled: > > > > >>> ma.masked.filled() > > array([999999]) > > > > and so on. > > > > The code contains numerous "if x is masked" statements to support all > > this logic. > > > > I would like to suggest a somewhat radical change for your review. > > > > There are two main uses of ma.masked: > > > > 1. To set mask: > > > > >>> x = ma.zeros(5) > > >>> x[2:4] = ma.masked > > >>> print x > > [0 0 -- -- 0] > > > > > > 2. To check if an element is masked: > > > > >>> x[2] is ma.masked > > True > > > > > > The second property allows a cute looking idiom "x[i] is masked", but > > only works for single element access: > > > > >>> x[2:4] is ma.masked > > False > > > > It also leads to strange results: > > > > >>> x[2].shape > > (0,) > > > > > > My proposal is to eliminate the property #2 from ma and redefine masked > > as None. Single element getitem will return a rank zero MaskedArray. We > > can also add "is_masked" property, which will allow a convenient check > > in the form "x[i].is_masked". > > > > The main benefit of this proposal is that x[i] will be duck typing > > compatible with numpy scalars. > > > > -- sasha > > > > > > ------------------------------------------------------- > > This SF.net email is sponsored by: Splunk Inc. Do you grep through log > > files > > for problems? Stop! Download the new AJAX search engine that makes > > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > > http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > From gruben at bigpond.net.au Fri Jan 6 14:46:36 2006 From: gruben at bigpond.net.au (Gary Ruben) Date: Fri Jan 6 14:46:36 2006 Subject: [Numpy-discussion] test() failures in 0.9.2 In-Reply-To: <43BDAAD3.6070705@ieee.org> References: <43BDAAD3.6070705@ieee.org> Message-ID: <43BE7B46.3030908@bigpond.net.au> Note: Running numpy.test() with the python23 installer version results in 2 errors on both my Win98 and my Win2k box: ERROR: check_basic (numpy.core.defmatrix.test_defmatrix.test_algebra) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\PYTHON23\Lib\site-packages\numpy\core\tests\test_defmatrix.py", line 111, in check_basic assert allclose((mA ** -i).A, B) File "C:\PYTHON23\Lib\site-packages\numpy\core\defmatrix.py", line 148, in __p ow__ x = self.I File "C:\PYTHON23\Lib\site-packages\numpy\core\defmatrix.py", line 203, in get I return matrix(inv(self)) File "C:\PYTHON23\Lib\site-packages\scipy\linalg\basic.py", line 176, in inv a1 = asarray_chkfinite(a) File "C:\PYTHON23\Lib\site-packages\scipy_base\function_base.py", line 32, in asarray_chkfinite if not all(isfinite(x)): TypeError: function not supported for these types, and can't coerce to supported types ====================================================================== ERROR: check_basic (numpy.core.defmatrix.test_defmatrix.test_properties) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\PYTHON23\Lib\site-packages\numpy\core\tests\test_defmatrix.py", line 34, in check_basic assert allclose(linalg.inv(A), mA.I) File "C:\PYTHON23\Lib\site-packages\numpy\core\defmatrix.py", line 203, in get I return matrix(inv(self)) File "C:\PYTHON23\Lib\site-packages\scipy\linalg\basic.py", line 176, in inv a1 = asarray_chkfinite(a) File "C:\PYTHON23\Lib\site-packages\scipy_base\function_base.py", line 32, in asarray_chkfinite if not all(isfinite(x)): TypeError: function not supported for these types, and can't coerce to supported types ---------------------------------------------------------------------- Ran 177 tests in 6.100s FAILED (errors=2) Gary R. From bsouthey at gmail.com Fri Jan 6 14:49:41 2006 From: bsouthey at gmail.com (Bruce Southey) Date: Fri Jan 6 14:49:41 2006 Subject: [Numpy-discussion] Re: [SciPy-user] [SciPy-dev] Long live to numpy (and its list as well) In-Reply-To: <43BD7A99.8070002@noaa.gov> References: <200601051625.k05GPixf021407@oobleck.astro.cornell.edu> <43BD5DFA.20906@colorado.edu> <43BD6109.9030402@sympatico.ca> <43BD7A99.8070002@noaa.gov> Message-ID: I am also +1 to keep numpy for the same reasons as Colin and Chris. So far there has been nothing in alpha SciPy versions that offer any advantage over Numarray for what I use or develop. Bruce On 1/5/06, Christopher Barker wrote: > > Colin J. Williams wrote: > > It seems to me that NumPy is the better way to go, the archives and > > downloads are more readily available there, but it's up to Todd Miller > > and the folk who have been maintaining NumPy. > > > > NumPy has been around for a while, probably longer than SciPy. > > I, for one, am only subscribed to the NumPy list, and have been for > years. I don't know how much non-NumPy stuff there is on the SciPy list, > but while I do use NumPy for Scientific stuff, I don't use the rest of > SciPy (because, frankly, it's always been a pain in the @#! to install) > and I don't need another high-traffic list. > > +1 numpy-discussion > > -Chris > > > -- > Christopher Barker, Ph.D. > Oceanographer > > NOAA/OR&R/HAZMAT (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log > files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Fri Jan 6 19:58:52 2006 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Fri Jan 6 19:58:52 2006 Subject: [Numpy-discussion] Installation of numpy header files Message-ID: Numpy include files get installed in /lib/pythonX.X/site- packages/numpy/base/include/numpy instead of /include/ pythonX.X/numpy. The attached patch fixes the problem. Does anyone rely on the current behavior? -- sasha PS: Using scipy core svn r1847 on a Linux system. -------------- next part -------------- A non-text attachment was scrubbed... Name: test_ma.py-patch Type: application/octet-stream Size: 688 bytes Desc: not available URL: From oliphant.travis at ieee.org Fri Jan 6 19:59:12 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri Jan 6 19:59:12 2006 Subject: [Numpy-discussion] numpy lists In-Reply-To: References: <200601051625.k05GPixf021407@oobleck.astro.cornell.edu> <43BD5DFA.20906@colorado.edu> <43BD6109.9030402@sympatico.ca> <43BD7A99.8070002@noaa.gov> Message-ID: <43BEFA6D.7090305@ieee.org> Bruce Southey wrote: > I am also +1 to keep numpy for the same reasons as Colin and Chris. So > far there has been nothing in alpha SciPy versions that offer any > advantage over Numarray for what I use or develop. There are two new mailing lists for NumPy numpy-devel at lists.sourceforge.net numpy-user at lists.sourceforge.net These are for developers and users to talk about only NumPy The SciPy lists can be for SciPy itself. Two packages deserve separate lists. -Travis From strawman at astraw.com Fri Jan 6 21:13:21 2006 From: strawman at astraw.com (Andrew Straw) Date: Fri Jan 6 21:13:21 2006 Subject: [Numpy-discussion] Installation of numpy header files In-Reply-To: References: Message-ID: <43BF4DD1.2050908@astraw.com> Alexander Belopolsky wrote: >Numpy include files get installed in /lib/pythonX.X/site- >packages/numpy/base/include/numpy instead of /include/ >pythonX.X/numpy. > > >Does anyone rely on the current behavior? > > I do. There was discussion about this a couple of months ago. To find the headers, use the following: import numpy numpy.get_numpy_include() >PS: Using scipy core svn r1847 on a Linux system. > > I guess you're not actually using scipy core, but numpy. > The attached patch fixes the problem. Actually, the attached patch seems entire unrelated. :) From ndarray at mac.com Fri Jan 6 21:24:45 2006 From: ndarray at mac.com (Sasha) Date: Fri Jan 6 21:24:45 2006 Subject: [Numpy-discussion] Re: Installation of numpy header files In-Reply-To: References: Message-ID: Sorry for the wrong attachment. Installing include files under lib breaks modules that use python distutils. Is it recommended that modules that depend on numpy use numpy distutils instead? Thanks. -- sasha On 1/6/06, Alexander Belopolsky wrote: > Numpy include files get installed in /lib/pythonX.X/site- > packages/numpy/base/include/numpy instead of /include/ > pythonX.X/numpy. > > The attached patch fixes the problem. Does anyone rely on the current behavior? > > -- sasha > > PS: Using scipy core svn r1847 on a Linux system. > > > -------------- next part -------------- A non-text attachment was scrubbed... Name: numpy-core-setup-patch Type: application/octet-stream Size: 1505 bytes Desc: not available URL: From ndarray at mac.com Fri Jan 6 21:46:49 2006 From: ndarray at mac.com (Sasha) Date: Fri Jan 6 21:46:49 2006 Subject: [Numpy-discussion] Installation of numpy header files In-Reply-To: <43BF4DD1.2050908@astraw.com> References: <43BF4DD1.2050908@astraw.com> Message-ID: On 1/7/06, Andrew Straw wrote: > ... There was discussion about this a couple of months ago. Could you, please give me some keywords for this discussion? I searched the archives and the only relevant thread was from 2002 (http://aspn.activestate.com/ASPN/Mail/Message/scipy-dev/1592700). I understand that at that time there was a solution involving sitecustomize.py, but that was replaced with site.cfg some time ago. The message that I cited above has a much better description of the problem that I am experiencing than what I wrote: > (1) include directories. > Distutils knows to include files from /usr/include/python2.2 (or > wherever it is installed) whenever building extension modules. > Numeric installs its header files inside this directory when installed > as root. However, when I install Numeric in /home/eric/linux, the > header files are installed in /home/eric/linux/python2.2. Distutils > doesn't know to look in hear for headers. To solve this, I'd have to > hack all the setup.py files for modules that rely on Numeric to use my > include_dirs. This isn't so nice. Thanks. -- sasha From alexander.belopolsky at gmail.com Fri Jan 6 21:47:27 2006 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Fri Jan 6 21:47:27 2006 Subject: [Numpy-discussion] Properties of ma.masked singleton In-Reply-To: <43BEB19F.6010107@pfdubois.com> References: <2988C214-E82E-4604-8697-989C4F6E3AB1@gmail.com> <43BEB19F.6010107@pfdubois.com> Message-ID: On 1/6/06, Paul F. Dubois wrote: > ... > I redid ma in a bit of a hurry and didn't pay any attention to the > issues you raise as to len(masked), which I agree should be 1. I did not suggest that len(masked) should be 1. It was 1 in Numeric MA because scallars had len 1 in Numeric, but in numpy len(scalar) is an error consistent with scalar.shape==() and len(x) == x.shape[0]. My proposal was not to return masked singleton from x[i] and return ma.array(x.data[i],mask=x.mask[i]), this will result in len(x[i]) being an error for rank 1 x and integer i, which would be consistent with numpy. -- sasha From ndarray at mac.com Fri Jan 6 22:05:31 2006 From: ndarray at mac.com (Sasha) Date: Fri Jan 6 22:05:31 2006 Subject: [Numpy-discussion] Installation of numpy header files In-Reply-To: References: <43BF4DD1.2050908@astraw.com> Message-ID: Ok, I've found it: http://www.scipy.org/documentation/mailman?fn=scipy-dev/2005-September/003238.html Sorry for the extra traffic. Let me paraphrase the solution here to hopefully make it more discoverable: """ Extension modules that need to compile against numpy should use get_numpy_include function to locate the appropriate include directory. Using distutils: import numpy Extension('extension_name', ... include_dirs=[numpy.get_numpy_include()]) """ -- sasha On 1/7/06, Sasha wrote: > On 1/7/06, Andrew Straw wrote: > > ... There was discussion about this a couple of months ago. > > Could you, please give me some keywords for this discussion? I > searched the archives and the only relevant thread was from 2002 > (http://aspn.activestate.com/ASPN/Mail/Message/scipy-dev/1592700). I > understand that at that time there was a solution involving > sitecustomize.py, but that was replaced with site.cfg some time ago. > > The message that I cited above has a much better description of the > problem that I am experiencing than what I wrote: > > > (1) include directories. > > Distutils knows to include files from /usr/include/python2.2 (or > > wherever it is installed) whenever building extension modules. > > Numeric installs its header files inside this directory when installed > > as root. However, when I install Numeric in /home/eric/linux, the > > header files are installed in /home/eric/linux/python2.2. Distutils > > doesn't know to look in hear for headers. To solve this, I'd have to > > hack all the setup.py files for modules that rely on Numeric to use my > > include_dirs. This isn't so nice. > > Thanks. > > -- sasha > From strawman at astraw.com Fri Jan 6 22:28:26 2006 From: strawman at astraw.com (Andrew Straw) Date: Fri Jan 6 22:28:26 2006 Subject: [Numpy-discussion] Re: [SciPy-user] numpy lists In-Reply-To: <43BEFA6D.7090305@ieee.org> References: <200601051625.k05GPixf021407@oobleck.astro.cornell.edu> <43BD5DFA.20906@colorado.edu> <43BD6109.9030402@sympatico.ca> <43BD7A99.8070002@noaa.gov> <43BEFA6D.7090305@ieee.org> Message-ID: <43BF14E6.8070507@astraw.com> Travis Oliphant wrote: >There are two new mailing lists for NumPy > >numpy-devel at lists.sourceforge.net >numpy-user at lists.sourceforge.net > >These are for developers and users to talk about only NumPy > > You can subscribe to these lists from http://sourceforge.net/mail/?group_id=1369 From Chris.Barker at noaa.gov Fri Jan 6 22:29:02 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri Jan 6 22:29:02 2006 Subject: [Numpy-discussion] Re: [SciPy-user] [SciPy-dev] Long live to numpy (and its list as well) In-Reply-To: References: <200601051625.k05GPixf021407@oobleck.astro.cornell.edu> <43BD5DFA.20906@colorado.edu> <43BD6109.9030402@sympatico.ca> <43BD7A99.8070002@noaa.gov> Message-ID: <43BF155C.20005@noaa.gov> Bruce Southey wrote: > I am also +1 to keep numpy for the same reasons as Colin and Chris. So > far there has been nothing in alpha SciPy versions that offer any > advantage over Numarray for what I use or develop. Just so it's clear: I was advocating that the numpy-discussion group be the primary place for NumPy-related ( as opposed to other scipy stuff) discussion. I was not advocating for or against the adaptation of the new scipy_base - scipy_core - now-numpy. As it happens, I am very excited by the new numpy, am really looking forward to its rapid adoption. -Chris > On 1/5/06, *Christopher Barker* +1 numpy-discussion -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From paul at pfdubois.com Fri Jan 6 23:19:08 2006 From: paul at pfdubois.com (Paul F. Dubois) Date: Fri Jan 6 23:19:08 2006 Subject: [Numpy-discussion] Properties of ma.masked singleton In-Reply-To: <2988C214-E82E-4604-8697-989C4F6E3AB1@gmail.com> References: <2988C214-E82E-4604-8697-989C4F6E3AB1@gmail.com> Message-ID: <43BEB19F.6010107@pfdubois.com> I'll look into your complaints. Your proposed solution does not work: if x[2] is masked: and if x[2] is None: become indistinguishable. Likewise for assignment. Your example if x[2:4] is masked is a red herring. Obviously this statement is false no matter how you define 'masked'. I redid ma in a bit of a hurry and didn't pay any attention to the issues you raise as to len(masked), which I agree should be 1. Alexander Belopolsky wrote: > In the current numpy version ma.masked singleton has the following > properties: > > >>> len(ma.masked) > 0 > > which is a change from old MA, where > >>> len(MA.masked) > 1 > > Similarly: > > >>> ma.masked.shape > (0,) > >>> MA.masked.shape > () > > It changes shape when filled: > > >>> ma.masked.filled() > array([999999]) > > and so on. > > The code contains numerous "if x is masked" statements to support all > this logic. > > I would like to suggest a somewhat radical change for your review. > > There are two main uses of ma.masked: > > 1. To set mask: > > >>> x = ma.zeros(5) > >>> x[2:4] = ma.masked > >>> print x > [0 0 -- -- 0] > > > 2. To check if an element is masked: > > >>> x[2] is ma.masked > True > > > The second property allows a cute looking idiom "x[i] is masked", but > only works for single element access: > > >>> x[2:4] is ma.masked > False > > It also leads to strange results: > > >>> x[2].shape > (0,) > > > My proposal is to eliminate the property #2 from ma and redefine masked > as None. Single element getitem will return a rank zero MaskedArray. We > can also add "is_masked" property, which will allow a convenient check > in the form "x[i].is_masked". > > The main benefit of this proposal is that x[i] will be duck typing > compatible with numpy scalars. > > -- sasha > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log > files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From vukweer at conservatek.com Sun Jan 8 13:47:45 2006 From: vukweer at conservatek.com (Adkins Sherrie) Date: Sun Jan 8 13:47:45 2006 Subject: [Numpy-discussion] wanted info Message-ID: <001001c6149c$ff8d93c0$72acc756@bleuazurb2nx0k> Current Price: 1.75 Expected 5 day: 2.30 COMPANY OVERVIEW Aggressive and energetic, Infinex boasts a dynamic and diversified portfolio of operations across North America, with an eye on international expansion. Grounded in natural resource exploration, Inifinex also offers investors access to exciting new developments in the high-tech sector and the booming international real estate market. Our market based experience, tenacious research techniques, and razor sharp analytical skills allow us to leverage opportunities in emerging markets and developing technologies. Identifying these opportunities in the earliest stages allows us to accelerate business development and fully realize the company??s true potential. Maximizing overall profitability and in turn enhancing shareholder value. Current Press Release Infinex Ventures Inc. (IFNX - News) is pleased to announce the appointment of Mr. Stefano Masullo, to its Board of Directors. Mr. Michael De Rosa the President says, "Mr. Masullo's varied background in finance, engineering and economics, as well as his experience of over 10 years as a Board member of a vast number of International companies, will make him a valuable addition to the Infinex Board. His appointment will show our commitment to the financial, engineering and business structure of our Company." Mr. Masullo attended the University of Luigi Bocconi, in Milan Italy, where he graduated in industrial, economic and financial sciences. Mr. Masullo first began his well rounded career during one of his years at University (1986-1987), where he assisted the Director of Faculty of Finance in finance and investment. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: yvybu.gif Type: image/gif Size: 2036 bytes Desc: not available URL: From cjw at sympatico.ca Sun Jan 8 14:43:56 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Sun Jan 8 14:43:56 2006 Subject: [Numpy-discussion] Re: [SciPy-user] numpy lists In-Reply-To: <43BF14E6.8070507@astraw.com> References: <200601051625.k05GPixf021407@oobleck.astro.cornell.edu> <43BD5DFA.20906@colorado.edu> <43BD6109.9030402@sympatico.ca> <43BD7A99.8070002@noaa.gov> <43BEFA6D.7090305@ieee.org> <43BF14E6.8070507@astraw.com> Message-ID: <43C15693.2060601@sympatico.ca> Andrew Straw wrote: > Travis Oliphant wrote: > >> There are two new mailing lists for NumPy >> >> numpy-devel at lists.sourceforge.net >> numpy-user at lists.sourceforge.net >> >> These are for developers and users to talk about only NumPy >> >> > You can subscribe to these lists from > http://sourceforge.net/mail/?group_id=1369 I send my postings to numpy-discussion at lists.sourceforge.net. Colin W. From lefeil at email.arizona.edu Sun Jan 8 20:19:10 2006 From: lefeil at email.arizona.edu (lefeil at email.arizona.edu) Date: Sun Jan 8 20:19:10 2006 Subject: [Numpy-discussion] Why I cannot import NumPy? Message-ID: <20060108211842.phbmjs0oggogcckg@www.email.arizona.edu> After installation, I cannot import NumPy. I tried to run setup.py in the numpy directory. But the following error occurs: IDLE 1.1.2 >>> ================================ RESTART ================================ >>> Assuming default configuration (distutils\command/{setup_command,setup}.py was not found) Appending numpy.distutils.command configuration to numpy.distutils Assuming default configuration (distutils\fcompiler/{setup_fcompiler,setup}.py was not found) Appending numpy.distutils.fcompiler configuration to numpy.distutils Appending numpy.distutils configuration to numpy Appending numpy.testing configuration to numpy F2PY Version 2_1830 Appending numpy.f2py configuration to numpy Traceback (most recent call last): File "C:\Python24\lib\site-packages\numpy\setup.py", line 27, in -toplevel- setup(**configuration(top_path='')) File "C:\Python24\lib\site-packages\numpy\setup.py", line 10, in configuration config.add_subpackage('core') File "C:\Python24\Lib\site-packages\numpy\distutils\misc_util.py", line 399, in add_subpackage config = self.get_subpackage(subpackage_name,subpackage_path) File "C:\Python24\Lib\site-packages\numpy\distutils\misc_util.py", line 389, in get_subpackage config = setup_module.configuration(*args) File "C:\Python24\Lib\site-packages\numpy\core\setup.py", line 19, in configuration open(generate_umath_py,'U'),generate_umath_py, IOError: [Errno 2] No such file or directory: 'core\\code_generators\\generate_umath.py' Can someone tell me how to solve this problem? There is no directory names code_generators in the code directory. Thanks Lefei From faltet at carabos.com Mon Jan 9 03:55:02 2006 From: faltet at carabos.com (Francesc Altet) Date: Mon Jan 9 03:55:02 2006 Subject: [Numpy-discussion] Re: [SciPy-dev] How to handle a[...] in numpy? In-Reply-To: References: Message-ID: <200601091254.10861.faltet@carabos.com> A Diumenge 08 Gener 2006 05:10, Sasha va escriure: > In Numeric a[...] would return an array unless a was 0-rank and a > python type othewise. What is the right way to do the same in numpy? [snip] > Proposal: Although I like a lot that 0-rank arrays and numpy scalar > types non-iterable, it may be reasonable to allow a[...]. This way > ellipsis can be interpereted as any number of ":"s including zero. > Another subscript operation that makes sense for scalars would be > a[...,newaxis] or even a[{newaxis, }* ..., {newaxis,}*], where > {newaxis,}* stands for any number of comma-separated newaxis tokens. > This will allow one to use ellipsis in generic code that would work on > any numpy type. I will contribute code if there is any interest. +1 More specifically, provided that: In [65]: type(numpy.array([0])[...]) Out[65]: In [66]: type(numpy.array([0])[0]) Out[66]: In [67]: type(numpy.array([0]).item()) Out[67]: I'd propose the next behaviour for 0-rank arrays: In [65]: type(numpy.array(0)[...]) Out[65]: In [66]: type(numpy.array(0)[()]) # Indexing a la numarray Out[66]: In [67]: type(numpy.array(0).item()) # already works Out[67]: -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From ndarray at mac.com Mon Jan 9 10:20:02 2006 From: ndarray at mac.com (Sasha) Date: Mon Jan 9 10:20:02 2006 Subject: [Numpy-discussion] Re: [SciPy-dev] How to handle a[...] in numpy? In-Reply-To: <200601091254.10861.faltet@carabos.com> References: <200601091254.10861.faltet@carabos.com> Message-ID: On 1/9/06, Francesc Altet wrote: > ... > I'd propose the next behaviour for 0-rank arrays: > > In [65]: type(numpy.array(0)[...]) > Out[65]: > > In [66]: type(numpy.array(0)[()]) # Indexing a la numarray > Out[66]: > I like the idea of supporting [()] on zero rank ndarrays, but I think it should be equivalent to [...]. I view [...] as [(slice(None),) * rank], and thus for rank=0, [...] is the same as [()]. Furthermore, I don't see any use for [...] that always returns the same array. As far as I remember in some old version of Numeric, [...] was a way to make a contiguous copy, but in numpy this is not the case (one needs to use copy method for that). >>> x[::2][...].flags {'WRITEABLE': True, 'UPDATEIFCOPY': False, 'CONTIGUOUS': False, 'FORTRAN': False, 'ALIGNED': True, 'OWNDATA': False} -- sasha From Chris.Barker at noaa.gov Mon Jan 9 10:51:06 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon Jan 9 10:51:06 2006 Subject: [Numpy-discussion] Re: [SciPy-user] numpy lists In-Reply-To: <43BF14E6.8070507@astraw.com> References: <200601051625.k05GPixf021407@oobleck.astro.cornell.edu> <43BD5DFA.20906@colorado.edu> <43BD6109.9030402@sympatico.ca> <43BD7A99.8070002@noaa.gov> <43BEFA6D.7090305@ieee.org> <43BF14E6.8070507@astraw.com> Message-ID: <43C2B09F.9040205@noaa.gov> Andrew Straw wrote: > Travis Oliphant wrote: >> numpy-devel at lists.sourceforge.net >> numpy-user at lists.sourceforge.net > You can subscribe to these lists from > http://sourceforge.net/mail/?group_id=1369 I don't see numpy-devel there. At least it's not obvious. I see: numpy-discussion (which is the list I'm posting too now, and has been around a long time for Numeric, then numarray, and now numpy) and: numpy-user (which does look like a new one) where can I find numpy-devel ? -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From strawman at astraw.com Mon Jan 9 10:55:02 2006 From: strawman at astraw.com (Andrew Straw) Date: Mon Jan 9 10:55:02 2006 Subject: [Numpy-discussion] Re: [SciPy-user] numpy lists In-Reply-To: <43C2B09F.9040205@noaa.gov> References: <200601051625.k05GPixf021407@oobleck.astro.cornell.edu> <43BD5DFA.20906@colorado.edu> <43BD6109.9030402@sympatico.ca> <43BD7A99.8070002@noaa.gov> <43BEFA6D.7090305@ieee.org> <43BF14E6.8070507@astraw.com> <43C2B09F.9040205@noaa.gov> Message-ID: <43C2B161.10009@astraw.com> Christopher Barker wrote: > Andrew Straw wrote: > >> Travis Oliphant wrote: >> >>> numpy-devel at lists.sourceforge.net >>> numpy-user at lists.sourceforge.net >> > >> You can subscribe to these lists from >> http://sourceforge.net/mail/?group_id=1369 > > > I don't see numpy-devel there. At least it's not obvious. I see: > > numpy-discussion (which is the list I'm posting too now, and has been > around a long time for Numeric, then numarray, and now numpy) > > and: > > numpy-user (which does look like a new one) > > where can I find numpy-devel ? It seems to be that numpy-devel has been (silently) deleted. I subscribed immediately after it was announced, tried posting a few days later, and got an error message. I can't remember exactly what the error was, but it seemed to be directly from sourceforge indicating problems with the numpy-devel list. So, I assume the list has simply been deleted. Is this so? From faltet at carabos.com Mon Jan 9 11:04:08 2006 From: faltet at carabos.com (Francesc Altet) Date: Mon Jan 9 11:04:08 2006 Subject: [Numpy-discussion] Re: [SciPy-dev] How to handle a[...] in numpy? In-Reply-To: References: <200601091254.10861.faltet@carabos.com> Message-ID: <200601092003.35687.faltet@carabos.com> A Dilluns 09 Gener 2006 19:19, Sasha va escriure: > On 1/9/06, Francesc Altet wrote: > > ... > > I'd propose the next behaviour for 0-rank arrays: > > > > In [65]: type(numpy.array(0)[...]) > > Out[65]: > > > > In [66]: type(numpy.array(0)[()]) # Indexing a la numarray > > Out[66]: > > I like the idea of supporting [()] on zero rank ndarrays, but I think > it should be equivalent to [...]. I view [...] as [(slice(None),) * > rank], and thus for rank=0, [...] is the same as [()]. However, the original aim of the "..." (ellipsis) operator is [taken for the numarray manual]: """ One final way of slicing arrays is with the keyword `...' This keyword is somewhat complicated. It stands for "however many `:' I need depending on the rank of the object I'm indexing, so that the indices I do specify are at the end of the index list as opposed to the usual beginning". So, if one has a rank-3 array A, then A[...,0] is the same thing as A[:,:,0], but if B is rank-4, then B[...,0] is the same thing as: B[:,:,:,0]. Only one `...' is expanded in an index expression, so if one has a rank-5 array C, then C[...,0,...] is the same thing as C[:,:,:,0,:]. """ > Furthermore, I don't see any use for [...] that always returns the > same array. As far as I remember in some old version of Numeric, > [...] was a way to make a contiguous copy, but in numpy this is not > the case (one needs to use copy method for that). I don't know for old versions of Numeric, but my impression is that the ellipsis meaning is clearly stated above. In fact, in a 4-dimensional array, say a, a[...] should be equivalent to a[:,:,:,:] and this does not necessarily implies a copy. Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From oliphant at ee.byu.edu Mon Jan 9 11:10:05 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Jan 9 11:10:05 2006 Subject: [Numpy-discussion] Re: [SciPy-user] numpy lists In-Reply-To: <43C2B09F.9040205@noaa.gov> References: <200601051625.k05GPixf021407@oobleck.astro.cornell.edu> <43BD5DFA.20906@colorado.edu> <43BD6109.9030402@sympatico.ca> <43BD7A99.8070002@noaa.gov> <43BEFA6D.7090305@ieee.org> <43BF14E6.8070507@astraw.com> <43C2B09F.9040205@noaa.gov> Message-ID: <43C2B4D9.8010307@ee.byu.edu> Christopher Barker wrote: > Andrew Straw wrote: > >> Travis Oliphant wrote: >> >>> numpy-devel at lists.sourceforge.net >>> numpy-user at lists.sourceforge.net >> > >> You can subscribe to these lists from >> http://sourceforge.net/mail/?group_id=1369 > > > I don't see numpy-devel there. At least it's not obvious. I see: > > numpy-discussion (which is the list I'm posting too now, and has been > around a long time for Numeric, then numarray, and now numpy) > > and: > > numpy-user (which does look like a new one) > > where can I find numpy-devel ? Sorry for the noise. I decided one numpy list was enough for now and hid the numpy-devel list. -Travis From alexander.belopolsky at gmail.com Mon Jan 9 12:37:05 2006 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Mon Jan 9 12:37:05 2006 Subject: [SciPy-dev] [Numpy-discussion] Re: How to handle a[...] in numpy? In-Reply-To: <200601092003.35687.faltet@carabos.com> References: <200601091254.10861.faltet@carabos.com> <200601092003.35687.faltet@carabos.com> Message-ID: On 1/9/06, Francesc Altet wrote: > However, the original aim of the "..." (ellipsis) operator is [taken > for the numarray manual]: > > """ > One final way of slicing arrays is with the keyword `...' This keyword > is somewhat complicated. It stands for "however many `:' I need This is precisely my motivation for making a[...] the same as a[()] for zero rank a. In this case "however many" is zero. In other words, a[...] is a short-hand for a[(slice(None),)*len(a.shape)]. Specifically, if a.shape = (), then a[...] = a[()]. > I don't know for old versions of Numeric, but my impression is that > the ellipsis meaning is clearly stated above. In fact, in a > 4-dimensional array, say a, a[...] should be equivalent to a[:,:,:,:] > and this does not necessarily implies a copy. I am not proposing any change for rank > 0 arrays, nor for the new numpy scalars. For a = array(0), why would you want a[...] have different type from a[()]? If as for rank-4 array a, a[...] should be equivalent to a[:,:,:,:] why would you expect a[...] for a rank-0 a be different from a[()]? -- sasha PS: There seems to be a terminological difficulty discussing this type of things. You call an array that takes 4 indices a 4-dimensional array, but in algebra 4-dimensional vector is a sequence of 4 numbers (array of shape (4,)). An object that is indexed by 4 numbers is a tensor of rank 4 (array of shape (n1, n2, n3, n4)). From ndarray at mac.com Mon Jan 9 19:23:03 2006 From: ndarray at mac.com (Sasha) Date: Mon Jan 9 19:23:03 2006 Subject: [SciPy-dev] [Numpy-discussion] Re: How to handle a[...] in numpy? In-Reply-To: References: <200601091254.10861.faltet@carabos.com> <200601092003.35687.faltet@carabos.com> Message-ID: Attached patch implements indexing of zero rank arrays: >>> x = array(42) >>> x[...] 42 >>> x[()] 42 >>> x[()].dtype >>> type(x[()]) Ellipsis and empty tuple are the only two valid indices. So far there was only one +1 vote for the feature (from Francesc Altet, who also suggested that x[()] be allowed). Does anyone object to this feature? I have also proposed to support x[...,newaxis] (see comment by sdementen at http://www.scipy.org/wikis/numdesign/). I will add this feature if there is interest. Finally, if we make x[...] valid for rank-0 arrays, should x[...] = value be allowed as well? I think it should, applying the principle of least surprized. An additional consideration is that rank-0 arrays are unhashable, but there is no obvious way to change their values. The following example show that rank-0 arrays are in fact mutable (in a non-obvious way): >>> x = array(1) >>> x.shape=(1,); x[0] = 42; x.shape=() >>> x Please comment on the following proposals: 1. x[...] 2. x[...] = value 3. x[..., newaxis] -- sasha From alexander.belopolsky at gmail.com Mon Jan 9 19:25:01 2006 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Mon Jan 9 19:25:01 2006 Subject: [SciPy-dev] [Numpy-discussion] Re: How to handle a[...] in numpy? In-Reply-To: References: <200601091254.10861.faltet@carabos.com> <200601092003.35687.faltet@carabos.com> Message-ID: Sorry forgot to attach. On 1/9/06, Sasha wrote: > Attached patch implements indexing of zero rank arrays: > > >>> x = array(42) > >>> x[...] > 42 > >>> x[()] > 42 > >>> x[()].dtype > > >>> type(x[()]) > > > Ellipsis and empty tuple are the only two valid indices. > > So far there was only one +1 vote for the feature (from Francesc > Altet, who also suggested that x[()] be allowed). Does anyone object > to this feature? > > I have also proposed to support x[...,newaxis] (see comment by > sdementen at http://www.scipy.org/wikis/numdesign/). I will add this > feature if there is interest. > > Finally, if we make x[...] valid for rank-0 arrays, should x[...] = > value be allowed as well? I think it should, applying the principle of > least surprized. An additional consideration is that rank-0 arrays > are unhashable, but there is no obvious way to change their values. > > The following example show that rank-0 arrays are in fact mutable (in > a non-obvious way): > > >>> x = array(1) > >>> x.shape=(1,); x[0] = 42; x.shape=() > >>> x > > Please comment on the following proposals: > > 1. x[...] > 2. x[...] = value > 3. x[..., newaxis] > > > -- sasha > -------------- next part -------------- Index: numpy/core/src/arrayobject.c =================================================================== --- numpy/core/src/arrayobject.c (revision 1863) +++ numpy/core/src/arrayobject.c (working copy) @@ -1796,6 +1796,9 @@ return NULL; } if (self->nd == 0) { + if (op == Py_Ellipsis || + PyTuple_Check(op) && 0 == PyTuple_GET_SIZE(op)) + return PyArray_ToScalar(self->data, self); PyErr_SetString(PyExc_IndexError, "0-d arrays can't be indexed."); return NULL; Index: numpy/core/tests/test_multiarray.py =================================================================== --- numpy/core/tests/test_multiarray.py (revision 1863) +++ numpy/core/tests/test_multiarray.py (working copy) @@ -69,5 +69,32 @@ d2 = dtypedescr('f8') assert_equal(d2, dtypedescr(float64)) +class test_zero_rank(ScipyTestCase): + def setUp(self): + self.d = array(0), array('x', object) - + def check_ellipsis_subscript(self): + a,b = self.d + + self.failUnlessEqual(a[...], 0) + self.failUnlessEqual(b[...].item(), 'x') + self.failUnless(type(a[...]) is a.dtype) + self.failUnless(type(b[...]) is b.dtype) + + def check_empty_subscript(self): + a,b = self.d + + self.failUnlessEqual(a[()], 0) + self.failUnlessEqual(b[()].item(), 'x') + self.failUnless(type(a[()]) is a.dtype) + self.failUnless(type(b[()]) is b.dtype) + + def check_invalid_subscript(self): + a,b = self.d + self.failUnlessRaises(IndexError, lambda x: x[0], a) + self.failUnlessRaises(IndexError, lambda x: x[0], b) + self.failUnlessRaises(IndexError, lambda x: x[array([], int)], a) + self.failUnlessRaises(IndexError, lambda x: x[array([], int)], b) + +if __name__ == "__main__": + ScipyTest('numpy.core.multiarray').run() From ndarray at mac.com Mon Jan 9 19:29:02 2006 From: ndarray at mac.com (Sasha) Date: Mon Jan 9 19:29:02 2006 Subject: [SciPy-dev] [Numpy-discussion] Re: How to handle a[...] in numpy? In-Reply-To: References: <200601091254.10861.faltet@carabos.com> <200601092003.35687.faltet@carabos.com> Message-ID: Sorry, forgot to attach. On 1/9/06, Sasha wrote: > Attached patch implements indexing of zero rank arrays: > > >>> x = array(42) > >>> x[...] > 42 > >>> x[()] > 42 > >>> x[()].dtype > > >>> type(x[()]) > > > Ellipsis and empty tuple are the only two valid indices. > > So far there was only one +1 vote for the feature (from Francesc > Altet, who also suggested that x[()] be allowed). Does anyone object > to this feature? > > I have also proposed to support x[...,newaxis] (see comment by > sdementen at http://www.scipy.org/wikis/numdesign/). I will add this > feature if there is interest. > > Finally, if we make x[...] valid for rank-0 arrays, should x[...] = > value be allowed as well? I think it should, applying the principle of > least surprized. An additional consideration is that rank-0 arrays > are unhashable, but there is no obvious way to change their values. > > The following example show that rank-0 arrays are in fact mutable (in > a non-obvious way): > > >>> x = array(1) > >>> x.shape=(1,); x[0] = 42; x.shape=() > >>> x > > Please comment on the following proposals: > > 1. x[...] > 2. x[...] = value > 3. x[..., newaxis] > > > -- sasha > -------------- next part -------------- Index: numpy/core/src/arrayobject.c =================================================================== --- numpy/core/src/arrayobject.c (revision 1863) +++ numpy/core/src/arrayobject.c (working copy) @@ -1796,6 +1796,9 @@ return NULL; } if (self->nd == 0) { + if (op == Py_Ellipsis || + PyTuple_Check(op) && 0 == PyTuple_GET_SIZE(op)) + return PyArray_ToScalar(self->data, self); PyErr_SetString(PyExc_IndexError, "0-d arrays can't be indexed."); return NULL; Index: numpy/core/tests/test_multiarray.py =================================================================== --- numpy/core/tests/test_multiarray.py (revision 1863) +++ numpy/core/tests/test_multiarray.py (working copy) @@ -69,5 +69,32 @@ d2 = dtypedescr('f8') assert_equal(d2, dtypedescr(float64)) +class test_zero_rank(ScipyTestCase): + def setUp(self): + self.d = array(0), array('x', object) - + def check_ellipsis_subscript(self): + a,b = self.d + + self.failUnlessEqual(a[...], 0) + self.failUnlessEqual(b[...].item(), 'x') + self.failUnless(type(a[...]) is a.dtype) + self.failUnless(type(b[...]) is b.dtype) + + def check_empty_subscript(self): + a,b = self.d + + self.failUnlessEqual(a[()], 0) + self.failUnlessEqual(b[()].item(), 'x') + self.failUnless(type(a[()]) is a.dtype) + self.failUnless(type(b[()]) is b.dtype) + + def check_invalid_subscript(self): + a,b = self.d + self.failUnlessRaises(IndexError, lambda x: x[0], a) + self.failUnlessRaises(IndexError, lambda x: x[0], b) + self.failUnlessRaises(IndexError, lambda x: x[array([], int)], a) + self.failUnlessRaises(IndexError, lambda x: x[array([], int)], b) + +if __name__ == "__main__": + ScipyTest('numpy.core.multiarray').run() From faltet at carabos.com Tue Jan 10 00:02:00 2006 From: faltet at carabos.com (Francesc Altet) Date: Tue Jan 10 00:02:00 2006 Subject: [SciPy-dev] [Numpy-discussion] Re: How to handle a[...] in numpy? In-Reply-To: References: <200601092003.35687.faltet@carabos.com> Message-ID: <200601100900.50025.faltet@carabos.com> A Dilluns 09 Gener 2006 21:36, Alexander Belopolsky va escriure: > On 1/9/06, Francesc Altet wrote: > > However, the original aim of the "..." (ellipsis) operator is [taken > > for the numarray manual]: > > > > """ > > One final way of slicing arrays is with the keyword `...' This keyword > > is somewhat complicated. It stands for "however many `:' I need > > This is precisely my motivation for making a[...] the same as a[()] for > zero rank a. In this case "however many" is zero. In other words, a[...] > is a short-hand > for a[(slice(None),)*len(a.shape)]. Specifically, if a.shape = (), > then a[...] = a[()]. I see your point. I was just proposing a way to return different objects for [...] and [()] that were consistent with higher dimensionality. So if one has a non-0 dimensional array (tensor) called a, then a[...] will return exactly the same object, so the same should be done with 0-d arrays (tensors) IMO. Now, the () index in 0d objects is just a way to substitute integer indexes in higher dimensional objects, and should return a scalar type, as it does in the later case. I recognize that this is my interpretation of the facts and chances are that I'm completely wrong with this. Anyway, sometimes I tend to think that this kind of issues are not very important, but from time to time they will certainly bit us, so perhaps this discussion would be useful for the community in the future. > > I don't know for old versions of Numeric, but my impression is that > > the ellipsis meaning is clearly stated above. In fact, in a > > 4-dimensional array, say a, a[...] should be equivalent to a[:,:,:,:] > > and this does not necessarily implies a copy. > > I am not proposing any change for rank > 0 arrays, nor for the new numpy > scalars. For a = array(0), why would you want a[...] have different > type from a[()]? > If as for rank-4 array a, a[...] should be equivalent to a[:,:,:,:] > why would you expect > a[...] for a rank-0 a be different from a[()]? For the reasons mentioned above: just to be consistent with higher dimensions (although as it turns out to be, the interpretation of "consistency" is different between both of us) and to have two different objects returned from [...] and [()]. > PS: There seems to be a terminological difficulty discussing this type > of things. You call an array that takes 4 indices a 4-dimensional > array, but in algebra 4-dimensional vector is a sequence of 4 numbers > (array of shape (4,)). An object that is indexed by 4 numbers is a > tensor of rank 4 (array of shape (n1, n2, n3, n4)). That's true. However, in the Numeric/numarray/numpy literature I think N-dimensional arrays are generally understood as tensors of rank-N. From the scipy manual ("Object Essentials" chapter): """ NumPy provides two fundamental objects: an N-dimensional array object (ndarray) and... """ But anyway, I think that, provided the context is clearly defined, the probability of being misled is (hopefully) quite low. Regards, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From humufr at yahoo.fr Tue Jan 10 07:22:01 2006 From: humufr at yahoo.fr (Humufr) Date: Tue Jan 10 07:22:01 2006 Subject: [Numpy-discussion] initialise an array Message-ID: <43C3D080.7060507@yahoo.fr> Hello, I have a function like this: def test(x,xall=None): if xall == None: xall =x I obtain this error message when I'm doing this with numpy (I don't have this problem with numarray and Numeric). ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all How can I do something similar? Do I have to use the function suggest in the error message? The function is something inside a long script with a lot of condition similar to this one so it's why I ask if it's normal to not have the special None parameter take in count. Thanks, N. From tim.hochberg at cox.net Tue Jan 10 08:47:05 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Tue Jan 10 08:47:05 2006 Subject: [Numpy-discussion] initialise an array In-Reply-To: <43C3D080.7060507@yahoo.fr> References: <43C3D080.7060507@yahoo.fr> Message-ID: <43C3E4CF.10206@cox.net> Humufr wrote: > Hello, > > I have a function like this: > > def test(x,xall=None): > if xall == None: xall =x > > I obtain this error message when I'm doing this with numpy (I don't > have this problem with numarray and Numeric). > > ValueError: The truth value of an array with more than one element is > ambiguous. Use a.any() or a.all > > How can I do something similar? Do I have to use the function suggest > in the error message? The function is something inside a long script > with a lot of condition similar to this one so it's why I ask if it's > normal to not have the special None parameter take in count. Use 'is', for example: if xall is None: xall = x -tim > > Thanks, > > N. > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log > files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > From faltet at carabos.com Tue Jan 10 10:59:13 2006 From: faltet at carabos.com (Francesc Altet) Date: Tue Jan 10 10:59:13 2006 Subject: [Numpy-discussion] Conversion of heterogeneous arrays in numpy<-->numarray Message-ID: <200601101958.32245.faltet@carabos.com> Hi, Is there any plan to provide a conversion between recarrays in numpy and numarray?. For the moment it does not seem to work: In [83]: r=numarray.records.array([(1,11,'a'),(2,22,'b')], formats='u1,f4,a1') In [84]: a=numpy.array(r, dtype='u1,f4,a1') In [85]: numarray.records.array(a) --------------------------------------------------------------------------- exceptions.ValueError Traceback (most recent call last) /tmp/ /usr/lib/python2.3/site-packages/numarray-1.5.0-py2.3-linux-i686.egg/numarray/records.py in array(buffer, formats, shape, names, byteorder, aligned) 406 byteorder=byteorder, aligned=aligned) 407 else: --> 408 raise ValueError("Unknown input type") 409 410 def _RecGetType(name): ValueError: Unknown input type In [86]: numpy.array(r) Out[86]: array([0Aa, ?Ab], dtype=(void,6)) However, if one specifies the format in for the numarray-->numpy it does seem to work: In [87]: numpy.array(r, dtype='u1,f4,a1') Out[87]: array([(1, 11.0, 'a'), (2, 22.0, 'b')], dtype=(void,6)) But not in the sense numpy-->numarray: In [88]: numarray.records.array(a, formats='u1,f4,a1') --------------------------------------------------------------------------- exceptions.ValueError Traceback (most recent call last) /tmp/ /usr/lib/python2.3/site-packages/numarray-1.5.0-py2.3-linux-i686.egg/numarray/records.py in array(buffer, formats, shape, names, byteorder, aligned) 406 byteorder=byteorder, aligned=aligned) 407 else: --> 408 raise ValueError("Unknown input type") 409 410 def _RecGetType(name): ValueError: Unknown input type Having this would be great for supporting heterogeneous arrays in PyTables. Regards, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From perry at stsci.edu Tue Jan 10 11:21:06 2006 From: perry at stsci.edu (Perry Greenfield) Date: Tue Jan 10 11:21:06 2006 Subject: [Numpy-discussion] Conversion of heterogeneous arrays in numpy<-->numarray In-Reply-To: <200601101958.32245.faltet@carabos.com> References: <200601101958.32245.faltet@carabos.com> Message-ID: <7d1c2d9b042dbfd73ac3e7ac273545b5@stsci.edu> I think it would be a good thing, but right now we are more focused on seeing what is involved in porting some of our libraries first. I'll try to find out how much work it would be though. Perry On Jan 10, 2006, at 1:58 PM, Francesc Altet wrote: > Hi, > > Is there any plan to provide a conversion between recarrays in numpy > and numarray?. For the moment it does not seem to work: > > In [83]: r=numarray.records.array([(1,11,'a'),(2,22,'b')], > formats='u1,f4,a1') > > In [84]: a=numpy.array(r, dtype='u1,f4,a1') > > In [85]: numarray.records.array(a) > ----------------------------------------------------------------------- > ---- > exceptions.ValueError Traceback (most > recent > call last) > > /tmp/ > > /usr/lib/python2.3/site-packages/numarray-1.5.0-py2.3-linux-i686.egg/ > numarray/records.py > in array(buffer, formats, shape, names, byteorder, aligned) > 406 byteorder=byteorder, aligned=aligned) > 407 else: > --> 408 raise ValueError("Unknown input type") > 409 > 410 def _RecGetType(name): > > ValueError: Unknown input type > > In [86]: numpy.array(r) > Out[86]: array([0Aa, ?Ab], dtype=(void,6)) > > However, if one specifies the format in for the numarray-->numpy it > does seem to work: > > In [87]: numpy.array(r, dtype='u1,f4,a1') > Out[87]: array([(1, 11.0, 'a'), (2, 22.0, 'b')], dtype=(void,6)) > > But not in the sense numpy-->numarray: > > In [88]: numarray.records.array(a, formats='u1,f4,a1') > ----------------------------------------------------------------------- > ---- > exceptions.ValueError Traceback (most > recent > call last) > > /tmp/ > > /usr/lib/python2.3/site-packages/numarray-1.5.0-py2.3-linux-i686.egg/ > numarray/records.py > in array(buffer, formats, shape, names, byteorder, aligned) > 406 byteorder=byteorder, aligned=aligned) > 407 else: > --> 408 raise ValueError("Unknown input type") > 409 > 410 def _RecGetType(name): > > ValueError: Unknown input type > > Having this would be great for supporting heterogeneous arrays in > PyTables. > > Regards, > > -- >> 0,0< Francesc Altet ? ? http://www.carabos.com/ > V V C?rabos Coop. V. ??Enjoy Data > "-" > > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log > files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://ads.osdn.com/?ad_idv37&alloc_id865&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From ndarray at mac.com Wed Jan 11 10:50:05 2006 From: ndarray at mac.com (Sasha) Date: Wed Jan 11 10:50:05 2006 Subject: [Numpy-discussion] Who will use numpy.ma? Message-ID: MA is intended to be a drop-in replacement for Numeric arrays that can explicitely handle missing observations. With the recent improvements to the array object in NumPy, the MA library has fallen behind. There are more than 50 methods in the ndarray object that are not present in ma.array. I would like to hear from people who work with datasets with missing observations? Do you use MA? Do you think with the support for nan's and replaceable mathematical operations, should missing observations be handled in numpy using special values rather than an array of masks? Thanks. -- sasha From konrad.hinsen at laposte.net Thu Jan 12 02:53:05 2006 From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net) Date: Thu Jan 12 02:53:05 2006 Subject: [Numpy-discussion] Re: [SciPy-user] [SciPy-dev] Long live to numpy (and its list as well) In-Reply-To: <200601052303.04502.faltet@carabos.com> References: <43BD6109.9030402@sympatico.ca> <43BD7A99.8070002@noaa.gov> <200601052303.04502.faltet@carabos.com> Message-ID: <6E760C38-F0F8-4137-B2A0-1FCF42F79FD3@laposte.net> On 05.01.2006, at 23:03, Francesc Altet wrote: > Accordingly, my vote is also: > > +1 numpy-discussion +1 from me as well. Same reason. Konrad. -- ------------------------------------------------------------------------ ------- Konrad Hinsen Laboratoire Leon Brillouin (CEA-CNRS), CEA Saclay, 91191 Gif-sur-Yvette Cedex, France Tel.: +33-1 69 08 79 25 Fax: +33-1 69 08 82 61 E-Mail: khinsen at cea.fr ------------------------------------------------------------------------ ------- From faltet at carabos.com Thu Jan 12 03:23:01 2006 From: faltet at carabos.com (Francesc Altet) Date: Thu Jan 12 03:23:01 2006 Subject: [Numpy-discussion] Re: [SciPy-user] [SciPy-dev] Long live to numpy (and its list as well) In-Reply-To: <6E760C38-F0F8-4137-B2A0-1FCF42F79FD3@laposte.net> References: <200601052303.04502.faltet@carabos.com> <6E760C38-F0F8-4137-B2A0-1FCF42F79FD3@laposte.net> Message-ID: <200601121221.55759.faltet@carabos.com> A Dijous 12 Gener 2006 11:52, konrad.hinsen at laposte.net va escriure: > On 05.01.2006, at 23:03, Francesc Altet wrote: > > Accordingly, my vote is also: > > > > +1 numpy-discussion > > +1 from me as well. Same reason. The outcome of the discussion is that it has been created a new list called numpy-user at lists.sf.net for discussing the new NumPy package. Apparently, the numpy-discussion will still be the place for discussing Numeric/numarray issues. Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From pjssilva at ime.usp.br Thu Jan 12 03:39:01 2006 From: pjssilva at ime.usp.br (Paulo J. S. Silva) Date: Thu Jan 12 03:39:01 2006 Subject: [Numpy-discussion] Two bugs in numpy 0.9.2 and a suggestion. Message-ID: <1137065869.16214.15.camel@localhost.localdomain> Hello, I have just installed numpy 0.9.2 in my system. Thanks for the good work. I think I got two bugs: 1) In the numpy.linalg method the cholesky decomposition doesn't work because the function triu is not available from numpy.core. If you add a line "from numpy import triu" on the top of the linalg.py file (with the other imports), it starts working. 2) If you try to instantiate a ndarray with weird parameters it can segfault and abort the interpreter. For example try to do the following: a = arange(10.) b = ndarray(a) Now the suggestion. What about adding a "transpose" property (in the Python parlance) to ndarray objects? I would call it "t". The following Python code explains what I mean: from numpy import * class Myndarray(ndarray): t = property(transpose, None, None) If you add this 3 lines to a file, import the Myndarray class, and create one of such arrays you will get transpose(a) by simply typing a.t. This allow for the following type of code (which I prefer): cost = dot(c.t, x) instead of cost = dot(transpose(c), x) or cost = dot(c.transpose(), x) Such a compact notation can be a blessing in code that needs lots of transpose (like the BFGS formula for example). If I knew more about writing C extensions for Python I would try to do it myself, but I don't. Is it possible to define properties in a C extension? Best, Paulo -- Paulo Jos? da Silva e Silva Professor Assistente do Dep. de Ci?ncia da Computa??o (Assistant Professor of the Computer Science Dept.) Universidade de S?o Paulo - Brazil e-mail: rsilva at ime.usp.br Web: http://www.ime.usp.br/~rsilva Teoria ? o que n?o entendemos o (Theory is something we don't) suficiente para chamar de pr?tica. (understand well enough to call practice) From joris at ster.kuleuven.be Thu Jan 12 04:44:03 2006 From: joris at ster.kuleuven.be (Joris De Ridder) Date: Thu Jan 12 04:44:03 2006 Subject: [Numpy-discussion] Who will use numpy.ma? Message-ID: <200601121343.13487.joris@ster.kuleuven.be> > With the recent improvements to the array object in NumPy, > the MA library has fallen behind. There are more than 50 methods > in the ndarray object that are not present in ma.array. I guess maintaining MA means double work, doesn't it? So, even if MA is updated now, in the future it's likely to be always somewhat behind. This is perhaps an argument against the use of this library? I wouldn't like to extensively use a library that will be phased out in a couple of years because it turns out to be somewhat redundant and/or behind w.r.t. the rest of numpy. The fact that with numpy you can do things like >>> from numpy import * >>> x = sin(1./arange(3.)) # arbitrary example >>> x[1] = nan # 'masking' an element >>> x = where(isnan(x), 10., x) # replacing masks by a number >>> x = where(x >= 10, nan, x) # clipping with nan replacement >>> y = sin(x) # sin(nan) = nan fulfils most of my needs. However, I haven't compared execution times for large arrays. Note that using NaNs, we don't distinguish between NaN and NA (not available). I am not sure this won't bite us somewhere in the future. I have a related question. numpy introduces the functions nanargmax(), nanargmin(), nanmax(), nanmin(), and nansum(). Was there a special reason to introduce extra nan... functions rather than adding an option like ignore_nan = True to the normal functions? Is this a performance issue? Will similar nan... equivalents be introduced for the functions mean() and reduce()? A side remark: >>> y = array([1, nan, 0.47]) >>> sort(y) array([ 1. , nan, 0.47 ]) No exception, no sorting. Is this a feature or a bug? :) J. Disclaimer: http://www.kuleuven.be/cwis/email_disclaimer.htm From bsouthey at gmail.com Thu Jan 12 06:01:10 2006 From: bsouthey at gmail.com (Bruce Southey) Date: Thu Jan 12 06:01:10 2006 Subject: [Numpy-discussion] Who will use numpy.ma? In-Reply-To: References: Message-ID: Hi, For any data collection in the real world, actual missing values occur very frequently - almost a certainity. For various operations there is probably no difference in what is really used, the main thing that comes to mind is the ability to separate those values that are actually missing i.e. unobserved from those that are obtained from mathematical functions like division by zero. However, it has been some time since I looked at the options so I am out-of-date. Perhaps the approach of the R language ( http://wwwr-project.org) may provide suitable approach to this. A second aspect of the masked arrays that is very neat is to be able to choose a masking value and it can be changed. This is a really feature that you don't realize how great it really is unless you have it! . It is very easy to identify and work with elements of the array that meet changing criteria just by changing the mask rather than a series of complex boolean operations and steps to get the same results. Regards Bruce On 1/11/06, Sasha wrote: > > MA is intended to be a drop-in replacement for Numeric arrays that can > explicitely handle missing observations. With the recent improvements > to the array object in NumPy, the MA library has fallen behind. There > are more than 50 methods in the ndarray object that are not present in > ma.array. > > I would like to hear from people who work with datasets with missing > observations? Do you use MA? Do you think with the support for nan's > and replaceable mathematical operations, should missing observations > be handled in numpy using special values rather than an array of > masks? > > Thanks. > > -- sasha > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log > files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://ads.osdn.com/?ad_idv37&alloc_id865&opclick > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cjw at sympatico.ca Thu Jan 12 06:30:04 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Thu Jan 12 06:30:04 2006 Subject: [Numpy-discussion] Two bugs in numpy 0.9.2 and a suggestion. In-Reply-To: <1137065869.16214.15.camel@localhost.localdomain> References: <1137065869.16214.15.camel@localhost.localdomain> Message-ID: <43C667AE.1010506@sympatico.ca> Paulo J. S. Silva wrote: >Hello, > >I have just installed numpy 0.9.2 in my system. Thanks for the good >work. I think I got two bugs: > >1) In the numpy.linalg method the cholesky decomposition doesn't work >because the function triu is not available from numpy.core. If you add a >line "from numpy import triu" on the top of the linalg.py file (with the >other imports), it starts working. > >2) If you try to instantiate a ndarray with weird parameters it can >segfault and abort the interpreter. For example try to do the following: > >a = arange(10.) >b = ndarray(a) > >Now the suggestion. What about adding a "transpose" property (in the >Python parlance) to ndarray objects? I would call it "t". > Or "T" to distinguish properties, which require no parentheses, from methods, whch do require parentheses? Colin W. From tim.hochberg at cox.net Thu Jan 12 06:51:09 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Thu Jan 12 06:51:09 2006 Subject: [Numpy-discussion] Two bugs in numpy 0.9.2 and a suggestion. In-Reply-To: <1137065869.16214.15.camel@localhost.localdomain> References: <1137065869.16214.15.camel@localhost.localdomain> Message-ID: <43C66CA5.9050005@cox.net> Paulo J. S. Silva wrote: [SNIP] >Now the suggestion. What about adding a "transpose" property (in the >Python parlance) to ndarray objects? I would call it "t". The following >Python code explains what I mean: > >from numpy import * >class Myndarray(ndarray): > t = property(transpose, None, None) > >If you add this 3 lines to a file, import the Myndarray class, and >create one of such arrays you will get transpose(a) by simply typing >a.t. This allow for the following type of code (which I prefer): > >cost = dot(c.t, x) > >instead of > >cost = dot(transpose(c), x) > >or > >cost = dot(c.transpose(), x) > >Such a compact notation can be a blessing in code that needs lots of >transpose (like the BFGS formula for example). > >If I knew more about writing C extensions for Python I would try to do >it myself, but I don't. Is it possible to define properties in a C >extension? > > Wouldn't this be more appropriate in Matrix (I'm assuming numpy has an equivalent to Numeric/Numarray's Matrix class)? For general N-D arrays I rarely find transpose without axes arguments to be useful, so for me this would would just amount to extra cruft. -tim >Best, > >Paulo > > From aisaac at american.edu Thu Jan 12 06:54:03 2006 From: aisaac at american.edu (Alan G Isaac) Date: Thu Jan 12 06:54:03 2006 Subject: [Numpy-discussion] Two bugs in numpy 0.9.2 and a suggestion. In-Reply-To: <43C667AE.1010506@sympatico.ca> References: <1137065869.16214.15.camel@localhost.localdomain> <43C667AE.1010506@sympatico.ca> Message-ID: Paulo wrote: >> Now the suggestion. What about adding a "transpose" property (in the >> Python parlance) to ndarray objects? I would call it "t". On Thu, 12 Jan 2006, "Colin J. Williams" apparently wrote: > Or "T" to distinguish properties, You mean like this? Or something else? Alan Isaac >>> import numpy as N >>> x = N.mat([[1,2,3],[4,5,6]]) >>> y = x.T >>> y matrix([[1, 4], [2, 5], [3, 6]]) From service at inl.ebay.com Thu Jan 12 08:39:10 2006 From: service at inl.ebay.com (eBay) Date: Thu Jan 12 08:39:10 2006 Subject: [Numpy-discussion] Important eBay! Account Information. Message-ID: An HTML attachment was scrubbed... URL: From haase at msg.ucsf.edu Thu Jan 12 09:14:02 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Thu Jan 12 09:14:02 2006 Subject: [Numpy-discussion] Re: [SciPy-user] [SciPy-dev] Long live to numpy (and its list as well) In-Reply-To: <200601121221.55759.faltet@carabos.com> References: <6E760C38-F0F8-4137-B2A0-1FCF42F79FD3@laposte.net> <200601121221.55759.faltet@carabos.com> Message-ID: <200601120913.25890.haase@msg.ucsf.edu> On Thursday 12 January 2006 03:21, Francesc Altet wrote: > A Dijous 12 Gener 2006 11:52, konrad.hinsen at laposte.net va escriure: > > On 05.01.2006, at 23:03, Francesc Altet wrote: > > > Accordingly, my vote is also: > > > > > > +1 numpy-discussion > > > > +1 from me as well. Same reason. > > The outcome of the discussion is that it has been created a new list > called numpy-user at lists.sf.net for discussing the new NumPy package. > Apparently, the numpy-discussion will still be the place for > discussing Numeric/numarray issues. > I'm sorry to disagree, I really think there are to many lists now ! The start of this "lists discussion" was about _reducing_ the total number of scipy/numpy lists, now we have two more ! Obviously there are going to be lots of posts regarding "transition from Numeric/numarray to NumPy" -- are they all supposed to cross post !? My vote (and that's how I read Konrad's - correct my if I'm wrong) is to continue using numpy-discussion and NOT have another lists (numpy-user) . My two cents, Sebastian Haase From konrad.hinsen at laposte.net Thu Jan 12 09:41:04 2006 From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net) Date: Thu Jan 12 09:41:04 2006 Subject: [Numpy-discussion] Re: [SciPy-user] [SciPy-dev] Long live to numpy (and its list as well) In-Reply-To: <200601120913.25890.haase@msg.ucsf.edu> References: <6E760C38-F0F8-4137-B2A0-1FCF42F79FD3@laposte.net> <200601121221.55759.faltet@carabos.com> <200601120913.25890.haase@msg.ucsf.edu> Message-ID: <24C944F6-95F8-4160-B3FD-4C3709591D2C@laposte.net> On 12.01.2006, at 18:13, Sebastian Haase wrote: > Obviously there are going to be lots of posts regarding "transition > from > Numeric/numarray to NumPy" -- are they all supposed to cross post !? > > My vote (and that's how I read Konrad's - correct my if I'm wrong) > is to > continue using numpy-discussion and NOT have another lists (numpy- > user) . I agree. It makes sense to have a separate developer list for those who work *on* NumPy, but please let's stick to a single list for users of all Python array packages. Separate lists will encourage a split in the community, and create lots of crosspostings. Konrad. -- ------------------------------------------------------------------------ ------- Konrad Hinsen Laboratoire Leon Brillouin (CEA-CNRS), CEA Saclay, 91191 Gif-sur-Yvette Cedex, France Tel.: +33-1 69 08 79 25 Fax: +33-1 69 08 82 61 E-Mail: khinsen at cea.fr ------------------------------------------------------------------------ ------- From cjw at sympatico.ca Thu Jan 12 09:52:11 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Thu Jan 12 09:52:11 2006 Subject: [Numpy-discussion] Two bugs in numpy 0.9.2 and a suggestion. In-Reply-To: References: <1137065869.16214.15.camel@localhost.localdomain> <43C667AE.1010506@sympatico.ca> Message-ID: <43C6970F.4080209@sympatico.ca> Alan G Isaac wrote: >Paulo wrote: > > >>>Now the suggestion. What about adding a "transpose" property (in the >>>Python parlance) to ndarray objects? I would call it "t". >>> >>> > > >On Thu, 12 Jan 2006, "Colin J. Williams" apparently wrote: > > >>Or "T" to distinguish properties, >> >> > >You mean like this? Or something else? >Alan Isaac > > > >>>>import numpy as N >>>>x = N.mat([[1,2,3],[4,5,6]]) >>>>y = x.T >>>>y >>>> >>>> >matrix([[1, 4], > [2, 5], > [3, 6]]) > > > > Thanks. Yes, exactly. Colin W. > >------------------------------------------------------- >This SF.net email is sponsored by: Splunk Inc. Do you grep through log files >for problems? Stop! Download the new AJAX search engine that makes >searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! >http://ads.osdn.com/?ad_idv37&alloc_id865&op=click >_______________________________________________ >Numpy-discussion mailing list >Numpy-discussion at lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > From strawman at astraw.com Thu Jan 12 09:56:21 2006 From: strawman at astraw.com (Andrew Straw) Date: Thu Jan 12 09:56:21 2006 Subject: [Numpy-discussion] Re: [SciPy-user] [SciPy-dev] Long live to numpy (and its list as well) In-Reply-To: <24C944F6-95F8-4160-B3FD-4C3709591D2C@laposte.net> References: <6E760C38-F0F8-4137-B2A0-1FCF42F79FD3@laposte.net> <200601121221.55759.faltet@carabos.com> <200601120913.25890.haase@msg.ucsf.edu> <24C944F6-95F8-4160-B3FD-4C3709591D2C@laposte.net> Message-ID: <43C6982B.2040407@astraw.com> konrad.hinsen at laposte.net wrote: > On 12.01.2006, at 18:13, Sebastian Haase wrote: > >> Obviously there are going to be lots of posts regarding "transition >> from >> Numeric/numarray to NumPy" -- are they all supposed to cross post !? >> >> My vote (and that's how I read Konrad's - correct my if I'm wrong) >> is to >> continue using numpy-discussion and NOT have another lists (numpy- >> user) . > > > I agree. It makes sense to have a separate developer list for those > who work *on* NumPy, but please let's stick to a single list for > users of all Python array packages. Separate lists will encourage a > split in the community, and create lots of crosspostings. I agree as well. In the last days, there have been several posts by authors who clearly missed ongoing discussions following from an original cross-post because further discussion took place exclusively on an unsubscribed list. Can we shut down numpy-user? It's only been around a couple weeks, and I bet everyone subscribed there is also on numpy-discussion. Cheers! Andrew From Chris.Barker at noaa.gov Thu Jan 12 09:59:02 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu Jan 12 09:59:02 2006 Subject: [Numpy-discussion] Re: [SciPy-user] [SciPy-dev] Long live to numpy (and its list as well) In-Reply-To: <200601120913.25890.haase@msg.ucsf.edu> References: <6E760C38-F0F8-4137-B2A0-1FCF42F79FD3@laposte.net> <200601121221.55759.faltet@carabos.com> <200601120913.25890.haase@msg.ucsf.edu> Message-ID: <43C69895.4050102@noaa.gov> Sebastian Haase wrote: > now we have two more ! I think it's only one: numpy-dev got dropped. > Obviously there are going to be lots of posts regarding "transition from > Numeric/numarray to NumPy" -- are they all supposed to cross post !? No, they should post to numpy-user. If it's about numpy, post there. Period. If it's about Numeric or numarray, and NOT numpy, then post to the old list. As I imagine there are a lot of folks that are not interested in the latest developments, and are "just using" one of the older packages, then the old list has a valuable place. Personally, I filter them both to the same place, so it makes little difference to me, though all that cross-posting gets annoying. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Fernando.Perez at colorado.edu Thu Jan 12 10:20:06 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Thu Jan 12 10:20:06 2006 Subject: [Numpy-discussion] Re: [SciPy-user] [SciPy-dev] Long live to numpy (and its list as well) In-Reply-To: <43C69895.4050102@noaa.gov> References: <6E760C38-F0F8-4137-B2A0-1FCF42F79FD3@laposte.net> <200601121221.55759.faltet@carabos.com> <200601120913.25890.haase@msg.ucsf.edu> <43C69895.4050102@noaa.gov> Message-ID: <43C69D96.603@colorado.edu> Christopher Barker wrote: > Sebastian Haase wrote: > >>now we have two more ! > > > I think it's only one: numpy-dev got dropped. > > >>Obviously there are going to be lots of posts regarding "transition from >>Numeric/numarray to NumPy" -- are they all supposed to cross post !? > > > No, they should post to numpy-user. If it's about numpy, post there. > Period. If it's about Numeric or numarray, and NOT numpy, then post to > the old list. As I imagine there are a lot of folks that are not > interested in the latest developments, and are "just using" one of the > older packages, then the old list has a valuable place. I don't really have an opinion on this matter, and have run out of mental bandwith to follow it. But I'd like to ask that, when the dust settles, could the powers-that-be please put up a clear notice on the new wiki as to what the lists are, and the purpose of each, with links to the relevant subscription pages? Cheers, f From pjssilva at ime.usp.br Thu Jan 12 10:41:07 2006 From: pjssilva at ime.usp.br (Paulo J. S. Silva) Date: Thu Jan 12 10:41:07 2006 Subject: [Numpy-discussion] Two bugs in numpy 0.9.2 and a suggestion. In-Reply-To: References: <1137065869.16214.15.camel@localhost.localdomain> <43C667AE.1010506@sympatico.ca> Message-ID: <1137091204.16214.26.camel@localhost.localdomain> Thank Alan and David for calling my attention back to the matrix object in numpy/numarray. I usually avoided using it because my first contact comes from Numeric where the Matrix objects don't play nicely with ufunctions: In [1]:from Numeric import * In [2]:from Matrix import Matrix In [3]:a = Matrix([1.,2,3]) In [4]:sin(a) Out[4]:array([ [ 0.84147098, 0.90929743, 0.14112001]]) See... The sine of a Matrix was not a Matrix anymore. I didn't realize this was fixed in numarray: In [1]:from numarray import * In [2]:from numarray.matrix import Matrix In [3]:a = Matrix([1.,2,3]) In [4]:sin(a) Out[4]:_Matrix([ 0.84147098, 0.90929743, 0.14112001]) And numpy keeps the nice behavior: In [1]:from numpy import * In [2]:a = matrix([1.,2,3]) In [3]:sin(a) Out[3]:matrix([[ 0.84147098, 0.90929743, 0.14112001]]) Great! Best, Paulo From cookedm at physics.mcmaster.ca Thu Jan 12 12:41:07 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Thu Jan 12 12:41:07 2006 Subject: [Numpy-discussion] Long live to numpy (and its list as well) In-Reply-To: <43C6982B.2040407@astraw.com> References: <6E760C38-F0F8-4137-B2A0-1FCF42F79FD3@laposte.net> <200601121221.55759.faltet@carabos.com> <200601120913.25890.haase@msg.ucsf.edu> <24C944F6-95F8-4160-B3FD-4C3709591D2C@laposte.net> <43C6982B.2040407@astraw.com> Message-ID: On Jan 12, 2006, at 12:55 , Andrew Straw wrote: > konrad.hinsen at laposte.net wrote: > >> On 12.01.2006, at 18:13, Sebastian Haase wrote: >> >>> Obviously there are going to be lots of posts regarding "transition >>> from >>> Numeric/numarray to NumPy" -- are they all supposed to cross post !? >>> >>> My vote (and that's how I read Konrad's - correct my if I'm wrong) >>> is to >>> continue using numpy-discussion and NOT have another lists (numpy- >>> user) . >> >> >> I agree. It makes sense to have a separate developer list for those >> who work *on* NumPy, but please let's stick to a single list for >> users of all Python array packages. Separate lists will encourage a >> split in the community, and create lots of crosspostings. > > I agree as well. In the last days, there have been several posts by > authors who clearly missed ongoing discussions following from an > original cross-post because further discussion took place > exclusively on > an unsubscribed list. Can we shut down numpy-user? It's only been > around > a couple weeks, and I bet everyone subscribed there is also on > numpy-discussion. I'd have to agree too. For one thing, the names are too similar: what really is the difference between -discussion and -user? numpy- discussion has a long history: this includes archives on sourceforge.net, gmane.org, and elsewhere, which gives a one-stop shop for searching archives, instead of having to remember to look up (or forgetting to look up) in several lists. It's also better for newcomers to only have one obvious list to post to. At the moment, I'd also say numpy is still closely linked with scipy in terms of developers, so a separate numpy-dev list would be premature. -- |>|\/|< /------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From oliphant.travis at ieee.org Thu Jan 12 13:00:06 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu Jan 12 13:00:06 2006 Subject: [Numpy-discussion] Re: [SciPy-user] [SciPy-dev] Long live to numpy (and its list as well) In-Reply-To: <24C944F6-95F8-4160-B3FD-4C3709591D2C@laposte.net> References: <6E760C38-F0F8-4137-B2A0-1FCF42F79FD3@laposte.net> <200601121221.55759.faltet@carabos.com> <200601120913.25890.haase@msg.ucsf.edu> <24C944F6-95F8-4160-B3FD-4C3709591D2C@laposte.net> Message-ID: <43C6C335.3030903@ieee.org> konrad.hinsen at laposte.net wrote: > Separate lists will encourage a split in the community, and create > lots of crosspostings. This convinced me... I closed the numpy-user list (already closed the numpy-devel list). Please make all numpy-related posts to numpy-discussion at lists.sourceforge.net It would help if users would make clear in the subject if they are not posting regarding current NumPy, but that their post is specific to Numarray and/or Numeric. -Travis From oliphant.travis at ieee.org Thu Jan 12 13:23:05 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu Jan 12 13:23:05 2006 Subject: [Numpy-discussion] Re: [Numpy-user] Optional shape in array description In-Reply-To: <200601121945.06205.faltet@carabos.com> References: <200601121945.06205.faltet@carabos.com> Message-ID: <43C6C8A5.1050208@ieee.org> Francesc Altet wrote: >Hi, > >I went over a weirdness on interpretation of the optional shape in the >descriptor of types when declaring records. > >In [176]: numpy.zeros((2,), dtype=[('c1', 'f4')]) >Out[176]: array([(0.0,), (0.0,)], dtype=(void,4)) > >So far, so good. However: > >In [177]: numpy.zeros((2,), dtype=[('c1', 'f4', 1)]) >Out[177]: array([(array([ 0.], dtype=float32),), (array([ 0.], >dtype=float32),)], dtype=(void,4)) > >And: > >In [178]: numpy.zeros((2,), dtype=[('c1', 'f4', (1,))]) >Out[178]: array([(array([ 0.], dtype=float32),), (array([ 0.], >dtype=float32),)], dtype=(void,4)) > >i.e. the very same value is obtained specifying shape=1 and shape=(1,). > >OTOH, in the point 3 of the description of the __array_descr__ of the >array protocol (http://numeric.scipy.org/array_interface.html), one >can read: > >""" >An optional shape tuple providing how many times this part of the >record should be repeated. Assumed (1,) if not given. >""" > >But this is clearly not what it is implemented. > >IMO, the correct thing (and what the numarray does) would be: > >In [176]: numpy.zeros((2,), dtype=[('c1', 'f4')]) >Out[176]: array([(0.0,), (0.0,)], dtype=(void,4)) >In [192]: numpy.zeros((2,), dtype=[('c1', 'f4')])[0][0] >Out[192]: 0.0 > >In [177]: numpy.zeros((2,), dtype=[('c1', 'f4', 1)]) >Out[177]: array([(0.0,), (0.0,)], dtype=(void,4)) >In [193]: numpy.zeros((2,), dtype=[('c1', 'f4', 1)])[0][0] >Out[193]: 0.0 > >In [178]: numpy.zeros((2,), dtype=[('c1', 'f4', (1,))]) >Out[178]: array([(array([ 0.], dtype=float32),), (array([ 0.], >dtype=float32),)], dtype=(void,4)) >In [194]: numpy.zeros((2,), dtype=[('c1', 'f4', (1,))])[0][0] >Out[194]: array([ 0.], dtype=float32) > >i.e. that shape=1 would be equivalent to don't specify it (scalar >case), and shape=(1,) would be equivalent to declare the field as >unidimensional but with one element. > > This is what is now done in SVN... Thanks for catching the corner case. We should also change the verbage on the array_protocol page. -Travis From ndarray at mac.com Thu Jan 12 13:30:05 2006 From: ndarray at mac.com (Sasha) Date: Thu Jan 12 13:30:05 2006 Subject: [Numpy-discussion] Who will use numpy.ma? In-Reply-To: References: <43C5E722.2040007@pfdubois.com> Message-ID: With Paul's permission I am posting his arguments and my responses. Numpy.ma will follow Paul's design and there is now a wiki page dedicated to the effort to make ma work better in numpy. (See http://projects.scipy.org/scipy/numpy/wiki/MaskedArray). -- sasha On 1/12/06, Sasha wrote: > Paul, > > Thank you very much for your insights and once again thanks for all > the great work that you've done. I've noticed that your reply was not > posted on any list, do you mind if I forward it to numpy-user? Please > see more below. > > On 1/12/06, Paul F. Dubois wrote: > > What special values? Are you sure this works on any platform? What, for > > example, is the special value for integer arrays? For arrays of objects? > > > Yes, these are hard questions. For floats nan is an obvious choice and > IEEE support is getting better on the new hardware. For objects None is > a fine choice. For integers some may argue for sys.maxint, and given that > numpy integer arithmetics is already handling overflow a check for maxint will > not add much complexity. Yet don't get me wrong: I don't see any replacement > for ma myself. > > > How do replaceable mathematical operations make any difference? The > > fundamental problem is that if array x has special values in some places > > and array y has them in some other places, how do you create a result > > that has special values in the correct places AND is of a type for which > > those special values are still treated as 'missing'. How do you do this? > > > > Replaceable operations would allow one to redefine all operations on integer > arrays to treat say sys.maxint as invariant and cast it to nan in floating point > conversions without changing the logic of main line numpy. > > > I converted MA to ma but did not have time to flesh out all the > > differences with the new ndarray. I was hoping the community would do > > that. > > Me too. That's was the point of my post - to find out the size of the comunity > rather than to suggest an alternative. > > > I am retired. > > > You deserve it. > > > It is my belief that the approach you outline is not workable, but > > perhaps I am not understanding it properly. > > > I don't have any workable approach other than enchancing ma to work > better with numpy. This is what I am doing right now. > > > If I, who have thought about this a lot, do not know for sure, what > > information can you derive from a poll of the general public, who will > > not think through these issues very carefully? > > > I was trying to poll numpy community to find out how many people actually > use ma in real projects. This would determine how well tested the new features > will be and how quickly any bugs will be discovered and fixed. > Unfortunately, I have > not seen a single response saying - I've been using MA for X years on > Y projects and > plan on using it as we upgrade to numpy. There was a lot of > theoretical discussions > and a pointer to a plotting library that has recently added MA > support, but no testimony > from end users. > > > I am close to absolutely positive that subclassing won't particularly > > ease the task. > > > I thought about this a little, and I think you are right. Subclassing > may improve speed a little, but all methods will need to be adapted > the same ways as it is done without subclassing. > > > For the reason I indicated, I don't care to engage in public discussions > > of complex technical issues so I have not cc'd this to the group. > > > I respect that, but please allow me to forward at least portions of > this correspondence > to the community. Your insights are invaluable. > > -- sasha > > > > > Sasha wrote: > > > MA is intended to be a drop-in replacement for Numeric arrays that can > > > explicitely handle missing observations. With the recent improvements > > > to the array object in NumPy, the MA library has fallen behind. There > > > are more than 50 methods in the ndarray object that are not present in > > > ma.array. > > > > > > I would like to hear from people who work with datasets with missing > > > observations? Do you use MA? Do you think with the support for nan's > > > and replaceable mathematical operations, should missing observations > > > be handled in numpy using special values rather than an array of > > > masks? > > > > > > Thanks. > > > > > > -- sasha > > > > > > > > > ------------------------------------------------------- > > > This SF.net email is sponsored by: Splunk Inc. Do you grep through log files > > > for problems? Stop! Download the new AJAX search engine that makes > > > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > > > http://ads.osdn.com/?ad_idv37&alloc_id865&op=click > > > _______________________________________________ > > > Numpy-discussion mailing list > > > Numpy-discussion at lists.sourceforge.net > > > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > > > > From archetype at virtualcountries.com Thu Jan 12 16:36:05 2006 From: archetype at virtualcountries.com (=?windows-1251?B?MjYtMjcg/+3i4PD/IDIwMDYg4+7k4A==?=) Date: Thu Jan 12 16:36:05 2006 Subject: [Numpy-discussion] =?windows-1251?B?1Ojt4O3x+yDk6/8g8u7vLezl7eXk5uXw7uI=?= Message-ID: ??????? ??? ???-?????????? 26-27 ?????? 2006 ???? ?????? ???????????? ???????? ? ?? ????????? ????????????? ?????????? ? ?????????? ???????? ??????? ?????? ?????, ????? ????????? ??????. ????? ???, ??? ?????????? ?????????? ????????? ??????? ?????? ? ????? ??????? ??????????? ??????????. ?????? ??????? ???????? ???????? ????? ??????????? ???????????, ? ????? ??? ????????????? ????????????, ??????? ????????? ?????????? ????????? ?????????? ????????? ????????, ??????????? ?????????? ??????, ????????? ?????????? ??????? ? - ??? ????????? - ????? ???????? ? ???????? ??????. ? ????????? ??????? ???????? ????? ??????????? ?????? ?????? ??????? ??????????? ??????? ????????, ?????? ??????? ?????????? ???????????; ?????????? ??????? ??????????????? ?????, ?????????? ??????????? ???????? ?? ???? ?????? ???????? ????????. ????????? ???????????? ? ???????? ?????????? ????????? ? ???????, ???????? ?????? ?????????? ?????????, ?????? ???????? ?????? ??????????????, ??????????? ???????????? ? ??????? ??? ???????? ????????? ?????????????? ???????. ?????????? ?????? ????? ?????????? ?? ??????? ???????????? ???????? (??????) ? ? ??????? ?????. ????????? ???????? ?????????? ??????: - ?????????? ?????????? ????? ???????, ?????????? ?????????? ?????????, - ??????????? ? ???????????? ???????? ???? ???????, - ?????? ? ???????? ????????, ??????? ? ???????? ? ???????, - ?????????? ????????? ???????? ???????, - ???????????? ? ??????? ?????????? ???????????, - ???????? ?????????????? ??????? ? ???????? ???????????????? ?????, - ????????? ???????????? ???????? ?? ???????? ?????????? ??????????, - ????????????? ???????? ?????????? ??? ???????? ???????? ??????? ???????, - ????????????? ????????????? ????? ????? ??????????, - ??????????????? ??????? ?????? ????? ?????????? ??????? ? ?????????????? ?? ??????????. ?????????? ?????????: 1. ??????? ?????????????? ??????????? ???????????. 2. ?????????????? ???? ? ????? ? ????????. 3. ???????????? ? ?????????? ????????? ?????. 4. ??????????? ?????????? ??????????. 5. ?????????? ????????? ????????. 6. ?????? ??????????? ????????? ?? ?????? ???????. 7. ???????? ?????? ???????. 8. ?????????? ????? ??????????? ???????. 9. ?????????? ? ????????????? ??????????. 10. ?????????? ????????? ????????. 11. ?????????? ?????????? ????????. 12. ????????? ????????????? ????????????? ???????? ?????????????. 13. ?????????? ??????? ? ????????. 14. ?????????????? ??? ???????? ???????? ??????????? ?????????? ? ????????????. ?????? ????????? ?? ?????? ???????? ? ????????????? ????????, ? ?????? ????????? ?????? ??????? ? ?????? ?????????? ??????. ???????? ????????? ?????????? ?? ?????? ?? ?????????: (095) 980-65-?9, 980-65-?6 -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Thu Jan 12 16:47:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu Jan 12 16:47:02 2006 Subject: [Numpy-discussion] Who will use numpy.ma? In-Reply-To: <200601121343.13487.joris@ster.kuleuven.be> References: <200601121343.13487.joris@ster.kuleuven.be> Message-ID: <43C6F860.20701@ieee.org> Joris De Ridder wrote: >>>>y = array([1, nan, 0.47]) >>>>sort(y) >>>> >>>> >array([ 1. , nan, 0.47 ]) > >No exception, no sorting. Is this a feature or a bug? :) > > It's a "problem" that the user needs to be aware of, but I'm not sure how to fix. Notice that the same problem exists with python try: b =y.tolist() b.sort() Same effect. The problem may be that nans are always False on comparison with another object and this ends up confusing the sort functions. To fix it we would have to write a special nan-sort or else remove the nans entirely, but I don't think we should check for nans for every sort of floating-point values, because it would be slower. So, the user needs to do that. -Travis From cjw at sympatico.ca Thu Jan 12 18:30:01 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Thu Jan 12 18:30:01 2006 Subject: [Numpy-discussion] Overlapping websites? Message-ID: <43C7106E.1070103@sympatico.ca> There are now: http://www.numpy.org and http://sourceforge.net/projects/numpy/ These seem to be synonyms. Are two URI's needed? Numeric 24, not that I use it, seems to have been lost. Colin W. From ariciputi at pito.com Fri Jan 13 01:04:02 2006 From: ariciputi at pito.com (Andrea Riciputi) Date: Fri Jan 13 01:04:02 2006 Subject: [Numpy-discussion] Numpy and the old scipy Message-ID: <7875CF3B-37D2-4BC1-9757-DAFFA9DF8B9A@pito.com> Hi all, I'm a long time Numeric and SciPy user and I've been following with interest the development of the new Numpy package since last year when Travis started rewriting Numeric (thanks a lot Travis!). Now that Numpy seems in a quite-ready state I'd like to give it a try (and hopefully help the developer in the bug fixing cycle). However I remember that in its early days (when it was still named scipy_core) it was not possible to install both the old scipy version (I mean version 0.3.2) and the new one. Since I use both Numeric and SciPy (the old one) on a daily basis for my job, I'd like to know if it's still the case, or if it is now possible to install the new numpy package side by side with the old scipy (and Numeric as well). Thanks, Andrea. From oliphant.travis at ieee.org Fri Jan 13 01:37:00 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri Jan 13 01:37:00 2006 Subject: [Numpy-discussion] Numpy and the old scipy In-Reply-To: <7875CF3B-37D2-4BC1-9757-DAFFA9DF8B9A@pito.com> References: <7875CF3B-37D2-4BC1-9757-DAFFA9DF8B9A@pito.com> Message-ID: <43C7747B.50302@ieee.org> Andrea Riciputi wrote: > > Since I use both Numeric and SciPy (the old one) on a daily basis for > my job, I'd like to know if it's still the case, or if it is now > possible to install the new numpy package side by side with the old > scipy (and Numeric as well). You can install NumPy and Numeric side-by-side just fine. They use different name spaces. In order to install new and old scipy you will definitely need to install one of them to another location besides site-packages (probably new scipy). Then you will need to make sure your sys.path is set up properly to find the scipy you are interested in using for that session. In principle new scipy is almost identical to old scipy except internally it uses NumPy instead of Numeric. So, once you convert from Numeric to NumPy, converting to the new scipy is done (it depends on if you regularly imported scipy_base or scipy_distutils. If you did those are gone. scipy_base--->numpy and scipy_distutils--->scipy.distutils. Keep us posted as to the difficulties as they arise... -Travis From annale at wp.pl Fri Jan 13 03:03:05 2006 From: annale at wp.pl (=?windows-1251?B?x+D35ewg6/7k6CDw4OHu8uD+8g==?=) Date: Fri Jan 13 03:03:05 2006 Subject: [Numpy-discussion] =?windows-1251?B?yuDqIOL75/vi4PL8IPMg6/7k5ekg5uXr4O3o5SDw4OHu8uDy/D8=?= Message-ID: "??? ?? ????? - ???? ???????, ??? ????? - ???? ???????????:" (???????? ????????) ?????? ???? ????????? ????? ???? ????????? ?????? ???? ?????? - ? ??????, ? ?????? - ????? ????? ??? ???????? ??????? ? ??? ?????? ???????????? ????????? ? ?????????? ? ???????? ?????????? ??? ???????? ??? ???????? ?????????? ?????? ??? ?????????? ??? ??????? "?????????????? ?????????"? ?????? ???? "?? ????? ????????????"? ?????? ????????? ?? ?????? ?????? ??????? ?????????? ????? ???????? ?? ???? ??????????? ???? ? ??? ???????? ?? ?? ??????. ?? ?????? ??? ???????? ? ???????? - ?????? ???????? ??? ??????? ? ? ?????? ?????. ? ?????? ??? ???????? ?????? ? ???? ??? ????????? ????????? ? ??????????? ??? ???????? ? ????? ??????? ????????? ????????? "?????? ????" ????????????? ? ?????????? ???? ? ????????? ?????: ??? ????? ????????? ??????????? ??? ?? ????????????? ??? ????????? ????? ? ??????? ??????????? ??? ????? ?????????? ????????? ???? ????????? ??????????? ? ?????? - ???????, ??????? ??????? ??? "????????? ??????", ? ??? ?????????? ? ??????? ????? ????? "?????? - ?? ??????" - ? ???????? ??? ??????????? ???????????. ??? ????? ?????????? ??????????? ??? ???????? ? ???????? ???????? ??? ??????? ? ???????? ??????? ?????????? ??? ?????????? ?????????????, ?????????????? ? ?????????? ?????? ? ??????? ??? ?????????? ????????????? ? ????????????? ???????????? ????????? - ???????????????????? ???????. ? ??? ?????????? ????????? ??????? ???????????????????? ????????? ?????? ??????????? ? ??????????? ??????? ???????? a. ?????????? ???????????, b. ????????????, c. ????????? ?????????????? ??????? ????????????????? ????????: 8 ????? ????????? ???????: 7700 ??????. ??? ?? ??????????. ???????? ????????? ?????????? ? ?????????? ?? ??????? ?? ?????? ?? ???????? (095) 518-05-75, 544-88-14 -------------- next part -------------- An HTML attachment was scrubbed... URL: From pjssilva at ime.usp.br Fri Jan 13 03:49:05 2006 From: pjssilva at ime.usp.br (Paulo J. S. Silva) Date: Fri Jan 13 03:49:05 2006 Subject: [Numpy-discussion] Should dot return 1x1 matrix as scalar? Message-ID: <1137152864.9521.30.camel@localhost.localdomain> Hello, I was playing with the matrix type in numpy and I felt the "need" to have the "dot" function returning a 1x1 matrix as a scalar. This would allow for expressions like: x = matrix(arange(10.)) versorX = x / sqrt(x.T*x) Right now, I have to write: versorX = x / sqrt(x.T*x).item() Actually, a.T*b can not always be used as the inner product with the current behavior. Note that dot already returns scalars when we multiply rank-1 arrays, but matrices are always rank-2. Best, Paulo From aisaac at american.edu Fri Jan 13 06:41:05 2006 From: aisaac at american.edu (Alan G Isaac) Date: Fri Jan 13 06:41:05 2006 Subject: [Numpy-discussion] Should dot return 1x1 matrix as scalar? In-Reply-To: <1137152864.9521.30.camel@localhost.localdomain> References: <1137152864.9521.30.camel@localhost.localdomain> Message-ID: On Fri, 13 Jan 2006, "Paulo J. S. Silva" apparently wrote: > I was playing with the matrix type in numpy and I felt the "need" to > have the "dot" function returning a 1x1 matrix as a scalar. Since a 1x1 matrix is not a scalar, the current behavior is desirable, IMO. Specifically, I want an error in the last example below, since otherwise matrix multiplication will not be associative. Cheers, Alan Isaac PS Current behavior: Python 2.4.1 (#65, Mar 30 2005, 09:13:57) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy as N >>> x = N.mat(range(10)) >>> x matrix([[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]]) >>> x*x.T matrix([[285]]) >>> 285*x matrix([[ 0, 285, 570, 855, 1140, 1425, 1710, 1995, 2280, 2565]]) >>> (x*x.T)*(x.T*x) Traceback (most recent call last): File "", line 1, in ? File "C:\Python24\Lib\site-packages\numpy\core\defmatrix.py", line 128, in __m ul__ return N.dot(self, other) ValueError: objects are not aligned From chanley at stsci.edu Fri Jan 13 07:34:02 2006 From: chanley at stsci.edu (Christopher Hanley) Date: Fri Jan 13 07:34:02 2006 Subject: [Numpy-discussion] numpy revision 1894 fails to build on Solaris Message-ID: <43C7C82C.4070002@stsci.edu> Hi Travis, Numpy failed to build in this morning's regression test. A cursory look seems to indicate that the build failure is related to the umathmodule changes. I'm looking into this now. I've included the build log in case you can recognize the problem right away. Chris -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: build.log URL: From pjssilva at ime.usp.br Fri Jan 13 07:46:01 2006 From: pjssilva at ime.usp.br (Paulo J. S. Silva) Date: Fri Jan 13 07:46:01 2006 Subject: [Numpy-discussion] Should dot return 1x1 matrix as scalar? In-Reply-To: References: <1137152864.9521.30.camel@localhost.localdomain> Message-ID: <1137167107.12551.41.camel@localhost.localdomain> I am sorry, but I don't agree with you. The matrix class seems to be there to help people with numerical linear algebra algorithms "a la matlab". Is this an incorrect assumption? The behavior I am asking can be restricted to the matrix class objects only. However as dot already makes a scalar out the multiplication of two rank-1 arrays (in which case it computes the inner product), I thought that this behavior could be extended to matrix objects. Another possibility (without touching dot) would be to change the functions __mul__, __rmul__, and __imul__ in the matrix class to deal with 1x1 matrices as scalars. That would be OK to me too. The current behavior is odd when writing numerical algorithms. Take a look at these two examples: 1) Computing the norm: The natural way: def norm(x): return sqrt(x.T*x) But in this case you can't write norm(v)*v! Then, you have then to define norm as def norm(x): return sqrt(x.T*x).item() 2) The conjugate gradient iteration: alpha = (r.T*r) / (p.T*A*p) x += alpha*p rold = rnew r += alpha*(A*p) beta = (r.T*r) / (rold.T*rold) p = -r + beta*p This returns an error on second, forth, and the last equation! In all these cases, you'll need a call to the "item" method. Finally, the example you gave (x*x.T)*(x.T*x) does not make sense as matrix multiplication formula unless you interpret the first part as a scalar. You seem to be happy that the compiler got "the mistake". But in any Mathematics book the formula above would be considered OK as the first part would be interpreted as an inner product (and the second an outer product). In my opinion, it is not a mistake in a linear algebra setting. My 2 cents, Paulo -- Paulo Jos? da Silva e Silva Professor Assistente do Dep. de Ci?ncia da Computa??o (Assistant Professor of the Computer Science Dept.) Universidade de S?o Paulo - Brazil e-mail: rsilva at ime.usp.br Web: http://www.ime.usp.br/~rsilva Teoria ? o que n?o entendemos o (Theory is something we don't) suficiente para chamar de pr?tica. (understand well enough to call practice) From oliphant at ee.byu.edu Fri Jan 13 09:07:02 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri Jan 13 09:07:02 2006 Subject: [Numpy-discussion] Should dot return 1x1 matrix as scalar? In-Reply-To: <1137155787.9521.43.camel@localhost.localdomain> References: <1137152864.9521.30.camel@localhost.localdomain> <43C798C1.2070401@ieee.org> <1137155787.9521.43.camel@localhost.localdomain> Message-ID: <43C7DDF9.3010409@ee.byu.edu> Paulo J. S. Silva wrote: >Ops... You are right. The example is not good. Look this though (now I >am actually copying my ipython session, instead of "remembering" it): > >--- Session copy here --- > >In [5]:x = matrix(arange(10.)).T >In [6]:x = matrix(arange(3.)).T >In [7]:A = matrix([[1.,2,3],[4,5,6],[7,8,9]]) >In [8]:b = x.T*x*A >--------------------------------------------------------------------------- >exceptions.ValueError Traceback (most >recent call last) > >/home/pjssilva/ > >/usr/local/lib/python2.4/site-packages/numpy/core/defmatrix.py in >__mul__(self, other) > 126 return N.multiply(self, other) > 127 else: >--> 128 return N.dot(self, other) > 129 > 130 def __rmul__(self, other): > >ValueError: matrices are not aligned > >--- End of copy --- > >You see, the inner product can not be used to mutiply by a matrix, which >is very odd in linear algebra. As the matrix class is supposed to >represent the linear algebra object we I see two options: > >1) Change the __mul__, __rmul__, __imul__ to deal with 1x1 matrices as >scalars. > >2) Change dot to convert 1x1 matrix to scalar at return. > > or 3) Return scalars instead of 1x1 matrices inside of __array_finalize__ (where the magic of ensuring matrices are rank-2 arrays is actually done). From oliphant at ee.byu.edu Fri Jan 13 09:10:01 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri Jan 13 09:10:01 2006 Subject: [Numpy-discussion] Should dot return 1x1 matrix as scalar? In-Reply-To: <1137155787.9521.43.camel@localhost.localdomain> References: <1137152864.9521.30.camel@localhost.localdomain> <43C798C1.2070401@ieee.org> <1137155787.9521.43.camel@localhost.localdomain> Message-ID: <43C7DEAB.8080806@ee.byu.edu> Paulo J. S. Silva wrote: >Obs: Actually I found the "problem" when implementing a QR decomposition >based on Householder reflections and comparing it to a Matlab code. >Numpy is only 10% slower than Matlab in this code. Man, I'll love to >give my numerical linear algebra course this year using Python instead >of Matlab/Octave. > > I would be interested to see how a raw-array solution compares. There is going to be some overhead of using a subclass, because of the attribute lookups that occur on all array creations and because the subclass is written in Python that could be on the order of 10%. Quantifying the subclass slow-down would be useful... Also, which BLAS are you using? -Travis From haase at msg.ucsf.edu Fri Jan 13 09:20:09 2006 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Fri Jan 13 09:20:09 2006 Subject: [Numpy-discussion] A proposal for dtype/dtypedescr in numpy objects In-Reply-To: <200601052240.58099.faltet@carabos.com> References: <200601052240.58099.faltet@carabos.com> Message-ID: <200601130919.41420.haase@msg.ucsf.edu> Hi, Following up: There was never any response to Francesc proposal ! I thought it sounded pretty good - as he argued: Still a good (late but acceptable) time to clean things up ! (I like just the fact that it removes the "ugly" doubling of having two: arr.dtype and arr.dtypecode ) Is this still on the table !? - Sebastian Haase On Thursday 05 January 2006 13:40, Francesc Altet wrote: > Hi, > > In my struggle for getting consistent behaviours with data types, I've > ended with a new proposal for treating them. The basic thing is that I > suggest to deprecate .dtype as being a first-class attribute and > replace it instead by the descriptor type container, which I find > quite more useful for end users. The current .dtype type will be still > accessible (mainly for developers) but buried in .dtype.type. > > Briefly stated: > > current proposed > ======= ======== > .dtypedescr --> moved into .dtype > .dtype --> moved into .dtype.type > .dtype.dtypestr --> moved into .dtype.str > new .dtype.name > > What is achieved with that? Well, not much, except easy of use and > > type comparison correctness. For example, with the next setup: > >>> import numpy > >>> a=numpy.arange(10,dtype='i') > >>> b=numpy.arange(10,dtype='l') > > we have currently: > >>> a.dtype > > > > >>> a.dtypedescr > > dtypedescr(' > >>> a.dtypedescr.dtypestr > > ' > >>> a.dtype.__name__[:-8] > > 'int32' > > >>> a.dtype == b.dtype > > False > > With the new proposal, we would have: > >>> a.dtype.type > > > > >>> a.dtype > > dtype(' > >>> a.dtype.str > > ' > >>> a.dtype.name > > 'int32' > > >>> a.dtype == b.dtype > > True > > The advantages of the new proposal are: > > - No more .dtype and .dtypedescr lying together, just current > .dtypedescr renamed to .dtype. I think that current .dtype does not > provide more useful information than current .dtypedesc, and giving > it a shorter name than .dtypedescr seems to indicate that it is more > useful to users (and in my opinion, it isn't). > > - Current .dtype is still accessible, but specifying and extra name in > path: .dtype.type (can be changed into .dtype.type_ or > whatever). This should be useful mainly for developers. > > - Added a useful dtype(descr).name so that one can quickly access to > the type name. > > - Comparison between data types works as it should now (without having > to create a metaclass for PyType_Type). > > Drawbacks: > > - Backward incompatible change. However, provided the advantages are > desirable, I think it is better changing now than later. > > - I don't specially like the string representation for the new .dtype > class. For example, I'd find dtype('Int32') much better than > dtype(' code, but they can be made later on (much less disruptive than the > proposed change). > > - Some other issues that I'm not aware of. > > > I'm attaching the patch for latest SVN. Once applied (please, pay > attention to the "XXX" signs in patched code), it passes all tests. > However, it may remain some gotchas (specially those cases that are > not checked in current tests). In case you are considering this change > to check in, please, tell me and I will revise much more carefully the > patch. If don't, never mind, it has been a good learning experience > anyway. > > Uh, sorry for proposing this sort of things in the hours previous to a > public release of numpy. From oliphant at ee.byu.edu Fri Jan 13 09:38:03 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri Jan 13 09:38:03 2006 Subject: [Numpy-discussion] A proposal for dtype/dtypedescr in numpy objects In-Reply-To: <200601130919.41420.haase@msg.ucsf.edu> References: <200601052240.58099.faltet@carabos.com> <200601130919.41420.haase@msg.ucsf.edu> Message-ID: <43C7E566.5090701@ee.byu.edu> Sebastian Haase wrote: >Hi, >Following up: There was never any response to Francesc proposal ! >I thought it sounded pretty good - as he argued: Still a good (late but >acceptable) time to clean things up ! >(I like just the fact that it removes the "ugly" doubling of having two: >arr.dtype and arr.dtypecode ) > > I think this proposal came during busy times and was not able to be looked at seriously. Times are still busy and so it is difficult to know what to do wih it. I think there is validity to what he is saying. The dtypedescr was only added in December while the dtype was there in March, so the reason for it is historical. I would not mind changing it so that .dtype actually returned the type-descriptor object. This would actually make things easier. It's only historical that it's not that way. One issue is that .dtypechar is a simple replacement for .typecode() but .dtype.char would involve two attribute lookups which may not be a good thing. But, this might not be a big deal because they should probably be using .dtype anyway. >Is this still on the table !? > > I'm willing to look at it, especially since I like the concept of the dtypedescr much better. >>In my struggle for getting consistent behaviours with data types, I've >>ended with a new proposal for treating them. The basic thing is that I >>suggest to deprecate .dtype as being a first-class attribute and >>replace it instead by the descriptor type container, which I find >>quite more useful for end users. >> I think this is true... I was just nervous to change it. But, prior to a 1.0 release I think we still could, if we do it quickly... >>The current .dtype type will be still >>accessible (mainly for developers) but buried in .dtype.type. >> >>Briefly stated: >> >>current proposed >>======= ======== >>.dtypedescr --> moved into .dtype >>.dtype --> moved into .dtype.type >>.dtype.dtypestr --> moved into .dtype.str >> new .dtype.name >> >> >> I actually like this proposal a lot as I think it gives proper place to the data-type descriptors. I say we do it, very soon, and put out another release quickly. -Travis From aisaac at american.edu Fri Jan 13 11:08:03 2006 From: aisaac at american.edu (Alan Isaac) Date: Fri Jan 13 11:08:03 2006 Subject: [Numpy-discussion] Should dot return 1x1 matrix as scalar? In-Reply-To: <1137167107.12551.41.camel@localhost.localdomain> References: <1137152864.9521.30.camel@localhost.localdomain><1137167107.12551.41.camel@localhost.localdomain> Message-ID: On Fri, 13 Jan 2006, "Paulo J. S. Silva" wrote: > as dot already makes a scalar out the multiplication of > two rank-1 arrays (in which case it computes the inner > product), I thought that this behavior could be extended > to matrix objects. Well, 'dot' for matrices is just a synonym for 'matrixmultiply', which it cannot be (in the same sense) for arrays. But I grant that I find it odd to enforce conformability for multiplication and not for addition. (See below.) I will also grant that GAUSS and Matlab behave as you wish, which might reflect a natural convenience and might reflect their impoverished types. Finally, I grant that I have not been able to quickly think up a use case where I want a 1x1 matrix for anything except error checking. Just to be clear, you do not want to get rid of 1x1 matrices, you just want to get rid of them as the result of *multiplication*, right. So [[1]]*[[2]]=2 but [[1]]+[[2]]=[[3]]. Right? And you would, I presume, find a 'scalar' function to be too clumsy. Cheers, Alan Isaac PS Conformability details: >>> t matrix([[2]]) >>> z matrix([[0, 0, 0], [0, 1, 2], [0, 2, 4]]) >>> t+z matrix([[2, 2, 2], [2, 3, 4], [2, 4, 6]]) >>> t*z Traceback (most recent call last): File "", line 1, in ? File "C:\Python24\Lib\site-packages\numpy\core\defmatrix.py", line 128, in __m ul__ return N.dot(self, other) ValueError: objects are not aligned >>> t.A*z.A array([[0, 0, 0], [0, 2, 4], [0, 4, 8]]) >>> From russel at appliedminds.com Fri Jan 13 11:08:13 2006 From: russel at appliedminds.com (Russel Howe) Date: Fri Jan 13 11:08:13 2006 Subject: [Numpy-discussion] question about index array behavior Message-ID: In the session below, I expected the for loop and the index array to have the same behavior. Is this behavior by design? Is there some other way to get the behavior of the for loop? The loop is too slow for my application ( len(ar1) == 18000). Russel Python 2.4.2 (#1, Nov 29 2005, 08:43:33) [GCC 4.0.1 (Apple Computer, Inc. build 5247)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> from numarray import * >>> import numarray.random_array as ra >>> print libnumarray.__version__ 1.5.0 >>> ar1=ra.random(10) >>> ar2=zeros(5, type=Float32) >>> ind=array([0,0,1,1,2,2,3,3,4,4]) >>> ar2[ind]+=ar1 >>> ar2 array([ 0.09791247, 0.26159889, 0.89386773, 0.32572687, 0.86001897], type=Float32) >>> ar1 array([ 0.49895534, 0.09791247, 0.424059 , 0.26159889, 0.29791802, 0.89386773, 0.44290054, 0.32572687, 0.53337622, 0.86001897]) >>> ar2*=0.0 >>> for x in xrange(len(ind)): ... ar2[ind[x]]+=ar1[x] ... >>> ar2 array([ 0.5968678 , 0.68565786, 1.19178581, 0.76862741, 1.39339519], type=Float32) >>> From aisaac at american.edu Fri Jan 13 11:09:04 2006 From: aisaac at american.edu (Alan Isaac) Date: Fri Jan 13 11:09:04 2006 Subject: [Numpy-discussion] Should dot return 1x1 matrix as scalar? In-Reply-To: <43C7DDF9.3010409@ee.byu.edu> References: <1137152864.9521.30.camel@localhost.localdomain> <43C798C1.2070401@ieee.org> <1137155787.9521.43.camel@localhost.localdomain><43C7DDF9.3010409@ee.byu.edu> Message-ID: On Fri, 13 Jan 2006, Travis Oliphant wrote: > 3) Return scalars instead of 1x1 matrices inside of __array_finalize__ > (where the magic of ensuring matrices are rank-2 arrays is actually done). Just for multiplication, or also for addition etc? Alan Isaac From Chris.Barker at noaa.gov Fri Jan 13 11:15:05 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri Jan 13 11:15:05 2006 Subject: [Numpy-discussion] Numpy and the old scipy In-Reply-To: <43C7747B.50302@ieee.org> References: <7875CF3B-37D2-4BC1-9757-DAFFA9DF8B9A@pito.com> <43C7747B.50302@ieee.org> Message-ID: <43C7FC25.9080607@noaa.gov> Travis Oliphant wrote: > In order to install new and old scipy you will definitely need to > install one of them to another location besides site-packages (probably > new scipy). > Then you will need to make sure your sys.path is set up properly to find > the scipy you are interested in using for that session. It sounds like SciPy could use a versioning scheme much like wxPythons: import wxversion wxversion.select("2.6") import wx See: http://wiki.wxpython.org/index.cgi/MultiVersionInstalls In fact, you could probably just grab the wxPython code and tweek it a little for SciPy. This was debated a lot on the wxPython lists before being implemented. After all you could "Just" write a few start-up scripts that manipulate PYTHONPATH, or re-name some directories, or put in a few sym links, or, or, or... Also, ideally wxPython major versions are compatible, etc, etc. However, when all was said and Dunn, this is a very nice system that works the same way on all platforms, and doesn't get in the way of anything. New versions get installed as the default, so if you never use wxversion, you never know it's there. If you do, then you can test new versions without having to break any old, running utilities, etc. Also, the infrastructure is in place for future major version changes, etc. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From pjssilva at ime.usp.br Fri Jan 13 11:18:01 2006 From: pjssilva at ime.usp.br (Paulo J. S. Silva) Date: Fri Jan 13 11:18:01 2006 Subject: [Numpy-discussion] Should dot return 1x1 matrix as scalar? In-Reply-To: <43C7DEAB.8080806@ee.byu.edu> References: <1137152864.9521.30.camel@localhost.localdomain> <43C798C1.2070401@ieee.org> <1137155787.9521.43.camel@localhost.localdomain> <43C7DEAB.8080806@ee.byu.edu> Message-ID: <1137179742.19206.9.camel@localhost.localdomain> > I would be interested to see how a raw-array solution compares. > There > is going to be some overhead of using a subclass, because of the > attribute lookups that occur on all array creations and because the > subclass is written in Python that could be on the order of 10%. > > Quantifying the subclass slow-down would be useful... Also, which > BLAS > are you using? I am using matlab 6.5.0 release 13 in a Athlon tbird 1.2Ghz Blas with numpy is ATLAS optimized for 3dnow. I m getting very interesting results. First the time for a QR factorization of a 1000x1000 random matrix: Matlab: >> t = cputime; R = houseqr(A); cputime - t ans = 67.1000 Numpy: In [9]:i = time.clock(); R = houseqr.houseqr(A); time.clock() - i Out[9]:79.009999999999991 If I code numpy naively making an extra matrix slice this drops to: In [14]:i = time.clock(); R = houseqr.houseqr(A); time.clock() - i Out[14]:114.34999999999999 (See the code below). I have then decided to campare the blas and here things get funny: Matrix multiplication: Matlab: >> t = cputime; A*A; cputime - t ans = 2.0600 Numpy: In [32]:i = time.clock(); C=A*A; time.clock() - i Out[32]:1.3600000000000136 It looks like numpy (ATLAS) is much better.... However in the QR code the most important part is a matrix times vector multiplication and an outer product. If I benchmark this operations I get: Matlab: >> t = cputime; bench(A,b); cputime - t ans = 5.7500 Numpy: In [27]:i = time.clock(); bench(A,b); time.clock() - i Out[27]:10.610000000000014 Why is numpy so slow?????? Paulo code --- houseqr.py --- from numpy import * def norm(x): return sqrt(x.T*x).item() def sinal(x): if x < 0.0: return -1.0 else: return 1.0 def houseqr(A): m, n = A.shape R = A.copy() bigE1 = matrix(zeros((m, 1), float)) bigE1[0] = 1.0 for j in range(n): x = R[j:,j] v = sinal(x[0])*norm(x)*bigE1[:m-j] + x v = v / norm(v) # Slower version. #R[j:,j:] -= 2.0*v*(v.T*R[j:,j:]) # Faster version: avoids the extra slicing. upRight = R[j:,j:] upRight -= 2.0*v*(v.T*upRight) return R --- houseqr.m --- function [A] = houseqr(A) [m, n] = size(A); bigE1 = zeros(m,1); bigE1(1) = 1.0; for j = 1:n, x = A(j:m,j); v = sinal(x(1))*norm(x)*bigE1(1:m-j+1) + x; v = v / (norm(v)); upRight = A(j:m,j:n); A(j:m,j:n) = upRight - 2*v*(v'*upRight); end --- bench.py --- from numpy import * def bench(A, b): for i in range(100): c = b.T*A d = b*c --- bench.m --- function bench(A, b) for i=1:100, c = b'*A; d = b*c; end From perry at stsci.edu Fri Jan 13 11:39:01 2006 From: perry at stsci.edu (Perry Greenfield) Date: Fri Jan 13 11:39:01 2006 Subject: [Numpy-discussion] question about index array behavior In-Reply-To: References: Message-ID: <1619fd20730e3d62f2b91aa3a28753f1@stsci.edu> On Jan 13, 2006, at 2:07 PM, Russel Howe wrote: > In the session below, I expected the for loop and the index array to > have the same behavior. Is this behavior by design? Is there some > other way to get the behavior of the for loop? The loop is too slow > for my application ( len(ar1) == 18000). > Russel This sort of usage of index arrays is always going to be a bit confusing and this is a common example of that. Anytime you are using repeated indices for index assignment, you are not going to get what you would naively think. It's useful to think of what is going on in a little more detail. Your use of index arrays is resulting in the elements you selected generating a 10 element array which is added to the random elements. Initially it is a 10 element array with all zero elements, and after the addition, it equals the random array elements. Then, the index assignment takes place. First, the first element of the summed array is assigned to 0, then the second element of the summed array is assigned to 0, and that is the problem. The summing is done before the assignment. Generally the last index of a repeated set is what is assigned as the final value. It is possible to do what you want without a for loop, but perhaps not as fast as it would be in C. One way to do it is to sort the indices in increasing order, generate the corresponding selected value array and then use accumulated sums to derive the sums corresponding to each index. It's a bit complicated, but can be much faster than a for loop. See example 3.7.4 to see the details of how this is done in our tutorial: http://www.scipy.org/wikis/topical_software/Tutorial Maybe someone has a more elegant, faster or clever way to do this that I've overlooked. I've seen this come up enough that it may be useful to provide a special function to make this easier to do. Perry Greenfield > Python 2.4.2 (#1, Nov 29 2005, 08:43:33) > [GCC 4.0.1 (Apple Computer, Inc. build 5247)] on darwin > Type "help", "copyright", "credits" or "license" for more information. > >>> from numarray import * > >>> import numarray.random_array as ra > >>> print libnumarray.__version__ > 1.5.0 > >>> ar1=ra.random(10) > >>> ar2=zeros(5, type=Float32) > >>> ind=array([0,0,1,1,2,2,3,3,4,4]) > >>> ar2[ind]+=ar1 > >>> ar2 > array([ 0.09791247, 0.26159889, 0.89386773, 0.32572687, > 0.86001897], type=Float32) > >>> ar1 > array([ 0.49895534, 0.09791247, 0.424059 , 0.26159889, 0.29791802, > 0.89386773, 0.44290054, 0.32572687, 0.53337622, > 0.86001897]) > >>> ar2*=0.0 > >>> for x in xrange(len(ind)): > ... ar2[ind[x]]+=ar1[x] > ... > >>> ar2 > array([ 0.5968678 , 0.68565786, 1.19178581, 0.76862741, > 1.39339519], type=Float32) > >>> > > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log > files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From strawman at astraw.com Fri Jan 13 19:17:01 2006 From: strawman at astraw.com (Andrew Straw) Date: Fri Jan 13 19:17:01 2006 Subject: [Numpy-discussion] Numpy and the old scipy In-Reply-To: <43C7FC25.9080607@noaa.gov> References: <7875CF3B-37D2-4BC1-9757-DAFFA9DF8B9A@pito.com> <43C7747B.50302@ieee.org> <43C7FC25.9080607@noaa.gov> Message-ID: <1497.4.240.165.38.1137208614.squirrel@webmail.astraw.com> Python eggs also support runtime version selection. It's possible we may need to backport egg support to old scipy, but it already works great with new numpy/scipy. I'd suggest this over re-implementing another wheel. > Travis Oliphant wrote: > >> In order to install new and old scipy you will definitely need to >> install one of them to another location besides site-packages (probably >> new scipy). >> Then you will need to make sure your sys.path is set up properly to find >> the scipy you are interested in using for that session. > > It sounds like SciPy could use a versioning scheme much like wxPythons: > > import wxversion > wxversion.select("2.6") > import wx > > See: > > http://wiki.wxpython.org/index.cgi/MultiVersionInstalls > > > In fact, you could probably just grab the wxPython code and tweek it a > little for SciPy. > > This was debated a lot on the wxPython lists before being implemented. > After all you could "Just" write a few start-up scripts that manipulate > PYTHONPATH, or re-name some directories, or put in a few sym links, > or, or, or... Also, ideally wxPython major versions are compatible, etc, > etc. > > However, when all was said and Dunn, this is a very nice system that > works the same way on all platforms, and doesn't get in the way of > anything. New versions get installed as the default, so if you never use > wxversion, you never know it's there. If you do, then you can test new > versions without having to break any old, running utilities, etc. > > Also, the infrastructure is in place for future major version changes, > etc. > > -Chris > > > -- > Christopher Barker, Ph.D. > Oceanographer > > NOAA/OR&R/HAZMAT (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log > files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From oliphant.travis at ieee.org Fri Jan 13 20:41:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri Jan 13 20:41:02 2006 Subject: [Numpy-discussion] Should dot return 1x1 matrix as scalar? In-Reply-To: <1137179742.19206.9.camel@localhost.localdomain> References: <1137152864.9521.30.camel@localhost.localdomain> <43C798C1.2070401@ieee.org> <1137155787.9521.43.camel@localhost.localdomain> <43C7DEAB.8080806@ee.byu.edu> <1137179742.19206.9.camel@localhost.localdomain> Message-ID: <43C880B2.1050002@ieee.org> Paulo J. S. Silva wrote: >Numpy: > >In [27]:i = time.clock(); bench(A,b); time.clock() - i >Out[27]:10.610000000000014 > > >Why is numpy so slow?????? > > > I think the problem here is that using the properties here to take advantage of the nice matrix math stuff is slower than just computing the dot product in the fastest possible way with raw arrays. I've been concerned about this for awhile. The benchmark below makes my point. While a matrix is a nice thing, I think it will always be slower.... It might be possible to speed it up and I'm open to suggestions... To see what I mean, try this.... import timeit t1 = timeit.Timer('c = b.T*A; d=c*b','from numpy import rand,mat; A = mat(rand(1000,1000));b = mat(rand(1000,1))') t2 = timeit.Timer('c = dot(b,A); d=dot(b,c)','from numpy import rand, dot; A = rand(1000,1000);b = rand(1000)') >>> t1.timeit(100) 6.0398328304290771 >>> t2.timeit(100) 1.2430641651153564 So, using raw arrays and dot product is 5x faster in this case..... -Travis From pjssilva at ime.usp.br Sat Jan 14 05:52:06 2006 From: pjssilva at ime.usp.br (Paulo J. S. Silva) Date: Sat Jan 14 05:52:06 2006 Subject: [Numpy-discussion] Should dot return 1x1 matrix as scalar? In-Reply-To: <43C880B2.1050002@ieee.org> References: <1137152864.9521.30.camel@localhost.localdomain> <43C798C1.2070401@ieee.org> <1137155787.9521.43.camel@localhost.localdomain> <43C7DEAB.8080806@ee.byu.edu> <1137179742.19206.9.camel@localhost.localdomain> <43C880B2.1050002@ieee.org> Message-ID: <1137246317.7490.15.camel@localhost.localdomain> Em Sex, 2006-01-13 ?s 21:40 -0700, Travis Oliphant escreveu: > Paulo J. S. Silva wrote: > > >Numpy: > > > >In [27]:i = time.clock(); bench(A,b); time.clock() - i > >Out[27]:10.610000000000014 > > > > > >Why is numpy so slow?????? > > > > > > > > I think the problem here is that using the properties here to take > advantage of the nice matrix math stuff is slower than just computing > the dot product in the fastest possible way with raw arrays. I've been > concerned about this for awhile. The benchmark below makes my point. > > While a matrix is a nice thing, I think it will always be slower.... It > might be possible to speed it up and I'm open to suggestions... > > To see what I mean, try this.... > Travis, I think I got the gotcha! The problem is the in the dot function. I have only skimmed on the _dotblas code, but I assume it try to find the right blas function based on the arguments ranks. If both arguments are matrices then both are rank two and the matrix-multiply blas function is called. This is "very" suboptimal if some of the matrices objects actually represent a vector or a scalar. The same problem would appear with pure arrays if one insists on using a column vector: In [3]:t1 = timeit.Timer('c = dot(A,b)','from numpy import rand, dot; A = rand(1000,1000);b = rand(1000)') In [4]:t2 = timeit.Timer('c = A*b','from numpy import rand, dot, mat; A = mat(rand(1000,1000));b = mat(rand(1000,1))') In [5]:t3 = timeit.Timer('c = dot(A,b)','from numpy import rand, dot, transpose; A = rand(1000,1000);b = rand(1000,1)') In [6]:t1.timeit(100) Out[6]:0.69448995590209961 In [7]:t2.timeit(100) Out[7]:1.1080958843231201 You see? The third test with pure arrays is as slow as the matrix based test. The problem is even more dramatic if you are trying to compute an inner product: In [13]:t1 = timeit.Timer('c = dot(b,b)','from numpy import rand, dot; b = rand(1000)') In [14]:t2 = timeit.Timer('c = b.T*b','from numpy import rand, mat; b = mat(rand(1000,1))') In [15]:t3 = timeit.Timer('c = dot(transpose(b),b)','from numpy import rand, dot, transpose; b = rand(1000,1)') In [16]:t1.timeit(10000) Out[16]:0.053219079971313477 In [17]:t2.timeit(10000) Out[17]:0.65550899505615234 In [18]:t3.timeit(10000) Out[18]:0.62446498870849609 Note that this is a very serious drawback for matrices objects as the most usual operation, matrix-vector multiplication, in numerical linear algebra algorithms is always suboptimal. The solution may be a "smarter" analysis in _dotblas.c that not only looks at the rank but that also "collapses" dimensions with only one element to decide which blas function to call. Note that this approach may be used to solve the problem with 1x1 matrices if properly coded. Some special care has to be taken to identify outer products to. I may try to look at this myself if you want. But certainly only in three weeks. I am preparing for an exam to confirm (make permanent) my position at the university that will take place in the last days of January. Best, Paulo From baccarat at albawaba.com Sat Jan 14 16:07:00 2006 From: baccarat at albawaba.com (=?windows-1251?B?MjQgLSAyNSD/7eLg8P8gMjAwNiDj7uTg?=) Date: Sat Jan 14 16:07:00 2006 Subject: [Numpy-discussion] =?windows-1251?B?y87DyNHSyMrAINHKy8DEwDogwcjHzcXRLc/LwM3I0M7CwM3IxSwgz9DOxcrSyNDO?= =?windows-1251?B?wsDNyMUgyCDQxc7Qw8DNyMfA1sjf?= Message-ID: <9d5001c618ea$0e435f2c$4e8f2a6e@albawaba.com> ?????????? ????????????? ????????????? ?????, ??????????? ???????, ??????? ?????????? ??????? ??????? ? ????? ?????????? ????????: ????????? ??????: ??????-????????????, ?????????????? ? ?????????????24 - 25 ?????? 2006 ???? ?? ????????? ???????? ?? ???????? ???? ???????????? ? ???????? ?????????????? ? ????????????? ??????, ??????????? ????????? ?????? ? ??????????, ??????? ?????? ????????????? ??????????? ???????????? ? ????? ? ???????? ?????????????? ? ?????????????, ? ????? ??????? ??????????? ?????????? ??????-????? ???????? ?????? ? ??????? ??????, ??????????? ??? ???????? ??????? ? ?????????? ? ???????????. ????????? ????????: ??????-???? - ?????? ???????? ?????????????? ??????. ??????? ??????????????, ????????????? ? ??????? ??????. ??????-???? - ?????? ???????? ????????????. ????????? ??????????? ?????????. ???????? ????????????? ?????????. ????????? ? ??????????? ?????? ???????. ??????????? ?????? - ?? ????? ? ?????????. ???? ?????????? ????????? ????????????? ?? ??????????? ???????. ???? ?????????? ????????? ????????????. ????? ???????????? ??? ?????? ? ??????????? ?? ???? ??????????. ??????????????? ???? ??????. ????? ??? ? ??????????? ?? ???? ???????????????? ???????? ???????????? ??? ???????? ???????. ????????????? ????? ??????????? ????????????. ??????????? ???????????? ??? ??????. ?????????????????? ?????????? ??????? ????????, ???????? ? ?????????????? ????? ???? ROTOMAT, LEANLIFT. ??????????? ?????????? ??????? ??????? ???????. ??????????? ???????? ?????? ? ?????????. ????????-??????????? ? ?????????????????? ???????. ????????????? ? ?????????????? ????????-????????????? ????????????. ????? ?????? ? ?????????????? ??????. ?????? ???????? ? ????? ?????? ??????????????. ??????????????????. ??????????? ??????? ? ??? ??????? ?? ??????. ??????? ???????? ??????? ? ??? ??????? ?? ??????? ?????????????? ??????. ????? ????????? ??????? ????????? ??????? ?? ?????? 3 - 5 ???. ??????????? ???????. ?????????? ???? ???????? ???????. ??????? ??????????????? ??????. ??????????????? ??????? ?? ??????. ?????????????? ?????????? ???????? ??????????????? ????????. ???????. ?????????????. ?????????? ?? ????????. ?????. ???????????? ???????. ???????? ??????? ???????. ??????????????. ??????????? ?????????? ?????????????? ??????. ???????????? ????????????? - ?????? ??????? ???????????. ?????????????? ????????????? ??????????????? ?????????. ????? ????????? ???????????. ?????? ????????????? ????????????? ?????? ? ?????? ??????? ???????? ??????????. ??????????? ? ???????????????? ??????? ?? ???????? ???????. ?????? ????????????? ????????????? ??????. ??????? ??? ????????? ????? ??????????? ??????. ?????????????? ????????????? ? ??????????? ???????? ??????????. ????????? ???????????. ?????? ? ??????????. ?????? ?????? ? ???????? ???????? ??????? ?? ???????????. ??????? ?????????? ???????????????? ?????????? ?? ?????? ????? ???????? - ????????? 7 ??? ?????????? ??????????? ??????????????? ???????? ?????????? ????????? ???????????????? ??????????? ? ????????? ?????? ????????? ???????. ? ????????? ????? ????? ?????????? ???????? ????? ????????? ?????????? ??? ???? ??????????, ????? ??? AVON, ???????? ???, ?????, ???????? Computerized Storage Systems Corp (??????? ???????? ??????? - ?????????), ???????? ???????. ?????? ????????? ?? ?????? ???????? ? ????????????? ????????,? ?????? ????????? ?????? ??????? ? ?????? ?????????? ??????. ???????? ????????? ?????????? ?? ?????? ?? ?????????: (495) 980-65-?9, 98?-65-36 -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Sat Jan 14 19:47:00 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sat Jan 14 19:47:00 2006 Subject: [Numpy-discussion] Changed attributes .dtypedescr, .dtypechar, .dtypestr and .dtype Message-ID: <43C9C573.2020809@ieee.org> There was some cruft left over from the change to making data-type descriptors real Python objects. This left lots of .dtype related attributes on the array object --- too many as Francesc Altet graciously pointed out. In the latest SVN, I've cleaned things up (thanks to a nice patch from Francesc to get it started). Basically, there is now only one attribute on the array object dealing with the data-type (arr.dtype). This attribute returns the data-type descriptor object for the array. This object itself has the attributes .char, .str, and .type (among others). I think this will lead to less confusion long term. The cruft was due to the fact that my understanding of the data-type descriptor came in December while seriously looking at records module. This will have some backward-compatibility issues (we are still pre-1.0 and early enough that I hope this is not too difficult to deal with). The compatibility to numpy-0.9.2 issues I can see are: 1) Replacing attributes that are now gone: .dtypechar --> .dtype.char .dtypestr --> .dtype.str .dtypedescr --> .dtype 2) Changing old .dtype -> .dtype.type This is only necessary if you were using a.dtype as a *typeobject* as in issubclass(a.dtype, ) If you were using .dtype as a parameter to dtype= then that usage will still work great (in fact a little faster) because now .dtype returns a "descriptor object" 3) The dtypedescr constructor is now called dtype. This change should have gone into the 0.9.2 release, but things got too hectic with all the name changes. I will quickly release 0.9.4 with these changes unless I hear strong disagreements within the next few days. -Travis P.S. SciPy SVN has been updated and fixed with the changes. Numeric compatibility now implies that .typecode() --> .dtype.char although if .typecode() was used as an argument to a function, then .dtype will very likely work. -Travis From oliphant.travis at ieee.org Sat Jan 14 21:59:01 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sat Jan 14 21:59:01 2006 Subject: [Numpy-discussion] Speed ups to matrix-vector multiplication In-Reply-To: <43C880B2.1050002@ieee.org> References: <1137152864.9521.30.camel@localhost.localdomain> <43C798C1.2070401@ieee.org> <1137155787.9521.43.camel@localhost.localdomain> <43C7DEAB.8080806@ee.byu.edu> <1137179742.19206.9.camel@localhost.localdomain> <43C880B2.1050002@ieee.org> Message-ID: <43C9E457.3020109@ieee.org> Travis Oliphant wrote: > Paulo J. S. Silva wrote: > >> Numpy: >> >> In [27]:i = time.clock(); bench(A,b); time.clock() - i >> Out[27]:10.610000000000014 >> >> >> Why is numpy so slow?????? >> >> >> > > I think the problem here is that using the properties here to take > advantage of the nice matrix math stuff is slower than just computing > the dot product in the fastest possible way with raw arrays. I've > been concerned about this for awhile. The benchmark below makes my > point. > > While a matrix is a nice thing, I think it will always be slower.... > It might be possible to speed it up and I'm open to suggestions... Indeed it was possible to speed things up as Paulo pointed out, by using the correct blas calls for the real size of the array. With some relatively simple modifications, I was able to get significant speed-ups in time using matrices: import timeit t1 = timeit.Timer('c = b.T*A; d=c*b','from numpy import rand,mat; A = mat(rand(1000,1000));b = mat(rand(1000,1))') t2 = timeit.Timer('c = dot(b,A); d=dot(b,c)','from numpy import rand, dot; A = rand(1000,1000);b = rand(1000)') t1.timeit(100) 1.4369449615478516 t2.timeit(100) 1.2983191013336182 Now, that looks more like a 10% overhead for the matrix class. -Travis From zpincus at stanford.edu Sun Jan 15 00:24:01 2006 From: zpincus at stanford.edu (Zachary Pincus) Date: Sun Jan 15 00:24:01 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location Message-ID: <936E82E6-E8C0-44A5-AB78-22FAEC4F4622@stanford.edu> Hello folks, I'm quite excited about the new release of numpy -- thanks for all the work. One question: where should I expect the arrayobject.h file to be installed? My old installation of Numeric placed the arrayobject.h (etc.) header files here: /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/ Numeric/ I'm on OS X, using the standard 'framework' build of python, which accounts for the particular install path. However, this is the proper include path for python-related C headers. When I installed numpy, arrayobject.h was placed here, in the site- packages directory: /Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site- packages/numpy/core/include/numpy/arrayobject.h My question: is this an error (possibly specific to OS X) and the numpy headers should have been placed with the other C headers? Or is this a policy change which puts the headers in the site-packages directory? I'm writing some code which I hope to be compatible with Numeric, numarray, and numpy; being able to reliably find arrayobject.h is pretty important for this. Thus, it would be good to know where, in general, this file will be installed by numpy. Thanks, Zach Pincus Program in Biomedical Informatics and Department of Biochemistry Stanford University School of Medicine From pearu at cens.ioc.ee Sun Jan 15 04:24:01 2006 From: pearu at cens.ioc.ee (pearu at cens.ioc.ee) Date: Sun Jan 15 04:24:01 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: <936E82E6-E8C0-44A5-AB78-22FAEC4F4622@stanford.edu> Message-ID: On Sun, 15 Jan 2006, Zachary Pincus wrote: > One question: where should I expect the arrayobject.h file to be > installed? numpy/arrayobject.h is installed to numpy.get_numpy_include(). > My old installation of Numeric placed the arrayobject.h (etc.) header > files here: > /Library/Frameworks/Python.framework/Versions/2.4/include/python2.4/ > Numeric/ > > I'm on OS X, using the standard 'framework' build of python, which > accounts for the particular install path. However, this is the proper > include path for python-related C headers. > > When I installed numpy, arrayobject.h was placed here, in the site- > packages directory: > /Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site- > packages/numpy/core/include/numpy/arrayobject.h > > My question: is this an error (possibly specific to OS X) and the > numpy headers should have been placed with the other C headers? Or is > this a policy change which puts the headers in the site-packages > directory? This is not an error. Use numpy.get_numpy_include() to retrive the directory of numpy header files. See numpy.get_numpy_include.__doc__ for more information. HTH, Pearu From aisaac at american.edu Sun Jan 15 05:14:02 2006 From: aisaac at american.edu (Alan G Isaac) Date: Sun Jan 15 05:14:02 2006 Subject: [Numpy-discussion] round_ vs round Message-ID: numpy's use of `round_` rather than `round` feels like a blemish. Was this done just to protect the built-in? If so, is that a good enough reason? fwiw, Alan Isaac From zpincus at stanford.edu Sun Jan 15 11:25:03 2006 From: zpincus at stanford.edu (Zachary Pincus) Date: Sun Jan 15 11:25:03 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: References: Message-ID: <2D771DF0-1A60-463B-A643-AD6E2980AB94@stanford.edu> >> Or is this a policy change which puts the headers in the site- >> packages >> directory? > > This is not an error. Use numpy.get_numpy_include() to retrive the > directory of numpy header files. See numpy.get_numpy_include.__doc__ > for more information. Thanks for the information. Unfortunately, the package I'm working on can't be built with distutils, so it may wind up being something of a contortion to call numpy.get_numpy_include() from the build system I need to use. Understanding that the optimal, preferred method of finding the include dir will always be numpy.get_numpy_include(), if I must resort to a suboptimal method of guessing likely locations, what will those locations be? Are they generally within the site-packages directory? Sometimes in the python include directory? Will they be ever-changing? I will endeavor to use the more proper methods in my system, but this information would be useful as a fallback. Thanks, Zach From xkgbpbxwo at amtote.com Sun Jan 15 13:00:03 2006 From: xkgbpbxwo at amtote.com (Ali Hattie) Date: Sun Jan 15 13:00:03 2006 Subject: [Numpy-discussion] Top news from Kristie Message-ID: <001001c61a16$8adf23c0$402c1c53@xr2xz8y0ujs4zq> Palmer Sara -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: dbdctphdmw.gif Type: image/gif Size: 11162 bytes Desc: not available URL: From ndarray at mac.com Sun Jan 15 19:17:01 2006 From: ndarray at mac.com (Sasha) Date: Sun Jan 15 19:17:01 2006 Subject: [Numpy-discussion] NumPy's integer arithmetics differs from Numeric Message-ID: It looks like NumPy and Numeric handle invalid integer operations differently: >>> from numpy import * >>> import Numeric as n >>> array(2)**40 2147483647 >>> array(2)/0 0 In numeric both fail: >>> n.array(2)**40 Traceback (most recent call last): File "", line 1, in ? ArithmeticError: Integer overflow in power. >>> n.array(2)/0 Traceback (most recent call last): File "", line 1, in ? ZeroDivisionError: divide by zero I don't know if the change was intentional, but if it was, I think it should be advertised. -- sasha From ndarray at mac.com Sun Jan 15 21:40:03 2006 From: ndarray at mac.com (Sasha) Date: Sun Jan 15 21:40:03 2006 Subject: [Numpy-discussion] NumPy's integer arithmetics differs from Numeric In-Reply-To: References: Message-ID: It looks like NumPy and Numeric handle invalid integer operations differently: >>> from numpy import * >>> import Numeric as n >>> array(2)**40 2147483647 >>> array(2)/0 0 In numeric both fail: >>> n.array(2)**40 Traceback (most recent call last): File "", line 1, in ? ArithmeticError: Integer overflow in power. >>> n.array(2)/0 Traceback (most recent call last): File "", line 1, in ? ZeroDivisionError: divide by zero I don't know if the change was intentional, but if it was, I think it should be advertised. -- sasha From cookedm at physics.mcmaster.ca Sun Jan 15 23:08:02 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Sun Jan 15 23:08:02 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: <2D771DF0-1A60-463B-A643-AD6E2980AB94@stanford.edu> References: <2D771DF0-1A60-463B-A643-AD6E2980AB94@stanford.edu> Message-ID: On Jan 15, 2006, at 14:23 , Zachary Pincus wrote: >>> Or is this a policy change which puts the headers in the site- >>> packages >>> directory? >> >> This is not an error. Use numpy.get_numpy_include() to retrive the >> directory of numpy header files. See numpy.get_numpy_include.__doc__ >> for more information. > > Thanks for the information. Unfortunately, the package I'm working > on can't be built with distutils, so it may wind up being something > of a contortion to call numpy.get_numpy_include() from the build > system I need to use. You don't say what type of build system it is. If it's autoconf or Makefile based, you could do something like NUMPY_HEADERS=$(python -c 'import numpy; print numpy.get_numpy_include ()') > Understanding that the optimal, preferred method of finding the > include dir will always be numpy.get_numpy_include(), if I must > resort to a suboptimal method of guessing likely locations, what > will those locations be? > > Are they generally within the site-packages directory? Sometimes in > the python include directory? Will they be ever-changing? For now, they're in the site-packages directory, under numpy/core/ include. That position might change (it used to be numpy/base/ include, for instance), if we rename modules again. They probably won't migrate to the python include directory for the reason they're not in there now (non-root users can't install there). -- |>|\/|< /------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From arnd.baecker at web.de Sun Jan 15 23:25:00 2006 From: arnd.baecker at web.de (Arnd Baecker) Date: Sun Jan 15 23:25:00 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: References: <2D771DF0-1A60-463B-A643-AD6E2980AB94@stanford.edu> Message-ID: On Mon, 16 Jan 2006, David M. Cooke wrote: > On Jan 15, 2006, at 14:23 , Zachary Pincus wrote: > > >>> Or is this a policy change which puts the headers in the site- > >>> packages directory? > >> > >> This is not an error. Use numpy.get_numpy_include() to retrive the > >> directory of numpy header files. See numpy.get_numpy_include.__doc__ > >> for more information. > > > > Thanks for the information. Unfortunately, the package I'm working > > on can't be built with distutils, It seems that distutils is not loved that much (E.g. Wax no longer uses distutils http://zephyrfalcon.org/weblog2/arch_e10_00870.html#e878 ) > > so it may wind up being something > > of a contortion to call numpy.get_numpy_include() from the build > > system I need to use. > > You don't say what type of build system it is. If it's autoconf or > Makefile based, you could do something like > > NUMPY_HEADERS=$(python -c 'import numpy; print numpy.get_numpy_include > ()') > > > Understanding that the optimal, preferred method of finding the > > include dir will always be numpy.get_numpy_include(), if I must > > resort to a suboptimal method of guessing likely locations, what > > will those locations be? > > > > Are they generally within the site-packages directory? Sometimes in > > the python include directory? Will they be ever-changing? > > For now, they're in the site-packages directory, under numpy/core/ > include. That position might change (it used to be numpy/base/ > include, for instance), if we rename modules again. They probably > won't migrate to the python include directory for the reason they're > not in there now (non-root users can't install there). If I do a `python setup.py --prefix=$HOME/my_numpy install`, this leads to ~/my_numpy/bin/f2py ~/my_numpy/lib/python2.3/site-packages/numpy/* Wouldn't then a ~/my_numpy/include/numpy/* be reasonable? Presumably I am overlooking something, and I know that this has been discussed in much detail before, but still ... Best, Arnd From faltet at carabos.com Mon Jan 16 03:38:06 2006 From: faltet at carabos.com (Francesc Altet) Date: Mon Jan 16 03:38:06 2006 Subject: [Numpy-discussion] Maximum available dimensions in numpy Message-ID: <1137411430.3273.5.camel@localhost.localdomain> Hi, I've downloaded a fresh SVN version of numpy (0.9.4.1914) and got a bit surprised becuase of this: >>> numpy.reshape(numpy.array((22,)), (1,)*20) array([[[[[[[[[[[[[[[[[[[[22]]]]]]]]]]]]]]]]]]]]) >>> numpy.reshape(numpy.array((22,)), (1,)*21) Traceback (most recent call last): File "", line 1, in ? File "/usr/lib/python2.3/site-packages/numpy/core/oldnumeric.py", line 164, in reshape return asarray(a).reshape(newshape) ValueError: sequence too large; must be smaller than 20 Before, I think numpy supported up to 32 dimensions. Is there any reason for this new limit? Just curious. Cheers, -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From faltet at carabos.com Mon Jan 16 04:44:12 2006 From: faltet at carabos.com (Francesc Altet) Date: Mon Jan 16 04:44:12 2006 Subject: [Numpy-discussion] Changed attributes .dtypedescr, .dtypechar, .dtypestr and .dtype In-Reply-To: <43C9C573.2020809@ieee.org> References: <43C9C573.2020809@ieee.org> Message-ID: <1137415387.10345.4.camel@localhost.localdomain> Hi Travis, El ds 14 de 01 del 2006 a les 20:45 -0700, en/na Travis Oliphant va escriure: > 1) Replacing attributes that are now gone: > > .dtypechar --> .dtype.char > .dtypestr --> .dtype.str > .dtypedescr --> .dtype > > 2) Changing old .dtype -> .dtype.type > > This is only necessary if you were using a.dtype as a *typeobject* > as in > issubclass(a.dtype, ) > > If you were using .dtype as a parameter to dtype= then that usage > will still work > great (in fact a little faster) because now .dtype returns a > "descriptor object" > > 3) The dtypedescr constructor is now called dtype. > > This change should have gone into the 0.9.2 release, but things got too > hectic with all the name changes. I will quickly release 0.9.4 with > these changes unless I hear strong disagreements within the next few days. Glad that you liked the suggestion :-). BTW, if you are going to release 0.9.4 soon, do not forget to correct this typo: --- python.nobackup/numpy/trunk/numpy/core/numerictypes.py 2006-01-16 11:07:13.000000000 +0100 +++ /usr/lib/python2.4/site-packages/numpy/core/numerictypes.py 2006-01-16 11:59:28.000000000 +0100 @@ -167,7 +167,7 @@ typeDict[tmpstr] = typeobj na_name = tmpstr elif base == 'complex': - na_num = '%s%d' % (base.capitalize(), bit/2) + na_name = '%s%d' % (base.capitalize(), bit/2) elif base == 'bool': na_name = base.capitalize() typeDict[na_name] = typeobj Thanks, -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From faltet at carabos.com Mon Jan 16 08:57:24 2006 From: faltet at carabos.com (Francesc Altet) Date: Mon Jan 16 08:57:24 2006 Subject: [Numpy-discussion] Inconsistency in dtype.arrdescr? Message-ID: <1137430595.10345.55.camel@localhost.localdomain> Hi, I find the next a bit misleading: >>> numpy.array((3,), dtype='u4').dtype.arrdescr [('', '>> numpy.zeros((3,), dtype='u4,u2').dtype.arrdescr [('f1', '0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From efiring at hawaii.edu Mon Jan 16 10:51:02 2006 From: efiring at hawaii.edu (Eric Firing) Date: Mon Jan 16 10:51:02 2006 Subject: [Numpy-discussion] bug in shape() Message-ID: <43CBEAEB.9000402@hawaii.edu> Travis et al., Jeff Whitaker pointed out to me that masked array support in matplotlib contouring does not work with numpy. The problem is that the shape function does not work with masked arrays. One easy solution is to return to the older Numeric version of shape via the attached patch against svn (edited to remove irrelevant chunks). Thanks. Eric -------------- next part -------------- A non-text attachment was scrubbed... Name: shape.patch Type: text/x-patch Size: 402 bytes Desc: not available URL: From gerard.vermeulen at grenoble.cnrs.fr Mon Jan 16 12:07:17 2006 From: gerard.vermeulen at grenoble.cnrs.fr (Gerard Vermeulen) Date: Mon Jan 16 12:07:17 2006 Subject: [Numpy-discussion] buglet in numpy-0.9.4.1915 Message-ID: <20060116210641.225a13fa.gerard.vermeulen@grenoble.cnrs.fr> PyChecker found this: /home/packer/usr/lib/python2.4/site-packages/numpy/core/numeric.py:311: No global (math) found math must be probably replaced by umath. I did not verify all of PyChecker's noise, but this one is obviously a bug. Gerard From oliphant.travis at ieee.org Mon Jan 16 12:08:01 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Jan 16 12:08:01 2006 Subject: [Numpy-discussion] Inconsistency in dtype.arrdescr? In-Reply-To: <1137430595.10345.55.camel@localhost.localdomain> References: <1137430595.10345.55.camel@localhost.localdomain> Message-ID: <43CBFCD2.1070602@ieee.org> Francesc Altet wrote: >Hi, > >I find the next a bit misleading: > > >>>>numpy.array((3,), dtype='u4').dtype.arrdescr >>>> >>>> >[('', ' >but... > > > >>>>numpy.zeros((3,), dtype='u4,u2').dtype.arrdescr >>>> >>>> >[('f1', ' >I think it would be more consistent if the former expression would give >as output: > >[('f1', ' > > The problem is these are two separate circumstances. In the first case you have no fields defined and so no field names. In the second case, you have two fields which receive default field names. -Travis From ndarray at mac.com Mon Jan 16 13:16:06 2006 From: ndarray at mac.com (Sasha) Date: Mon Jan 16 13:16:06 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: References: <2D771DF0-1A60-463B-A643-AD6E2980AB94@stanford.edu> Message-ID: > Wouldn't then a > ~/my_numpy/include/numpy/* > be reasonable? > I've asked the same question before: http://www.scipy.net/pipermail/scipy-dev/2006-January/004952.html I understand that this was done for the benefit of people who have permissions to write to site-packages, but not to the include directory. I don't see why such users cannot just provide --prefix and why would any administrator allow users without administrative privileges to write to the system site-packages directory. For what it's worth, my +1 on keeping includes in include. -- sasha From pearu at cens.ioc.ee Mon Jan 16 13:45:01 2006 From: pearu at cens.ioc.ee (pearu at cens.ioc.ee) Date: Mon Jan 16 13:45:01 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: <2D771DF0-1A60-463B-A643-AD6E2980AB94@stanford.edu> Message-ID: On Sun, 15 Jan 2006, Zachary Pincus wrote: > >> Or is this a policy change which puts the headers in the site- > >> packages > >> directory? > > > > This is not an error. Use numpy.get_numpy_include() to retrive the > > directory of numpy header files. See numpy.get_numpy_include.__doc__ > > for more information. > > Thanks for the information. Unfortunately, the package I'm working on > can't be built with distutils, so it may wind up being something of a > contortion to call numpy.get_numpy_include() from the build system I > need to use. > > Understanding that the optimal, preferred method of finding the > include dir will always be numpy.get_numpy_include(), if I must > resort to a suboptimal method of guessing likely locations, what will > those locations be? > > Are they generally within the site-packages directory? Sometimes in > the python include directory? Will they be ever-changing? If you install numpy, it's header files, e.g. numpy/arrayobject.h, always endup in numpy/core/include directory. So, using import numpy os.path.join(os.path.dirname(numpy.__file__),'core','include') should always return correct path to numpy header files. Pearu From pearu at cens.ioc.ee Mon Jan 16 13:57:01 2006 From: pearu at cens.ioc.ee (pearu at cens.ioc.ee) Date: Mon Jan 16 13:57:01 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: Message-ID: On Mon, 16 Jan 2006, Arnd Baecker wrote: > On Mon, 16 Jan 2006, David M. Cooke wrote: > > > On Jan 15, 2006, at 14:23 , Zachary Pincus wrote: > > > > >>> Or is this a policy change which puts the headers in the site- > > >>> packages directory? > > >> > > >> This is not an error. Use numpy.get_numpy_include() to retrive the > > >> directory of numpy header files. See numpy.get_numpy_include.__doc__ > > >> for more information. > > > > > > Thanks for the information. Unfortunately, the package I'm working > > > on can't be built with distutils, > > It seems that distutils is not loved that much > (E.g. Wax no longer uses distutils > http://zephyrfalcon.org/weblog2/arch_e10_00870.html#e878 ) What Wax does, works well with pure Python packages, so I guess Wax is a pure Python package. As soon as it would need extension modules, one should need to use distutils or some other extension building utility. > > > Understanding that the optimal, preferred method of finding the > > > include dir will always be numpy.get_numpy_include(), if I must > > > resort to a suboptimal method of guessing likely locations, what > > > will those locations be? > > > > > > Are they generally within the site-packages directory? Sometimes in > > > the python include directory? Will they be ever-changing? > > > > For now, they're in the site-packages directory, under numpy/core/ > > include. That position might change (it used to be numpy/base/ > > include, for instance), if we rename modules again. They probably > > won't migrate to the python include directory for the reason they're > > not in there now (non-root users can't install there). The location of numpy header files will not change in future, at least I don't see any reason for doing so.. > If I do a `python setup.py --prefix=$HOME/my_numpy install`, > this leads to > > ~/my_numpy/bin/f2py > ~/my_numpy/lib/python2.3/site-packages/numpy/* > > Wouldn't then a > ~/my_numpy/include/numpy/* > be reasonable? > > Presumably I am overlooking something, and I know that this > has been discussed in much detail before, but still ... Yes, this has been discussed before and the discussion is in scipy-dev archives somewhere.. The location of numpy header files is well-documented (we have numpy/doc/CAPI.txt and numpy.get_numpy_include()), so I don't understand what is the problem with the current situation. Pearu From oliphant.travis at ieee.org Mon Jan 16 13:57:04 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Jan 16 13:57:04 2006 Subject: [Numpy-discussion] Re: [SciPy-user] numpy.dft.real+fft problem? In-Reply-To: <43CBBD45.4050306@gmail.com> References: <43CBBD45.4050306@gmail.com> Message-ID: <43CC1686.3000401@ieee.org> Andrew Jaffe wrote: >Hi All, > >[This has already appeared on python-scientific-devel; apologies to >those of you seeing it twice...] > >There seems to be a problem with the real_fft routine; starting with a >length n array, it should give a length n/2+1 array with real numbers >in the first and last positions (for n even). However, note that the >last entry on the first row has a real part, as opposed to the >complex routine fft, which has it correct. > >The weirdness of the real part seems to imply it's due to uninitialized >memory. > > Thanks for the catch. Yes it was due to uninitalized memory. PyArray_FromDims always initializes the memory to zero (but it uses int * arguments). The new routine was using PyArray_SimpleNew (using intp* arguments) which does not initialize the memory to zero. Changing PyArray_SimpleNew to PyArray_Zeros fixed the problem. -Travis From alexander.belopolsky at gmail.com Mon Jan 16 14:14:03 2006 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Mon Jan 16 14:14:03 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: References: Message-ID: > Yes, this has been discussed before and the discussion is in scipy-dev > archives somewhere.. The location of numpy header files is well-documented > (we have numpy/doc/CAPI.txt and numpy.get_numpy_include()), so I don't > understand what is the problem with the current situation. The problem with the current situation is that the include files end up in a rather obscure place. Is there any other python extension package that installs its include files in site-packages? Can someone summarize the reasons for the change so that this problem can find its resting place in a FAQ somewhere? What problem does the new location solve that cannot be solved with the --prefix option? -- sasha From ndarray at mac.com Mon Jan 16 14:15:02 2006 From: ndarray at mac.com (Sasha) Date: Mon Jan 16 14:15:02 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: References: Message-ID: > Yes, this has been discussed before and the discussion is in scipy-dev > archives somewhere.. The location of numpy header files is well-documented > (we have numpy/doc/CAPI.txt and numpy.get_numpy_include()), so I don't > understand what is the problem with the current situation. The problem with the current situation is that the include files end up in a rather obscure place. Is there any other python extension package that installs its include files in site-packages? Can someone summarize the reasons for the change so that this problem can find its resting place in a FAQ somewhere? What problem does the new location solve that cannot be solved with the --prefix option? -- sasha From cjw at sympatico.ca Mon Jan 16 14:23:04 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Mon Jan 16 14:23:04 2006 Subject: [Numpy-discussion] Inconsistency in dtype.arrdescr? In-Reply-To: <1137430595.10345.55.camel@localhost.localdomain> References: <1137430595.10345.55.camel@localhost.localdomain> Message-ID: <43CC1C9E.40009@sympatico.ca> Francesc Altet wrote: >Hi, > >I find the next a bit misleading: > > > >>>>numpy.array((3,), dtype='u4').dtype.arrdescr >>>> >>>> >[('', ' >but... > > > >>>>numpy.zeros((3,), dtype='u4,u2').dtype.arrdescr >>>> >>>> >[('f1', ' >I think it would be more consistent if the former expression would give >as output: > >[('f1', ' >Cheers, > > > Why not respond with a simple character string, such as f1, References: <1137430595.10345.55.camel@localhost.localdomain> <43CC1C9E.40009@sympatico.ca> Message-ID: <43CC1D7A.5030404@ieee.org> Colin J. Williams wrote: > Francesc Altet wrote: > >> Hi, >> >> I find the next a bit misleading: >> >> >> >>>>> numpy.array((3,), dtype='u4').dtype.arrdescr >>>>> >>>> >> [('', '> >> but... >> >> >> >>>>> numpy.zeros((3,), dtype='u4,u2').dtype.arrdescr >>>>> >>>> >> [('f1', '> >> I think it would be more consistent if the former expression would give >> as output: >> >> [('f1', '> >> Cheers, >> >> >> > Why not respond with a simple character string, such as f1, References: <1137411430.3273.5.camel@localhost.localdomain> Message-ID: > Before, I think numpy supported up to 32 dimensions. Is there any reason > for this new limit? Just curious. It was actually 40 until recently. I don't know the answer to your question (Travis?), but I am curious why would anyone need more than say 4? Solving PDEs by finite differences method in more than 4 dimensional spaces, anyone? I know I sound like some very well known person, but really 20 "ought to be enough for everyone" (TM) :-). -- sasha From oliphant.travis at ieee.org Mon Jan 16 14:48:05 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Jan 16 14:48:05 2006 Subject: [Numpy-discussion] Maximum available dimensions in numpy In-Reply-To: <1137411430.3273.5.camel@localhost.localdomain> References: <1137411430.3273.5.camel@localhost.localdomain> Message-ID: <43CC228B.6020703@ieee.org> Francesc Altet wrote: >Hi, > >I've downloaded a fresh SVN version of numpy (0.9.4.1914) and got a bit >surprised becuase of this: > > > >>>>numpy.reshape(numpy.array((22,)), (1,)*20) >>>> >>>> >array([[[[[[[[[[[[[[[[[[[[22]]]]]]]]]]]]]]]]]]]]) > > >>>>numpy.reshape(numpy.array((22,)), (1,)*21) >>>> >>>> >Traceback (most recent call last): > File "", line 1, in ? > File "/usr/lib/python2.3/site-packages/numpy/core/oldnumeric.py", line >164, in reshape > return asarray(a).reshape(newshape) >ValueError: sequence too large; must be smaller than 20 > >Before, I think numpy supported up to 32 dimensions. Is there any reason >for this new limit? Just curious. > > No, just trying to be a bit more conservative. If there is really a need for larger dimensions, it can be changed using a simple define in arrayobject.h -Travis From pebarrett at gmail.com Mon Jan 16 14:52:02 2006 From: pebarrett at gmail.com (Paul Barrett) Date: Mon Jan 16 14:52:02 2006 Subject: [Numpy-discussion] Maximum available dimensions in numpy In-Reply-To: References: <1137411430.3273.5.camel@localhost.localdomain> Message-ID: <40e64fa20601161451w659a1c1tfbe8c8afdb453610@mail.gmail.com> On 1/16/06, Sasha wrote: > > > Before, I think numpy supported up to 32 dimensions. Is there any reason > > for this new limit? Just curious. > > It was actually 40 until recently. I don't know the answer to your > question (Travis?), but I am curious why would anyone need more than > say 4? Solving PDEs by finite differences method in more than 4 > dimensional spaces, anyone? I know I sound like some very well known > person, but really 20 "ought to be enough for everyone" (TM) :-). > How about setting the default case to 3 or 4 dimensions and then special casing the rare higher dimensional arrays, i.e. using malloc for these situations. The default dimension size could be a compile time option for those who routinely exceed the default size of 3 or 4. -- Paul -- Paul Barrett, PhD Johns Hopkins University Assoc. Research Scientist Dept of Physics and Astronomy Phone: 410-516-5190 Baltimore, MD 21218 -------------- next part -------------- An HTML attachment was scrubbed... URL: From pearu at cens.ioc.ee Mon Jan 16 15:14:00 2006 From: pearu at cens.ioc.ee (pearu at cens.ioc.ee) Date: Mon Jan 16 15:14:00 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: Message-ID: On Mon, 16 Jan 2006, Sasha wrote: > > Yes, this has been discussed before and the discussion is in scipy-dev > > archives somewhere.. The location of numpy header files is well-documented > > (we have numpy/doc/CAPI.txt and numpy.get_numpy_include()), so I don't > > understand what is the problem with the current situation. > > The problem with the current situation is that the include files end > up in a rather obscure > place. Is there any other python extension package that installs its > include files in site-packages? Can someone summarize the reasons for > the change so that this problem > can find its resting place in a FAQ somewhere? What problem does the > new location solve that cannot be solved with the --prefix option? See the threads starting at http://www.scipy.org/mailinglists/mailman?fn=scipy-dev/2005-October/003420.html http://www.scipy.org/mailinglists/mailman?fn=scipy-user/2005-October/005454.html Basically, the main argument behind the current decision is that there are systems where people don't have permissions to install header files to std python include directory but they can install to python site-packages (e.g. when installing prebuilt distributions). Also note that not all potential numpy users are able/willing/.. to use --prefix option. Why we should scare them away when we can provide defaults that work for all situations, though the defaults are not standard and require some tolerance from developers to follow numpy docs.. Pearu From wbaxter at gmail.com Mon Jan 16 15:43:01 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Mon Jan 16 15:43:01 2006 Subject: [Numpy-discussion] The Low-down on numpy/scipy? Message-ID: Just this past week I thought I'd check out NumPy/SciPy to see if I might be able to port some of my matlab code over to that, easier than I was finding porting it to C++. But I find the current situation somewhat confusing. Travis' page says that scipy_core is now the word, and numarray/numeric are dead, but I can't find any links for a scipy_core download anywhere. Where should someone new to numpy and scipy start, if they are just beginning to look into it as of today? Would I be better off just waiting a month or two for things to settle down? Thanks, Bill Baxter -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Mon Jan 16 16:14:09 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Jan 16 16:14:09 2006 Subject: [Numpy-discussion] The Low-down on numpy/scipy? In-Reply-To: References: Message-ID: <43CC36B2.5050806@ieee.org> Bill Baxter wrote: > Just this past week I thought I'd check out NumPy/SciPy to see if I > might be able to port some of my matlab code over to that, easier than > I was finding porting it to C++. But I find the current situation > somewhat confusing. Travis' page says that scipy_core is now the word, Which page is that? Please help me make it less confusing... The word is numpy. > > Where should someone new to numpy and scipy start, if they are just > beginning to look into it as of today? Go to http://numeric.scipy.org (which is the homepage of the numpy project page at sourceforge) For scipy, go to http://new.scipy.org which should be moved over to http://www.scipy.org in the near future. -Travis From ndarray at mac.com Mon Jan 16 16:20:01 2006 From: ndarray at mac.com (Sasha) Date: Mon Jan 16 16:20:01 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: References: Message-ID: On 1/16/06, pearu at cens.ioc.ee wrote: > ... > Basically, the main argument behind the current decision is that there are > systems where people don't have permissions to install header files to std > python include directory but they can install to python site-packages > (e.g. when installing prebuilt distributions). > That's the argument that I mentioned. It also seems to be not just the "main", but the *only* argument. I am still not convinced. What kind of systems do you have in mind? If you are talking about Linux distributions that prepackage python, there is no reason for a sysadimin to let users write to /usr/lib/pythonX.Y/site-packages and not to /usr/include/pythonX.Y/. It is much easier to mess up a say Red Hat system by placing wrong files into /usr/lib/pythonX.Y/site-packages than by messing with /usr/include/pythonX.Y/. A site that would like to allow users to install python packages should either recommend them to use --prefix or provide an installation in say /usr/local that has more liberal permissions to pythonX.Y directories. > Also note that not all potential numpy users are able/willing/.. to use > --prefix option. Why we should scare them away when we can provide > defaults that work for all situations, though the defaults are not > standard and require some tolerance from developers to follow numpy docs. Personally, I can tolerate any location less that 10 levels deep under install prefix and current location is only 8 :-). However, if there is a valid reason to change the location of include files, this functionality should be implemented as a (possibly default) option to distutils, not by treating includes as "data_files" rather than "headers". If the alternative location is specified as an option in site.cfg and is accompanied by a comment describing how to disable it, I am sure there will be fewer posts asking where did the headers go in the future. -- sasha From ndarray at mac.com Mon Jan 16 16:24:04 2006 From: ndarray at mac.com (Sasha) Date: Mon Jan 16 16:24:04 2006 Subject: [Numpy-discussion] The Low-down on numpy/scipy? In-Reply-To: <43CC36B2.5050806@ieee.org> References: <43CC36B2.5050806@ieee.org> Message-ID: On 1/16/06, Travis Oliphant wrote: > For scipy, go to http://new.scipy.org ... That should be http://new.scipy.org/Wiki -- sasha From wbaxter at gmail.com Mon Jan 16 19:38:01 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Mon Jan 16 19:38:01 2006 Subject: [Numpy-discussion] The Low-down on numpy/scipy? In-Reply-To: References: <43CC36B2.5050806@ieee.org> Message-ID: Ok. Thanks for trying to clear this up. Top hit on a google for numpy is here: http://numeric.scipy.org/ In the text it says that "the new scipy used to be called scipy core but not anymore". However, right below that in big bold letters it says "New Features of SciPy Core". Then the page goes on to consistently refer to it as scipy core, despite that apparently no longer being the name. So it's easy to miss that the name has changed. Also the only file download link on that page is for the sourceforge site http://sourceforge.net/project/showfiles.php?group_id=1369&package_id=175103 Which has downloads for NumPy, but also has a prominent news item announcing the release of scipy_core. The presence of the news release there led me to belive that the download for scipy_core would also be located there. So it would be helpful if the numeric.scipy.org page had a link for downloading scipy. The links to the scipy_core presentations just make it seem all the more like you should be able to download scipy_core from there. This page: http://new.scipy.org/Wiki/Download could perhaps mention that the SciPy they are offering is the package formerly known as scipy_core, to additionally eliminate confusion. There was another page I was able to get to previously, perhaps www.scipy.org, but it's not responding now. Anyway, there was one scipy page that said basically "information on these pages may be out of date, there's something new afoot" but I don't seem to recall any handy link taking me directly to the proper downloads. Or maybe I was still thinking I was looking for something called scipy_core, rather than scipy. Hope that information can help you make the web pages clearer. Thanks for your help. --Bill On 1/17/06, Sasha wrote: > > On 1/16/06, Travis Oliphant wrote: > > For scipy, go to http://new.scipy.org ... > > That should be http://new.scipy.org/Wiki > > -- sasha > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.hochberg at cox.net Mon Jan 16 20:00:03 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Mon Jan 16 20:00:03 2006 Subject: [Numpy-discussion] Maximum available dimensions in numpy In-Reply-To: <40e64fa20601161451w659a1c1tfbe8c8afdb453610@mail.gmail.com> References: <1137411430.3273.5.camel@localhost.localdomain> <40e64fa20601161451w659a1c1tfbe8c8afdb453610@mail.gmail.com> Message-ID: <43CC6B58.80302@cox.net> Paul Barrett wrote: > On 1/16/06, *Sasha* > wrote: > > > Before, I think numpy supported up to 32 dimensions. Is there > any reason > > for this new limit? Just curious. > > It was actually 40 until recently. I don't know the answer to your > question (Travis?), but I am curious why would anyone need more than > say 4? Solving PDEs by finite differences method in more than 4 > dimensional spaces, anyone? I know I sound like some very well known > person, but really 20 "ought to be enough for everyone" (TM) :-). > > > How about setting the default case to 3 or 4 dimensions and then > special casing the rare higher dimensional arrays, i.e. using malloc > for these situations. The default dimension size could be a compile > time option for those who routinely exceed the default size of 3 or 4. This seems like premature optimization. In most cases, if you're in a situation where the dimensional overhead matters (lot's of small arrays) you are using Numeric/Numarray/NumPy poorly and your code is going to be slow and bloated anyway. The way to get good efficiency with these extensions is to do block operations on large matrices. This often involves a little trickery and several extra dimensions. Reducing the default matrix size down to 3 or 4 makes efficient code slower since going through malloc will involve an extra dereference and probably some extra branches. There's also no point in setting the default max matrix size to if the typical default allignment (8 byte, IIRC) is going to leave some bytes unused due to allignment. If one were to pursue some sort of hybrid scheme as proposed above, I'd say at a minimum a default dimension size would be 6; larger depending on how the allignment works out. I am also leary of this sort of hybrid scheme since the code for dimensions larger the the code would be little tested and would thus be a good place for bugs to lurk. I don't see anything wrong with making the maximum dimension size a compile time option though. However, since in the common case the extra dimensions are unlikely to affect performance in any meaningful, I'd recomend that the maximum number of default dimensions stay large by default. Thus people who need to conserve bytes, which I'd consider the rare case, have the option of reducing the max dimensions while the arrays behave in an unsuprising manner when compiled in the normal manner. If someone has the extra time, it would be interesting to see some data about how always mallocing the extra dimensions, so there was no maximum dimensions limit, affects performance. I'd also be interested in seeing cases where the extra dimensions actually affect performance before doing to stuff complicate^H^H^H^H^H^H^H^H fix things. -tim > > -- Paul > > -- > Paul Barrett, PhD Johns Hopkins University > Assoc. Research Scientist Dept of Physics and Astronomy > Phone: 410-516-5190 Baltimore, MD 21218 From pebarrett at gmail.com Mon Jan 16 20:31:02 2006 From: pebarrett at gmail.com (Paul Barrett) Date: Mon Jan 16 20:31:02 2006 Subject: [Numpy-discussion] Maximum available dimensions in numpy In-Reply-To: <43CC6B58.80302@cox.net> References: <1137411430.3273.5.camel@localhost.localdomain> <40e64fa20601161451w659a1c1tfbe8c8afdb453610@mail.gmail.com> <43CC6B58.80302@cox.net> Message-ID: <40e64fa20601162030v67941d24t62288147507c9822@mail.gmail.com> On 1/16/06, Tim Hochberg wrote: > > Paul Barrett wrote: > > > On 1/16/06, *Sasha* > wrote: > > > > > Before, I think numpy supported up to 32 dimensions. Is there > > any reason > > > for this new limit? Just curious. > > > > It was actually 40 until recently. I don't know the answer to your > > question (Travis?), but I am curious why would anyone need more than > > say 4? Solving PDEs by finite differences method in more than 4 > > dimensional spaces, anyone? I know I sound like some very well known > > person, but really 20 "ought to be enough for everyone" (TM) :-). > > > > > > How about setting the default case to 3 or 4 dimensions and then > > special casing the rare higher dimensional arrays, i.e. using malloc > > for these situations. The default dimension size could be a compile > > time option for those who routinely exceed the default size of 3 or 4. > > This seems like premature optimization. In most cases, if you're in a > situation where the dimensional overhead matters (lot's of small arrays) > you are using Numeric/Numarray/NumPy poorly and your code is going to be > slow and bloated anyway. The way to get good efficiency with these > extensions is to do block operations on large matrices. This often > involves a little trickery and several extra dimensions. Reducing the > default matrix size down to 3 or 4 makes efficient code slower since > going through malloc will involve an extra dereference and probably some > extra branches. It also avoids the possibility of running up against the maximum number of dimensions, while conserving memory. For those users that create a multitude of small arrays, the wasted memory might become important. I only suggested 3 or 4 dimensions, because it would appear to cover 99% of the cases. I hope those users creating the other 1%, know what they are doing. There's also no point in setting the default max matrix size to if > the typical default allignment (8 byte, IIRC) is going to leave some > bytes unused due to allignment. If one were to pursue some sort of > hybrid scheme as proposed above, I'd say at a minimum a default > dimension size would be 6; larger depending on how the allignment works > out. I am also leary of this sort of hybrid scheme since the code for > dimensions larger the the code would be little tested and would thus > be a good place for bugs to lurk. Agreed. But 20 or 30 extra dimensions also seems rather a waste and ad hoc. I don't see anything wrong with making the maximum dimension size a > compile time option though. However, since in the common case the extra > dimensions are unlikely to affect performance in any meaningful, I'd > recomend that the maximum number of default dimensions stay large by > default. Thus people who need to conserve bytes, which I'd consider the > rare case, have the option of reducing the max dimensions while the > arrays behave in an unsuprising manner when compiled in the normal manner. > > If someone has the extra time, it would be interesting to see some data > about how always mallocing the extra dimensions, so there was no maximum > dimensions limit, affects performance. I'd also be interested in seeing > cases where the extra dimensions actually affect performance before > doing to stuff complicate^H^H^H^H^H^H^H^H fix things. > I'd venture to guess that mallocing the extra dimensions would give a small, though noticeable, decrease in performance. Of course, as the array becomes larger, this overhead will decrease in relation to the time it takes to perform the operation, much like the overhead seen in numarray, though not as large. As you mentioned, arrays with many dimensions are often large arrays, so the malloc overhead will probably not be too significant. -- Paul -- Paul Barrett, PhD Johns Hopkins University Assoc. Research Scientist Dept of Physics and Astronomy Phone: 410-516-5190 Baltimore, MD 21218 -------------- next part -------------- An HTML attachment was scrubbed... URL: From Fernando.Perez at colorado.edu Mon Jan 16 20:52:14 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Mon Jan 16 20:52:14 2006 Subject: [Numpy-discussion] Maximum available dimensions in numpy In-Reply-To: <40e64fa20601162030v67941d24t62288147507c9822@mail.gmail.com> References: <1137411430.3273.5.camel@localhost.localdomain> <40e64fa20601161451w659a1c1tfbe8c8afdb453610@mail.gmail.com> <43CC6B58.80302@cox.net> <40e64fa20601162030v67941d24t62288147507c9822@mail.gmail.com> Message-ID: <43CC77B4.6050503@colorado.edu> Paul Barrett wrote: >>>How about setting the default case to 3 or 4 dimensions and then >>>special casing the rare higher dimensional arrays, i.e. using malloc >>>for these situations. The default dimension size could be a compile >>>time option for those who routinely exceed the default size of 3 or 4. >> >>This seems like premature optimization. In most cases, if you're in a >>situation where the dimensional overhead matters (lot's of small arrays) >>you are using Numeric/Numarray/NumPy poorly and your code is going to be >>slow and bloated anyway. The way to get good efficiency with these >>extensions is to do block operations on large matrices. This often >>involves a little trickery and several extra dimensions. Reducing the >>default matrix size down to 3 or 4 makes efficient code slower since >>going through malloc will involve an extra dereference and probably some >>extra branches. > > > > It also avoids the possibility of running up against the maximum number of > dimensions, while conserving memory. For those users that create a multitude > of small arrays, the wasted memory might become important. I only suggested > 3 or 4 dimensions, because it would appear to cover 99% of the cases. I > hope those users creating the other 1%, know what they are doing. How about following blitz convention (since we use blitz for looping in C++ fast via weave)? Blitz sets the max at 11 (I don't know if this comes from a Spinal Tap reference, string theory, or pure chance :). I agree that 32 is perhaps excessive, but 4 is certainly too low a number, I think. Given that most regular numerical algorithms scale exponentially with the number of dimensions, it's hard to think of anything useful you can do with a 32-dimensional array. But there are plenty of algorithms that brute-force their way through problems with k^d scaling, for d=10 or so. Cheers, f From pearu at cens.ioc.ee Mon Jan 16 21:07:03 2006 From: pearu at cens.ioc.ee (pearu at cens.ioc.ee) Date: Mon Jan 16 21:07:03 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: Message-ID: On Mon, 16 Jan 2006, Sasha wrote: > On 1/16/06, pearu at cens.ioc.ee wrote: > > ... > > Basically, the main argument behind the current decision is that there are > > systems where people don't have permissions to install header files to std > > python include directory but they can install to python site-packages > > (e.g. when installing prebuilt distributions). > > > > That's the argument that I mentioned. It also seems to be not just > the "main", but the *only* argument. I am still not convinced. What > kind of systems do you have in mind? If you are talking about Linux > distributions that prepackage python, ... No, not Linux. It was more like OSX issue, the exact platform should be mentioned somewhere in the discussion that I sent. Hopefully someone whom the header location change to std location would affect, can step in and give better arguments here than me.. > > Also note that not all potential numpy users are able/willing/.. to use > > --prefix option. Why we should scare them away when we can provide > > defaults that work for all situations, though the defaults are not > > standard and require some tolerance from developers to follow numpy docs. > > Personally, I can tolerate any location less that 10 levels deep under > install prefix and current location is only 8 :-). However, if there > is a valid reason to change the location of include files, this > functionality should be implemented as a (possibly default) option to > distutils, not by treating includes as "data_files" rather than > "headers". > > If the alternative location is specified as an option in site.cfg and > is accompanied by a comment describing how to disable it, I am sure > there will be fewer posts asking where did the headers go in the > future. I'm note sure that giving more options would be better. For me the exact location of numpy header files is unimportant as long as I have get_numpy_include(), or when using numpy.distutils that would use correct numpy include_dirs automatically when needed. Installing headers as data files has worked well sofar... except those posts:) Enhancing distutils would be another topic.. but the process would be too slow for the scipy project, we need numpy.distutils features now. If someone could start PEP'ing std distutils with numpy.distutils features, that would be great. Pearu From rudolphv at gmail.com Tue Jan 17 04:25:02 2006 From: rudolphv at gmail.com (Rudolph van der Merwe) Date: Tue Jan 17 04:25:02 2006 Subject: [Numpy-discussion] Use of numpy, scipy and matplotlib (pylab) together? Message-ID: <97670e910601170424w206793c7qa7c84e957cbb4882@mail.gmail.com> What is the correct importing order if one wants to use the full capabilities of numpy+scipy+matplotlib? I have all three packages installed and all of them import correctly. For Python code that makes use of all of these, should the import order be: from numpy import * from scipy import * from pylab import * ? I assume a certain amount of re-importing will occur since both scipy and matplotlib (pylab) uses numpy. How does one ensure that the correct objects don't get overwritten? -- Rudolph van der Merwe From list at jsaul.de Tue Jan 17 04:49:01 2006 From: list at jsaul.de (Joachim Saul) Date: Tue Jan 17 04:49:01 2006 Subject: [Numpy-discussion] Probable bug Message-ID: <20060117124739.GA23620@st18.gfz-potsdam.de> Hello all, I have observed a weird behaviour with 24.2, which actually can break existing code: >>> import Numeric >>> Numeric.__version__ '23.8' >>> x=Numeric.zeros(5) >>> type(x[0]) >>> x=Numeric.zeros(5,'f') >>> type(x[0]) >>> This is what I would expect. However, using 24.2: >>> import Numeric >>> Numeric.__version__ '24.2' >>> x=Numeric.zeros(5) >>> type(x[0]) >>> x=Numeric.zeros(5, 'f') >>> type(x[0]) >>> print type(1*x[0]) Strange, isn't it? Is there any rationale behind this inconsistent behaviour or is it just a bug? Cheers, Joachim From konrad.hinsen at laposte.net Tue Jan 17 06:45:11 2006 From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net) Date: Tue Jan 17 06:45:11 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: References: Message-ID: On Jan 17, 2006, at 1:18, Sasha wrote: >> Also note that not all potential numpy users are able/willing/.. >> to use >> --prefix option. Why we should scare them away when we can provide >> defaults that work for all situations, though the defaults are not >> standard and require some tolerance from developers to follow >> numpy docs. > > Personally, I can tolerate any location less that 10 levels deep under > install prefix > and current location is only 8 :-). However, if there is a valid > reason to change the location of include files, this functionality > should be implemented as a (possibly default) option to distutils, not > by treating includes as "data_files" rather than "headers". I tend to agree, and I am a bit less tolerant about playing with header file location. The argument "just follow the NumPy docs, it's not difficult" ignores that NumPy, like many other Python library, is a software component that many people install without reading any documentation, just because some other component requires it. If NumPy header files end up in a non-standard place, the authors of all Python packages that depend on NumPy, no matter by how many levels of indirection, have to add obscure code to their installation scrips. If every popular Python library did the same, we'd soon have a big mess. Moreover, it is possible that other libraries implement the same API as NumPy, but use different header conventions (in fact, Numeric and Numarray are candidates). Then every author of a package that indirectly depends on this API must not only add NumPy-specific code to his setup.py, but also check which of the multiple implementations is actually installed. Standard locations, be it for Python code or for header files, have been created to avoid this kind of mess. The standard for Python is defined by distutils, and at the moment it's putting header files into $prefix/include/pythonX.Y. If that turns out to be a bad decision on some platform, then distutils should be modified. If individual packages start trying to outsmart distutils, we lose the advantages of a standard installation system. Konrad. From hetland at tamu.edu Tue Jan 17 06:55:14 2006 From: hetland at tamu.edu (Robert Hetland) Date: Tue Jan 17 06:55:14 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: References: Message-ID: <22F30CE0-7362-44A7-8DA1-79C6FE9E70D0@tamu.edu> On Jan 16, 2006, at 5:12 PM, wrote: > Also note that not all potential numpy users are able/willing/.. to > use > --prefix option. Why we should scare them away when we can provide > defaults that work for all situations, though the defaults are not > standard and require some tolerance from developers to follow numpy > docs.. Why not just have the install script 'try:' to put a symlink in the standard place. That way, the files will be installed in any case, but (if successful) they will also be accessible from the standard places. If the symlinks are not created, a warning could be issued, but the install would not fail. I have used this solution -- putting the simlink in by hand -- and it seems to work fine. -Rob ----- Rob Hetland, Assistant Professor Dept of Oceanography, Texas A&M University p: 979-458-0096, f: 979-845-6331 e: hetland at tamu.edu, w: http://pong.tamu.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From konrad.hinsen at laposte.net Tue Jan 17 07:08:01 2006 From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net) Date: Tue Jan 17 07:08:01 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: <22F30CE0-7362-44A7-8DA1-79C6FE9E70D0@tamu.edu> References: <22F30CE0-7362-44A7-8DA1-79C6FE9E70D0@tamu.edu> Message-ID: <96D6BEBF-8A8D-4B69-9A6C-BC2A86BDA9B0@laposte.net> On Jan 17, 2006, at 15:55, Robert Hetland wrote: > Why not just have the install script 'try:' to put a symlink in the > standard place. That way, the files will be installed in any case, > but (if successful) they will also be accessible from the standard > places. If the symlinks are not created, a warning could be > issued, but the install would not fail. What good is that symlink if no one can rely on its presence? Konrad. -- --------------------------------------------------------------------- Konrad Hinsen Laboratoire L?on Brillouin, CEA Saclay, 91191 Gif-sur-Yvette Cedex, France Tel.: +33-1 69 08 79 25 Fax: +33-1 69 08 82 61 E-Mail: khinsen at cea.fr --------------------------------------------------------------------- From hetland at tamu.edu Tue Jan 17 07:15:08 2006 From: hetland at tamu.edu (Robert Hetland) Date: Tue Jan 17 07:15:08 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: <96D6BEBF-8A8D-4B69-9A6C-BC2A86BDA9B0@laposte.net> References: <22F30CE0-7362-44A7-8DA1-79C6FE9E70D0@tamu.edu> <96D6BEBF-8A8D-4B69-9A6C-BC2A86BDA9B0@laposte.net> Message-ID: <89D76E1D-BE44-43FF-AD68-D0A8FACD795D@tamu.edu> On Jan 17, 2006, at 9:08 AM, konrad.hinsen at laposte.net wrote: > What good is that symlink if no one can rely on its presence? True. I think putting the include files in a non-standard place should be the exception, rather than the rule. The standard install should behave itself, and put things in the right place. -Rob ----- Rob Hetland, Assistant Professor Dept of Oceanography, Texas A&M University p: 979-458-0096, f: 979-845-6331 e: hetland at tamu.edu, w: http://pong.tamu.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From byrnes at bu.edu Tue Jan 17 07:55:03 2006 From: byrnes at bu.edu (John Byrnes) Date: Tue Jan 17 07:55:03 2006 Subject: [Numpy-discussion] Use of numpy, scipy and matplotlib (pylab) together? In-Reply-To: <97670e910601170424w206793c7qa7c84e957cbb4882@mail.gmail.com> References: <97670e910601170424w206793c7qa7c84e957cbb4882@mail.gmail.com> Message-ID: <20060117155407.GA10790@localhost.localdomain> On Tue, Jan 17, 2006 at 02:24:33PM +0200, Rudolph van der Merwe wrote: > What is the correct importing order if one wants to use the full > capabilities of numpy+scipy+matplotlib? I believe the correct way to import the packages is to not use from blah import * but rather to use: import numpy as np import scipy as S import pylab as pl This keeps the namespace clean and prevents function collisions. HTH, John -- Courage and perseverance have a magical talisman, before which difficulties disappear and obstacles vanish into air. -- John Quincy Adams From hetland at tamu.edu Tue Jan 17 08:07:01 2006 From: hetland at tamu.edu (Robert Hetland) Date: Tue Jan 17 08:07:01 2006 Subject: [Numpy-discussion] Use of numpy, scipy and matplotlib (pylab) together? In-Reply-To: <20060117155407.GA10790@localhost.localdomain> References: <97670e910601170424w206793c7qa7c84e957cbb4882@mail.gmail.com> <20060117155407.GA10790@localhost.localdomain> Message-ID: <6C42166F-0E8E-4A3A-B951-D12F8A82680E@tamu.edu> On Jan 17, 2006, at 9:54 AM, John Byrnes wrote: > but rather to use: > import numpy as np > import scipy as S > import pylab as pl This may indeed be better (although I am generally lazy, and just import * them). I would certainly be willing to make the effort not to be lazy if there was a standard convention. Are there standard names for importing these packages? I have seen quite a few different conventions in various peoples code. E..g, why not import scipy as sp (instead of S)? From the Numeric days, people often import NumPy (et al.) as N, matplotlib as mpl, etc. -Rob ----- Rob Hetland, Assistant Professor Dept of Oceanography, Texas A&M University p: 979-458-0096, f: 979-845-6331 e: hetland at tamu.edu, w: http://pong.tamu.edu -------------- next part -------------- An HTML attachment was scrubbed... URL: From faltet at carabos.com Tue Jan 17 08:21:01 2006 From: faltet at carabos.com (Francesc Altet) Date: Tue Jan 17 08:21:01 2006 Subject: [Numpy-discussion] Some inconsistency in numarray.records format support Message-ID: <200601171719.40974.faltet@carabos.com> Hi, When building recarrays in numpy following the numarray convention: In [2]: numpy.zeros((3,), dtype='u1') Out[2]: array([0, 0, 0], dtype=uint8) In [3]: numpy.zeros((3,), dtype='u1,f4') Out[3]: array([(0, 0.0), (0, 0.0), (0, 0.0)], dtype=(void,5)) In [4]: numpy.zeros((3,), dtype='2u1,f4') Out[4]: array([(array([0, 0], dtype=uint8), 0.0), (array([0, 0], dtype=uint8), 0.0), (array([0, 0], dtype=uint8), 0.0)], dtype=(void,6)) In [7]: numpy.zeros((3,), dtype='(2,)u1,f4') Out[7]: array([(array([0, 0], dtype=uint8), 0.0), (array([0, 0], dtype=uint8), 0.0), (array([0, 0], dtype=uint8), 0.0)], dtype=(void,6)) So far so good, but In [5]: numpy.zeros((3,), dtype='2u1') --------------------------------------------------------------------------- exceptions.TypeError Traceback (most recent call last) /home/faltet/python.nobackup/numpy/ TypeError: data type not understood Also: In [6]: numpy.zeros((3,), dtype='(2,)u1') --------------------------------------------------------------------------- exceptions.ValueError Traceback (most recent call last) /home/faltet/python.nobackup/numpy/ /usr/lib/python2.4/site-packages/numpy/core/_internal.py in _commastring(astr) 296 res = _split(astr) 297 if (len(res)) == 1: --> 298 raise ValueError, "no commas present" 299 result = [] 300 for k,item in enumerate(res): ValueError: no commas present Perhaps this is the same case as defining tables with just one column? -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From jh at oobleck.astro.cornell.edu Tue Jan 17 08:26:06 2006 From: jh at oobleck.astro.cornell.edu (Joe Harrington) Date: Tue Jan 17 08:26:06 2006 Subject: [Numpy-discussion] Maximum available dimensions in numpy In-Reply-To: <20060117145603.DABE013765@sc8-sf-spam2.sourceforge.net> (numpy-discussion-request@lists.sourceforge.net) References: <20060117145603.DABE013765@sc8-sf-spam2.sourceforge.net> Message-ID: <200601171625.k0HGPde7007309@oobleck.astro.cornell.edu> For reasons that were laid out when this topic came up a month or so ago (perhaps on another of our infinitude of email lists), I'm opposed to reducing the number of dimensions to even 10. A sizeable chunk of the community does use many dimensions. I routinely use more than 5 dimensions, and I'm pretty good at reducing the number for performance where it's practical. There's an atmospheric sciences tool called vis5d (x,y,z,t,field parameter vector), which shows just how commonly meteorologists use 5 dimensions. I don't think of the atmospheric sciences as being a particularly dimension-heavy group, but you can easily think of adding a few more dimensions that let you choose model runs with varied initial parameters. Try tracking the variables of a complex medical trial in just 5 dimensions. IDL's limit is 7 dimensions, and many don't use it as a result. Having more in our default binary installs is a selling point. This is a pretty limiting change to be suggesting, so if there's a change to be made, it should come only after implementing and benchmarking to see what the actual performance benefits are, and then polling the community. If there is a really big improvement in having 4 or 8 dimensions max, and if "malloc and unlimit everything" isn't fast, then it may be worth supporting low-dim and high-dim versions of the binary installs. But, let's wait until we have more volunteers before doing that. --jh-- From cjw at sympatico.ca Tue Jan 17 08:38:01 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Tue Jan 17 08:38:01 2006 Subject: [Numpy-discussion] Use of numpy, scipy and matplotlib (pylab) together? In-Reply-To: <20060117155407.GA10790@localhost.localdomain> References: <97670e910601170424w206793c7qa7c84e957cbb4882@mail.gmail.com> <20060117155407.GA10790@localhost.localdomain> Message-ID: <43CD1D14.4080309@sympatico.ca> John Byrnes wrote: >On Tue, Jan 17, 2006 at 02:24:33PM +0200, Rudolph van der Merwe wrote: > > >>What is the correct importing order if one wants to use the full >>capabilities of numpy+scipy+matplotlib? >> >> >I believe the correct way to import the packages is to not use > >from blah import * > >but rather to use: >import numpy as np >import scipy as S >import pylab as pl > >This keeps the namespace clean and prevents function collisions. > >HTH, >John > > +1 From ndarray at mac.com Tue Jan 17 09:10:07 2006 From: ndarray at mac.com (Sasha) Date: Tue Jan 17 09:10:07 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: <89D76E1D-BE44-43FF-AD68-D0A8FACD795D@tamu.edu> References: <22F30CE0-7362-44A7-8DA1-79C6FE9E70D0@tamu.edu> <96D6BEBF-8A8D-4B69-9A6C-BC2A86BDA9B0@laposte.net> <89D76E1D-BE44-43FF-AD68-D0A8FACD795D@tamu.edu> Message-ID: My main concern is not with the non-standard default location, but with the way distutils are tricked into placing headers into that location. Currently this is done by mislabeling the header files as "data files" in setup.py as follows: config.add_data_files(join('include','numpy','*.h')) This looks like a hack that just happens to work on platforms where headers do not require any special treatement such as end-of-line conversion or pre-compilation. This is not guaranteed to work on every python platform now and may certainly break in the future. -- sasha On 1/17/06, Robert Hetland wrote: > > > On Jan 17, 2006, at 9:08 AM, konrad.hinsen at laposte.net wrote: > > > What good is that symlink if no one can rely on its presence? > > True. > > I think putting the include files in a non-standard place should be the > exception, rather than the rule. The standard install should behave itself, > and put things in the right place. > > -Rob > > > ----- > Rob Hetland, Assistant Professor > Dept of Oceanography, Texas A&M University > p: 979-458-0096, f: 979-845-6331 > e: hetland at tamu.edu, w: http://pong.tamu.edu > From strawman at astraw.com Tue Jan 17 09:30:14 2006 From: strawman at astraw.com (Andrew Straw) Date: Tue Jan 17 09:30:14 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: References: <22F30CE0-7362-44A7-8DA1-79C6FE9E70D0@tamu.edu> <96D6BEBF-8A8D-4B69-9A6C-BC2A86BDA9B0@laposte.net> <89D76E1D-BE44-43FF-AD68-D0A8FACD795D@tamu.edu> Message-ID: <43CD296C.7000700@astraw.com> The main advantage of the current situation for me is that it's easy to use with .eggs (for the reasons given). The symlink idea won't break this, but the "let's make it like it was" solution would break it (which was why it was changed). Can anyone think of a solution that would please both the old-way-preferers and the .egg users? Also, as for as I know, the only packages that install things into /usr/include/python2.x/packagename are Numeric and numarray, so I would argue it's not "standard", (although it may have lots of history). Sasha wrote: >My main concern is not with the non-standard default location, but >with the way distutils are tricked into placing headers into that >location. Currently this is done by mislabeling the header files as >"data files" in setup.py as follows: > > config.add_data_files(join('include','numpy','*.h')) > >This looks like a hack that just happens to work on platforms where >headers do not require any special treatement such as end-of-line >conversion or pre-compilation. This is not guaranteed to work on every >python platform now and may certainly break in the future. > >-- sasha > >On 1/17/06, Robert Hetland wrote: > > >>On Jan 17, 2006, at 9:08 AM, konrad.hinsen at laposte.net wrote: >> >> >>What good is that symlink if no one can rely on its presence? >> >>True. >> >>I think putting the include files in a non-standard place should be the >>exception, rather than the rule. The standard install should behave itself, >>and put things in the right place. >> >>-Rob >> >> >>----- >>Rob Hetland, Assistant Professor >>Dept of Oceanography, Texas A&M University >>p: 979-458-0096, f: 979-845-6331 >>e: hetland at tamu.edu, w: http://pong.tamu.edu >> >> >> > > >------------------------------------------------------- >This SF.net email is sponsored by: Splunk Inc. Do you grep through log files >for problems? Stop! Download the new AJAX search engine that makes >searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! >http://sel.as-us.falkag.net/sel?cmd=k&kid3432&bid#0486&dat1642 >_______________________________________________ >Numpy-discussion mailing list >Numpy-discussion at lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > From sj at cs.umb.edu Tue Jan 17 10:24:10 2006 From: sj at cs.umb.edu (Szymon Jaroszewicz) Date: Tue Jan 17 10:24:10 2006 Subject: [Numpy-discussion] Re: Maximum available dimensions in numpy Message-ID: <1137522194.9752.18.camel@localhost> I'm using multidimensional arrays (numarray currently) for storing conditional probability distributions in Bayesian networks, in which case a number of dimensions in the range of 20 or more can happen easily. Obviously for this reason I think its not a good idea to limit the dimensionality, especially that its not clear if the gain is worth it. Also I don't think my application to be particularly unusual, see for example Python Bayesian Network toolkit http://elliotpbnt.blogspot.com/ funded by Google summer of code. -- Szymon Jaroszewicz From alexander.belopolsky at gmail.com Tue Jan 17 10:27:12 2006 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Tue Jan 17 10:27:12 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: <43CD296C.7000700@astraw.com> References: <22F30CE0-7362-44A7-8DA1-79C6FE9E70D0@tamu.edu> <96D6BEBF-8A8D-4B69-9A6C-BC2A86BDA9B0@laposte.net> <89D76E1D-BE44-43FF-AD68-D0A8FACD795D@tamu.edu> <43CD296C.7000700@astraw.com> Message-ID: On 1/17/06, Andrew Straw wrote: > ... > Also, as for as I know, the only packages that install things into > /usr/include/python2.x/packagename are Numeric and numarray, so I would > argue it's not "standard", (although it may have lots of history). I don't know what would qualify as "standard", but Numeric's header location is specifically mentioned in the distutils manual: """ If you need to include header files from some other Python extension, you can take advantage of the fact that header files are installed in a consistent way by the Distutils install_header command. For example, the Numerical Python header files are installed (on a standard Unix installation) to /usr/local/include/python1.5/Numerical. (The exact location will differ according to your platform and Python installation.) Since the Python include directory--/usr/local/include/python1.5 in this case--is always included in the search path when building Python extensions, the best approach is to write C code like #include """ (see http://docs.python.org/dist/describing-extensions.html#SECTION002320000000000000000) The same section also criticises the idea of specifying include path explicitely: """ If you must put the Numerical include directory right into your header search path, though, you can find that directory using the Distutils distutils.sysconfig module: from distutils.sysconfig import get_python_inc incdir = os.path.join(get_python_inc(plat_specific=1), 'Numerical') setup(..., Extension(..., include_dirs=[incdir]), ) Even though this is quite portable--it will work on any Python installation, regardless of platform--it's probably easier to just write your C code in the sensible way. """ -- sasha From faltet at carabos.com Tue Jan 17 10:50:04 2006 From: faltet at carabos.com (Francesc Altet) Date: Tue Jan 17 10:50:04 2006 Subject: [Numpy-discussion] Re: Maximum available dimensions in numpy In-Reply-To: <1137522194.9752.18.camel@localhost> References: <1137522194.9752.18.camel@localhost> Message-ID: <200601171949.27382.faltet@carabos.com> A Dimarts 17 Gener 2006 19:23, Szymon Jaroszewicz va escriure: > I'm using multidimensional arrays (numarray currently) for storing > conditional probability distributions in Bayesian networks, in which > case a number of dimensions in the range of 20 or more can happen > easily. Obviously for this reason I think its not a good idea to limit > the dimensionality, especially that its not clear if the gain is worth > it. Also I don't think my application to be particularly unusual, see > for example Python Bayesian Network toolkit > http://elliotpbnt.blogspot.com/ funded by Google summer of code. Provided your example and that the minimum dimension size is 2 (1 does not increase the space requeriments) for such arrays, that a boolean would be saved on each element of the array, and that 2**32 = 4 GB (an amount of memory that is getting common in modern machines) I would say that allowing 32 dimensions (at least) would sound reasonable. Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From pearu at cens.ioc.ee Tue Jan 17 11:01:11 2006 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Tue Jan 17 11:01:11 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: References: <22F30CE0-7362-44A7-8DA1-79C6FE9E70D0@tamu.edu> <96D6BEBF-8A8D-4B69-9A6C-BC2A86BDA9B0@laposte.net> <89D76E1D-BE44-43FF-AD68-D0A8FACD795D@tamu.edu> <43CD296C.7000700@astraw.com> Message-ID: On Tue, 17 Jan 2006, Alexander Belopolsky wrote: > On 1/17/06, Andrew Straw wrote: >> ... >> Also, as for as I know, the only packages that install things into >> /usr/include/python2.x/packagename are Numeric and numarray, so I would >> argue it's not "standard", (although it may have lots of history). > > I don't know what would qualify as "standard", but Numeric's header > location is specifically mentioned in the distutils manual: > """ > If you need to include header files from some other Python extension, > you can take advantage of the fact that header files are installed in > a consistent way by the Distutils install_header command. For example, > the Numerical Python header files are installed (on a standard Unix > installation) to /usr/local/include/python1.5/Numerical. (The exact > location will differ according to your platform and Python > installation.) Since the Python include > directory--/usr/local/include/python1.5 in this case--is always > included in the search path when building Python extensions, the best > approach is to write C code like > > #include > """ As I read this section, I have two thoughts: (i) it was written in Python 1.5 days when distutils was first introduced, and back then I doubt that anyone thought about emergence of setuptools kind of tools - maybe future distutils will support setuptools better so that we treat header files like above, until then we can use the current working solution; and (ii) it all sounds like a recommendation, not a must-follow-or-something-will-break. Pearu From pearu at cens.ioc.ee Tue Jan 17 11:10:03 2006 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Tue Jan 17 11:10:03 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: References: <22F30CE0-7362-44A7-8DA1-79C6FE9E70D0@tamu.edu> <96D6BEBF-8A8D-4B69-9A6C-BC2A86BDA9B0@laposte.net> <89D76E1D-BE44-43FF-AD68-D0A8FACD795D@tamu.edu> Message-ID: On Tue, 17 Jan 2006, Sasha wrote: > My main concern is not with the non-standard default location, but > with the way distutils are tricked into placing headers into that > location. Currently this is done by mislabeling the header files as > "data files" in setup.py as follows: > > config.add_data_files(join('include','numpy','*.h')) > > This looks like a hack that just happens to work on platforms where > headers do not require any special treatement such as end-of-line > conversion or pre-compilation. This is not guaranteed to work on every > python platform now and may certainly break in the future. This is new to me. Sofar I know that numpy has been succesfully built on various unix systems, on Windows, and on OSX. The end-of-lines of numpy headers have never caused any breakage afaik and numpy header files do not require pre-compilation. Could you give more specific examples of python platforms where building numpy extension modules could fail? Pearu From ndarray at mac.com Tue Jan 17 12:19:07 2006 From: ndarray at mac.com (Sasha) Date: Tue Jan 17 12:19:07 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: References: <22F30CE0-7362-44A7-8DA1-79C6FE9E70D0@tamu.edu> <96D6BEBF-8A8D-4B69-9A6C-BC2A86BDA9B0@laposte.net> <89D76E1D-BE44-43FF-AD68-D0A8FACD795D@tamu.edu> Message-ID: > This is new to me. Sofar I know that numpy has been succesfully built on > various unix systems, on Windows, and on OSX. The end-of-lines of numpy > headers have never caused any breakage afaik and numpy header files do not > require pre-compilation. Could you give more specific examples of python > platforms where building numpy extension modules could fail? > No, I cannot and as I've just checked as of python 2.4 distutils do not process headers in any way. In fact install_data and install_headers commands are almost identical (both just copy files) and that's why your code happens to work. I did not say your code does not work on all platforms, I said it is not guaranteed to work. Any change to distutils that makes install_data and install_headers commands differ will break your code. When I first installled numpy (it was still scipy_core back then) my build scripts stopped working once I've changed #includes in my source. I tried to restore the old include location by changing install_headers setting, but that had no effect. Then I've dicovered that numpy does not use install_headers command at all. Surely everything was fixed by adding "include_dirs = [numpy.get_numpy_include()]" to my setup scripts, but it took several posts to various forums to discover that option. (I started by filing a bug report on SF.) Given that distutils manual specifically discouraged setting custom include_dirs, many future users of numpy are bound to relive my experience. Also, imagine if more third party packages follow numpy lead --- will I have to append an entry for each package to include_dirs? -- sasha From oliphant.travis at ieee.org Tue Jan 17 13:11:01 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue Jan 17 13:11:01 2006 Subject: ***[Possible UCE]*** Re: [Numpy-discussion] Speed ups to matrix-vector multiplication In-Reply-To: <1137508539.18730.18.camel@localhost.localdomain> References: <1137152864.9521.30.camel@localhost.localdomain> <43C798C1.2070401@ieee.org> <1137155787.9521.43.camel@localhost.localdomain> <43C7DEAB.8080806@ee.byu.edu> <1137179742.19206.9.camel@localhost.localdomain> <43C880B2.1050002@ieee.org> <43C9E457.3020109@ieee.org> <1137454543.14286.13.camel@localhost.localdomain> <43CC5F06.3060608@ieee.org> <1137508539.18730.18.camel@localhost.localdomain> Message-ID: <43CD5D22.3030506@ieee.org> Paulo J. S. Silva wrote: >Travis, > >It is good that my code was helpful. > >I have just tested your code and it seems to be OK now. I have written a >script to test the dot function. I am sending it attached. Think of the >script as a unit test for dot. (It tests correctness of the computations >and verify the dimensional integrity check). > > Thanks for the tests. I'll make them unit tests and put them in the test suite.. >I am using the svn version of numpy now (revision 1923). I am still >seeing slow matrix times vector multiplications if the matrix is >transposed: > >What is going on? > > Hmm.. I'm not sure. What I expect to get called is the cblas_Xgemv function with Order CblasColMajor if A has been transposed... Perhaps this is a slower operation? -Travis From chanley at stsci.edu Tue Jan 17 13:25:02 2006 From: chanley at stsci.edu (Christopher Hanley) Date: Tue Jan 17 13:25:02 2006 Subject: [Numpy-discussion] numpy.fromfile method modification? Message-ID: <43CD6086.7070204@stsci.edu> Hi Travis, Would it be possible to allow numpy's fromfile method to accept a shape parameter? This would make it similar to numarray's fromfile method and be one less thing for people making the switch to worry about. Chris From ndarray at mac.com Tue Jan 17 13:34:01 2006 From: ndarray at mac.com (Sasha) Date: Tue Jan 17 13:34:01 2006 Subject: [Numpy-discussion] Changing "nomask" in numpy.ma Message-ID: I would like to propose to change the default value of the mask array of the ma objects from None to bool_(0). This will allow ma to take advantage of numpy scalars providing array interface. For example a check that now has to be written as "a.mask is not None and a.mask.any()" can become just a.mask.any(). Ma will still use a singleton value for an empty mask so that the code that relies on "a.mask is None" check can be changed to "a.mask is nomask". Any objections? -- sasha PS: Somewhat related: >>> from numpy import * >>> bool_(0) is bool_(0) False >>> bool(0) is bool(0) True Is there any reason not to intern numpy's bool_'s? -- sasha From oliphant.travis at ieee.org Tue Jan 17 13:46:30 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue Jan 17 13:46:30 2006 Subject: [Numpy-discussion] Changing "nomask" in numpy.ma In-Reply-To: References: Message-ID: <43CD6573.2030305@ieee.org> Sasha wrote: >>>>bool_(0) is bool_(0) >>>> >>>> >False > > >>>>bool(0) is bool(0) >>>> >>>> >True > >Is there any reason not to intern numpy's bool_'s? > > No, it just hasn't been done. It would be a good idea... but require some code in PyArray_Scalar and in the bool_new method.. -Travis From oliphant.travis at ieee.org Tue Jan 17 13:49:01 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue Jan 17 13:49:01 2006 Subject: [Numpy-discussion] numpy.fromfile method modification? In-Reply-To: <43CD6442.4080303@stsci.edu> References: <43CD6086.7070204@stsci.edu> <43CD6375.3050008@ieee.org> <43CD6442.4080303@stsci.edu> Message-ID: <43CD6613.3060904@ieee.org> Christopher Hanley wrote: > Travis Oliphant wrote: > >> Yes, I think it would be possible. Please keep me informed of other >> possible compatibility improvements. >> >> Does the shape parameter just control the size of the resulting >> array? Is it equivalent to a >> .reshape(newshape) at the end? >> >> -Travis >> > > The shape parameter does control the size of the resulting array (so > you wouldn't need any information from the existing count parameter). > > The resulting array is the equivalent of what you would get if you > were to use the reshape method on the array currently returned by > fromfile. I prefer to us the array.shape = (newshape) since it is an > inplace operation instead of reshape. At least this is what I am > doing in the pyfits port. > What do you think about the sizing paramter of numpy? Is that important? If so, I would suggest a wrapper function in Python that uses the current fromfile function. We could rename the builtin to _fromfile in that case.. -Travis From oliphant.travis at ieee.org Tue Jan 17 13:49:03 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue Jan 17 13:49:03 2006 Subject: [Numpy-discussion] numpy.fromfile method modification? In-Reply-To: <43CD6442.4080303@stsci.edu> References: <43CD6086.7070204@stsci.edu> <43CD6375.3050008@ieee.org> <43CD6442.4080303@stsci.edu> Message-ID: <43CD661E.6020204@ieee.org> Christopher Hanley wrote: > Travis Oliphant wrote: > >> Yes, I think it would be possible. Please keep me informed of other >> possible compatibility improvements. >> >> Does the shape parameter just control the size of the resulting >> array? Is it equivalent to a >> .reshape(newshape) at the end? >> >> -Travis >> > > The shape parameter does control the size of the resulting array (so > you wouldn't need any information from the existing count parameter). > > The resulting array is the equivalent of what you would get if you > were to use the reshape method on the array currently returned by > fromfile. I prefer to us the array.shape = (newshape) since it is an > inplace operation instead of reshape. At least this is what I am > doing in the pyfits port. > What do you think about the sizing paramter of Numarray? Is that important? If so, I would suggest a wrapper function in Python that uses the current fromfile function. We could rename the builtin to _fromfile in that case.. -Travis From ndarray at mac.com Tue Jan 17 14:27:05 2006 From: ndarray at mac.com (Sasha) Date: Tue Jan 17 14:27:05 2006 Subject: [Numpy-discussion] Interesting timing results Message-ID: > python -m timeit -s "x=bool(0)" "x or x" 10000000 loops, best of 3: 0.111 usec per loop > python -m timeit -s "from numpy import bool_, logical_or as or_; x=bool_(0)" "or_(x, x)" 100000 loops, best of 3: 3.25 usec per loop Numpy is 32x slower than python -- not very good. Interestingly, > python -m timeit -s "from numpy import bool_; x=bool_(0)" "x or x" 1000000 loops, best of 3: 0.388 usec per loop -- sasha From chanley at stsci.edu Tue Jan 17 14:51:02 2006 From: chanley at stsci.edu (Christopher Hanley) Date: Tue Jan 17 14:51:02 2006 Subject: [Numpy-discussion] numpy.fromfile method modification? In-Reply-To: <43CD661E.6020204@ieee.org> References: <43CD6086.7070204@stsci.edu> <43CD6375.3050008@ieee.org> <43CD6442.4080303@stsci.edu> <43CD661E.6020204@ieee.org> Message-ID: <43CD73E7.1020909@stsci.edu> Travis Oliphant wrote: > What do you think about the sizing paramter of Numarray? Is that > important? The sizing parameter is apparently a recently contributed patch to numarray. We currently have no dependencies on it. I do not know how widely it is used outside of STScI. Chris From oliphant.travis at ieee.org Tue Jan 17 15:18:01 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue Jan 17 15:18:01 2006 Subject: [Numpy-discussion] Re: [Fwd: Re: _byteorder question] In-Reply-To: <43CD6F38.8070001@stsci.edu> References: <43CD6F38.8070001@stsci.edu> Message-ID: <43CD7AE4.3010700@ieee.org> Christopher Hanley wrote: > Hi Travis, > > This message is completely unrelated to the fromfile discussion. I > have forwarded your previous message to Todd for clarification of our > needs. > > I have a question regarding byteorder in numpy. In pyfits, I have a > need to change the "sense" of a byteorder flag on an array without > actually performing the byteswap operation. I cannot find a mechanism > in numpy to actually do this. Does one exist? > Yes, you want to change the array data-type object (which holds byte-order information). Previously NOTSWAPPED was a flag on the array itself which could have been toggled similarly to Numarray. Now, however, byte-order information is a property of the data-type object itself. On SVN version of NumPy (where the data-type object is now a data-type descriptor --- instead of the typeobject of the array scalar), you use either a.dtype = a.dtype.newbyteorder() # swaps byteorder (you can also specify 'big', 'little', or 'native') or if you want a new view with the new byteorder then, b = a.view(a.dtype.newbyteorder()) Notice: a = arange(5) b = a.view(a.dtype.newbyteorder('swap')) print a print b print a.tostring() == b.tostring() This results in [0 1 2 3 4] [ 0 16777216 33554432 50331648 67108864] True Notice that the actual data is exactly the same... For fun, try adding 1 to the b array and notice how the a array is changed :-) -Travis From aisaac at american.edu Tue Jan 17 15:28:04 2006 From: aisaac at american.edu (Alan G Isaac) Date: Tue Jan 17 15:28:04 2006 Subject: [Numpy-discussion] Re: Maximum available dimensions in numpy In-Reply-To: <1137522194.9752.18.camel@localhost> References: <1137522194.9752.18.camel@localhost> Message-ID: On Tue, 17 Jan 2006, Szymon Jaroszewicz apparently wrote: > see for example Python Bayesian Network toolkit > http://elliotpbnt.blogspot.com/ funded by Google summer of > code. Where is the code? Cheers, Alan Isaac From oliphant.travis at ieee.org Tue Jan 17 15:32:03 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue Jan 17 15:32:03 2006 Subject: [Numpy-discussion] Interesting timing results In-Reply-To: References: Message-ID: <43CD7E3A.8070404@ieee.org> Sasha wrote: >>python -m timeit -s "x=bool(0)" "x or x" >> >> >10000000 loops, best of 3: 0.111 usec per loop > > >>python -m timeit -s "from numpy import bool_, logical_or as or_; x=bool_(0)" "or_(x, x)" >> >> >100000 loops, best of 3: 3.25 usec per loop > >Numpy is 32x slower than python -- not very good. > > This is a known fact. Currently all array scalar math goes through the entire ufunc machinery. This is clearly sub-optimal. It is the reason for the scalarmath module that I've started in the src directory. Eventually, we should be able to have scalar math as fast as Python scalars. Anybody wanting to help out. This is a good place. One issue is how to handle scalar math division-by-zero, and overflow errors. Ideally these should be handled the same way that ufuncs do. But, this will necessarily cause some slow-down to look up the local and-or module-level attribute that can alter the behavior. -Travis >Interestingly, > > > >>python -m timeit -s "from numpy import bool_; x=bool_(0)" "x or x" >> >> >1000000 loops, best of 3: 0.388 usec per loop > > Again, not too surprising given that logical_or goes through the ufunc machinery... -Travis From oliphant.travis at ieee.org Tue Jan 17 15:37:07 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue Jan 17 15:37:07 2006 Subject: [Numpy-discussion] Maximum available dimensions in numpy In-Reply-To: <43CC6B58.80302@cox.net> References: <1137411430.3273.5.camel@localhost.localdomain> <40e64fa20601161451w659a1c1tfbe8c8afdb453610@mail.gmail.com> <43CC6B58.80302@cox.net> Message-ID: <43CD7F87.1010003@ieee.org> Tim Hochberg wrote: > > I don't see anything wrong with making the maximum dimension size a > compile time option though. However, since in the common case the > extra dimensions are unlikely to affect performance in any meaningful, > I'd recomend that the maximum number of default dimensions stay large > by default. Thus people who need to conserve bytes, which I'd > consider the rare case, have the option of reducing the max dimensions > while the arrays behave in an unsuprising manner when compiled in the > normal manner. > > If someone has the extra time, it would be interesting to see some > data about how always mallocing the extra dimensions, so there was no > maximum dimensions limit, affects performance. I'd also be interested > in seeing cases where the extra dimensions actually affect performance > before doing to stuff complicate^H^H^H^H^H^H^H^H fix things. Right now, the array has dimensions and strides arrays malloced as necessary. So, as far as the array-object itself is concerned there is no upper limit. The dimension limit only comes from the fact that for ease in coding, static arrays for dimensions and/or strides are used in quite a few places in the code. If these were all converted to use dynamic memory from the heap, then the dimension limit would go away. If anybody wants to try and do this, they are certainly welcome... -Travis From stephen.walton at csun.edu Tue Jan 17 16:02:01 2006 From: stephen.walton at csun.edu (Stephen Walton) Date: Tue Jan 17 16:02:01 2006 Subject: [Numpy-discussion] tracing bdist_rpm Message-ID: <43CD8524.6080307@csun.edu> I am still trying, fitfully, to figure out why numpy and scipy setup.py do not respect an alternative fortran compiler with config_fc when bdist_rpm is used as the build command, but do if the normal build command is used. Is there any easy way to get a trace of what Python is actually being called as a result of "python setup.py config_fc --fcompiler=absoft bdist_rpm"? From Chris.Barker at noaa.gov Tue Jan 17 17:20:03 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue Jan 17 17:20:03 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: <89D76E1D-BE44-43FF-AD68-D0A8FACD795D@tamu.edu> References: <22F30CE0-7362-44A7-8DA1-79C6FE9E70D0@tamu.edu> <96D6BEBF-8A8D-4B69-9A6C-BC2A86BDA9B0@laposte.net> <89D76E1D-BE44-43FF-AD68-D0A8FACD795D@tamu.edu> Message-ID: <43CD9776.4050002@noaa.gov> > On Jan 17, 2006, at 9:08 AM, konrad.hinsen at laposte.net wrote: > >> What good is that symlink if no one can rely on its presence? That way, on OS-X (or whatever) when someone has a problem compiling an extension, all they have to do is get their sysadmin to put in the symlink, rather than re-install the whole numpy package, or have the numpy install fail in the first place. A small improvemnet, I suppose. As a Linux+OS-X user, I've run into a number of these Apple "Think Different" issues, but I don't think creating an ugly, non-standard system to accommodate the fact that Apple like to do things strangely is the right solution. Apple doesn't maintain their python distribution well, or really play well with the python community, so many of us end up installing a new Python anyway. Maybe we could even get Apple to fix this. As someone pointed out it doesn't really make any sense to let regular users write to site-packages, but not to include. +1 the standard include location. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From wbaxter at gmail.com Tue Jan 17 17:48:04 2006 From: wbaxter at gmail.com (Bill Baxter) Date: Tue Jan 17 17:48:04 2006 Subject: [Numpy-discussion] The Low-down on numpy/scipy? In-Reply-To: References: <43CC36B2.5050806@ieee.org> Message-ID: I see the numeric.scipy.org page has been updated. Looks much less confusing now. Great. One point that I think might still be slightly confusing is that scipy_core was never intending to replace functionality in scipy at all. Is that right? I wasn't sure for a while there whether "scipy_core is now NumPy" meant that SciPy functionality had been subsumed into NumPy. --bb On 1/17/06, Bill Baxter wrote: > > Ok. Thanks for trying to clear this up. > > Top hit on a google for numpy is here: http://numeric.scipy.org/ > > In the text it says that "the new scipy used to be called scipy core but > not anymore". However, right below that in big bold letters it says "New > Features of SciPy Core". Then the page goes on to consistently refer to it > as scipy core, despite that apparently no longer being the name. So it's > easy to miss that the name has changed. > > Also the only file download link on that page is for the sourceforge site http://sourceforge.net/project/showfiles.php?group_id=1369&package_id=175103 > > Which has downloads for NumPy, but also has a prominent news item > announcing the release of scipy_core. The presence of the news release > there led me to belive that the download for scipy_core would also be > located there. > > So it would be helpful if the numeric.scipy.org page had a link for > downloading scipy. The links to the scipy_core presentations just make it > seem all the more like you should be able to download scipy_core from there. > > > This page: http://new.scipy.org/Wiki/Download could perhaps mention that > the SciPy they are offering is the package formerly known as scipy_core, to > additionally eliminate confusion. > > There was another page I was able to get to previously, perhaps > www.scipy.org, but it's not responding now. Anyway, there was one scipy > page that said basically "information on these pages may be out of date, > there's something new afoot" but I don't seem to recall any handy link > taking me directly to the proper downloads. Or maybe I was still thinking I > was looking for something called scipy_core, rather than scipy. > > Hope that information can help you make the web pages clearer. > > Thanks for your help. > > --Bill > > > On 1/17/06, Sasha < ndarray at mac.com> wrote: > > > > On 1/16/06, Travis Oliphant < oliphant.travis at ieee.org> wrote: > > > For scipy, go to http://new.scipy.org ... > > > > That should be http://new.scipy.org/Wiki > > > > -- sasha > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndarray at mac.com Tue Jan 17 17:51:04 2006 From: ndarray at mac.com (Sasha) Date: Tue Jan 17 17:51:04 2006 Subject: [Numpy-discussion] Interesting timing results In-Reply-To: <43CD7E3A.8070404@ieee.org> References: <43CD7E3A.8070404@ieee.org> Message-ID: On 1/17/06, Travis Oliphant wrote: > Anybody wanting to help out. This is a good place. One issue is how to > handle scalar math division-by-zero, and overflow errors. Ideally these > should be handled the same way that ufuncs do. But, this will > necessarily cause some slow-down to look up the local and-or > module-level attribute that can alter the behavior. > Let me try to fill in the functions that take and return bools. Since this is the only type with a finite domain, it deserves to be handled separately. Once the two values are interned, the ops can be implemented as real fast table lookups. -- sasha From oliphant.travis at ieee.org Tue Jan 17 18:23:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue Jan 17 18:23:02 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: <43CD296C.7000700@astraw.com> References: <22F30CE0-7362-44A7-8DA1-79C6FE9E70D0@tamu.edu> <96D6BEBF-8A8D-4B69-9A6C-BC2A86BDA9B0@laposte.net> <89D76E1D-BE44-43FF-AD68-D0A8FACD795D@tamu.edu> <43CD296C.7000700@astraw.com> Message-ID: <43CDA64E.8090902@ieee.org> Andrew Straw wrote: >The main advantage of the current situation for me is that it's easy to >use with .eggs (for the reasons given). The symlink idea won't break >this, but the "let's make it like it was" solution would break it (which >was why it was changed). Can anyone think of a solution that would >please both the old-way-preferers and the .egg users? > > This is the important point in this discussion. There was a reason it was changed. I remember opposing the change, originally. However, it solved a real problem and only required people to give up their arbitrary notions of where include files *should* be. I don't see this as a real issue since you can get where the include files are very easily. But more than that, I don't see how we can change the current behavior until a solution is found that will also work for the people who forged the current solution in the first place. With that, however, I think it would be helpful for one of those people who use .eggs to clarify why it solves their problem to have the include files in their current location. -Travis From alexander.belopolsky at gmail.com Tue Jan 17 18:55:01 2006 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Tue Jan 17 18:55:01 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: <43CDA64E.8090902@ieee.org> References: <22F30CE0-7362-44A7-8DA1-79C6FE9E70D0@tamu.edu> <96D6BEBF-8A8D-4B69-9A6C-BC2A86BDA9B0@laposte.net> <89D76E1D-BE44-43FF-AD68-D0A8FACD795D@tamu.edu> <43CD296C.7000700@astraw.com> <43CDA64E.8090902@ieee.org> Message-ID: On 1/17/06, Travis Oliphant wrote: > Andrew Straw wrote: > ... > This is the important point in this discussion. There was a reason it > was changed. I remember opposing the change, originally. However, it > solved a real problem and only required people to give up their > arbitrary notions of where include files *should* be. I don't see this > as a real issue since you can get where the include files are very easily. > I don't have any "arbitrary notions of where include files *should* be" and I have no opposition to the current location. The only thing I dislike is that the needs of a particular site situation were addressed by misdiscribing the package structure to distutils rather than by adding a default site.cfg file to the distribution that users with different needs can easily edit. Another concern is that if numpy will ever be considered for inclusion in the standard "batteries included" python distribution, the same issue will rise again. -- sasha From pearu at scipy.org Wed Jan 18 00:42:00 2006 From: pearu at scipy.org (Pearu Peterson) Date: Wed Jan 18 00:42:00 2006 Subject: [Numpy-discussion] tracing bdist_rpm In-Reply-To: <43CD8524.6080307@csun.edu> References: <43CD8524.6080307@csun.edu> Message-ID: On Tue, 17 Jan 2006, Stephen Walton wrote: > I am still trying, fitfully, to figure out why numpy and scipy setup.py do > not respect an alternative fortran compiler with config_fc when bdist_rpm is > used as the build command, but do if the normal build command is used. Is > there any easy way to get a trace of what Python is actually being called as > a result of "python setup.py config_fc --fcompiler=absoft bdist_rpm"? This is not scipy/numpy setup.py specific issue. bdist_rpm calls `python build` and ignores all other commands in command line. See the code in std distutils/commands/bdist_rpm.py around lines #411, def_build variable.. To resolve this issue, numpy.distutils.command.bdist_rpm could add a wrapper around _make_spec_file that fixes bdist_rpm build command. Pearu From sj at cs.umb.edu Wed Jan 18 01:26:04 2006 From: sj at cs.umb.edu (Szymon Jaroszewicz) Date: Wed Jan 18 01:26:04 2006 Subject: [Numpy-discussion] Re: Re: Maximum available dimensions in numpy Message-ID: <1137576326.4048.6.camel@localhost> It's in https://developer.berlios.de/projects/pbnt/. Cheers, Szymon > On Tue, 17 Jan 2006, Szymon Jaroszewicz apparently wrote: > > see for example Python Bayesian Network toolkit > > http://elliotpbnt.blogspot.com/ funded by Google summer of > > code. > > Where is the code? > > Cheers, > Alan Isaac > -- Szymon Jaroszewicz From pjssilva at ime.usp.br Wed Jan 18 03:45:02 2006 From: pjssilva at ime.usp.br (Paulo Jose da Silva e Silva) Date: Wed Jan 18 03:45:02 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks Message-ID: <1137584647.19613.17.camel@localhost.localdomain> Hello, Travis asked me to benchmark numpy versus matlab in some basic linear algebra operations. Here are the resuts for matrices/vectors of dimensions 5, 50 and 500: Operation x'*y x*y' A*x A*B A'*x Half 2in2 Dimension 5 Array 0.94 0.7 0.22 0.28 1.12 0.98 1.1 Matrix 7.06 1.57 0.66 0.79 1.6 3.11 4.56 Matlab 1.88 0.44 0.41 0.35 0.37 1.2 0.98 Dimension 50 Array 9.74 3.09 0.56 18.12 13.93 4.2 4.33 Matrix 81.99 3.81 1.04 19.13 14.58 6.3 7.88 Matlab 16.98 1.94 1.07 17.86 0.73 1.57 1.77 Dimension 500 Array 1.2 8.97 2.03 166.59 20.34 3.99 4.31 Matrix 17.95 9.09 2.07 166.62 20.67 4.11 4.45 Matlab 2.09 6.07 2.17 169.45 2.1 2.56 3.06 Obs: The operation Half is actually A*x using only the lower half of the matrix and vector. The operation 2in2 is A*x using only the even indexes. Of course there are many repetitions of the same operation: 100000 for dim 5 and 50 and 1000 for dim 500. The inner product is number of repetitions is multiplied by dimension (it is very fast). The software is numpy svn version 1926 Matlab 6.5.0.180913a Release 13 (Jun 18 2002) Both softwares are using the *same* BLAS and LAPACK (ATLAS for sse). As you can see, numpy array looks very competitive. The matrix class in numpy has too much overhead for small dimension though. This overhead is very small for medium size arrays. Looking at the results above (specially the small dimensions ones, for higher dimensions the main computations are being performed by the same BLAS) I believe we can say: 1) Numpy array is faster on usual operations but outerproduct (I believe the reason is that the dot function uses the regular matrix multiplication to compute outer-products, instead of using a special function. This can "easily" changes). In particular numpy was faster in matrix times vector operations, which is the most usual in numerical linear algebra. 2) Any operation that involves transpose suffers a very big penalty in numpy. Compare A'*x and A*x, it is 10 times slower. In contrast Matlab deals with transpose quite well. Travis is already aware of this and it can be probably solved. 3) When using subarrays, numpy is a slower. The difference seems acceptable. Travis, can this be improved? Best, Paulo Obs: Latter on (in a couple of days) I may present less synthetic benchmarks (a QR factorization and a Modified Cholesky). From bsouthey at gmail.com Wed Jan 18 05:57:03 2006 From: bsouthey at gmail.com (Bruce Southey) Date: Wed Jan 18 05:57:03 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: <1137584647.19613.17.camel@localhost.localdomain> References: <1137584647.19613.17.camel@localhost.localdomain> Message-ID: Hi, Thanks for doing this as it helps determine which approach to take when coding problems. Could you add the Numeric and numarray to these benchmarks? If for no other reason to show the advantage of the new numpy. I am curious in your code because you get very different results for matrix class depending on whether x or y is transposed. Do you first transpose the x or y first before the multiplication or is the multiplication done in place by just switching the indices? Also, for x'*y, is the results for Dimension 50 and Dimension 500 switched? Thanks Bruce On 1/18/06, Paulo Jose da Silva e Silva wrote: > > Hello, > > Travis asked me to benchmark numpy versus matlab in some basic linear > algebra operations. Here are the resuts for matrices/vectors of > dimensions 5, 50 and 500: > > Operation x'*y x*y' A*x A*B A'*x Half 2in2 > > Dimension 5 > Array 0.94 0.7 0.22 0.28 1.12 0.98 1.1 > Matrix 7.06 1.57 0.66 0.79 1.6 3.11 4.56 > Matlab 1.88 0.44 0.41 0.35 0.37 1.2 0.98 > > Dimension 50 > Array 9.74 3.09 0.56 18.12 13.93 4.2 4.33 > Matrix 81.99 3.81 1.04 19.13 14.58 6.3 7.88 > Matlab 16.98 1.94 1.07 17.86 0.73 1.57 1.77 > > Dimension 500 > Array 1.2 8.97 2.03 166.59 20.34 3.99 4.31 > Matrix 17.95 9.09 2.07 166.62 20.67 4.11 4.45 > Matlab 2.09 6.07 2.17 169.45 2.1 2.56 3.06 > > Obs: The operation Half is actually A*x using only the lower half of the > matrix and vector. The operation 2in2 is A*x using only the even > indexes. > > Of course there are many repetitions of the same operation: 100000 for > dim 5 and 50 and 1000 for dim 500. The inner product is number of > repetitions is multiplied by dimension (it is very fast). > > The software is > > numpy svn version 1926 > Matlab 6.5.0.180913a Release 13 (Jun 18 2002) > > Both softwares are using the *same* BLAS and LAPACK (ATLAS for sse). > > As you can see, numpy array looks very competitive. The matrix class in > numpy has too much overhead for small dimension though. This overhead is > very small for medium size arrays. Looking at the results above > (specially the small dimensions ones, for higher dimensions the main > computations are being performed by the same BLAS) I believe we can say: > > 1) Numpy array is faster on usual operations but outerproduct (I believe > the reason is that the dot function uses the regular matrix > multiplication to compute outer-products, instead of using a special > function. This can "easily" changes). In particular numpy was faster in > matrix times vector operations, which is the most usual in numerical > linear algebra. > > 2) Any operation that involves transpose suffers a very big penalty in > numpy. Compare A'*x and A*x, it is 10 times slower. In contrast Matlab > deals with transpose quite well. Travis is already aware of this and it > can be probably solved. > > 3) When using subarrays, numpy is a slower. The difference seems > acceptable. Travis, can this be improved? > > Best, > > Paulo > > Obs: Latter on (in a couple of days) I may present less synthetic > benchmarks (a QR factorization and a Modified Cholesky). > > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log > files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From svetosch at gmx.net Wed Jan 18 07:47:08 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Wed Jan 18 07:47:08 2006 Subject: [Numpy-discussion] array vs. matrix for matlab-compatible stuff Message-ID: <43CE631D.1000006@gmx.net> Hi, I've spent a couple of weeks with scipy/numpy and the old-to-new transition; now that the transition is over (?) but some confusion is remaining (on my side) I feel the need to ask a basic question about matlab compatibility in terms of matrix (linear algebra) programming. Take "eye" and "identity" for example; given that "eye" supposedly exists to facilitate transiton from matlab to numpy/scipy (correct?), I expected eye to be/return a matrix. However (as you guys know better than I do), it's not a matrix: >>> import numpy >>> type(numpy.eye(2)) >>> type(numpy.identity(2)) >>> type(numpy.mat(numpy.eye(2))) Why is that so? There doesn't seem to be any value added of eye over identity, and the downside is inconvenience of having to explicitly convert those arrays to matrices (to use them in multiplication, inverting them, avoid unexpected broadcasting behavior if I mistakenly write down a non-defined matrix product etc.). Surely I'm missing something here, but what? Thanks for your help, Sven From matthew.brett at gmail.com Wed Jan 18 08:08:03 2006 From: matthew.brett at gmail.com (Matthew Brett) Date: Wed Jan 18 08:08:03 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: <1137584647.19613.17.camel@localhost.localdomain> References: <1137584647.19613.17.camel@localhost.localdomain> Message-ID: <1e2af89e0601180738x3979fa9dyed6616c116333680@mail.gmail.com> Hi, > Travis asked me to benchmark numpy versus matlab in some basic linear > algebra operations. Here are the resuts for matrices/vectors of > dimensions 5, 50 and 500: This is really excellent, thanks. Is there any chance we can make these and other benchmarks part of the pre-release testing? Apart from testing for bottlenecks, if we could show that we were in the ballpark of matlab for speed for each release, this would be very helpful for those us trying to persuade our matlab colleagues to switch. Best, Matthew From aghast at thirdage.com Wed Jan 18 08:37:03 2006 From: aghast at thirdage.com (=?windows-1251?B?NiAtIDExINTFwtDAy98gMjAwNiDj7uTg?=) Date: Wed Jan 18 08:37:03 2006 Subject: [Numpy-discussion] =?windows-1251?B?z9DAytLI18XRysjJIMrT0NEgxMvfINTIzcDN0c7CzsPOIMTI0MXK0s7QwA==?= Message-ID: <926601c61c4c$21d57af4$87c47dbb@thirdage.com> ? ??? ????? ?? ??????? ??????????? ?????????: ? ???? ??! ?????????? ??????????? ?? ???? ?????? ???????? ?????? ??????????? ???????????? ? ???????????? ?????????? ????????? ????????????? ?????????? ???????????? ???????? ? ??? ???? ??????????. ???????????? ???? ??? ??????????? ????????? 6 - 11 ??????? 2006 ???? ? ????? ?????????? ???????????? ???????? ?????, ?????????? ???????? SRC ??????? ????? ?????????? ??????????, ??????? ???????? ??????? ????? ??? ???? ?????????????, ?????????? ??? ?????????, ??????? ? ???????????. ?????? ??? ??????? ?????????? ?????, ??????? ????? ?????? ?????????? ??????????, ?? ??????? ?????????? ?????????? ? ???????????, ????????? ?? ? ????????? ?????????? ??????????????????? ????????-??????????: ? ????????? ?????: ???? ? ????? ??????????? ????????? ? ????????? ??????????? ??????????? ?????????? ????????? ???????? ?????????? ?????????? ???????? ???????? ??????? ??????????????? ????? ?????????? ? ???????? ?????????? ???????? ????????? ???? ?????????? ???????? ?????? ??????????? ? ???????? ???????? ?????? ????????-????????????? ???????????? ????????? ???????? ???????? ?????????? ?????? ? ???????? ?????? ?????????? ?????? ?????? ???? ???? ????????, ?????????? ?????????? ? ????????, ??????????? ??? ??????????? ?????????? ? ??????????????? ???????????? ????????. ?????? ????????? ?? ?????? ???????? ? ????????????? ????????, ? ?????? ????????? ?????? ??????? ? ?????? ?????????? ??????. ????? ??????? ??????? ? ????? ??? ???????? ????????? ??????????, ????????? ??? ?? ?????????: (095) 98O-65-16, 98O-65-49 -------------- next part -------------- An HTML attachment was scrubbed... URL: From pjssilva at ime.usp.br Wed Jan 18 09:04:00 2006 From: pjssilva at ime.usp.br (Paulo J. S. Silva) Date: Wed Jan 18 09:04:00 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: References: <1137584647.19613.17.camel@localhost.localdomain> Message-ID: <1137603780.19613.44.camel@localhost.localdomain> Em Qua, 2006-01-18 ?s 07:56 -0600, Bruce Southey escreveu: > Hi, > Thanks for doing this as it helps determine which approach to take > when coding problems. Could you add the Numeric and numarray to these > benchmarks? If for no other reason to show the advantage of the new > numpy. I add a new table with the requested benchmarks below. As you can see, numpy is always faster then numarray but slower than Numeric in indexing. Numeric also seems to suffer less from the transpose phenomenon. > > I am curious in your code because you get very different results for > matrix class depending on whether x or y is transposed. Do you first > transpose the x or y first before the multiplication or is the > multiplication done in place by just switching the indices? I think the transpose problem is that the code makes a copy of the array before calling blas (to make it fit the blas call). Actually if you change the code from dot(transpose(A), b) to M = transpose(A).copy() dot(M, b) the time spend in the operation doesn't change. I am also sending my Python benchmark code attached to this message. Anyone can used as you want (it requires numpy, Numeric and numarray installed). > Also, for x'*y, is the results for Dimension 50 and Dimension 500 > switched? No the inner product results have different number of calls for all dimensions. Hence you can't compare time between dimensions. Paulo --- New table --- Obs: I have fixed a error in my old code that made the outer product for numpy look bad. The way it was coded it was forcing an extra array copy before the BLAS call. Tests x.T*y x*y.T A*x A*B A.T*x half 2in2 Dimension: 5 Array 0.94 0.25 0.22 0.27 1.08 0.93 1.10 Matrix 6.93 1.56 0.64 0.75 1.64 3.08 4.48 NumArr 2.87 0.63 0.62 0.68 8.24 7.22 1 1.37 Numeri 1.15 0.33 0.29 0.36 0.68 0.61 0.72 Matlab 1.88 0.44 0.41 0.35 0.37 1.20 0.98 Dimension: 50 Array 9.64 2.03 0.57 18.74 13.75 4.09 4.29 Matrix 82.98 3.70 1.04 19.87 14.58 6.35 7.91 NumArr 29.72 2.58 0.95 18.40 12.88 8.50 12.90 Numeri 11.97 2.21 0.61 17.71 9.98 1.04 3.22 Matlab 16.98 1.94 1.07 17.86 0.73 1.57 1.77 Dimension: 500 Array 1.22 8.97 2.02 165.03 19.99 3.96 4.25 Matrix 18.13 9.20 2.01 164.90 20.29 4.06 4.35 NumArr 3.16 9.02 2.09 164.83 21.59 4.29 5.72 Numeri 1.46 8.99 2.03 165.01 19.22 3.24 4.50 Matlab 2.09 6.07 2.17 169.45 2.10 2.56 3.06 From Fernando.Perez at colorado.edu Wed Jan 18 09:05:07 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Wed Jan 18 09:05:07 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: <1e2af89e0601180738x3979fa9dyed6616c116333680@mail.gmail.com> References: <1137584647.19613.17.camel@localhost.localdomain> <1e2af89e0601180738x3979fa9dyed6616c116333680@mail.gmail.com> Message-ID: <43CE7531.2080708@colorado.edu> Matthew Brett wrote: > Hi, > > >>Travis asked me to benchmark numpy versus matlab in some basic linear >>algebra operations. Here are the resuts for matrices/vectors of >>dimensions 5, 50 and 500: > > > This is really excellent, thanks. Is there any chance we can make > these and other benchmarks part of the pre-release testing? Apart > from testing for bottlenecks, if we could show that we were in the > ballpark of matlab for speed for each release, this would be very > helpful for those us trying to persuade our matlab colleagues to > switch. +1 It might not be part of test(1), but at test(10) these tests could be automatically run, activating each line as each package is found (or not) in the system (Numeric, numarray, matlab). This way, people who have matlab on their box can even get a real-time check of how their fresh-off-svn numpy fares against matlab that day. cheers, f From pjssilva at ime.usp.br Wed Jan 18 09:08:05 2006 From: pjssilva at ime.usp.br (Paulo J. S. Silva) Date: Wed Jan 18 09:08:05 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: References: <1137584647.19613.17.camel@localhost.localdomain> Message-ID: <1137603902.19613.47.camel@localhost.localdomain> Ops... Forgot to send the benchmark code... (I know it is ugly and very synthetic, I'll improve it with time). -------------- next part -------------- A non-text attachment was scrubbed... Name: benchmark.py Type: text/x-python Size: 4317 bytes Desc: not available URL: From pjssilva at ime.usp.br Wed Jan 18 09:11:06 2006 From: pjssilva at ime.usp.br (Paulo J. S. Silva) Date: Wed Jan 18 09:11:06 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: <1e2af89e0601180738x3979fa9dyed6616c116333680@mail.gmail.com> References: <1137584647.19613.17.camel@localhost.localdomain> <1e2af89e0601180738x3979fa9dyed6616c116333680@mail.gmail.com> Message-ID: <1137604057.19613.51.camel@localhost.localdomain> Em Qua, 2006-01-18 ?s 15:38 +0000, Matthew Brett escreveu: > > This is really excellent, thanks. Is there any chance we can make > these and other benchmarks part of the pre-release testing? Apart > from testing for bottlenecks, if we could show that we were in the > ballpark of matlab for speed for each release, this would be very > helpful for those us trying to persuade our matlab colleagues to > switch. > Sure. As soon as I add some less synthetic benchmark it can be even more interesting. I'll try to make the code easily callable from anyone that has both numpy and matlab on his own machine. As soon as it is polished enough I'll make it available to this comunity. It may take some weeks though. Best, Paulo From oliphant.travis at ieee.org Wed Jan 18 09:51:05 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed Jan 18 09:51:05 2006 Subject: [Numpy-discussion] Re: [SciPy-dev] numpy/scipy some remarks In-Reply-To: <43CE3E27.5080207@gmail.com> References: <43CE3E27.5080207@gmail.com> Message-ID: <43CE7FDF.5030506@ieee.org> Andrew Jaffe wrote: >I have some related numpy/scipy questions: > > - I recently upgraded to the most recent SVN of numpy, without doing >likewise for scipy. I found that the numpy.test() calls failed in a >couple of places -- because *scipy* hadn't been updated with the latest >dtype updates! (I can't reproduce the errors since I've since updated >scipy.) I thought the whole point of the numpy/scipy split was to avoid >'implicit' calls of scipy from numpy, wasn't it? > > Not entirely. The issue was namespace clashes between packages. This issue has come up before and is localized to the numpy.dual module. If somebody would like to re-write the module to contain a register function that scipy hooks into to register new functions for the ones listed there, then that would be a better solution. But, I haven't had time to do it. -Travis From robert.kern at gmail.com Wed Jan 18 10:08:03 2006 From: robert.kern at gmail.com (Robert Kern) Date: Wed Jan 18 10:08:03 2006 Subject: [Numpy-discussion] Re: [SciPy-dev] numpy/scipy some remarks In-Reply-To: <43CE8077.8020706@colorado.edu> References: <43CE3E27.5080207@gmail.com> <43CE8077.8020706@colorado.edu> Message-ID: <43CE83D5.4060105@gmail.com> Fernando Perez wrote: > Anyway, I won't belabor this point any longer. I'd just like to hear from > others their opinion on this matter, and if a decision is made to go ahead > with the overwriting, at least I think the rationale for it should be well > justified (and be more than "it's convenient"). The fact that over the last > few weeks we've had several surprised questions on this is, to me, an > indicator that I'm not the one uncomfortable with this decision. I haven't followed this discussion in great detail, but I believe the current situation is this: 1) If you use numpy.dft and numpy.linalg directly, you will always get the numpy versions no matter what else is installed. 2) If you want to optionally use optimized scipy versions if they are available and regular numpy versions otherwise, then you use the functions exposed in numpy.dual. You do so at your own risk. 3) pkgload() exists to support the loading of subpackages. It does not reach into numpy.dft or numpy.linalg at all. It is not relevant to this issue. 4) There are some places in numpy that use numpy.dual. I think we can address all of your concerns by changing #4. -- Robert Kern robert.kern at gmail.com "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From oliphant.travis at ieee.org Wed Jan 18 10:16:03 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed Jan 18 10:16:03 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: <1137584647.19613.17.camel@localhost.localdomain> References: <1137584647.19613.17.camel@localhost.localdomain> Message-ID: <43CE85AD.1090904@ieee.org> Paulo Jose da Silva e Silva wrote: >Hello, > >Travis asked me to benchmark numpy versus matlab in some basic linear >algebra operations. Here are the resuts for matrices/vectors of >dimensions 5, 50 and 500: > > Hi Paulo, Will you run these again with the latest SVN version of numpy. I couldn't figure out why a copy was being made on transpose (because it shouldn't have been). Then, I dug deep into the PyArray_FromAny code and found bad logic in when a copy was needed that was causing an inappropriate copy. I fixed that and now wonder how things will change. Because presumably, the dotblas function should handle the situation now... -Travis From oliphant.travis at ieee.org Wed Jan 18 10:20:08 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed Jan 18 10:20:08 2006 Subject: [Numpy-discussion] Re: [SciPy-dev] numpy/scipy some remarks In-Reply-To: <43CE83D5.4060105@gmail.com> References: <43CE3E27.5080207@gmail.com> <43CE8077.8020706@colorado.edu> <43CE83D5.4060105@gmail.com> Message-ID: <43CE86C5.1050702@ieee.org> Robert Kern wrote: >Fernando Perez wrote: > > > >>Anyway, I won't belabor this point any longer. I'd just like to hear from >>others their opinion on this matter, and if a decision is made to go ahead >>with the overwriting, at least I think the rationale for it should be well >>justified (and be more than "it's convenient"). The fact that over the last >>few weeks we've had several surprised questions on this is, to me, an >>indicator that I'm not the one uncomfortable with this decision. >> >> > >I haven't followed this discussion in great detail, but I believe the current >situation is this: > >1) If you use numpy.dft and numpy.linalg directly, you will always get the numpy >versions no matter what else is installed. > >2) If you want to optionally use optimized scipy versions if they are available >and regular numpy versions otherwise, then you use the functions exposed in >numpy.dual. You do so at your own risk. > >3) pkgload() exists to support the loading of subpackages. It does not reach >into numpy.dft or numpy.linalg at all. It is not relevant to this issue. > >4) There are some places in numpy that use numpy.dual. > >I think we can address all of your concerns by changing #4. > > > This is an accurate assessment. However, I do not want to eliminate number 4 as I've mentioned before. I think there is a place for having functions that can be over-written with better versions. I agree that it could be implemented better, however, with some kind of register function instead of automatically looking in scipy... -Travis From robert.kern at gmail.com Wed Jan 18 10:32:01 2006 From: robert.kern at gmail.com (Robert Kern) Date: Wed Jan 18 10:32:01 2006 Subject: [SciPy-dev] [Numpy-discussion] Re: numpy/scipy some remarks In-Reply-To: <43CE86C5.1050702@ieee.org> References: <43CE3E27.5080207@gmail.com> <43CE8077.8020706@colorado.edu> <43CE83D5.4060105@gmail.com> <43CE86C5.1050702@ieee.org> Message-ID: <43CE8983.2050804@gmail.com> Travis Oliphant wrote: > Robert Kern wrote: >>4) There are some places in numpy that use numpy.dual. >> >>I think we can address all of your concerns by changing #4. And actually, I think we can eat our cake and have it, too, by providing a way to restrict numpy.dual to only use numpy versions. We won't provide a way to force numpy.dual to only use some_other_version. I think Fernando's examples won't be problematic, then. > This is an accurate assessment. However, I do not want to eliminate > number 4 as I've mentioned before. I think there is a place for having > functions that can be over-written with better versions. I agree that > it could be implemented better, however, with some kind of register > function instead of automatically looking in scipy... Like egg entry_points. Please, let's not reinvent this wheel again. http://peak.telecommunity.com/DevCenter/PkgResources#entry-points -- Robert Kern robert.kern at gmail.com "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From chanley at stsci.edu Wed Jan 18 10:47:06 2006 From: chanley at stsci.edu (Christopher Hanley) Date: Wed Jan 18 10:47:06 2006 Subject: [Numpy-discussion] numpy / numarray chararray incompatibility Message-ID: <43CE8CF1.7000802@stsci.edu> Hi Travis, The following works in numarray but fails in numpy: In [23]: a1 = numarray.strings.array(['abc','def','xx']) In [24]: a1 Out[24]: CharArray(['abc', 'def', 'xx']) In [25]: from numpy.core import char In [26]: a = char.array(['abc', 'def', 'xx']) --------------------------------------------------------------------------- exceptions.TypeError Traceback (most recent call last) /data/sparty1/dev/pyfits-numpy/test/ /data/sparty1/dev/site-packages/lib/python/numpy/core/defchararray.py in array(obj, itemsize, copy, unicode, fortran) 321 dtype += str(itemsize) 322 --> 323 if isinstance(obj, str) or isinstance(obj, unicode): 324 if itemsize is None: 325 itemsize = len(obj) TypeError: isinstance() arg 2 must be a class, type, or tuple of classes and types Chris From schofield at ftw.at Wed Jan 18 11:01:01 2006 From: schofield at ftw.at (Ed Schofield) Date: Wed Jan 18 11:01:01 2006 Subject: [Numpy-discussion] [SciPy-dev] importing madness In-Reply-To: <43CE83D5.4060105@gmail.com> References: <43CE3E27.5080207@gmail.com> <43CE8077.8020706@colorado.edu> <43CE83D5.4060105@gmail.com> Message-ID: <43CE9178.3050602@ftw.at> Robert Kern wrote: >Fernando Perez wrote: > > > >>Anyway, I won't belabor this point any longer. I'd just like to hear from >>others their opinion on this matter, and if a decision is made to go ahead >>with the overwriting, at least I think the rationale for it should be well >>justified (and be more than "it's convenient"). The fact that over the last >>few weeks we've had several surprised questions on this is, to me, an >>indicator that I'm not the one uncomfortable with this decision. >> >> > >I haven't followed this discussion in great detail, but I believe the current >situation is this: > >1) If you use numpy.dft and numpy.linalg directly, you will always get the numpy >versions no matter what else is installed. > >2) If you want to optionally use optimized scipy versions if they are available >and regular numpy versions otherwise, then you use the functions exposed in >numpy.dual. You do so at your own risk. > >3) pkgload() exists to support the loading of subpackages. It does not reach >into numpy.dft or numpy.linalg at all. It is not relevant to this issue. > >4) There are some places in numpy that use numpy.dual. > >I think we can address all of your concerns by changing #4. > > I've been battling for half an hour trying to import scipy.linalg. Is this one of Fernando's scary predictions coming true? I get: >>> from scipy import linalg >>> dir(linalg) ['Heigenvalues', 'Heigenvectors', 'LinAlgError', 'ScipyTest', '__builtins__', '__doc__', '__file__', '__name__', '__path__', 'cholesky', 'cholesky_decomposition', 'det', 'determinant', 'eig', 'eigenvalues', 'eigenvectors', 'eigh', 'eigvals', 'eigvalsh', 'generalized_inverse', 'inv', 'inverse', 'lapack_lite', 'linalg', 'linear_least_squares', 'lstsq', 'pinv', 'singular_value_decomposition', 'solve', 'solve_linear_equations', 'svd', 'test'] >>> linalg.__file__ '/home/schofield/Tools/lib/python2.4/site-packages/numpy/linalg/__init__.pyc' This is the linalg package from numpy, not scipy. It's missing my favourite pinv2 function. What is going on?! I've just cleaned everything out and built from the latest SVN revisions. Using pkgload('linalg') doesn't seem to help: >>> import scipy >>> scipy.pkgload('linalg') >>> linalg.__file__ Traceback (most recent call last): File "", line 1, in ? NameError: name 'linalg' is not defined >>> scipy.linalg.__file__ '/home/schofield/Tools/lib/python2.4/site-packages/numpy/linalg/__init__.pyc' The only thing that helps is calling pkgload() with no arguments: >>> import scipy >>> scipy.pkgload() Overwriting lib= from /home/schofield/Tools/lib/python2.4/site-packages/scipy/lib/__init__.pyc (was from /home/schofield/Tools/lib/python2.4/site-packages/numpy/lib/__init__.pyc) >>> scipy.linalg.__file__ '/home/schofield/Tools/lib/python2.4/site-packages/scipy/linalg/__init__.pyc' Unless I'm doing something very stupid, there seem to be multiple sources of evil here. First, numpy's linalg package is available from the scipy namespace, which seems like a recipe for Ed's madness, since he can't find his pinv2() function. Second, scipy.pkgload('linalg') silently fails to make any visible difference. This is probably a simple bug. Third, ..., there is no third. But bad things usually come in threes. -- Ed From Fernando.Perez at colorado.edu Wed Jan 18 11:03:00 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Wed Jan 18 11:03:00 2006 Subject: [SciPy-dev] [Numpy-discussion] Re: numpy/scipy some remarks In-Reply-To: <43CE86C5.1050702@ieee.org> References: <43CE3E27.5080207@gmail.com> <43CE8077.8020706@colorado.edu> <43CE83D5.4060105@gmail.com> <43CE86C5.1050702@ieee.org> Message-ID: <43CE90AA.2020104@colorado.edu> Travis Oliphant wrote: >>3) pkgload() exists to support the loading of subpackages. It does not reach >>into numpy.dft or numpy.linalg at all. It is not relevant to this issue. >> >>4) There are some places in numpy that use numpy.dual. >> >>I think we can address all of your concerns by changing #4. >> >> >> > > > This is an accurate assessment. However, I do not want to eliminate > number 4 as I've mentioned before. I think there is a place for having > functions that can be over-written with better versions. I agree that > it could be implemented better, however, with some kind of register > function instead of automatically looking in scipy... Mmh, I think I'm confused then: it seemed to me that pkgload() WAS overwriting numpy names, from the messages which the environment variable controls. Is that not true? Here's a recent thread: http://aspn.activestate.com/ASPN/Mail/Message/scipy-dev/2974044 where this was shown: In [3]: import scipy Overwriting fft= from scipy.fftpack.basic (was from numpy.dft.fftpack) Overwriting ifft= from scipy.fftpack.basic (was from numpy.dft.fftpack) I understood this as 'scipy.fftpack.basic.fft overwrote numpy.dft.fftpack.fft'. Does this then not affect the numpy namespace at all? I also would like to propose that, rather than using an environment variable, pkgload() takes a 'verbose=' keyword (or 'quiet='). I think it's much cleaner to say pkgload(quiet=1) or pkgload(verbose=0) than relying on users configuring env. variables for something like this. Cheers, f From pjssilva at ime.usp.br Wed Jan 18 11:03:02 2006 From: pjssilva at ime.usp.br (Paulo J. S. Silva) Date: Wed Jan 18 11:03:02 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: <43CE85AD.1090904@ieee.org> References: <1137584647.19613.17.camel@localhost.localdomain> <43CE85AD.1090904@ieee.org> Message-ID: <1137610937.19613.65.camel@localhost.localdomain> Em Qua, 2006-01-18 ?s 11:15 -0700, Travis Oliphant escreveu: > Will you run these again with the latest SVN version of numpy. I > couldn't figure out why a copy was being made on transpose (because it > shouldn't have been). Then, I dug deep into the PyArray_FromAny code > and found bad logic in when a copy was needed that was causing an > inappropriate copy. > > I fixed that and now wonder how things will change. Because presumably, > the dotblas function should handle the situation now... > Good work Travis :-) Tests x.T*y x*y.T A*x A*B A.T*x half 2in2 Dimension: 5 Array 0.9000 0.2400 0.2000 0.2600 0.7100 0.9400 1.1600 Matrix 4.7800 1.5700 0.6200 0.7600 1.0600 3.0400 4.6500 NumArr 3.2900 0.7400 0.6800 0.7800 8.4800 7.4200 11.6600 Numeri 1.3300 0.3900 0.3100 0.4200 0.7900 0.6800 0.7600 Matlab 1.88 0.44 0.41 0.35 0.37 1.20 0.98 Dimension: 50 Array 9.0000 2.1400 0.5500 18.9500 1.4100 4.2700 4.4500 Matrix 48.7400 3.9200 1.0100 20.2000 1.8000 6.5000 8.1900 NumArr 32.3900 2.6800 1.0000 18.9700 13.0300 8.6300 13.0700 Numeri 13.1000 2.2600 0.6500 18.2700 10.1500 1.0400 3.2600 Matlab 16.98 1.94 1.07 17.86 0.73 1.57 1.77 Dimension: 500 Array 1.1400 9.2300 2.0100 168.2700 2.1800 4.0200 4.2900 Matrix 5.0300 9.3500 2.1500 167.5300 2.1700 4.1100 4.4200 NumArr 3.4400 9.1000 2.1000 168.7100 21.8400 4.3900 5.8900 Numeri 1.5800 9.2700 2.0700 167.5600 20.0500 3.4000 4.6800 Matlab 2.09 6.07 2.17 169.45 2.10 2.56 3.06 Note the 10-fold speed-up for higher dimensions :-) It looks like that now that numpy only looses to matlab in small dimensions. Probably, the problem is the creation of the object to represent the transposed object. Probably Matlab creation of objects is very lightweight (they only have matrices objects to deal with). Probably this phenomenon explains the behavior for the indexing operations too. Paulo From eric at enthought.com Wed Jan 18 11:04:05 2006 From: eric at enthought.com (eric jones) Date: Wed Jan 18 11:04:05 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: <43CE7531.2080708@colorado.edu> References: <1137584647.19613.17.camel@localhost.localdomain> <1e2af89e0601180738x3979fa9dyed6616c116333680@mail.gmail.com> <43CE7531.2080708@colorado.edu> Message-ID: <43CE90E6.2070107@enthought.com> In scipy, we talked about having a benchmark_xyz methods that could be added to the test classes. These weren't run during unit tests (scipy.test()) but would could be run using scipy.benchmark() or something like that. I can't remember if Pearu got the machinery in place, but it seems to me it wouldn't be so hard. You would have to add guards around benchmarks that compare to 3rd party tools, obviously, so that people without them could still run the benchmark suite. Adding a regression process that checks against results from previous builds to flag potential problems when a slow down is noted would be good -- that is more work. Anyway, something flagging these "tests" as benchmarks instead of standard correctness tests seems like a good idea. eric Fernando Perez wrote: > Matthew Brett wrote: >> Hi, >> >> >>> Travis asked me to benchmark numpy versus matlab in some basic linear >>> algebra operations. Here are the resuts for matrices/vectors of >>> dimensions 5, 50 and 500: >> >> >> This is really excellent, thanks. Is there any chance we can make >> these and other benchmarks part of the pre-release testing? Apart >> from testing for bottlenecks, if we could show that we were in the >> ballpark of matlab for speed for each release, this would be very >> helpful for those us trying to persuade our matlab colleagues to >> switch. > > +1 > > It might not be part of test(1), but at test(10) these tests could be > automatically run, activating each line as each package is found (or > not) in the system (Numeric, numarray, matlab). This way, people who > have matlab on their box can even get a real-time check of how their > fresh-off-svn numpy fares against matlab that day. > > cheers, > > f > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log > files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From ndarray at mac.com Wed Jan 18 11:22:00 2006 From: ndarray at mac.com (Sasha) Date: Wed Jan 18 11:22:00 2006 Subject: [Numpy-discussion] Shouldn't bool_ derive from bool? Message-ID: >>> from numpy import * >>> isinstance(bool_(True), bool) False From ndarray at mac.com Wed Jan 18 11:29:05 2006 From: ndarray at mac.com (Sasha) Date: Wed Jan 18 11:29:05 2006 Subject: [Numpy-discussion] Re: Shouldn't bool_ derive from bool? In-Reply-To: References: Message-ID: Oops, bool cannot be subclassed in python, but bool is a subclass of int, so it makes sense to derive from int_, only which int? In python bool inherits all number methods from int and overrides only and, or, and xor. Maybe bool_ can do the same ... -- sasha On 1/18/06, Sasha wrote: > >>> from numpy import * > >>> isinstance(bool_(True), bool) > False > From strawman at astraw.com Wed Jan 18 11:36:05 2006 From: strawman at astraw.com (Andrew Straw) Date: Wed Jan 18 11:36:05 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: <1137610937.19613.65.camel@localhost.localdomain> References: <1137584647.19613.17.camel@localhost.localdomain> <43CE85AD.1090904@ieee.org> <1137610937.19613.65.camel@localhost.localdomain> Message-ID: <43CE985E.9050406@astraw.com> Paulo J. S. Silva wrote: >Em Qua, 2006-01-18 ?s 11:15 -0700, Travis Oliphant escreveu: > > > >>Will you run these again with the latest SVN version of numpy. I >>couldn't figure out why a copy was being made on transpose (because it >>shouldn't have been). Then, I dug deep into the PyArray_FromAny code >>and found bad logic in when a copy was needed that was causing an >>inappropriate copy. >> >>I fixed that and now wonder how things will change. Because presumably, >>the dotblas function should handle the situation now... >> >> >> > >Good work Travis :-) > >Tests x.T*y x*y.T A*x A*B A.T*x half 2in2 > >Dimension: 5 >Array 0.9000 0.2400 0.2000 0.2600 0.7100 0.9400 1.1600 >Matrix 4.7800 1.5700 0.6200 0.7600 1.0600 3.0400 4.6500 >NumArr 3.2900 0.7400 0.6800 0.7800 8.4800 7.4200 11.6600 >Numeri 1.3300 0.3900 0.3100 0.4200 0.7900 0.6800 0.7600 >Matlab 1.88 0.44 0.41 0.35 0.37 1.20 0.98 > >Dimension: 50 >Array 9.0000 2.1400 0.5500 18.9500 1.4100 4.2700 4.4500 >Matrix 48.7400 3.9200 1.0100 20.2000 1.8000 6.5000 8.1900 >NumArr 32.3900 2.6800 1.0000 18.9700 13.0300 8.6300 13.0700 >Numeri 13.1000 2.2600 0.6500 18.2700 10.1500 1.0400 3.2600 >Matlab 16.98 1.94 1.07 17.86 0.73 1.57 1.77 > >Dimension: 500 >Array 1.1400 9.2300 2.0100 168.2700 2.1800 4.0200 4.2900 >Matrix 5.0300 9.3500 2.1500 167.5300 2.1700 4.1100 4.4200 >NumArr 3.4400 9.1000 2.1000 168.7100 21.8400 4.3900 5.8900 >Numeri 1.5800 9.2700 2.0700 167.5600 20.0500 3.4000 4.6800 >Matlab 2.09 6.07 2.17 169.45 2.10 2.56 3.06 > >Note the 10-fold speed-up for higher dimensions :-) > >It looks like that now that numpy only looses to matlab in small >dimensions. Probably, the problem is the creation of the object to >represent the transposed object. Probably Matlab creation of objects is >very lightweight (they only have matrices objects to deal with). >Probably this phenomenon explains the behavior for the indexing >operations too. > >Paulo > > > > Here's an idea Fernando and I have briefly talked about off-list, but which perhaps bears talking about here: Is there speed to be gained by an alternative, very simple, very optimized ndarray constructor? The idea would be a special-case constructor with very limited functionality designed purely for speed. It wouldn't support (m)any of the fantastic things Travis has done, but would be useful only in specialized use cases, such as creating indices. I'm not familiar enough with what the normal constructor does to know if we could implement something, (in C, perhaps) that would do nothing but create a simple, contiguous array significantly faster than what is currently done. Or does the current constructor create a new instance about as fast as possible? I know Travis has optimized it, but it's a general purpose constructor, and I'm thinking these extra features may take some extra CPU cycles. Cheers! Andrew From oliphant at ee.byu.edu Wed Jan 18 11:49:03 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed Jan 18 11:49:03 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: <43CE985E.9050406@astraw.com> References: <1137584647.19613.17.camel@localhost.localdomain> <43CE85AD.1090904@ieee.org> <1137610937.19613.65.camel@localhost.localdomain> <43CE985E.9050406@astraw.com> Message-ID: <43CE9B65.5040803@ee.byu.edu> Andrew Straw wrote: > Here's an idea Fernando and I have briefly talked about off-list, but > which perhaps bears talking about here: Is there speed to be gained by > an alternative, very simple, very optimized ndarray constructor? The > idea would be a special-case constructor with very limited > functionality designed purely for speed. It wouldn't support (m)any of > the fantastic things Travis has done, but would be useful only in > specialized use cases, such as creating indices. The general purpose constructor is PyArray_NewFromDescr(...) I suspect this could be special cased for certain circumstances and the special-case called occasionally. Their are checks on the dimensions that could be avoided in certain circumstances (like when we are getting the dimensions from another arrayobject already...) We could also inline the __array_from_strides code... Besides that, I'm not sure what else to optimize... -Travis From oliphant at ee.byu.edu Wed Jan 18 11:56:07 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed Jan 18 11:56:07 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: <43CE985E.9050406@astraw.com> References: <1137584647.19613.17.camel@localhost.localdomain> <43CE85AD.1090904@ieee.org> <1137610937.19613.65.camel@localhost.localdomain> <43CE985E.9050406@astraw.com> Message-ID: <43CE9D1D.6090509@ee.byu.edu> >> > Here's an idea Fernando and I have briefly talked about off-list, but > which perhaps bears talking about here: Is there speed to be gained by > an alternative, very simple, very optimized ndarray constructor? The > idea would be a special-case constructor with very limited > functionality designed purely for speed. It wouldn't support (m)any of > the fantastic things Travis has done, but would be useful only in > specialized use cases, such as creating indices. > > I'm not familiar enough with what the normal constructor does to know > if we could implement something, (in C, perhaps) that would do nothing > but create a simple, contiguous array significantly faster than what > is currently done. Or does the current constructor create a new > instance about as fast as possible? I know Travis has optimized it, > but it's a general purpose constructor, and I'm thinking these extra > features may take some extra CPU cycles. I think the indexing code will be slower because it is more sophisticated than Numeric's. Basically, it has to check for fancy indexing before defaulting to the old way. I see this as more of a slow-down than array creation. It might be possible to improve it --- more eyeballs are always helpful. But, I'm not sure how at this point. -Travis From robert.kern at gmail.com Wed Jan 18 12:07:02 2006 From: robert.kern at gmail.com (Robert Kern) Date: Wed Jan 18 12:07:02 2006 Subject: [SciPy-dev] [Numpy-discussion] Re: numpy/scipy some remarks In-Reply-To: <43CE90AA.2020104@colorado.edu> References: <43CE3E27.5080207@gmail.com> <43CE8077.8020706@colorado.edu> <43CE83D5.4060105@gmail.com> <43CE86C5.1050702@ieee.org> <43CE90AA.2020104@colorado.edu> Message-ID: <43CE9FA2.3000705@gmail.com> Fernando Perez wrote: > Travis Oliphant wrote: > >>>3) pkgload() exists to support the loading of subpackages. It does not reach >>>into numpy.dft or numpy.linalg at all. It is not relevant to this issue. >>> >>>4) There are some places in numpy that use numpy.dual. >>> >>>I think we can address all of your concerns by changing #4. >> >>This is an accurate assessment. However, I do not want to eliminate >>number 4 as I've mentioned before. I think there is a place for having >>functions that can be over-written with better versions. I agree that >>it could be implemented better, however, with some kind of register >>function instead of automatically looking in scipy... > > Mmh, I think I'm confused then: So am I, now! > it seemed to me that pkgload() WAS overwriting > numpy names, from the messages which the environment variable controls. Is > that not true? Here's a recent thread: > > http://aspn.activestate.com/ASPN/Mail/Message/scipy-dev/2974044 > > where this was shown: > > In [3]: import scipy > Overwriting fft= from > scipy.fftpack.basic (was from > numpy.dft.fftpack) > Overwriting ifft= from > scipy.fftpack.basic (was from > numpy.dft.fftpack) [~]$ python Python 2.4.1 (#2, Mar 31 2005, 00:05:10) [GCC 3.3 20030304 (Apple Computer, Inc. build 1666)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import scipy scipy.>>> scipy.pkgload(verbose=2) Imports to 'scipy' namespace ---------------------------- __all__.append('io') import lib -> success Overwriting lib= from /Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/scipy-0.4.4.1307-py2.4-macosx-10.4-ppc.egg/scipy/lib/__init__.pyc (was from /Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/numpy-0.9.4.1849-py2.4-macosx-10.4-ppc.egg/numpy/lib/__init__.pyc) __all__.append('signal') __all__.append('interpolate') __all__.append('lib.lapack') import cluster -> success __all__.append('montecarlo') __all__.append('fftpack') __all__.append('sparse') __all__.append('integrate') __all__.append('optimize') __all__.append('special') import lib.blas -> success __all__.append('linalg') __all__.append('stats') >>> import numpy >>> numpy.lib >>> scipy.lib [Ignore the SVN version numbers. They are faked. I was using checkouts from last night.] > I understood this as 'scipy.fftpack.basic.fft overwrote > numpy.dft.fftpack.fft'. Does this then not affect the numpy namespace at all? If it does, then I agree with you that this should change. > I also would like to propose that, rather than using an environment variable, > pkgload() takes a 'verbose=' keyword (or 'quiet='). I think it's much cleaner > to say > > pkgload(quiet=1) or pkgload(verbose=0) > > than relying on users configuring env. variables for something like this. It does take a verbose keyword argument. -- Robert Kern robert.kern at gmail.com "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From ndarray at mac.com Wed Jan 18 12:09:07 2006 From: ndarray at mac.com (Sasha) Date: Wed Jan 18 12:09:07 2006 Subject: [Numpy-discussion] Singleton bool_ values Message-ID: As of svn version 1931, numpy bool_ values are singletons: >>> from numpy import * >>> bool_(0) is array([True,False])[1] This change makes bool_ values behavior more similar to python bools and will allow much faster implementation of scalar boolean algebra. In order to allow other modules to take advantage of this property, I would like to propose several additions to python and c interfaces. At the Python level: define True_ and False_ constants. At the C-API level: PyArrayScalar_True PyArrayScalar_False PyArrayScalar_BoolFromLong PyArrayScalar_RETURN_TRUE PyArrayScalar_RETURN_FALSE PyArrayScalar_RETURN_BOOL_FROM_LONG Any objections? -- sasha From robert.kern at gmail.com Wed Jan 18 12:09:09 2006 From: robert.kern at gmail.com (Robert Kern) Date: Wed Jan 18 12:09:09 2006 Subject: [SciPy-dev] [Numpy-discussion] importing madness In-Reply-To: <43CE9178.3050602@ftw.at> References: <43CE3E27.5080207@gmail.com> <43CE8077.8020706@colorado.edu> <43CE83D5.4060105@gmail.com> <43CE9178.3050602@ftw.at> Message-ID: <43CEA042.20108@gmail.com> Ed Schofield wrote: > Unless I'm doing something very stupid, there seem to be multiple > sources of evil here. First, numpy's linalg package is available from > the scipy namespace, which seems like a recipe for Ed's madness, since > he can't find his pinv2() function. Second, scipy.pkgload('linalg') > silently fails to make any visible difference. This is probably a > simple bug. It's certainly not intended behavior. The process by which pkgload() determines where it is being called from is messing up. pkgload lives in numpy._import_tools, but it is also exposed in the scipy namespace, too. Please file a bug report and assign it to Pearu. -- Robert Kern robert.kern at gmail.com "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From Fernando.Perez at colorado.edu Wed Jan 18 14:01:02 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Wed Jan 18 14:01:02 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: <43CE9B65.5040803@ee.byu.edu> References: <1137584647.19613.17.camel@localhost.localdomain> <43CE85AD.1090904@ieee.org> <1137610937.19613.65.camel@localhost.localdomain> <43CE985E.9050406@astraw.com> <43CE9B65.5040803@ee.byu.edu> Message-ID: <43CEBA67.8000104@colorado.edu> Travis Oliphant wrote: > Andrew Straw wrote: > > >>Here's an idea Fernando and I have briefly talked about off-list, but >>which perhaps bears talking about here: Is there speed to be gained by >>an alternative, very simple, very optimized ndarray constructor? The >>idea would be a special-case constructor with very limited >>functionality designed purely for speed. It wouldn't support (m)any of >>the fantastic things Travis has done, but would be useful only in >>specialized use cases, such as creating indices. > > > The general purpose constructor is > > PyArray_NewFromDescr(...) > > I suspect this could be special cased for certain circumstances and the > special-case called occasionally. Their are checks on the dimensions > that could be avoided in certain circumstances (like when we are getting > the dimensions from another arrayobject already...) > > We could also inline the __array_from_strides code... > > Besides that, I'm not sure what else to optimize... Just to give some context: this came to my mind inspired by Blitz++'s TinyVector and TinyMatrix objects. In Blitz, arrays have compile-time rank, but run-time size in all dimensions. Since this introduces some overhead, Blitz offers also the Tiny* classes, which are compile-time fixed _both_ in rank and in size. This allows a number of optimizations to be made on them, at the cost of some flexibility lost. Some info on these guys: http://www.oonumerics.org/blitz/manual/blitz07.html What Andrew and I discussed was the idea of writing some object which would only support the most basic operations: element-wise arithmetic, slicing, linear algebra calls on them (matrix-matrix, matrix-vector and vector operations), and little else. I'd be OK losing fancy indexing, byteswapping, memory-mapping, reshaping, and anything else which costs either: 1. initialization-time CPU cycles 2. memory footprint 3. runtime element access and arithmetic. Such objects could be very useful in many contexts. I'd even like an immutable version, so they could be used as dictionary keys without having to make a tuple out of them. This would allow algorithms which use small arrays as multidimensional indices in sparse tree structures to be used without the hoops one must jump through today, and with higher performance. I wonder if giving up reshaping would allow the indexing code to be faster, as specialized versions could be hard-coded for each rank, with only say ranks 1-4 offered for this kind of object (I know we recently had a discussion about large ranks, but this object would be geared towards pure performance, and certainly working in 32 dimensions is a 'flexibility-driven' case, where the generic objects are called for). Note that I had never mentioned this in public, because I think it may be a slight specialization that isn't needed early on, and currently the library's priority was to get off the ground. But having such objects could be very handy, and now that the C API is starting to stabilize, maybe someone can play with this as a side project. Once they prove their worth, these beasts could be folded as part of the official distribution. I am not really qualified to judge whether there are enough areas for optimization where the sacrifices indicated could really pay off, both in terms of memory and performance. Cheers, f From pearu at scipy.org Wed Jan 18 14:15:01 2006 From: pearu at scipy.org (Pearu Peterson) Date: Wed Jan 18 14:15:01 2006 Subject: [SciPy-dev] [Numpy-discussion] Re: numpy/scipy some remarks In-Reply-To: <43CE90AA.2020104@colorado.edu> References: <43CE3E27.5080207@gmail.com> <43CE8077.8020706@colorado.edu> <43CE83D5.4060105@gmail.com> <43CE86C5.1050702@ieee.org> <43CE90AA.2020104@colorado.edu> Message-ID: On Wed, 18 Jan 2006, Fernando Perez wrote: > Mmh, I think I'm confused then: it seemed to me that pkgload() WAS > overwriting numpy names, from the messages which the environment variable > controls. Is that not true? Here's a recent thread: > > http://aspn.activestate.com/ASPN/Mail/Message/scipy-dev/2974044 pkgload() is overwriting numpy names in *scipy namespace*. To be explicit, the following is going on in scipy/__init__.py when pkgload is called, pseudocode follows: from numpy import * # imports fft old_fft = fft from scipy.fftpack import fft print 'Overwriting',old_fft,'with',fft del old_fft And nothing else! So, scipy.fft, that was numpy.fft, is set to scipy.fftpack.fft. numpy.fft remains the same. ... > I understood this as 'scipy.fftpack.basic.fft overwrote > numpy.dft.fftpack.fft'. Does this then not affect the numpy namespace at > all? No! See above. > I also would like to propose that, rather than using an environment variable, > pkgload() takes a 'verbose=' keyword (or 'quiet='). I think it's much > cleaner to say > > pkgload(quiet=1) or pkgload(verbose=0) > > than relying on users configuring env. variables for something like this. pkgload has already verbose kwd argumend. See ?pkgload for more information. Pearu From Fernando.Perez at colorado.edu Wed Jan 18 14:29:11 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Wed Jan 18 14:29:11 2006 Subject: [SciPy-dev] [Numpy-discussion] Re: numpy/scipy some remarks In-Reply-To: References: <43CE3E27.5080207@gmail.com> <43CE8077.8020706@colorado.edu> <43CE83D5.4060105@gmail.com> <43CE86C5.1050702@ieee.org> <43CE90AA.2020104@colorado.edu> Message-ID: <43CEC128.30601@colorado.edu> Pearu Peterson wrote: >>I understood this as 'scipy.fftpack.basic.fft overwrote >>numpy.dft.fftpack.fft'. Does this then not affect the numpy namespace at >>all? > > > No! See above. Ah, OK. Thanks for the clarification, I misunderstood the message. >>I also would like to propose that, rather than using an environment variable, >>pkgload() takes a 'verbose=' keyword (or 'quiet='). I think it's much >>cleaner to say >> >>pkgload(quiet=1) or pkgload(verbose=0) >> >>than relying on users configuring env. variables for something like this. > > > > pkgload has already verbose kwd argumend. See ?pkgload for more > information. What is this '?foo' syntax you speak of? Certainly not python... ;-) Sorry for not checking, I just don't have new-scipy on my work machine, only at home. Cheers, f From perry at stsci.edu Wed Jan 18 14:38:01 2006 From: perry at stsci.edu (Perry Greenfield) Date: Wed Jan 18 14:38:01 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: <43CEBA67.8000104@colorado.edu> References: <1137584647.19613.17.camel@localhost.localdomain> <43CE85AD.1090904@ieee.org> <1137610937.19613.65.camel@localhost.localdomain> <43CE985E.9050406@astraw.com> <43CE9B65.5040803@ee.byu.edu> <43CEBA67.8000104@colorado.edu> Message-ID: It's not a new idea. I raised it some time ago and I don't think it was new then either. I have to believe that if you allowed only Float64 (and perhaps a complex variant) and used other restrictions then it would be much faster for small arrays. One would think it would be much easier to implement than Numeric/numarray/numpy... I've always thought that those looking for really fast small array performance would be better served by something like this. But you'd really have to fight off feature creep. ("This almost meets my needs. If it could only do xxx") Perry On Jan 18, 2006, at 5:00 PM, Fernando Perez wrote: > Travis Oliphant wrote: >> Andrew Straw wrote: >>> Here's an idea Fernando and I have briefly talked about off-list, >>> but which perhaps bears talking about here: Is there speed to be >>> gained by an alternative, very simple, very optimized ndarray >>> constructor? The idea would be a special-case constructor with very >>> limited functionality designed purely for speed. It wouldn't support >>> (m)any of the fantastic things Travis has done, but would be useful >>> only in specialized use cases, such as creating indices. >> The general purpose constructor is >> PyArray_NewFromDescr(...) >> I suspect this could be special cased for certain circumstances and >> the special-case called occasionally. Their are checks on the >> dimensions that could be avoided in certain circumstances (like when >> we are getting the dimensions from another arrayobject already...) >> We could also inline the __array_from_strides code... >> Besides that, I'm not sure what else to optimize... > > Just to give some context: this came to my mind inspired by Blitz++'s > TinyVector and TinyMatrix objects. In Blitz, arrays have compile-time > rank, but run-time size in all dimensions. Since this introduces some > overhead, Blitz offers also the Tiny* classes, which are compile-time > fixed _both_ in rank and in size. This allows a number of > optimizations to be made on them, at the cost of some flexibility > lost. Some info on these guys: > > http://www.oonumerics.org/blitz/manual/blitz07.html > > What Andrew and I discussed was the idea of writing some object which > would only support the most basic operations: element-wise arithmetic, > slicing, linear algebra calls on them (matrix-matrix, matrix-vector > and vector operations), and little else. I'd be OK losing fancy > indexing, byteswapping, memory-mapping, reshaping, and anything else > which costs either: > > 1. initialization-time CPU cycles > 2. memory footprint > 3. runtime element access and arithmetic. > > Such objects could be very useful in many contexts. I'd even like an > immutable version, so they could be used as dictionary keys without > having to make a tuple out of them. This would allow algorithms which > use small arrays as multidimensional indices in sparse tree structures > to be used without the hoops one must jump through today, and with > higher performance. > > I wonder if giving up reshaping would allow the indexing code to be > faster, as specialized versions could be hard-coded for each rank, > with only say ranks 1-4 offered for this kind of object (I know we > recently had a discussion about large ranks, but this object would be > geared towards pure performance, and certainly working in 32 > dimensions is a 'flexibility-driven' case, where the generic objects > are called for). > > Note that I had never mentioned this in public, because I think it may > be a slight specialization that isn't needed early on, and currently > the library's priority was to get off the ground. But having such > objects could be very handy, and now that the C API is starting to > stabilize, maybe someone can play with this as a side project. Once > they prove their worth, these beasts could be folded as part of the > official distribution. > > I am not really qualified to judge whether there are enough areas for > optimization where the sacrifices indicated could really pay off, both > in terms of memory and performance. > > Cheers, > > f > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log > files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://sel.as-us.falkag.net/sel? > cmd=lnk&kid=103432&bid=230486&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From cookedm at physics.mcmaster.ca Wed Jan 18 14:42:06 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Wed Jan 18 14:42:06 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: <43CEBA67.8000104@colorado.edu> (Fernando Perez's message of "Wed, 18 Jan 2006 15:00:07 -0700") References: <1137584647.19613.17.camel@localhost.localdomain> <43CE85AD.1090904@ieee.org> <1137610937.19613.65.camel@localhost.localdomain> <43CE985E.9050406@astraw.com> <43CE9B65.5040803@ee.byu.edu> <43CEBA67.8000104@colorado.edu> Message-ID: Fernando Perez writes: > Travis Oliphant wrote: >> Andrew Straw wrote: >> >>> Here's an idea Fernando and I have briefly talked about off-list, >>> but which perhaps bears talking about here: Is there speed to be >>> gained by an alternative, very simple, very optimized ndarray >>> constructor? The idea would be a special-case constructor with very >>> limited functionality designed purely for speed. It wouldn't >>> support (m)any of the fantastic things Travis has done, but would >>> be useful only in specialized use cases, such as creating indices. >> The general purpose constructor is >> PyArray_NewFromDescr(...) >> I suspect this could be special cased for certain circumstances and >> the special-case called occasionally. Their are checks on the >> dimensions that could be avoided in certain circumstances (like when >> we are getting the dimensions from another arrayobject already...) >> We could also inline the __array_from_strides code... >> Besides that, I'm not sure what else to optimize... > > Just to give some context: this came to my mind inspired by Blitz++'s > TinyVector and TinyMatrix objects. In Blitz, arrays have compile-time > rank, but run-time size in all dimensions. Since this introduces some > overhead, Blitz offers also the Tiny* classes, which are compile-time > fixed _both_ in rank and in size. This allows a number of > optimizations to be made on them, at the cost of some flexibility > lost. Some info on these guys: > > http://www.oonumerics.org/blitz/manual/blitz07.html > > What Andrew and I discussed was the idea of writing some object which > would only support the most basic operations: element-wise arithmetic, > slicing, linear algebra calls on them (matrix-matrix, matrix-vector > and vector operations), and little else. I'd be OK losing fancy > indexing, byteswapping, memory-mapping, reshaping, and anything else > which costs either: > > 1. initialization-time CPU cycles > 2. memory footprint > 3. runtime element access and arithmetic. > > Such objects could be very useful in many contexts. I'd even like an > immutable version, so they could be used as dictionary keys without > having to make a tuple out of them. This would allow algorithms which > use small arrays as multidimensional indices in sparse tree structures > to be used without the hoops one must jump through today, and with > higher performance. I've done a little bit of work along these lines. I have a module I call vector3 [*] which has 2- and 3-dimensional immutable vectors, using either ints or doubles. It's as fast as I could make it, while keeping it all written in Pyrex. I find it very convenient for anything vector-related. Konrad Hinsen has something similiar in the development version of his ScientificPython package. [*] http://arbutus.mcmaster.ca/dmc/software/vector3.html Also, I've also done some playing around with a n-dimensional vector type (restricted to doubles). My best attempts make it ~4-5x faster than numpy (and 2x faster than Numeric) for vectors of dimension 10 on simple ops like + and *, 2x faster than numpy for dimension 1000, and approaching 1x as you make the vectors larger. Indexing is about 3x faster than numpy, and 1.4x faster than Numeric. So that gives I think some idea of the maximum speed-up possible. I think the speedups mostly come from the utter lack of any polymorphism: it handles vectors of doubles only, and only as contiguous vectors (no strides). -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From Fernando.Perez at colorado.edu Wed Jan 18 14:52:08 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Wed Jan 18 14:52:08 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: References: <1137584647.19613.17.camel@localhost.localdomain> <43CE85AD.1090904@ieee.org> <1137610937.19613.65.camel@localhost.localdomain> <43CE985E.9050406@astraw.com> <43CE9B65.5040803@ee.byu.edu> <43CEBA67.8000104@colorado.edu> Message-ID: <43CEC67B.1090600@colorado.edu> David M. Cooke wrote: > I've done a little bit of work along these lines. I have a module I > call vector3 [*] which has 2- and 3-dimensional immutable vectors, > using either ints or doubles. It's as fast as I could make it, while > keeping it all written in Pyrex. I find it very convenient for > anything vector-related. Konrad Hinsen has something similiar in the > development version of his ScientificPython package. > > [*] http://arbutus.mcmaster.ca/dmc/software/vector3.html > > Also, I've also done some playing around with a n-dimensional vector > type (restricted to doubles). My best attempts make it ~4-5x faster > than numpy (and 2x faster than Numeric) for vectors of dimension 10 > on simple ops like + and *, 2x faster than numpy for dimension 1000, > and approaching 1x as you make the vectors larger. Indexing is about > 3x faster than numpy, and 1.4x faster than Numeric. So that gives I > think some idea of the maximum speed-up possible. > > I think the speedups mostly come from the utter lack of any > polymorphism: it handles vectors of doubles only, and only as > contiguous vectors (no strides). This is excellent, thanks for the pointer. I can see uses for vectors (still 1-d, no strides, etc) with more than 3 elements, and perhaps fixed-size (no reshaping, no striding) 2-d arrays (matrices), but this looks like a good starting point. Sandbox material? Cheers, f From oliphant at ee.byu.edu Wed Jan 18 14:59:03 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed Jan 18 14:59:03 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: <43CEC67B.1090600@colorado.edu> References: <1137584647.19613.17.camel@localhost.localdomain> <43CE85AD.1090904@ieee.org> <1137610937.19613.65.camel@localhost.localdomain> <43CE985E.9050406@astraw.com> <43CE9B65.5040803@ee.byu.edu> <43CEBA67.8000104@colorado.edu> <43CEC67B.1090600@colorado.edu> Message-ID: <43CEC7FC.1040408@ee.byu.edu> Fernando Perez wrote: > David M. Cooke wrote: > >> I've done a little bit of work along these lines. I have a module I >> call vector3 [*] which has 2- and 3-dimensional immutable vectors, >> using either ints or doubles. It's as fast as I could make it, while >> keeping it all written in Pyrex. I find it very convenient for >> anything vector-related. Konrad Hinsen has something similiar in the >> development version of his ScientificPython package. >> >> [*] http://arbutus.mcmaster.ca/dmc/software/vector3.html >> >> Also, I've also done some playing around with a n-dimensional vector >> type (restricted to doubles). My best attempts make it ~4-5x faster >> than numpy (and 2x faster than Numeric) for vectors of dimension 10 >> on simple ops like + and *, 2x faster than numpy for dimension 1000, >> and approaching 1x as you make the vectors larger. Indexing is about >> 3x faster than numpy, and 1.4x faster than Numeric. So that gives I >> think some idea of the maximum speed-up possible. >> >> I think the speedups mostly come from the utter lack of any >> polymorphism: it handles vectors of doubles only, and only as >> contiguous vectors (no strides). > > > This is excellent, thanks for the pointer. I can see uses for vectors > (still 1-d, no strides, etc) with more than 3 elements, and perhaps > fixed-size (no reshaping, no striding) 2-d arrays (matrices), but this > looks like a good starting point. Sandbox material? > With the array interface, these kinds of objects can play very nicely with full ndarray's as well... -Travis From Fernando.Perez at colorado.edu Wed Jan 18 15:22:10 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Wed Jan 18 15:22:10 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: References: <1137584647.19613.17.camel@localhost.localdomain> <43CE85AD.1090904@ieee.org> <1137610937.19613.65.camel@localhost.localdomain> <43CE985E.9050406@astraw.com> <43CE9B65.5040803@ee.byu.edu> <43CEBA67.8000104@colorado.edu> Message-ID: <43CECD88.6050809@colorado.edu> Perry Greenfield wrote: > It's not a new idea. I raised it some time ago and I don't think it was I wasn't claiming authorship, sorry if it sounded like that :) In fact, I remember specifically talking with you about this at scipy'03, in the context of small array performance issues for the at-the-time-nascent numarray, and I'm sure similar things have been done many times before. I've had it floating in my head since I first saw blitz, back in 2001, and blitz probably got it from... There's nothing really new under the sun ;) > new then either. I have to believe that if you allowed only Float64 > (and perhaps a complex variant) and used other restrictions then it > would be much faster for small arrays. One would think it would be much > easier to implement than Numeric/numarray/numpy... I've always thought > that those looking for really fast small array performance would be > better served by something like this. But you'd really have to fight > off feature creep. ("This almost meets my needs. If it could only do > xxx") Couldn't that last issue be well dealt with by the fact that today's numpy is fairly subclassing-friendly? (which, if I remember correctly, wasn't quite the case with at least old Numeric). Cheers, f From rowen at cesmail.net Wed Jan 18 16:02:09 2006 From: rowen at cesmail.net (Russell E. Owen) Date: Wed Jan 18 16:02:09 2006 Subject: [Numpy-discussion] Efficient way to handle None->nan? Message-ID: We're getting numeric data from a (MySQL) database. We'd like to use numarray or NumPy on the resulting data, but some values may be None. Is there a fast, efficient way to replace None with NaN? I'd hate to use a list comprehension on each data tuple before turning it into an array, but I haven't thought of anything else. numarray.array and numarray.where are both intolerant of None in the input data. -- Russell From oliphant.travis at ieee.org Wed Jan 18 16:10:11 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed Jan 18 16:10:11 2006 Subject: [Numpy-discussion] array vs. matrix for matlab-compatible stuff In-Reply-To: <43CE631D.1000006@gmx.net> References: <43CE631D.1000006@gmx.net> Message-ID: <43CE7E68.1010908@ieee.org> Sven Schreiber wrote: > Hi, > I've spent a couple of weeks with scipy/numpy and the old-to-new > transition; now that the transition is over (?) but some confusion is > remaining (on my side) I feel the need to ask a basic question about > matlab compatibility in terms of matrix (linear algebra) programming. > > Take "eye" and "identity" for example; given that "eye" supposedly > exists to facilitate transiton from matlab to numpy/scipy (correct?), > I expected eye to be/return a matrix. Historical is the only reason. Numeric always returned an array for eye not a matrix. We could return a matrix without difficulty especially if we put an eye --> identity transition in convertcode.py -Travis From ndarray at mac.com Wed Jan 18 16:11:13 2006 From: ndarray at mac.com (Sasha) Date: Wed Jan 18 16:11:13 2006 Subject: [Numpy-discussion] Efficient way to handle None->nan? In-Reply-To: References: Message-ID: >>> from numpy.core.ma import masked_values >>> from numpy import nan >>> masked_values([1.0,None,2.0],None).filled(nan).astype(float) array([ 1. , nan, 2. ]) On 1/18/06, Russell E. Owen wrote: > We're getting numeric data from a (MySQL) database. We'd like to use > numarray or NumPy on the resulting data, but some values may be None. Is > there a fast, efficient way to replace None with NaN? I'd hate to use a > list comprehension on each data tuple before turning it into an array, > but I haven't thought of anything else. > > numarray.array and numarray.where are both intolerant of None in the > input data. > > -- Russell > > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From perry at stsci.edu Wed Jan 18 17:00:01 2006 From: perry at stsci.edu (Perry Greenfield) Date: Wed Jan 18 17:00:01 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: <43CECD88.6050809@colorado.edu> References: <1137584647.19613.17.camel@localhost.localdomain> <43CE85AD.1090904@ieee.org> <1137610937.19613.65.camel@localhost.localdomain> <43CE985E.9050406@astraw.com> <43CE9B65.5040803@ee.byu.edu> <43CEBA67.8000104@colorado.edu> <43CECD88.6050809@colorado.edu> Message-ID: <863273026756969dc8715e00bf0776c7@stsci.edu> On Jan 18, 2006, at 6:21 PM, Fernando Perez wrote: > Perry Greenfield wrote: >> It's not a new idea. I raised it some time ago and I don't think it >> was > > I wasn't claiming authorship, sorry if it sounded like that :) In > fact, I remember specifically talking with you about this at scipy'03, > in the context of small array performance issues for the > at-the-time-nascent numarray, and I'm sure similar things have been > done many times before. I've had it floating in my head since I first > saw blitz, back in 2001, and blitz probably got it from... There's > nothing really new under the sun ;) > Really :-). I remember that conversation and wondered if it had something to do with that. (And I remember Paul Dubois talking to me about similar ideas). I think it is worth trying (and has been I see, though I would have expected perhaps even a greater speed improvement; somehow I think it should not take a lot of time if you don't need all the type, shape and striding flexibility). It just needs someone to do it. >> new then either. I have to believe that if you allowed only Float64 >> (and perhaps a complex variant) and used other restrictions then it >> would be much faster for small arrays. One would think it would be >> much easier to implement than Numeric/numarray/numpy... I've always >> thought that those looking for really fast small array performance >> would be better served by something like this. But you'd really have >> to fight off feature creep. ("This almost meets my needs. If it >> could only do xxx") > > Couldn't that last issue be well dealt with by the fact that today's > numpy is fairly subclassing-friendly? (which, if I remember correctly, > wasn't quite the case with at least old Numeric). Does that help? You aren't talking about the fast array subclassing numpy are you? I'm not sure what you mean here. Perry From Fernando.Perez at colorado.edu Wed Jan 18 17:09:02 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Wed Jan 18 17:09:02 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: <863273026756969dc8715e00bf0776c7@stsci.edu> References: <1137584647.19613.17.camel@localhost.localdomain> <43CE85AD.1090904@ieee.org> <1137610937.19613.65.camel@localhost.localdomain> <43CE985E.9050406@astraw.com> <43CE9B65.5040803@ee.byu.edu> <43CEBA67.8000104@colorado.edu> <43CECD88.6050809@colorado.edu> <863273026756969dc8715e00bf0776c7@stsci.edu> Message-ID: <43CEE67D.9000707@colorado.edu> Perry Greenfield wrote: > On Jan 18, 2006, at 6:21 PM, Fernando Perez wrote: > Really :-). I remember that conversation and wondered if it had > something to do with that. (And I remember Paul Dubois talking to me > about similar ideas). I think it is worth trying (and has been I see, > though I would have expected perhaps even a greater speed improvement; > somehow I think it should not take a lot of time if you don't need all > the type, shape and striding flexibility). It just needs someone to do > it. Maybe putting David's code into the sandbox would be a good starting point. >>>new then either. I have to believe that if you allowed only Float64 >>>(and perhaps a complex variant) and used other restrictions then it >>>would be much faster for small arrays. One would think it would be >>>much easier to implement than Numeric/numarray/numpy... I've always >>>thought that those looking for really fast small array performance >>>would be better served by something like this. But you'd really have >>>to fight off feature creep. ("This almost meets my needs. If it >>>could only do xxx") >> >>Couldn't that last issue be well dealt with by the fact that today's >>numpy is fairly subclassing-friendly? (which, if I remember correctly, >>wasn't quite the case with at least old Numeric). > > > Does that help? You aren't talking about the fast array subclassing > numpy are you? I'm not sure what you mean here. What I meant was that by having good subclassing functionality, it's easier to ward off requests for every feature under the sun. It's much easier to say: 'this basic object provides a very small, core set of array features where the focus is on raw speed rather than fancy features; if you need extra features, subclass it and add them yourself' when the subclassing is actually reasonably easy. Note that I haven't actually used array subclassing myself (haven't needed it), so I may be mistaken in my comments here, it's just an intuition. Cheers, f From oliphant at ee.byu.edu Wed Jan 18 17:21:09 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed Jan 18 17:21:09 2006 Subject: [Numpy-discussion] Re: Passing context in __array__ In-Reply-To: References: <43CEB431.3010405@ee.byu.edu> Message-ID: <43CEE946.1050906@ee.byu.edu> Alexander Belopolsky wrote: >On 1/18/06, Travis Oliphant wrote: > > >>This mail got placed in my junk folder so I did not see it until now. >>Sorry about the confusion... >> >> >> >To add to the confusion, I've interpreted your previous e-mail as "go >ahead" and commited the changes. Hopefuly your concerns can be >addressed and I will not have to retract the change. > > > That is actually fine. I don't mind having changes in SVN that we discuss and fix. It's what its for. As long as it builds, then great... I went ahead and altered the C-API based on your desired use and changed numpy to work with it. This means that the SVN version of NumPy requires a recompilation of extension modules.... Some of my previous comments did not understand how you had implemented the routine. Hopefully what's in SVN now works correctly (it passes your tests...) >Yes, I agree and confessed to this problem in the comments to my code. > However, I did not want to touch PyArray_FromAny because it is part >of published C API. The best solution would be to expose parts of >PyArray_FromAny in C API and use those in ufunc code. > > We are pre 1.0 and so I don't have too much problems adding an extra context argument to the PyArray_FromAny call :-) I've also simplified things a bit, too. I moved array_fromobject to PyArray_FromAny and renamed PyArray_FromAny to PyArray_CheckFromAny(...) as it has a couple more checks to ensure DEFAULT_FLAGS are set when ENSURECOPY is set and to check for the NOTSWAPPED requirement... Often flags is 0 or does not have these FLAGS (or we have a native byte-order descriptor anyway and so the NOTSWAPPED requirement is redundant) and so the simpler PyArray_FromAny can be used. Most people are still using PyArray_ContiguousFromObject and friends anyway and so the change doesn't affect them at all. -Travis From perry at stsci.edu Wed Jan 18 17:22:05 2006 From: perry at stsci.edu (Perry Greenfield) Date: Wed Jan 18 17:22:05 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: <43CEE67D.9000707@colorado.edu> References: <1137584647.19613.17.camel@localhost.localdomain> <43CE85AD.1090904@ieee.org> <1137610937.19613.65.camel@localhost.localdomain> <43CE985E.9050406@astraw.com> <43CE9B65.5040803@ee.byu.edu> <43CEBA67.8000104@colorado.edu> <43CECD88.6050809@colorado.edu> <863273026756969dc8715e00bf0776c7@stsci.edu> <43CEE67D.9000707@colorado.edu> Message-ID: <33e3b696b76c303ca00abe37ff0d48f3@stsci.edu> On Jan 18, 2006, at 8:08 PM, Fernando Perez wrote: >>> Couldn't that last issue be well dealt with by the fact that today's >>> numpy is fairly subclassing-friendly? (which, if I remember >>> correctly, wasn't quite the case with at least old Numeric). >> Does that help? You aren't talking about the fast array subclassing >> numpy are you? I'm not sure what you mean here. > > What I meant was that by having good subclassing functionality, it's > easier to ward off requests for every feature under the sun. It's > much easier to say: > > 'this basic object provides a very small, core set of array features > where the focus is on raw speed rather than fancy features; if you > need extra features, subclass it and add them yourself' > > when the subclassing is actually reasonably easy. Note that I haven't > actually used array subclassing myself (haven't needed it), so I may > be mistaken in my comments here, it's just an intuition. > Yeah, that makes sense. I was confused by the reference to numpy. Perry From oliphant.travis at ieee.org Wed Jan 18 18:19:06 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed Jan 18 18:19:06 2006 Subject: [Numpy-discussion] New SVN (>1939) requires a rebuild of extension modules Message-ID: <43CEF708.9070609@ieee.org> Just in case you missed the note in the middle of another message. The SVN version of NumPy has a few new C-API calls and requires a rebuild of extension modules. If you get mysterious segfaults all of a sudden, you haven't recompiled... -Travis From chris at trichech.us Wed Jan 18 19:10:01 2006 From: chris at trichech.us (Christopher Fonnesbeck) Date: Wed Jan 18 19:10:01 2006 Subject: [Numpy-discussion] numpy svn unit test failure Message-ID: Tonight's svn build of numpy on OS X 10.4 fails the unit test: In [2]: numpy.test(1,1) Found 3 tests for numpy.distutils.misc_util Found 10 tests for numpy.core.umath Found 8 tests for numpy.lib.arraysetops Found 42 tests for numpy.lib.type_check Found 9 tests for numpy.lib.twodim_base Found 15 tests for numpy.core.multiarray Found 25 tests for numpy.core.ma Found 6 tests for numpy.core.defmatrix Found 3 tests for numpy.lib.getlimits Found 33 tests for numpy.lib.function_base Found 3 tests for numpy.dft.helper Found 6 tests for numpy.core.records Found 8 tests for numpy.core.numeric Found 4 tests for numpy.lib.index_tricks Found 44 tests for numpy.lib.shape_base Found 0 tests for __main__ ........................................................................ ........................................E...E........................... ........................................................................ ..... ====================================================================== ERROR: check_basic (numpy.core.defmatrix.test_defmatrix.test_algebra) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/numpy-0.9.4.1940-py2.4-macosx-10.4-ppc.egg/ numpy/core/tests/test_defmatrix.py", line 111, in check_basic assert allclose((mA ** -i).A, B) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/numpy-0.9.4.1940-py2.4-macosx-10.4-ppc.egg/ numpy/core/defmatrix.py", line 155, in __pow__ x = self.I File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/numpy-0.9.4.1940-py2.4-macosx-10.4-ppc.egg/ numpy/core/defmatrix.py", line 209, in getI from numpy.dual import inv File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/numpy-0.9.4.1940-py2.4-macosx-10.4-ppc.egg/ numpy/dual.py", line 24, in ? import scipy.linalg as linpkg File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/scipy-0.4.4.1562-py2.4-macosx-10.4-ppc.egg/ scipy/linalg/__init__.py", line 8, in ? from basic import * File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/scipy-0.4.4.1562-py2.4-macosx-10.4-ppc.egg/ scipy/linalg/basic.py", line 17, in ? from flinalg import get_flinalg_funcs File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/scipy-0.4.4.1562-py2.4-macosx-10.4-ppc.egg/ scipy/linalg/flinalg.py", line 15, in ? from numpy.distutils.misc_util import PostponedException ImportError: cannot import name PostponedException ====================================================================== ERROR: check_basic (numpy.core.defmatrix.test_defmatrix.test_properties) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/numpy-0.9.4.1940-py2.4-macosx-10.4-ppc.egg/ numpy/core/tests/test_defmatrix.py", line 34, in check_basic assert allclose(linalg.inv(A), mA.I) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/numpy-0.9.4.1940-py2.4-macosx-10.4-ppc.egg/ numpy/core/defmatrix.py", line 209, in getI from numpy.dual import inv File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/numpy-0.9.4.1940-py2.4-macosx-10.4-ppc.egg/ numpy/dual.py", line 24, in ? import scipy.linalg as linpkg File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/scipy-0.4.4.1562-py2.4-macosx-10.4-ppc.egg/ scipy/linalg/__init__.py", line 8, in ? from basic import * File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/scipy-0.4.4.1562-py2.4-macosx-10.4-ppc.egg/ scipy/linalg/basic.py", line 17, in ? from flinalg import get_flinalg_funcs File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/scipy-0.4.4.1562-py2.4-macosx-10.4-ppc.egg/ scipy/linalg/flinalg.py", line 15, in ? from numpy.distutils.misc_util import PostponedException ImportError: cannot import name PostponedException ---------------------------------------------------------------------- Ran 221 tests in 2.990s FAILED (errors=2) Out[2]: -- Christopher J. Fonnesbeck Population Ecologist, Marine Mammal Section Fish & Wildlife Research Institute (FWC) St. Petersburg, FL Adjunct Assistant Professor Warnell School of Forest Resources University of Georgia Athens, GA T: 727.235.5570 E: chris at trichech.us -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2417 bytes Desc: not available URL: From strawman at astraw.com Wed Jan 18 20:04:03 2006 From: strawman at astraw.com (Andrew Straw) Date: Wed Jan 18 20:04:03 2006 Subject: [Numpy-discussion] New SVN (>1939) requires a rebuild of extension modules In-Reply-To: <43CEF708.9070609@ieee.org> References: <43CEF708.9070609@ieee.org> Message-ID: <43CF0F80.3060301@astraw.com> Travis Oliphant wrote: > > Just in case you missed the note in the middle of another message. > > The SVN version of NumPy has a few new C-API calls and requires a > rebuild of extension modules. If you get mysterious segfaults all of > a sudden, you haven't recompiled... > > -Travis > Hi Travis, I have an idea that might prevent people from accidentally getting mysterious errors due to mixing extensions compiled against different versions of numpy. Basically, the idea is that a version number is changed every time the C API changes. This version number is stored in each extension module at compile time, and at run time, each module init function that calls import_array() will trigger a test against the version number of the numpy core. I've implemented my idea in the included patch, and, as far as I can tell, it actually works. This is based on using the NDARRAY_VERSION define in arrayobject.h, which I'm completely unsure if this is a suitable use for. Furthermore, there's a lot going on that I don't understand in this part of the numpy code. Thus, I consider my patch only a proof of concept and hope you (or someone) will take this idea, perhaps the patch, and run with it. I hope this prevents many headaches in the future... What do you think? -------------- next part -------------- A non-text attachment was scrubbed... Name: API_version.patch Type: text/x-patch Size: 3071 bytes Desc: not available URL: From ndarray at mac.com Wed Jan 18 21:04:01 2006 From: ndarray at mac.com (Sasha) Date: Wed Jan 18 21:04:01 2006 Subject: [Numpy-discussion] New SVN (>1939) requires a rebuild of extension modules In-Reply-To: <43CF0F80.3060301@astraw.com> References: <43CEF708.9070609@ieee.org> <43CF0F80.3060301@astraw.com> Message-ID: +1 I suggest a little more descriptive exception: PyErr_Format(PyExc_ImportError, "extension module compiled against version %X of numpy API" ", not %X", NDARRAY_VERSION, PyArray_GetNDArrayCVersion()) On 1/18/06, Andrew Straw wrote: > Travis Oliphant wrote: > > > > > Just in case you missed the note in the middle of another message. > > > > The SVN version of NumPy has a few new C-API calls and requires a > > rebuild of extension modules. If you get mysterious segfaults all of > > a sudden, you haven't recompiled... > > > > -Travis > > > > Hi Travis, > > I have an idea that might prevent people from accidentally getting > mysterious errors due to mixing extensions compiled against different > versions of numpy. Basically, the idea is that a version number is > changed every time the C API changes. This version number is stored in > each extension module at compile time, and at run time, each module init > function that calls import_array() will trigger a test against the > version number of the numpy core. > > I've implemented my idea in the included patch, and, as far as I can > tell, it actually works. This is based on using the NDARRAY_VERSION > define in arrayobject.h, which I'm completely unsure if this is a > suitable use for. Furthermore, there's a lot going on that I don't > understand in this part of the numpy code. Thus, I consider my patch > only a proof of concept and hope you (or someone) will take this idea, > perhaps the patch, and run with it. I hope this prevents many headaches > in the future... > > What do you think? > > > From faltet at carabos.com Thu Jan 19 01:13:03 2006 From: faltet at carabos.com (Francesc Altet) Date: Thu Jan 19 01:13:03 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: <43CE9D1D.6090509@ee.byu.edu> References: <1137584647.19613.17.camel@localhost.localdomain> <43CE985E.9050406@astraw.com> <43CE9D1D.6090509@ee.byu.edu> Message-ID: <200601191012.14204.faltet@carabos.com> A Dimecres 18 Gener 2006 20:55, Travis Oliphant va escriure: > I think the indexing code will be slower because it is more > sophisticated than Numeric's. Basically, it has to check for fancy > indexing before defaulting to the old way. I see this as more of a > slow-down than array creation. It might be possible to improve it --- > more eyeballs are always helpful. But, I'm not sure how at this point. I'm sure you already know this: http://oprofile.sourceforge.net It is a nice way to do profiling at C level on Linux machines. Running the Paulo benchmarks through oprofile can surely bring some light. Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From arnd.baecker at web.de Thu Jan 19 01:29:03 2006 From: arnd.baecker at web.de (Arnd Baecker) Date: Thu Jan 19 01:29:03 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: <200601191012.14204.faltet@carabos.com> References: <1137584647.19613.17.camel@localhost.localdomain> <43CE985E.9050406@astraw.com> <43CE9D1D.6090509@ee.byu.edu> <200601191012.14204.faltet@carabos.com> Message-ID: On Thu, 19 Jan 2006, Francesc Altet wrote: > A Dimecres 18 Gener 2006 20:55, Travis Oliphant va escriure: > > I think the indexing code will be slower because it is more > > sophisticated than Numeric's. Basically, it has to check for fancy > > indexing before defaulting to the old way. I see this as more of a > > slow-down than array creation. It might be possible to improve it --- > > more eyeballs are always helpful. But, I'm not sure how at this point. > > I'm sure you already know this: > > http://oprofile.sourceforge.net > > It is a nice way to do profiling at C level on Linux machines. Running > the Paulo benchmarks through oprofile can surely bring some light. What might be nice is to use the output of oprofile and stick it into kcachegrind, http://kcachegrind.sourceforge.net/ to get a GUI to the results. I haven't done this, but according to http://linux.com.hk/penguin/man/1/op2calltree.html the tool is `op2calltree` - convert OProfile profiling data to KCachegrind calltree format and further "This utility is part of the KDE Software Development Kit." Best, Arnd From faltet at carabos.com Thu Jan 19 02:01:06 2006 From: faltet at carabos.com (Francesc Altet) Date: Thu Jan 19 02:01:06 2006 Subject: [Numpy-discussion] New SVN (>1939) requires a rebuild of extension modules In-Reply-To: References: <43CEF708.9070609@ieee.org> <43CF0F80.3060301@astraw.com> Message-ID: <200601191100.21715.faltet@carabos.com> A Dijous 19 Gener 2006 06:03, Sasha va escriure: > +1 > > I suggest a little more descriptive exception: > > PyErr_Format(PyExc_ImportError, > "extension module compiled against version %X of numpy > API" ", not %X", NDARRAY_VERSION, PyArray_GetNDArrayCVersion()) I also like this idea. I'd go a step further and add to the above message something like this: "...Nertheless, if you know what are you doing and want to allow this extension to run, set the environment variable FORCE_NUMPY_RUN to 1 (will continue warning you) or 2 (skip all warnings)." Of course, one should provide some code to check the environment variable, but this should be not very difficult. Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From svetosch at gmx.net Thu Jan 19 04:42:09 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Thu Jan 19 04:42:09 2006 Subject: [Numpy-discussion] array vs. matrix for matlab-compatible stuff In-Reply-To: <43CE7E68.1010908@ieee.org> References: <43CE631D.1000006@gmx.net> <43CE7E68.1010908@ieee.org> Message-ID: <43CF8942.3050304@gmx.net> Travis Oliphant schrieb: > Sven Schreiber wrote: > >> Hi, >> I've spent a couple of weeks with scipy/numpy and the old-to-new >> transition; now that the transition is over (?) but some confusion is >> remaining (on my side) I feel the need to ask a basic question about >> matlab compatibility in terms of matrix (linear algebra) programming. >> >> Take "eye" and "identity" for example; given that "eye" supposedly >> exists to facilitate transiton from matlab to numpy/scipy (correct?), >> I expected eye to be/return a matrix. > > > Historical is the only reason. Numeric always returned an array for eye > not a matrix. We could return a matrix without difficulty especially if > we put an eye --> identity transition in convertcode.py > I see, thanks for the quick answer. So wouldn't it be a good idea to have all the specifically matrix-related stuff (afaics, things in numpy/lib/twodim_base.py) return matrices? It seems my question (or misunderstanding) has a broader scope: Coming from matrix languages, I'm glad about short notations like A.I or A*B representing standard matrix operations. Much easier than linalg.inverse(A) or matrixmultiply(A,B). However, many matrix functions (decompositions etc.) seem to return arrays instead of matrices, even if you feed them matrices (is this assumption correct?). So you have to use mat(returned-result) again and again to be able to do return-result.I afterwards, which seems clumsy. So what's the best strategy here? I'm aware that this is very much a basic/newbie question, so a pointer to the answer(s) would also be very welcome. If you think I should purchase the numpy-book to get the authorative answer, feel free to tell me so (but only if it really has the answer ;-) Thanks for your help, Sven From pjssilva at ime.usp.br Thu Jan 19 04:58:02 2006 From: pjssilva at ime.usp.br (Paulo J. S. Silva) Date: Thu Jan 19 04:58:02 2006 Subject: [Numpy-discussion] Updated benchmark Message-ID: <1137675462.2961.7.camel@localhost.localdomain> Hello, I am sending an update benchmark table below. I have noticed that in my benchmarck code for numpy I was using transpose(A) instead of A.transpose() which introduces an extra function call. A very big penalty for tiny matrices. Now the transpose operation doesn't look that bad in numpy when compared to matlab. There is still something on this table that I can't understand. Why are the index operations so expensive in domension 50? If you look closely you'll see that numpy is faster in dimension 5, much slower in dimension 50 (2.6 times slower) and then the gap decreases in dimension 500 (1.6 slower). Best, Paulo Tests x.T*y x*y.T A*x A*B A.T*x half 2in2 Dimension: 5 Array 0.82 0.22 0.18 0.26 0.33 0.90 1.07 Matrix 4.54 1.41 0.56 0.66 0.99 2.87 4.34 NumArr 2.82 0.66 0.61 0.71 7.54 6.61 10.56 Numeri 1.15 0.34 0.28 0.38 0.66 0.60 0.71 Matlab 1.60 0.39 0.31 0.33 0.35 1.13 0.91 Dimension: 50 Array 9.03 1.95 0.48 16.60 0.98 3.89 4.07 Matrix 45.94 3.55 0.90 17.30 1.67 5.93 7.42 NumArr 29.18 2.47 0.90 18.23 11.86 7.87 11.88 Numeri 11.91 2.13 0.59 17.21 9.27 0.93 3.02 Matlab 16.12 1.81 0.99 16.28 0.70 1.46 1.66 Dimension: 500 Array 1.09 8.05 1.77 151.32 1.89 3.63 3.86 Matrix 4.87 8.16 1.82 151.44 1.94 4.22 4.35 NumArr 3.12 8.19 1.80 151.66 19.83 3.94 5.30 Numeri 1.43 8.18 1.81 151.68 17.52 2.94 4.08 Matlab 1.91 5.07 1.89 150.52 1.84 2.27 2.74 From konrad.hinsen at laposte.net Thu Jan 19 06:02:04 2006 From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net) Date: Thu Jan 19 06:02:04 2006 Subject: [Numpy-discussion] arrayobject.h (and friends) install location In-Reply-To: <43CD296C.7000700@astraw.com> References: <22F30CE0-7362-44A7-8DA1-79C6FE9E70D0@tamu.edu> <96D6BEBF-8A8D-4B69-9A6C-BC2A86BDA9B0@laposte.net> <89D76E1D-BE44-43FF-AD68-D0A8FACD795D@tamu.edu> <43CD296C.7000700@astraw.com> Message-ID: <79CFDD9A-7239-49EB-9722-4AB5EA36B87D@laposte.net> On 17.01.2006, at 18:29, Andrew Straw wrote: > Also, as for as I know, the only packages that install things into > /usr/include/python2.x/packagename are Numeric and numarray, so I > would > argue it's not "standard", (although it may have lots of history). Two of my own packages, ScientificPython and MMTK, do the same, so that's already four. In fact, I don't know any package that puts its include files elsewhere. Most packages have no include files at all of course. Another argument for this being the "standard" location is that it is in the include file search path when Python modules are built using distutils, and the path where Distutils puts them.. Yet another argument is that it's about the only location where include files can be placed without any necessity for client packages to know the habits of each of its dependencies. On 18.01.2006, at 02:18, Christopher Barker wrote: > As a Linux+OS-X user, I've run into a number of these Apple "Think > Different" issues, but I don't think creating an ugly, non-standard > system to accommodate the fact that Apple like to do things > strangely is the right solution. Apple doesn't maintain their > python distribution well, or really play well with the python > community, so many of us end up installing a new Python anyway. There isn't much difference between MacOS X and Linux: on both systems you need superuser rights to write to /usr/include. On 18.01.2006, at 03:22, Travis Oliphant wrote: > This is the important point in this discussion. There was a > reason it was changed. I remember opposing the change, > originally. However, it solved a real problem and only required > people to give up their arbitrary notions of where include files > *should* be. I don't see this as a real issue since you can get > where the include files are very easily. It's hardly arbitrary - see above. And I don't think anyone cares where include files are, as long as installing them via distutils will put them somewhere where another package installation using distutils will find them. > With that, however, I think it would be helpful for one of those > people who use .eggs to clarify why it solves their problem to have > the include files in their current location. Also what their problem exactly is - I don't remember seeing a description on this list. Perhaps we can come up with a different solution. Konrad. -- ------------------------------------------------------------------------ ------- Konrad Hinsen Laboratoire Leon Brillouin (CEA-CNRS), CEA Saclay, 91191 Gif-sur-Yvette Cedex, France Tel.: +33-1 69 08 79 25 Fax: +33-1 69 08 82 61 E-Mail: khinsen at cea.fr ------------------------------------------------------------------------ ------- From cjw at sympatico.ca Thu Jan 19 06:35:20 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Thu Jan 19 06:35:20 2006 Subject: [Numpy-discussion] An apparent bug in the Windows version of numpy 0.9.2 Message-ID: <43CFA2CB.3000806@sympatico.ca> Example: > PythonWin 2.4.2 (#67, Sep 28 2005, 12:41:11) [MSC v.1310 32 bit (Intel)] > on win32. > Portions Copyright 1994-2004 Mark Hammond (mhammond at skippinet.com.au) - > see 'Help/About PythonWin' for further copyright information. > >>> from numpy.core import umath > THIS CRASHES PYTHONWIN From svetosch at gmx.net Thu Jan 19 07:39:01 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Thu Jan 19 07:39:01 2006 Subject: [Numpy-discussion] An apparent bug in the Windows version of numpy 0.9.2 In-Reply-To: <43CFA2CB.3000806@sympatico.ca> References: <43CFA2CB.3000806@sympatico.ca> Message-ID: <43CFB299.1030505@gmx.net> Colin J. Williams schrieb: > Example: >> PythonWin 2.4.2 (#67, Sep 28 2005, 12:41:11) [MSC v.1310 32 bit (Intel)] >> on win32. >> Portions Copyright 1994-2004 Mark Hammond (mhammond at skippinet.com.au) - >> see 'Help/About PythonWin' for further copyright information. >> >>> from numpy.core import umath >> THIS CRASHES PYTHONWIN > not for me, so there must be some other ingredients as well -sven From strawman at astraw.com Thu Jan 19 09:33:01 2006 From: strawman at astraw.com (Andrew Straw) Date: Thu Jan 19 09:33:01 2006 Subject: [Numpy-discussion] include file location and .eggs Message-ID: <43CFCCF8.8000401@astraw.com> (We've had threads about this before. See http://www.scipy.org/documentation/mailman?fn=scipy-dev/2005-September/003238.html and http://sourceforge.net/mailarchive/forum.php?thread_id=9474853&forum_id=4890) There's been recent discussion regarding the location of the numpy include files. The pre-numpy way (Numeric and numarray) was to use distutils's install_headers, which places them in the long-familiar locations like /usr/local/include/python2.3/ (on *nix). This was a solution we all lived with for many years. Why then, is numpy rocking the boat and putting the include files somewhere else? For me, and I suspect for most others in favor of the new way, the answer boils down to using eggs with out-of-the box numpy. Eggs are the distributable form of Python package distributions made with setuptools (see http://peak.telecommunity.com/DevCenter/setuptools ). This isn't an email to convince everyone to use eggs, but I would like to say what issues eggs solve, what they make possible, and why the current include file location doesn't work with them. I _would_ like to convince people that supporting eggs is a valid thing for numpy to do, even if it means some change in the include file location and therefore some pain. (Although the amount of pain seems minimal to me.) I hope the developers of numpy-based extension modules, even if they never plan on using eggs, are willing to accomodate those of us who do. = Why can't eggs use traditional include locations? = Multiple version support. Using eggs, it is possible to have multiple versions of the same distribution, such as numpy, installed within a single Python installation. (I use "distribution" to mean collection of Python packages and modules created by a call to distutils' setup() and use that word because some distributions, such as matplotlib, include multiple top-level packages.) This means I can have a root-installed, site-wide installation of numpy on my unix machine for all users but I can easily play with a per-user installed numpy installation if I want to. With multiple versions of numpy installed on my system, the include files I need to use to compile an extension module are obviously going to depend on the right version of the headers. This simply isn't possible using the traditional include location, but requires the appropriate headers for the the version of numpy in use. I don't see any easier solution to this issue than what eggs already has implemented -- import the numpy package and ask it where its include files are (which will be in a sub-directory of it's top-level space within site-packages). = What issues do eggs solve? = I already mentioned multiple version support. eggs also offer dependency analysis and resolution. They also allow compatibility with easy_install, meaning automatic downloading and installation of dependencies (see http://peak.telecommunity.com/DevCenter/EasyInstall ). = What else do eggs make possible? = Eggs are distributed as a single ".egg" file, which is really just a specially formatted .zip file. If the package is "zip-safe", eggs are installed in site-packages as a single zip file. Even if they're not zip-safe, eggs are installed in site-packages in their own single directory. This makes package management much easier. Another of my favorite enhancements that setuptools brings to Python is runtime plugin support. If I have package A, which can optionally use plugins from module B, I can install A without installing B. When I install B, A will automatically be able to use B. If I write C to the same plugin interface that B uses, A can also use C, even if A never knew of the existance of C. There is lots more that setuptools enables. This is necessarily only a short list of what eggs can do and are useful for. Please see http://peak.telecommunity.com/DevCenter/setuptools for more information. I hope those with reservations about the change in include file locations now understand the reason for the change and, even more optimistically, I hope those folks are willing to accept the current state of affairs. I will attempt to answer any more questions on the issue. Cheers! Andrew From cjw at sympatico.ca Thu Jan 19 10:26:04 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Thu Jan 19 10:26:04 2006 Subject: [Numpy-discussion] An apparent bug in the Windows version of numpy 0.9.2 In-Reply-To: <43CFB299.1030505@gmx.net> References: <43CFA2CB.3000806@sympatico.ca> <43CFB299.1030505@gmx.net> Message-ID: <43CFD988.5090105@sympatico.ca> Sven Schreiber wrote: >Colin J. Williams schrieb: > > >>Example: >> >> >>>PythonWin 2.4.2 (#67, Sep 28 2005, 12:41:11) [MSC v.1310 32 bit (Intel)] >>>on win32. >>>Portions Copyright 1994-2004 Mark Hammond (mhammond at skippinet.com.au) - >>>see 'Help/About PythonWin' for further copyright information. >>> >>> from numpy.core import umath >>>THIS CRASHES PYTHONWIN >>> >>> > >not for me, so there must be some other ingredients as well >-sven > > > Thanks for this, I'm using Numpy 0.9.2. Does anyone have any suggestion as to what the the other ingredients might be. Similar problems arise with Boa-constructer and Python Scripter. The latter also reports a floating point error. Colin W. From oliphant.travis at ieee.org Thu Jan 19 11:26:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu Jan 19 11:26:02 2006 Subject: [Numpy-discussion] array vs. matrix for matlab-compatible stuff In-Reply-To: <43CF8942.3050304@gmx.net> References: <43CE631D.1000006@gmx.net> <43CE7E68.1010908@ieee.org> <43CF8942.3050304@gmx.net> Message-ID: <43CFE79B.9000509@ieee.org> Sven Schreiber wrote: >I see, thanks for the quick answer. So wouldn't it be a good idea to have all the specifically >matrix-related stuff (afaics, things in numpy/lib/twodim_base.py) return matrices? > >It seems my question (or misunderstanding) has a broader scope: Coming from matrix languages, I'm >glad about short notations like A.I or A*B representing standard matrix operations. Much easier than >linalg.inverse(A) or matrixmultiply(A,B). However, many matrix functions (decompositions etc.) seem >to return arrays instead of matrices, even if you feed them matrices (is this assumption correct?). >So you have to use mat(returned-result) again and again to be able to do return-result.I afterwards, >which seems clumsy. So what's the best strategy here? > > Yes, some of them do still return arrays. Matrices are longer lived in NumPy then the were in Numeric, for sure, but many functions still aren't friendly to matrices and convert all inputs to arrays before operation. Originally, I had the asarray(...) function not convert matrices by default, but this is too surprising because matrices change the '*' and '**' operators which could make your function not work. We should convert all the functions that don't handle matrices so they will. I'd like to see matrices survive longer than they do. There are some functions that try to do that -Travis From svetosch at gmx.net Thu Jan 19 11:47:01 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Thu Jan 19 11:47:01 2006 Subject: [Numpy-discussion] array vs. matrix for matlab-compatible stuff In-Reply-To: <43CF8942.3050304@gmx.net> References: <43CE631D.1000006@gmx.net> <43CE7E68.1010908@ieee.org> <43CF8942.3050304@gmx.net> Message-ID: <43CFECB1.20008@gmx.net> Sven Schreiber schrieb: > So you have to use mat(returned-result) again and again to be able to do return-result.I afterwards, Well I hope the fact that the above line is syntactically scandalous and a good reason to feel ashamed of myself is not the reason that nobody is answering my previous post .... ;-) But seriously, isn't there a preferred way to do matrix computations in numpy? Should I stick with arrays a la "linalg.inv(eye(3))", or does it make more sense to use matrices, as in "mat(eye(3)).I" ? Or, since both notations seem a little clumsy to me, isn't there an even better way that maybe even combines the best of both worlds? I definitely have the feeling that I'm missing something. _Any_ feedback is highly appreciated, thanks, Sven From oliphant.travis at ieee.org Thu Jan 19 12:12:04 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu Jan 19 12:12:04 2006 Subject: [Numpy-discussion] An apparent bug in the Windows version of numpy 0.9.2 In-Reply-To: <43CFD988.5090105@sympatico.ca> References: <43CFA2CB.3000806@sympatico.ca> <43CFB299.1030505@gmx.net> <43CFD988.5090105@sympatico.ca> Message-ID: <43CFE6A1.8060504@ieee.org> Colin J. Williams wrote: >> > Thanks for this, I'm using Numpy 0.9.2. Does anyone have any > suggestion as to what the the other ingredients might be. > Similar problems arise with Boa-constructer and Python Scripter. The > latter also reports a floating point error. I'm pretty sure it has to do with the fact that I'm using the mingw32 compiler to compile NumPy, while Pythonwin was compiled using something else. It might be related to Py_complex numbers which in Python are passed around as structures and my understanding is different compilers treat this differently. But, it might be something else as well. I've been surprised in the past by some of the quirks of compiling on windows. The safest thing to do is compile extensions with the same compiler used to compile Python (but I don't have that compiler, so I can't do it). -Travis From oliphant.travis at ieee.org Thu Jan 19 12:25:03 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu Jan 19 12:25:03 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: <200601191012.14204.faltet@carabos.com> References: <1137584647.19613.17.camel@localhost.localdomain> <43CE985E.9050406@astraw.com> <43CE9D1D.6090509@ee.byu.edu> <200601191012.14204.faltet@carabos.com> Message-ID: <43CFF596.2070007@ieee.org> Francesc Altet wrote: >http://oprofile.sourceforge.net > >It is a nice way to do profiling at C level on Linux machines. Running >the Paulo benchmarks through oprofile can surely bring some light. > > I ran the following code snippet (timed under a Timeit instance) through the oprofile profiler for both NumPy and Numeric, to look at indexing speeds. op = "b = A[::2,::2]; d = A[1:80,:]" This is what I found: overall 61242 53.9606 /usr/bin/python 17647 15.5488 /usr/lib/python2.4/site-packages/numpy/core/multiarray.so 15942 14.0466 /lib/tls/libc-2.3.3.so 7158 6.3069 /no-vmlinux 6995 6.1633 /usr/lib/python2.4/site-packages/Numeric/_numpy.so Showing that more time is spent in NumPy than in Numeric doing indexing... Here's the breakdown for NumPy samples % symbol name 2353 13.3337 PyArray_PyIntAsIntp # This is also slower --- called more often? 2060 11.6734 PyArray_MapIterNew # This calls fancy_indexing_check. 1980 11.2200 slice_GetIndices 1631 9.2424 parse_index 1149 6.5110 arraymapiter_dealloc # Interesting this is taking so long? 1142 6.4714 array_subscript 1121 6.3524 _IsAligned 1069 6.0577 array_dealloc 780 4.4200 fancy_indexing_check 684 3.8760 PyArray_NewFromDescr 627 3.5530 parse_subindex 538 3.0487 PyArray_DescrFromType 534 3.0260 array_subscript_nice 455 2.5783 _IsContiguous 370 2.0967 _IsFortranContiguous 334 1.8927 slice_coerce_index 294 1.6660 PyArray_UpdateFlags 234 1.3260 anonymous symbol from section .plt 161 0.9123 PyArray_Return 120 0.6800 array_alloc 2 0.0113 PyArray_Broadcast 2 0.0113 PyArray_IterNew 1 0.0057 LONG_setitem 1 0.0057 PyArray_EquivTypes 1 0.0057 PyArray_FromAny 1 0.0057 PyArray_FromStructInterface 1 0.0057 PyArray_IntpConverter 1 0.0057 PyArray_SetNumericOps 1 0.0057 initialize_numeric_types Here's the breakdown for Numeric: 1577 22.5447 slice_GetIndices 1155 16.5118 parse_index 912 13.0379 PyArray_FromDimsAndDataAndDescr 792 11.3224 array_subscript 675 9.6497 PyArray_IntegerAsInt 517 7.3910 parse_subindex 401 5.7327 array_dealloc 379 5.4182 slice_coerce_index 339 4.8463 array_subscript_nice 161 2.3016 anonymous symbol from section .plt 82 1.1723 PyArray_Return 5 0.0715 do_sliced_copy Anybody interested in optimization? -Travis From svetosch at gmx.net Thu Jan 19 12:54:21 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Thu Jan 19 12:54:21 2006 Subject: [Numpy-discussion] array vs. matrix for matlab-compatible stuff In-Reply-To: <43CFE79B.9000509@ieee.org> References: <43CE631D.1000006@gmx.net> <43CE7E68.1010908@ieee.org> <43CF8942.3050304@gmx.net> <43CFE79B.9000509@ieee.org> Message-ID: <43CFFC84.7050803@gmx.net> Travis Oliphant schrieb: > Yes, some of them do still return arrays. Matrices are longer lived in > NumPy then the were in Numeric, for sure, but many functions still > aren't friendly to matrices and convert all inputs to arrays before > operation. Originally, I had the asarray(...) function not convert > matrices by default, but this is too surprising because matrices change > the '*' and '**' operators which could make your function not work. > We should convert all the functions that don't handle matrices so they > will. I'd like to see matrices survive longer than they do. There are > some functions that try to do that > Thanks very much for this useful information! I totally agree, long live the matrices... In the ebook you are selling, are things like that (which functions preserve matrix-type) documented? Those kind of things would be a reason for me to buy it. cheers, Sven From ndarray at mac.com Thu Jan 19 12:57:12 2006 From: ndarray at mac.com (Sasha) Date: Thu Jan 19 12:57:12 2006 Subject: [Numpy-discussion] Interesting timing results In-Reply-To: <43CD7E3A.8070404@ieee.org> References: <43CD7E3A.8070404@ieee.org> Message-ID: On 1/17/06, Travis Oliphant wrote: > ... Currently all array scalar math goes through the > entire ufunc machinery. This is clearly sub-optimal. It is the reason > for the scalarmath module that I've started in the src directory. > Eventually, we should be able to have scalar math as fast as Python > scalars. I have implemented "nonzero", &, | and ^ for scalar bools. (See http://projects.scipy.org/scipy/numpy/changeset/1946). Here are the timings: Before: > python -m timeit -s "from numpy import bool_; x = bool_(0)" "x & x" 100000 loops, best of 3: 3.85 usec per loop Now: > python -m timeit -s "from numpy import bool_; x = bool_(0)" "x & x" 10000000 loops, best of 3: 0.174 usec per loop This is close to python bool: > python -m timeit -s "x = bool(0)" "x & x" 10000000 loops, best of 3: 0.142 usec per loop and faster than python int: > python -m timeit -s "from numpy import bool_; x = 0" "x & x" 10000000 loops, best of 3: 0.199 usec per loop But it is in fact all within the timing error now. I did not put it in the scalarmath module for two reasons: first, scalarmath is not hooked to numpy yet and second because C-API does not provide access to scalar bools yet. I have posted a proposal for C-API changes and did not hear any opposition (well, no support either), so I will implement that soon. There are a few issues with the new APIs that I proposed. First is a simple one: I proposed to expose boolean scalars as named constants to Python, the question is how to call them. 1. True_ and False_ 2. true and false The second issue is whether to add new numbers to _ARRAY_API or add a separate _ARRAY_SCALAR_API . I've only optimized scalar-scalar case in binary ops and fall back to old for everything else. I would like to optimize scalar-array and array-scalar cases as well, but I wonder if it is apropriate to make "(bool_(0) | x) is x" true when x is an array. Alternatively (bool_(0) | x) can become a new array that shares data with x. -- sasha From oliphant.travis at ieee.org Thu Jan 19 13:31:05 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu Jan 19 13:31:05 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: <43CFF596.2070007@ieee.org> References: <1137584647.19613.17.camel@localhost.localdomain> <43CE985E.9050406@astraw.com> <43CE9D1D.6090509@ee.byu.edu> <200601191012.14204.faltet@carabos.com> <43CFF596.2070007@ieee.org> Message-ID: <43D004FC.7090108@ieee.org> Travis Oliphant wrote: > > This is what I found: overall > > 61242 53.9606 /usr/bin/python > 17647 15.5488 > /usr/lib/python2.4/site-packages/numpy/core/multiarray.so > 15942 14.0466 /lib/tls/libc-2.3.3.so > 7158 6.3069 /no-vmlinux > 6995 6.1633 /usr/lib/python2.4/site-packages/Numeric/_numpy.so > > Showing that more time is spent in NumPy than in Numeric doing > indexing... > I optimized PyArray_PyIntAsIntp and changed things so that extra allocations are not done if the indexing is not fancy in the new svn of numpy and improved the performance of indexing a bit... The new numbers... samples % symbol name 551 18.4775 slice_GetIndices 482 16.1636 array_subscript 343 11.5023 parse_index 309 10.3622 PyArray_PyIntAsIntp 166 5.5667 fancy_indexing_check 164 5.4997 _IsAligned 140 4.6948 array_dealloc 134 4.4936 parse_subindex 133 4.4601 anonymous symbol from section .plt 127 4.2589 PyArray_NewFromDescr 121 4.0577 slice_coerce_index 79 2.6492 _IsFortranContiguous 67 2.2468 array_subscript_nice 56 1.8779 _IsContiguous 49 1.6432 array_alloc 31 1.0396 PyArray_UpdateFlags 29 0.9725 PyArray_Return 1 0.0335 array_itemsize_get For reference, the Numeric numbers are: samples % symbol name 375 25.9695 slice_GetIndices 255 17.6593 parse_index 198 13.7119 PyArray_IntegerAsInt 135 9.3490 array_subscript 130 9.0028 PyArray_FromDimsAndDataAndDescr 117 8.1025 parse_subindex 81 5.6094 slice_coerce_index 55 3.8089 array_subscript_nice 44 3.0471 array_dealloc 40 2.7701 anonymous symbol from section .plt 13 0.9003 PyArray_Return 1 0.0693 do_sliced_copy These look a bit better and the results show that simple indexing seems to be only slowed down by the fancy indexing check... -Travis From oliphant.travis at ieee.org Thu Jan 19 13:38:11 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu Jan 19 13:38:11 2006 Subject: [Numpy-discussion] Interesting timing results In-Reply-To: References: <43CD7E3A.8070404@ieee.org> Message-ID: <43D00699.10001@ieee.org> Sasha wrote: >On 1/17/06, Travis Oliphant wrote: > > >>... Currently all array scalar math goes through the >>entire ufunc machinery. This is clearly sub-optimal. It is the reason >>for the scalarmath module that I've started in the src directory. >>Eventually, we should be able to have scalar math as fast as Python >>scalars. >> >> > >I have implemented "nonzero", &, | and ^ for scalar bools. (See >http://projects.scipy.org/scipy/numpy/changeset/1946). Here are the >timings: > >Before: > > >>python -m timeit -s "from numpy import bool_; x = bool_(0)" "x & x" >> >> >100000 loops, best of 3: 3.85 usec per loop > >Now: > > >>python -m timeit -s "from numpy import bool_; x = bool_(0)" "x & x" >> >> >10000000 loops, best of 3: 0.174 usec per loop > >This is close to python bool: > > >>python -m timeit -s "x = bool(0)" "x & x" >> >> >10000000 loops, best of 3: 0.142 usec per loop > >and faster than python int: > > >>python -m timeit -s "from numpy import bool_; x = 0" "x & x" >> >> >10000000 loops, best of 3: 0.199 usec per loop > >But it is in fact all within the timing error now. > >I did not put it in the scalarmath module for two reasons: first, >scalarmath is not hooked to numpy yet and second because C-API does >not provide access to scalar bools yet. I have posted a proposal for >C-API changes and did not hear any opposition (well, no support >either), so I will implement that soon. > >There are a few issues with the new APIs that I proposed. First is a >simple one: I proposed to expose boolean scalars as named constants to >Python, the question is >how to call them. > >1. True_ and False_ > > +1 >2. true and false > > -1 >The second issue is whether to add new numbers to _ARRAY_API or add a >separate _ARRAY_SCALAR_API . > > No separate _SCALAR_API.... >I've only optimized scalar-scalar case in binary ops and fall back to >old for everything else. I would like to optimize scalar-array and >array-scalar cases as well, but I wonder if it is apropriate to make >"(bool_(0) | x) is x" true when x is an array. Alternatively >(bool_(0) | x) can become a new array that shares data with x. > > Other operations with scalars return a new array. The fact that this would be different would probably be a bad thing in the end. So, I vote for returning a new copy of the data... -Travis From oliphant.travis at ieee.org Thu Jan 19 15:21:11 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu Jan 19 15:21:11 2006 Subject: [Numpy-discussion] Time for a new release of NumPy Message-ID: <43D01EC7.3040203@ieee.org> I'd like to make a release of NumPy over the weekend. Please submit bugs to the list before Saturday night (GMT -7) NumPy 0.9.2 was based on SVN version 1829 and we are over 100 checkins from that point, including the significant change to the .dtype attribute.... -Travis From info at barclays.co.uk Thu Jan 19 15:30:06 2006 From: info at barclays.co.uk (info at barclays.co.uk) Date: Thu Jan 19 15:30:06 2006 Subject: [Numpy-discussion] Barclays IBank security measure ID 3342025 Message-ID: <20060119222017.15399.qmail@ns32968.ovh.net> An HTML attachment was scrubbed... URL: From ndarray at mac.com Thu Jan 19 16:53:03 2006 From: ndarray at mac.com (Sasha) Date: Thu Jan 19 16:53:03 2006 Subject: [Numpy-discussion] Time for a new release of NumPy In-Reply-To: <43D01EC7.3040203@ieee.org> References: <43D01EC7.3040203@ieee.org> Message-ID: On 1/19/06, Travis Oliphant wrote: > > ... Please submit > bugs to the list before Saturday night (GMT -7) Trac has bug report pages: http://projects.scipy.org/scipy/numpy/report/6 but they don't appear to be in use. Here is my bug report: http://projects.scipy.org/scipy/numpy/ticket/3 -- sasha From mark at mitre.org Thu Jan 19 19:22:05 2006 From: mark at mitre.org (Mark Heslep) Date: Thu Jan 19 19:22:05 2006 Subject: [Numpy-discussion] Speed performance on array constant set Message-ID: <43D0573E.4090301@mitre.org> Im doing some work with the OpenCv* project. Im using swig typemaps to convert the Cv data structures to numarray which works well. Id like to restrict Cv use to what its strengths: complicated vision processing algorithms like optical flow. For the case of simple Cv data manipulations, I'd rather use NumPy functions & methods but was surprised at the performance comparison. - A simple scalar constant fill with cvSet. 'im' here is a wrapped Cv image data structure. > python -m timeit -s "import opencv.cv as cv; im = > cv.cvCreateImage(cv.cvSize(1000,1000), 8, 1)" "cv.cvSet( im, > cv.cvRealScalar( 7 ) )" > 100 loops, best of 3: 2.58 msec per loop - If I try the equivalent with NumPy > python -m timeit -s "import numarray as na; a = na.zeros((1000,1000) > )" "a[:,:] = 7" > 10 loops, best of 3: 45.1 msec per loop A >10x hit. Am I using the preferred / optimal NumPy method here? I scanned the earlier Scalar posts but thought that was boolean type only issue. Mark *OpenCv is an computer vision library, open source, and is sponsored by Intel. It includes many video capable functions for application to motion analysis, tracking and the like. From oliphant.travis at ieee.org Thu Jan 19 19:49:01 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu Jan 19 19:49:01 2006 Subject: [Numpy-discussion] Speed performance on array constant set In-Reply-To: <43D0573E.4090301@mitre.org> References: <43D0573E.4090301@mitre.org> Message-ID: <43D05D99.1060200@ieee.org> Mark Heslep wrote: > Im doing some work with the OpenCv* project. Im using swig typemaps > to convert the Cv data structures to numarray which works well. Id > like to restrict Cv use to what its strengths: complicated vision > processing algorithms like optical flow. For the case of simple Cv > data manipulations, I'd rather use NumPy functions & methods but was > surprised at the performance comparison. > - A simple scalar constant fill with cvSet. 'im' here is a wrapped Cv > image data structure. > >> python -m timeit -s "import opencv.cv as cv; im = >> cv.cvCreateImage(cv.cvSize(1000,1000), 8, 1)" "cv.cvSet( im, >> cv.cvRealScalar( 7 ) )" >> 100 loops, best of 3: 2.58 msec per loop > > > - If I try the equivalent with NumPy > >> python -m timeit -s "import numarray as na; a = na.zeros((1000,1000) >> )" "a[:,:] = 7" > > >> 10 loops, best of 3: 45.1 msec per loop > > This is actually a bit surprising that opencv can create and fill so quickly. Perhaps they are using optimized SSE functions for the Intel platform, or something? -Travis From ndarray at mac.com Thu Jan 19 19:53:02 2006 From: ndarray at mac.com (Sasha) Date: Thu Jan 19 19:53:02 2006 Subject: [Numpy-discussion] Speed performance on array constant set In-Reply-To: References: <43D0573E.4090301@mitre.org> Message-ID: On 1/19/06, Mark Heslep wrote: > A >10x hit. Am I using the preferred / optimal NumPy method here? I > scanned the earlier Scalar posts but thought that was boolean type > only issue. Try a.fill(7). On my system: > python -m timeit -s "import numpy as na; a = na.zeros((1000,1000))" "a[:,:] = 7" 100 loops, best of 3: 17.1 msec per loop > python -m timeit -s "import numpy as na; a = na.zeros((1000,1000))" "a.fill(7)" 100 loops, best of 3: 6.28 msec per loop You can also use empty instead of zeros, but that wont affect the timings that you presented. -- sasha From mark at mitre.org Thu Jan 19 20:24:01 2006 From: mark at mitre.org (Mark Heslep) Date: Thu Jan 19 20:24:01 2006 Subject: [Numpy-discussion] Speed performance on array constant set In-Reply-To: <43D05D99.1060200@ieee.org> References: <43D0573E.4090301@mitre.org> <43D05D99.1060200@ieee.org> Message-ID: <43D065AF.3080307@mitre.org> Travis Oliphant wrote: > This is actually a bit surprising that opencv can create and fill so > quickly. Perhaps they are using optimized SSE functions for the > Intel platform, or something? > -Travis > Ah, sorry, Im an unintentional fraud. Yes I have Intel's optimization library IPP turned on and had forgotten about it. So one more time: With IPP on as before. UseOptimized = # of Cv functions available w/ IPP > python -m timeit -s "import opencv.cv as cv; print > cv.cvUseOptimized(1); im =cv.cvCreateImage(cv.cvSize(1000,1000), 8, > 1)" "cv.cvSet( im, cv.cvRealScalar( 7 ) )" > 305 > 305 > 305 > 305 > 305 > 100 loops, best of 3: 2.24 msec per loop And without: > python -m timeit -s "import opencv.cv as cv; print > cv.cvUseOptimized(0); im =cv.cvCreateImage(cv.cvSize(1000,1000), 8, > 1)" "cv.cvSet( im, cv.cvRealScalar( 7 ) )" > 0 > 0 > 0 > 0 > 0 > 100 loops, best of 3: 6.94 msec per loop So IPP gives me 3X, which leads me to ask about plans for IPP / SSE for NumPy, no offense intended to non Intel users. I believe I recall some post that auto code generation in NumArray was the road block? Mark From mark at mitre.org Thu Jan 19 20:38:00 2006 From: mark at mitre.org (Mark Heslep) Date: Thu Jan 19 20:38:00 2006 Subject: [Numpy-discussion] Speed performance on array constant set In-Reply-To: References: <43D0573E.4090301@mitre.org> Message-ID: <43D06908.9060306@mitre.org> Sasha wrote: >Try a.fill(7). > >On my system: > > >>python -m timeit -s "import numpy as na; a = na.zeros((1000,1000))" "a[:,:] = 7" >> >> >100 loops, best of 3: 17.1 msec per loop > > > >>python -m timeit -s "import numpy as na; a = na.zeros((1000,1000))" "a.fill(7)" >> >> >100 loops, best of 3: 6.28 msec per loop > >You can also use empty instead of zeros, but that wont affect the >timings that you presented. > >-- sasha > > > Thank you. Any equivalent to 'fill' for NumArray or Numeric? Well, I suppose its time to move to Sci/NumPy. Im on Fedora 4, any pointers to a NumPy package? Mean time Ill pull from SVN and try again to build. I have the updated gcc Fortran. Mark From mark at mitre.org Thu Jan 19 20:45:01 2006 From: mark at mitre.org (Mark Heslep) Date: Thu Jan 19 20:45:01 2006 Subject: [Numpy-discussion] Speed performance on array constant set In-Reply-To: <43D06908.9060306@mitre.org> References: <43D0573E.4090301@mitre.org> <43D06908.9060306@mitre.org> Message-ID: <43D06AA2.9080005@mitre.org> Scratch that - got it from SF. > Im on Fedora 4, any pointers to a NumPy package? Mean time Ill pull > from SVN and try again to build. I have the updated gcc Fortran. > > Mark > > From Norbert.Nemec.list at gmx.de Fri Jan 20 00:22:02 2006 From: Norbert.Nemec.list at gmx.de (Norbert Nemec) Date: Fri Jan 20 00:22:02 2006 Subject: [Numpy-discussion] Bug? numpy.matrix.sum() Message-ID: <43D09D6E.5070907@gmx.de> Is it just me who finds this confusing? There is some bug there, but I cannot even say what it is supposed to behave like... --------------------------------- In [1]: import numpy In [2]: numpy.__version__ Out[2]: '0.9.2' In [3]: a = numpy.matrix(numpy.eye(2)) In [4]: a.sum() Out[4]: matrix([[1, 0, 0, 1]]) In [5]: a.A.sum() Out[5]: 2 In [6]: sum(a) Out[6]: matrix([[1, 1]]) In [7]: sum(a.A) Out[7]: array([1, 1]) --------------------------------- From Norbert.Nemec.list at gmx.de Fri Jan 20 00:31:01 2006 From: Norbert.Nemec.list at gmx.de (Norbert Nemec) Date: Fri Jan 20 00:31:01 2006 Subject: [Numpy-discussion] Matrix/Vector norm? Message-ID: <43D09F8F.6080702@gmx.de> Hi there, im missing a norm function in NumPy. Writing one by hand is simple, but doing so both general and efficient is a lot more tricky. Ideas? Norbert From arnd.baecker at web.de Fri Jan 20 00:53:01 2006 From: arnd.baecker at web.de (Arnd Baecker) Date: Fri Jan 20 00:53:01 2006 Subject: [Numpy-discussion] Time for a new release of NumPy In-Reply-To: <43D01EC7.3040203@ieee.org> References: <43D01EC7.3040203@ieee.org> Message-ID: On Thu, 19 Jan 2006, Travis Oliphant wrote: > I'd like to make a release of NumPy over the weekend. Please submit > bugs to the list before Saturday night (GMT -7) > > NumPy 0.9.2 was based on SVN version 1829 and we are over 100 checkins > from that point, including the significant change to the .dtype > attribute.... I would like to get numpy working with intels compiler icc - However, as long as no one can tell me where to get information on my scipy distutils question/problem http://www.scipy.org/mailinglists/mailman?fn=scipy-dev/2006-January/005138.html I can't help further ;-) (Personally I don't need an icc compiled one at the moment, I can always install gcc, but ...) Best, Arnd P.S.: I just saw that on http://www.scipy.org/ there is still reference to new core/ new scipy etc: """ This site does not reflect the rapid growth of scipy that has taken place while a new core scipy array object has been built to replace Numeric. This link provides more information on where to get the new scipy core. A version of scipy that works on newcore is available for anyonymous check out from a subversion repostitory at http://svn.scipy.org/svn/scipy/trunk. File downloads are available on this site or from sourceforge here. """ It would be great if this could be corrected (BTW, what about a new round of name changing - it was so much fun ...;-) From august at attorney.com Fri Jan 20 01:40:01 2006 From: august at attorney.com (=?windows-1251?B?NyAtIDgg9OXi8ODr/yAyMDA2IOPu5OA=?=) Date: Fri Jan 20 01:40:01 2006 Subject: [Numpy-discussion] =?windows-1251?B?z9DAytLI18XRysjJIMzA0MrF0sjNww==?= Message-ID: ?????????? ???-?????????? ? ??????????? ????????????? ?????????? ? ???????, ??????, ??????????????? ???????? ? ?????????????? ?????????? ??????? ??????? ? ????? ?????????? ????????: ???????????? ????????? ????????? ????????? ??????????? ?? ?????????? ? ????? ??????????? ????????? ???????????????? ???? ? ?????????? ???????? 7 - 8 ??????? 2006 ???? ?? ??????? ?? ???????? ??????????? ???????????? ????????? ?????????? ???????? ? ?????? ?????????????? ? ??????? ?? ???????????????, ????? ??????????? ???? ?????????? ??????-????????? ?? ???? ???????? ???????????? ???????????? ? ???????? ?? ??????????. ???? ? ????? ?????? ?????????? ? ??????????????? ????????? ???????? ??????????? ????????? ?????????? ? ????????????????????? ????????. ?????? ? ?????????. ?????????? ????? ??????????. ???????? ???????????????? ?????? ?????????? ? ???????? ? ?? ?????????. ???????? ????? ? ???????????????? ???????? ?? ??? ??????? ????? ??????????. ????????? ?????????????? ? ???????????? ??????? ????????? ??????????. ????? ?????????????? ???????? ? ?????????????. ???? ???????????? ? ?? ????????. ???????? ?????? "??????????" ?????. ?????????? ? ???????? ???????? CRM. ?????????????? ???????? ? ????????? ????????????????????? ???????? ??????????????? ???????? ??????/??????. ????????? ? ?????????? ????? ????? ???????/?????. ????? ?????????? ?1 - ???????????? ???????. ??????????? ??????????? ???????? ??????????????? ? ????????????? ??? ????????????? ????????. ??????????????? ? ??????????? ????? ??????/???????????? ???? ? ???????? ????????/??????. ??????? ?????????? ????, ??????? ??????, ????? ?????? ? ????????????. ??????????? ????????/?????? ?? ????? ? ??????????? ??????????? ??????????????? ? ????????????? ? ?????????? ???????????????????????? ????????. ??????????? ???????????? ?????????????? ?????? ?????????? ? ????????. ????????????? ???????????? ? ?? ???? ? ????????? ????????????????????? ???????? ??????? ??? ?????????? ????????? ????????????????? ???????????? ? ??????/??????. ??? ???????? ??????????? ?????. ??????? ?????? ?????????? ????????? ????????. ????? ?????????? ?2 - ???????????? ???????. ?????????? ????????? ???????????? ???????????? ? ???????? ????????? ?? ???????????. ?????????? ????? ????????? ????????. ?????????????? ?????? ??? ??????????: CD ???? ? ?????? ???????? ??????????????? ?????????? ????? ? ????????????? ?????????, ????? ?????? ???????? "?????????: ????????? ??????-????????". ????? ???????? - ??????????? ???????? ?????????????? ????????. ???????????????? ??????????? ? ????? ????????????? ??????????, ???????????, PR, Direct-marketing, Sale promotions ? ????????????? ???????? ??????-????????? ??? ??????????? ? ???? ????????? ???? ????????????, ????? ???? ????????? ?????????????? ????????????? ? ????????? ????????, ???????? ?? ??????????? ?????????. ????? ???? "???????????? ?????????. ??????? ??????? ????????????", "???????????? ????????? 2. ??????????? ?? ?????????" ? "???????? ? ???????? ??????????", ?????????????? ?????? ? ?????????? ? ??????? ????????????? ???????????????? ????????, ???? ?????????????? ????????? ?????????? ?????????. ????? ??????? ??????? ? ???????? ??? ?????? ???????????, ????????? ?? ?????????: (495) 980-65-36, 980-65-39 ?????? ????????? ?? ?????? ???????? ? ????????????? ????????, ? ?????? ????????? ?????? ??????? ? ?????? ?????????? ??????. -------------- next part -------------- An HTML attachment was scrubbed... URL: From MAILER-DAEMON at ibspublic.ibs.fr Fri Jan 20 03:30:10 2006 From: MAILER-DAEMON at ibspublic.ibs.fr (Mail Delivery Subsystem) Date: Fri Jan 20 03:30:10 2006 Subject: [Numpy-discussion] Returned mail: see transcript for details Message-ID: <200601201129.k0KBTNa7005336@ibspublic.ibs.fr> The original message was received at Fri, 20 Jan 2006 12:29:12 +0100 from adsl196-62-32-217-196.adsl196-10.iam.net.ma [196.217.32.62] ----- The following addresses had permanent fatal errors ----- (reason: 550 5.1.1 ... User unknown) ----- Transcript of session follows ----- ... while talking to ibsprive.ibs.fr.: >>> RCPT To: <<< 550 5.1.1 ... User unknown 550 5.1.1 ... User unknown -------------- next part -------------- An embedded message was scrubbed... From: unknown sender Subject: no subject Date: no date Size: 38 URL: From numpy-discussion at lists.sourceforge.net Fri Jan 20 06:29:09 2006 From: numpy-discussion at lists.sourceforge.net (numpy-discussion at lists.sourceforge.net) Date: Fri, 20 Jan 2006 12:29:09 +0100 Subject: hello Message-ID: <200601201129.k0KBTBa7005322@ibspublic.ibs.fr> An embedded and charset-unspecified text was scrubbed... Name: warning1.txt URL: -------------- next part -------------- The message contains Unicode characters and has been sent as a binary attachment. From aisaac at american.edu Fri Jan 20 07:03:03 2006 From: aisaac at american.edu (Alan G Isaac) Date: Fri Jan 20 07:03:03 2006 Subject: [Numpy-discussion] Bug? numpy.matrix.sum() In-Reply-To: <43D09D6E.5070907@gmx.de> References: <43D09D6E.5070907@gmx.de> Message-ID: On Fri, 20 Jan 2006, Norbert Nemec apparently wrote: > In [1]: import numpy > In [2]: numpy.__version__ > Out[2]: '0.9.2' > In [3]: a = numpy.matrix(numpy.eye(2)) > In [4]: a.sum() > Out[4]: matrix([[1, 0, 0, 1]]) > In [5]: a.A.sum() > Out[5]: 2 > In [6]: sum(a) > Out[6]: matrix([[1, 1]]) > In [7]: sum(a.A) > Out[7]: array([1, 1]) Only [4] looks wrong. However it works "right" if you provide a numerical axis argument. (I put "right" in quotes because it is not clear to me that a row vector is expected as the result when summing columns.) The default is axis=None, which is not working as expected for matrices. Cheers, Alan Isaac From cjw at sympatico.ca Fri Jan 20 07:40:02 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Fri Jan 20 07:40:02 2006 Subject: [Numpy-discussion] Interesting timing results In-Reply-To: <43D00699.10001@ieee.org> References: <43CD7E3A.8070404@ieee.org> <43D00699.10001@ieee.org> Message-ID: <43D10404.5030107@sympatico.ca> Travis Oliphant wrote: > Sasha wrote: > >> On 1/17/06, Travis Oliphant wrote: >> >> >>> ... Currently all array scalar math goes >>> through the >>> entire ufunc machinery. This is clearly sub-optimal. It is the >>> reason >>> for the scalarmath module that I've started in the src directory. >>> Eventually, we should be able to have scalar math as fast as Python >>> scalars. >>> >> >> >> I have implemented "nonzero", &, | and ^ for scalar bools. (See >> http://projects.scipy.org/scipy/numpy/changeset/1946). Here are the >> timings: >> >> Before: >> >> >>> python -m timeit -s "from numpy import bool_; x = bool_(0)" "x & x" >>> >> >> 100000 loops, best of 3: 3.85 usec per loop >> >> Now: >> >> >>> python -m timeit -s "from numpy import bool_; x = bool_(0)" "x & x" >>> >> >> 10000000 loops, best of 3: 0.174 usec per loop >> >> This is close to python bool: >> >> >>> python -m timeit -s "x = bool(0)" "x & x" >>> >> >> 10000000 loops, best of 3: 0.142 usec per loop >> >> and faster than python int: >> >> >>> python -m timeit -s "from numpy import bool_; x = 0" "x & x" >>> >> >> 10000000 loops, best of 3: 0.199 usec per loop >> >> But it is in fact all within the timing error now. >> >> I did not put it in the scalarmath module for two reasons: first, >> scalarmath is not hooked to numpy yet and second because C-API does >> not provide access to scalar bools yet. I have posted a proposal for >> C-API changes and did not hear any opposition (well, no support >> either), so I will implement that soon. >> >> There are a few issues with the new APIs that I proposed. First is a >> simple one: I proposed to expose boolean scalars as named constants to >> Python, the question is >> how to call them. >> >> 1. True_ and False_ >> >> > +1 Why not True and False? >>> type(True) >>> Colin W. > >> 2. true and false >> >> > -1 > >> The second issue is whether to add new numbers to _ARRAY_API or add a >> separate _ARRAY_SCALAR_API . >> >> > No separate _SCALAR_API.... > >> I've only optimized scalar-scalar case in binary ops and fall back to >> old for everything else. I would like to optimize scalar-array and >> array-scalar cases as well, but I wonder if it is apropriate to make >> "(bool_(0) | x) is x" true when x is an array. Alternatively >> (bool_(0) | x) can become a new array that shares data with x. >> >> > Other operations with scalars return a new array. The fact that this > would be different would probably be a bad thing in the end. So, I > vote for returning a new copy of the data... > > -Travis > > > > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log > files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From cjw at sympatico.ca Fri Jan 20 07:43:06 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Fri Jan 20 07:43:06 2006 Subject: [Numpy-discussion] Time for a new release of NumPy In-Reply-To: <43D01EC7.3040203@ieee.org> References: <43D01EC7.3040203@ieee.org> Message-ID: <43D104EE.1070208@sympatico.ca> Travis Oliphant wrote: > > I'd like to make a release of NumPy over the weekend. Please submit > bugs to the list before Saturday night (GMT -7) > > NumPy 0.9.2 was based on SVN version 1829 and we are over 100 checkins > from that point, including the significant change to the .dtype > attribute.... Will there be some note with the release explaining this, or do we have to browse the discussion to get the detail? Colin W. From faltet at carabos.com Fri Jan 20 07:51:07 2006 From: faltet at carabos.com (Francesc Altet) Date: Fri Jan 20 07:51:07 2006 Subject: [Numpy-discussion] Time for a new release of NumPy In-Reply-To: <43D104EE.1070208@sympatico.ca> References: <43D01EC7.3040203@ieee.org> <43D104EE.1070208@sympatico.ca> Message-ID: <200601201650.41924.faltet@carabos.com> A Divendres 20 Gener 2006 16:42, Colin J. Williams va escriure: > Travis Oliphant wrote: > > I'd like to make a release of NumPy over the weekend. Please submit > > bugs to the list before Saturday night (GMT -7) > > > > NumPy 0.9.2 was based on SVN version 1829 and we are over 100 checkins > > from that point, including the significant change to the .dtype > > attribute.... > > Will there be some note with the release explaining this, or do we have > to browse the discussion to get the detail? Here you have the original explanation from Travis posted in this list a couple of days ago: ----------- There was some cruft left over from the change to making data-type descriptors real Python objects. This left lots of .dtype related attributes on the array object --- too many as Francesc Altet graciously pointed out. In the latest SVN, I've cleaned things up (thanks to a nice patch from Francesc to get it started). Basically, there is now only one attribute on the array object dealing with the data-type (arr.dtype). This attribute returns the data-type descriptor object for the array. This object itself has the attributes .char, .str, and .type (among others). I think this will lead to less confusion long term. The cruft was due to the fact that my understanding of the data-type descriptor came in December while seriously looking at records module. This will have some backward-compatibility issues (we are still pre-1.0 and early enough that I hope this is not too difficult to deal with). The compatibility to numpy-0.9.2 issues I can see are: 1) Replacing attributes that are now gone: .dtypechar --> .dtype.char .dtypestr --> .dtype.str .dtypedescr --> .dtype 2) Changing old .dtype -> .dtype.type This is only necessary if you were using a.dtype as a *typeobject* as in issubclass(a.dtype, ) If you were using .dtype as a parameter to dtype= then that usage will still work great (in fact a little faster) because now .dtype returns a "descriptor object" 3) The dtypedescr constructor is now called dtype. This change should have gone into the 0.9.2 release, but things got too hectic with all the name changes. I will quickly release 0.9.4 with these changes unless I hear strong disagreements within the next few days. -Travis P.S. SciPy SVN has been updated and fixed with the changes. Numeric compatibility now implies that .typecode() --> .dtype.char although if .typecode() was used as an argument to a function, then .dtype will very likely work. -Travis -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From oliphant.travis at ieee.org Fri Jan 20 08:42:01 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri Jan 20 08:42:01 2006 Subject: [Numpy-discussion] Speed performance on array constant set In-Reply-To: <43D065AF.3080307@mitre.org> References: <43D0573E.4090301@mitre.org> <43D05D99.1060200@ieee.org> <43D065AF.3080307@mitre.org> Message-ID: <43D06AEA.50707@ieee.org> Mark Heslep wrote: > Travis Oliphant wrote: > >> This is actually a bit surprising that opencv can create and fill so >> quickly. Perhaps they are using optimized SSE functions for the >> Intel platform, or something? >> -Travis >> > Ah, sorry, Im an unintentional fraud. Yes I have Intel's optimization > library IPP turned on and had forgotten about it. So one more time: > > With IPP on as before. UseOptimized = # of Cv functions available w/ > IPP > >> python -m timeit -s "import opencv.cv as cv; print >> cv.cvUseOptimized(1); im =cv.cvCreateImage(cv.cvSize(1000,1000), 8, >> 1)" "cv.cvSet( im, cv.cvRealScalar( 7 ) )" >> 305 >> 305 >> 305 >> 305 >> 305 >> 100 loops, best of 3: 2.24 msec per loop > > > And without: > >> python -m timeit -s "import opencv.cv as cv; print >> cv.cvUseOptimized(0); im =cv.cvCreateImage(cv.cvSize(1000,1000), 8, >> 1)" "cv.cvSet( im, cv.cvRealScalar( 7 ) )" >> 0 >> 0 >> 0 >> 0 >> 0 >> 100 loops, best of 3: 6.94 msec per loop > > > So IPP gives me 3X, which leads me to ask about plans for IPP / SSE > for NumPy, no offense intended to non Intel users. I believe I recall > some post that auto code generation in NumArray was the road block? There was some talk of using liboil for this (similar to what _dotblas does). There could definitely be some gains. I don't see any road block other than time and effort.... With my own tests of a ctypes-wrapped function that just mallocs memory and fills it, I put numpy at about 3x slower than that function. The scalar fill function of numpy could definitely be made faster. Right now, it's still pretty generic. -Travis From oliphant.travis at ieee.org Fri Jan 20 08:42:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri Jan 20 08:42:02 2006 Subject: [Numpy-discussion] Speed performance on array constant set In-Reply-To: <43D0573E.4090301@mitre.org> References: <43D0573E.4090301@mitre.org> Message-ID: <43D05BA0.7000700@ieee.org> Mark Heslep wrote: > Im doing some work with the OpenCv* project. Im using swig typemaps > to convert the Cv data structures to numarray which works well. Id > like to restrict Cv use to what its strengths: complicated vision > processing algorithms like optical flow. For the case of simple Cv > data manipulations, I'd rather use NumPy functions & methods but was > surprised at the performance comparison. > - A simple scalar constant fill with cvSet. 'im' here is a wrapped Cv > image data structure. > >> python -m timeit -s "import opencv.cv as cv; im = >> cv.cvCreateImage(cv.cvSize(1000,1000), 8, 1)" "cv.cvSet( im, >> cv.cvRealScalar( 7 ) )" >> 100 loops, best of 3: 2.58 msec per loop > > > - If I try the equivalent with NumPy > >> python -m timeit -s "import numarray as na; a = na.zeros((1000,1000) >> )" "a[:,:] = 7" > > >> 10 loops, best of 3: 45.1 msec per loop > > > A >10x hit. Am I using the preferred / optimal NumPy method here? I > scanned the earlier Scalar posts but thought that was boolean type > only issue. > First of all, try using NumPy instead of Numarray: import numpy as na Second: use math (i.e. a += 7) Third: There are specialized fill routines .fill() in numpy and a simliar routine in Numarray that can be faster then indexing. Best, -Travis From oliphant.travis at ieee.org Fri Jan 20 08:42:03 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri Jan 20 08:42:03 2006 Subject: [Numpy-discussion] Matrix/Vector norm? In-Reply-To: <43D09F8F.6080702@gmx.de> References: <43D09F8F.6080702@gmx.de> Message-ID: <43D0A290.8080600@ieee.org> Norbert Nemec wrote: >Hi there, > >im missing a norm function in NumPy. Writing one by hand is simple, but >doing so both general and efficient is a lot more tricky. > >Ideas? > > There's one in scipy (i'm not sure if it's the best of breed, but it's a starting point). Look in scipy.linalg.norm -Travis From oliphant.travis at ieee.org Fri Jan 20 08:42:04 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri Jan 20 08:42:04 2006 Subject: [Numpy-discussion] Bug? numpy.matrix.sum() In-Reply-To: <43D09D6E.5070907@gmx.de> References: <43D09D6E.5070907@gmx.de> Message-ID: <43D0A247.7000003@ieee.org> Norbert Nemec wrote: >Is it just me who finds this confusing? There is some bug there, but I >cannot even say what it is supposed to behave like... > >--------------------------------- >In [1]: import numpy > >In [2]: numpy.__version__ >Out[2]: '0.9.2' > >In [3]: a = numpy.matrix(numpy.eye(2)) > >In [4]: a.sum() >Out[4]: matrix([[1, 0, 0, 1]]) > > > Bug... Fixed in SVN. -Travis From Chris.Barker at noaa.gov Fri Jan 20 09:36:09 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri Jan 20 09:36:09 2006 Subject: [Numpy-discussion] Speed performance on array constant set In-Reply-To: <43D065AF.3080307@mitre.org> References: <43D0573E.4090301@mitre.org> <43D05D99.1060200@ieee.org> <43D065AF.3080307@mitre.org> Message-ID: <43D11F90.30602@noaa.gov> Sasha wrote: >>python -m timeit -s "import numpy as na; a = na.zeros((1000,1000))" "a.fill(7)" or use a += 7: $ python2.4 -m timeit -s "import numpy as na; a = na.zeros((1000,1000))" "a.fill(7)" 100 loops, best of 3: 6.95 msec per loop $ python2.4 -m timeit -s "import numpy as na; a = na.zeros((1000,1000))" "a += 7" 100 loops, best of 3: 3.24 msec per loop A factor of 2 speedup for me. I don't know why fill is slower. > So IPP gives me 3X, which leads me to ask about plans for IPP / SSE for > NumPy, no offense intended to non Intel users. I"ve wondered about this as as well, though not necessarily IPP / SEE. It seems that BLAS should provide some optimizations that could be used outside of the strictly linear algebra functions, like element-wise multiplication, array copying, etc. However, I haven't looked into it, and I suppose it would make for a lot of special-case code. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From cookedm at physics.mcmaster.ca Fri Jan 20 09:48:03 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Fri Jan 20 09:48:03 2006 Subject: [Numpy-discussion] Numpy x Matlab: some synthetic benchmarks In-Reply-To: <43CEC67B.1090600@colorado.edu> (Fernando Perez's message of "Wed, 18 Jan 2006 15:51:39 -0700") References: <1137584647.19613.17.camel@localhost.localdomain> <43CE85AD.1090904@ieee.org> <1137610937.19613.65.camel@localhost.localdomain> <43CE985E.9050406@astraw.com> <43CE9B65.5040803@ee.byu.edu> <43CEBA67.8000104@colorado.edu> <43CEC67B.1090600@colorado.edu> Message-ID: Fernando Perez writes: > David M. Cooke wrote: > >> I've done a little bit of work along these lines. I have a module I >> call vector3 [*] which has 2- and 3-dimensional immutable vectors, >> using either ints or doubles. It's as fast as I could make it, while >> keeping it all written in Pyrex. I find it very convenient for >> anything vector-related. Konrad Hinsen has something similiar in the >> development version of his ScientificPython package. >> [*] http://arbutus.mcmaster.ca/dmc/software/vector3.html >> Also, I've also done some playing around with a n-dimensional vector >> type (restricted to doubles). My best attempts make it ~4-5x faster >> than numpy (and 2x faster than Numeric) for vectors of dimension 10 >> on simple ops like + and *, 2x faster than numpy for dimension 1000, >> and approaching 1x as you make the vectors larger. Indexing is about >> 3x faster than numpy, and 1.4x faster than Numeric. So that gives I >> think some idea of the maximum speed-up possible. >> I think the speedups mostly come from the utter lack of any >> polymorphism: it handles vectors of doubles only, and only as >> contiguous vectors (no strides). > > This is excellent, thanks for the pointer. I can see uses for vectors > (still 1-d, no strides, etc) with more than 3 elements, and perhaps > fixed-size (no reshaping, no striding) 2-d arrays (matrices), but this > looks like a good starting point. Sandbox material? Well, I'd be pleased to donate vector3 to scipy as a sandbox (although I think it's very useful as a standalone package for others with no need of full scipy. The general vector stuff is still quite rough, and at this point is really only meant as a study on how fast I can make something like that run :-) -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From faltet at carabos.com Fri Jan 20 09:59:04 2006 From: faltet at carabos.com (Francesc Altet) Date: Fri Jan 20 09:59:04 2006 Subject: [Numpy-discussion] Segmentation fault when dealing with nested records Message-ID: <200601201858.34777.faltet@carabos.com> Hi, I'm starting to write a series of torture test for numpy and (nested) heterogeneous arrays. I've found a first problem that causes a segfault: In [7]: dtype=[('x', '0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From Ed.Schofield at ftw.at Fri Jan 20 10:17:02 2006 From: Ed.Schofield at ftw.at (Schofield, Ed) Date: Fri Jan 20 10:17:02 2006 Subject: [Numpy-discussion] Re: array vs. matrix for matlab-compatible stuff Message-ID: (Apologies if this is a duplicate. My mail isn't getting through ...) Travis Oliphant wrote: >> Sven Schreiber wrote: >> >> I see, thanks for the quick answer. So wouldn't it be a good idea to >> have all the specifically >> matrix-related stuff (afaics, things in numpy/lib/twodim_base.py) >> return matrices? >> >> It seems my question (or misunderstanding) has a broader scope: >> Coming from matrix languages, I'm >> glad about short notations like A.I or A*B representing standard >> matrix operations. Much easier than >> linalg.inverse(A) or matrixmultiply(A,B). However, many matrix >> functions (decompositions etc.) seem >> to return arrays instead of matrices, even if you feed them matrices >> (is this assumption correct?). >> So you have to use mat(returned-result) again and again to be able to >> do return-result.I afterwards, >> which seems clumsy. So what's the best strategy here? > > Yes, some of them do still return arrays. Matrices are longer lived > in NumPy then the were in Numeric, for sure, but many functions still > aren't friendly to matrices and convert all inputs to arrays before > operation. Originally, I had the asarray(...) function not convert > matrices by default, but this is too surprising because matrices > change the '*' and '**' operators which could make your function not > work. > We should convert all the functions that don't handle matrices so they > will. I'd like to see matrices survive longer than they do. There > are some functions that try to do that I agree, and I'm willing to help with this. We should also think about handling SciPy's spmatrix objects properly. In this case we don't want NumPy to have any dependency on a particular format for SciPy's spmatrix objects -- but we could design and publish a simple interface for sparse matrix objects (from SciPy or anywhere else) to conform to for basic NumPy compatibility. How should we go about this? Let's take linalg.solve_linear_equations() as an example. We could use isinstance(x, matrix) for dense matrix objects, because they're part of NumPy, then use asarray(x), process them as arrays, and wrap them up with asmatrix(x) at the end. But with sparse matrices this wouldn't work. What if we specify instead that objects passed to functions like solve_linear_equations() need to provide two members: x.toarray(), allowing x to represent itself as an array x.fromarray(a), creating another object of the same type as x out of the given array a We could choose other names, of course, maybe asarray() or whatever else. But this would simplify the code for all these functions while allowing us to support other array-like objects without needing to know any more about them. I'm also +1 on eye() returning a matrix :) -- Ed -------------- next part -------------- An HTML attachment was scrubbed... URL: From dd55 at cornell.edu Fri Jan 20 10:25:05 2006 From: dd55 at cornell.edu (Darren Dale) Date: Fri Jan 20 10:25:05 2006 Subject: [Numpy-discussion] numpy license file Message-ID: <200601201324.34947.dd55@cornell.edu> I also apologize if this is a duplicate, my mail isnt getting through either. I'm looking for the license file in svn numpy, but cant find it. a grep of "license" on the numpy directory turned up several references to terms of the SciPy license, and to the missing LICENSE.txt. Some folks are working on a gentoo ebuild for numpy/scipy in gentoo's bugzilla, and raised questions about numpy's license. Darren From ghall at research.dfci.harvard.edu Fri Jan 20 11:01:08 2006 From: ghall at research.dfci.harvard.edu (Giles Hall) Date: Fri Jan 20 11:01:08 2006 Subject: [Numpy-discussion] Bugfix for arraybase.h Message-ID: <43D13347.3050309@research.dfci.harvard.edu> Hi, This simple python code snippit used to core dump on my modern AMD 64 Linux servers: from numarray import * a = zeros( 10, Float32 ) b = transpose(a) dot( b,a ) This crash was caused by a corrupted "libnumarray_API". The pointer was initially assigned the correct value, but becomes corrupted almost immediately after its initial runtime binding. On my systems, line 725 of _dotblas.c would clobber the "libnumarray_API" pointer while initializing the "dotFunctions" vector table. the problem is "dotFunctions", an array of function pointers, is sized statically by "PyArray_NTYPES", an enumerated constant that reflects the value of "tMaxType" defined in arraybase.h like this: typedef enum { tAny, tBool, tInt8, tUInt8, tInt16, tUInt16, tInt32, tUInt32, tInt64, tUInt64, tFloat32, tFloat64, tComplex32, tComplex64, tObject, /* placeholder... does nothing */ tDefault = tFloat64, #if LP64 tLong = tInt64, #else tLong = tInt32, #endif tMaxType } NumarrayType; The problem is that "tMaxType" is not assigned 17, like we would expect, since it's at the end of the list. On my machine, "tMaxType" is assigned the value 9! The problem is the explicit assignment of "tLong" and "tDefault". Explicitly assigned constants in an enumerated list will modify the running count. Since the macro "LP64" is defined, "tLong" is assigned value of "tInt64", which is 8, and "tMaxType" is assigned the next value, 9. This means the vector table "dotFunctions" is shorter then we intend, and overflowing this table can corrupt important data. The fix is simple, just define tMaxType before the explicitly assigned aliases: typedef enum { tAny, tBool, tInt8, tUInt8, tInt16, tUInt16, tInt32, tUInt32, tInt64, tUInt64, tFloat32, tFloat64, tComplex32, tComplex64, tObject, /* placeholder... does nothing */ tMaxType, tDefault = tFloat64, #if LP64 tLong = tInt64, #else tLong = tInt32, #endif } NumarrayType; From faltet at carabos.com Fri Jan 20 11:06:06 2006 From: faltet at carabos.com (Francesc Altet) Date: Fri Jan 20 11:06:06 2006 Subject: [Numpy-discussion] Time for a new release of NumPy In-Reply-To: <43D104EE.1070208@sympatico.ca> References: <43D01EC7.3040203@ieee.org> <43D104EE.1070208@sympatico.ca> Message-ID: <200601202005.12065.faltet@carabos.com> A Divendres 20 Gener 2006 16:42, Colin J. Williams va escriure: > Travis Oliphant wrote: > > I'd like to make a release of NumPy over the weekend. Please submit > > bugs to the list before Saturday night (GMT -7) > > > > NumPy 0.9.2 was based on SVN version 1829 and we are over 100 checkins > > from that point, including the significant change to the .dtype > > attribute.... > > Will there be some note with the release explaining this, or do we have > to browse the discussion to get the detail? Colin told me that my previous message got truncated. I'm sending now a link to the original message from Travis: http://aspn.activestate.com/ASPN/Mail/Message/numpy-discussion/2979474 Hope this time it would look well. -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From ivazquez at ivazquez.net Fri Jan 20 11:51:05 2006 From: ivazquez at ivazquez.net (Ignacio Vazquez-Abrams) Date: Fri Jan 20 11:51:05 2006 Subject: [Numpy-discussion] numpy license file In-Reply-To: <200601201324.34947.dd55@cornell.edu> References: <200601201324.34947.dd55@cornell.edu> Message-ID: <1137786625.910.3.camel@ignacio.lan> On Fri, 2006-01-20 at 13:24 -0500, Darren Dale wrote: > I also apologize if this is a duplicate, my mail isnt getting through either. > > I'm looking for the license file in svn numpy, but cant find it. a grep of > "license" on the numpy directory turned up several references to terms of the > SciPy license, and to the missing LICENSE.txt. > > Some folks are working on a gentoo ebuild for numpy/scipy in gentoo's > bugzilla, and raised questions about numpy's license. The Numpy code itself is BSD but the included f2py is LGPL, so it seems that the entire package is LGPL. -- Ignacio Vazquez-Abrams http://fedora.ivazquez.net/ gpg --keyserver hkp://subkeys.pgp.net --recv-key 38028b72 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From oliphant at ee.byu.edu Fri Jan 20 14:05:02 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri Jan 20 14:05:02 2006 Subject: [Fwd: Re: [Fwd: Re: [Numpy-discussion] numpy license file]] Message-ID: <43D15D60.3020300@ee.byu.edu> -------------- next part -------------- An embedded message was scrubbed... From: unknown sender Subject: no subject Date: no date Size: 38 URL: From pearu at scipy.org Fri Jan 20 15:47:28 2006 From: pearu at scipy.org (Pearu Peterson) Date: Fri, 20 Jan 2006 14:47:28 -0600 (CST) Subject: [Fwd: Re: [Numpy-discussion] numpy license file] In-Reply-To: <43D1562C.5070103@ee.byu.edu> References: <43D1562C.5070103@ee.byu.edu> Message-ID: On Fri, 20 Jan 2006, Travis Oliphant wrote: > Can we release f2py in numpy under a BSD license so as not to encumber all of > numpy? Yes. Pearu --------------070907070601070307030400-- From dd55 at cornell.edu Fri Jan 20 14:08:30 2006 From: dd55 at cornell.edu (Darren Dale) Date: Fri Jan 20 14:08:30 2006 Subject: [Numpy-discussion] numpy license file In-Reply-To: <1137786625.910.3.camel@ignacio.lan> References: <200601201324.34947.dd55@cornell.edu> <1137786625.910.3.camel@ignacio.lan> Message-ID: <200601201702.28089.dd55@cornell.edu> On Friday 20 January 2006 14:50, Ignacio Vazquez-Abrams wrote: > On Fri, 2006-01-20 at 13:24 -0500, Darren Dale wrote: > > I also apologize if this is a duplicate, my mail isnt getting through > > either. > > > > I'm looking for the license file in svn numpy, but cant find it. a grep > > of "license" on the numpy directory turned up several references to terms > > of the SciPy license, and to the missing LICENSE.txt. > > > > Some folks are working on a gentoo ebuild for numpy/scipy in gentoo's > > bugzilla, and raised questions about numpy's license. > > The Numpy code itself is BSD but the included f2py is LGPL, so it seems > that the entire package is LGPL. I don't want this to seem disrespectful, but could I get a second opinion on this? From oliphant at ee.byu.edu Fri Jan 20 14:17:04 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri Jan 20 14:17:04 2006 Subject: [Numpy-discussion] numpy license file In-Reply-To: <1137786625.910.3.camel@ignacio.lan> References: <200601201324.34947.dd55@cornell.edu> <1137786625.910.3.camel@ignacio.lan> Message-ID: <43D1613F.6050308@ee.byu.edu> Ignacio Vazquez-Abrams wrote: >On Fri, 2006-01-20 at 13:24 -0500, Darren Dale wrote: > > >>I also apologize if this is a duplicate, my mail isnt getting through either. >> >>I'm looking for the license file in svn numpy, but cant find it. a grep of >>"license" on the numpy directory turned up several references to terms of the >>SciPy license, and to the missing LICENSE.txt. >> >>Some folks are working on a gentoo ebuild for numpy/scipy in gentoo's >>bugzilla, and raised questions about numpy's license. >> >> > >The Numpy code itself is BSD but the included f2py is LGPL, so it seems >that the entire package is LGPL. > > Pearu has agreed to release the numpy.f2py under the same license as NumPy. Thus, there should be no problem regardless of the veracity of this implication. -Travis From oliphant at ee.byu.edu Fri Jan 20 14:28:04 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri Jan 20 14:28:04 2006 Subject: [Numpy-discussion] Time for a new release of NumPy In-Reply-To: References: <43D01EC7.3040203@ieee.org> <43D036AC.7030503@ieee.org> <43D04E3C.4020108@ieee.org> Message-ID: <43D163E1.4050509@ee.byu.edu> Sasha wrote: >On 1/19/06, Travis Oliphant wrote: > > >>... >>I don't think this is right. zero-rank float arrays use Python's >>floating-point str or repr function to produce the string, so whatever >>Python is doing will be used. >> >> > >Well, it looks like they use repr for str: > > > >>>>from numpy import * >>>>len(str(1/3.)) >>>> >>>> >14 > > >>>>len(str(array(1/3.))) >>>> >>>> >19 > > >>>>len(repr(1/3.)) >>>> >>>> >19 > > > O.K. That is a bug. And it is now fixed (an oversight...) -Travis From rowen at cesmail.net Fri Jan 20 14:44:05 2006 From: rowen at cesmail.net (Russell E. Owen) Date: Fri Jan 20 14:44:05 2006 Subject: [Numpy-discussion] Re: Efficient way to handle None->nan? References: Message-ID: In article , Sasha wrote: > >>> from numpy.core.ma import masked values > >>> from numpy import nan > >>> masked values([1.0,None,2.0],None).filled(nan).astype(float) > array([ 1. , nan, 2. ]) Neat! Any idea if that's likely to keep working in future versions? It doesn't work in numarray.ma.masked_values and in general it seems like a lot of numpy and numarray raise exception when they get a list that contains None. -- Russell From bsouthey at gmail.com Fri Jan 20 14:48:03 2006 From: bsouthey at gmail.com (Bruce Southey) Date: Fri Jan 20 14:48:03 2006 Subject: [Numpy-discussion] numpy license file In-Reply-To: <200601201702.28089.dd55@cornell.edu> References: <200601201324.34947.dd55@cornell.edu> <1137786625.910.3.camel@ignacio.lan> <200601201702.28089.dd55@cornell.edu> Message-ID: Hi, > The Numpy code itself is BSD but the included f2py is LGPL, so it seems > that the entire package is LGPL. This is not really correct but I am not a lawyer or license expert nor a Numpy developer. Just because f2py is (or rather was) LGPL it does not automatically make all the Numpy code LGPL. If a piece of code is a derivative of f2py then it must be released as LGPL or with a more restrictive license that does not violate the LGPL but you can not release it under a less restrictive license such as BSD. But any piece of code that is not a derivative of f2py (such as being completely independent of f2py) can have it own license (that must not violate other licenses). Exactly what is a derivative is a real mess so seek legal advice. Regards Bruce On 1/20/06, Darren Dale wrote: > > On Friday 20 January 2006 14:50, Ignacio Vazquez-Abrams wrote: > > On Fri, 2006-01-20 at 13:24 -0500, Darren Dale wrote: > > > I also apologize if this is a duplicate, my mail isnt getting through > > > either. > > > > > > I'm looking for the license file in svn numpy, but cant find it. a > grep > > > of "license" on the numpy directory turned up several references to > terms > > > of the SciPy license, and to the missing LICENSE.txt. > > > > > > Some folks are working on a gentoo ebuild for numpy/scipy in gentoo's > > > bugzilla, and raised questions about numpy's license. > > > > The Numpy code itself is BSD but the included f2py is LGPL, so it seems > > that the entire package is LGPL. > > I don't want this to seem disrespectful, but could I get a second opinion > on > this? > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log > files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at ee.byu.edu Fri Jan 20 15:04:02 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri Jan 20 15:04:02 2006 Subject: [Numpy-discussion] Efficient way to handle None->nan? In-Reply-To: References: Message-ID: <43D16C31.3060209@ee.byu.edu> Russell E. Owen wrote: >We're getting numeric data from a (MySQL) database. We'd like to use >numarray or NumPy on the resulting data, but some values may be None. Is >there a fast, efficient way to replace None with NaN? I'd hate to use a >list comprehension on each data tuple before turning it into an array, >but I haven't thought of anything else. > > Current numpy SVN allows array([1.0,None,2.0],dtype=float) array([ 1. , nan, 2. ]) -Travis From dd55 at cornell.edu Fri Jan 20 15:30:03 2006 From: dd55 at cornell.edu (Darren Dale) Date: Fri Jan 20 15:30:03 2006 Subject: [Numpy-discussion] numpy license info in svn? Message-ID: <200601200901.44802.dd55@cornell.edu> I'm looking for the license file in svn numpy, but cant find it. a grep of "license" on the numpy directory turned up several references to terms of the SciPy license, and to the missing LICENSE.txt. Some folks are working on a gentoo ebuild for numpy/scipy in gentoo's bugzilla, and raised questions about numpy's license. Darren From dd55 at cornell.edu Fri Jan 20 18:41:04 2006 From: dd55 at cornell.edu (Darren Dale) Date: Fri Jan 20 18:41:04 2006 Subject: [Numpy-discussion] fromstring feature request Message-ID: <200601202140.26335.dd55@cornell.edu> Would be possible to add a feature to numpy's fromstring function, to make it consistent with fromfile? fromfile will return an array from a binary file or an ascii file, but fromstring will only work with binary data. I would put fromstring to good use if it dealt with ascii data. Darren From apices at etang.com Fri Jan 20 19:21:03 2006 From: apices at etang.com (=?windows-1251?B?MzAg/+3i4PD/IC0gMiD05eLw4Ov/IDIwMDYg4+7k4A==?=) Date: Fri Jan 20 19:21:03 2006 Subject: [Numpy-discussion] =?windows-1251?B?z9DAytLI18XRysjJIMrT0NEgz84gws3T0tDFzc3FzNMgys7N0tDOy94gyCDBztDc?= =?windows-1251?B?wcUg0SDKztDPztDA0sjCzdvMIMzO2MXNzcjXxdHSws7M?= Message-ID: ???????????? ???? ?? ??????????? ???????? ? ?????? ? ????????????? ?????????????? 30 ?????? - 2 ??????? 2006 ???? ???? ????????????? ?????????, ???????? ?????????? ???????????, ???????? ??????????? ? ?????? ??????? ???????????? ????????? ???? ?????????????? ?????????? ? ????????, ????? ?? ???????????? ???????? ???????? ?????????? ????????. ??? ??????? ????? ????????????? ? ?????????? ?????????? ??????????? ?? ???????????? ??????? ????????? ??? ???? ? ????? ? ????????? ???????. ?????? ?? ? ??? ???????? ????? ???????? ?????????? ?????? ??????????? ????????, ??????? ???????? ??????????? ??????? ???????????? ?????, ????????? ??????????????????? ? ???????????????? ?????? ? ?????? ? ????????? ????????????? ????????? ?????????? ???????, ???????? ??? ??????????????, ?????????? ? ????????????????? ?? ??????? ???????????. ??????? ????????? ?? ????????????? ?????????? ??????????? ???????, ?????? ??????? ?????????? ? ??????????? ????????, ? ????? ????????????? ?????????? ?????. ?????????? ??????????, ??????????? ? ???????????? ?????????? ? ????????? ??????????????? ??????????????????? ????????-??????????: ???????????? ???? ?? ??????????? ???????? ? ?????? ? ????????????? ?????????????? 30 ?????? - 2 ??????? 2006 ???? ???? ?????: ??????????? ??????????? ??????? ? ?????????? ??????????? ??????????? ??????????? ? ????????? ? ??????????? ???????? ??????????? ????????, ??????? ???????? ?????????? ????? ?????????????? ??????? ????? ???????? ?????? ??????????????? ???????????? ? ?????????????? ?????????. ????????? ?? ??? ?????????????? ????????????, ????? ???????? ???????? ???????? ???????? ???????? ? ??????? ??????????? ? ????? ??????? ???????? ? ??????????? ????????????? ????? ? ???????? ???????? ?????????? ???????? ?? ???????? ?????????? ? ?????????? ????????. ?? ???????? ????? ???????? ??????-????? "????????????? ??????????? ??????????? ???????? ??? ?????? ?? ????????????? ???????? ?????????????????? ? ?????????????? ???????", ? ????? ??????????? ? ?????? ?????????? ???????? ?? ???????????? ???????? ????????. ???????: (?95) 980-65-l6, 980-65-?9 -------------- next part -------------- An HTML attachment was scrubbed... URL: From borreguero at gmail.com Sat Jan 21 09:16:06 2006 From: borreguero at gmail.com (Jose Borreguero) Date: Sat Jan 21 09:16:06 2006 Subject: [Numpy-discussion] HELP! eig and eigvals hang the python session In-Reply-To: <7cced4ed0601210820q75545136y@mail.gmail.com> References: <7cced4ed0601210820q75545136y@mail.gmail.com> Message-ID: <7cced4ed0601210915v46d38cdev@mail.gmail.com> Hi all, I don't understand what's going on. Here's my python session: $ python >>> from numpy.random import rand >>> a=rand(3,3) >>> from numpy.linalg import det,eig >>> det(a) 0.070796819514446802 >>> eig(a) and the process freezes here (at least 18minutes from now). I checked with 'top' and python is using all CPU. Any ideas, please? jose From aisaac at american.edu Sat Jan 21 09:52:08 2006 From: aisaac at american.edu (Alan G Isaac) Date: Sat Jan 21 09:52:08 2006 Subject: [Numpy-discussion] HELP! eig and eigvals hang the python session In-Reply-To: <7cced4ed0601210915v46d38cdev@mail.gmail.com> References: <7cced4ed0601210820q75545136y@mail.gmail.com><7cced4ed0601210915v46d38cdev@mail.gmail.com> Message-ID: On Sat, 21 Jan 2006, Jose Borreguero apparently wrote: >>>> eig(a) > and the process freezes here (at least 18minutes from now). I checked with > 'top' and python is using all CPU. No problem here. fwiw, Alan Isaac Python 2.4.1 (#65, Mar 30 2005, 09:13:57) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy as N >>> a=N.rand(3,3) >>> from numpy.linalg import det,eig >>> det(a) -0.1313102666029379 >>> eig(a) (array([ 1.65540878, -0.16786818, 0.47252527]), array([[-0.55573211, -0.8712454 8, 0.25278726], [-0.52611295, 0.37246933, -0.5095001 ], [-0.64371343, 0.31968409, 0.82250122]])) >>> N.__version__ '0.9.2' >>> From ndarray at mac.com Sat Jan 21 09:54:02 2006 From: ndarray at mac.com (Sasha) Date: Sat Jan 21 09:54:02 2006 Subject: [Numpy-discussion] HELP! eig and eigvals hang the python session In-Reply-To: <7cced4ed0601210915v46d38cdev@mail.gmail.com> References: <7cced4ed0601210820q75545136y@mail.gmail.com> <7cced4ed0601210915v46d38cdev@mail.gmail.com> Message-ID: You may be hitting a known problem in lapack's _geev functions that rely on computations not being performed with extra precision. See https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=138683 -- sasha On 1/21/06, Jose Borreguero wrote: > Hi all, > I don't understand what's going on. Here's my python session: > $ python > >>> from numpy.random import rand > >>> a=rand(3,3) > >>> from numpy.linalg import det,eig > >>> det(a) > 0.070796819514446802 > >>> eig(a) > and the process freezes here (at least 18minutes from now). I checked with > 'top' and python is using all CPU. > Any ideas, please? > jose > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://sel.as-us.falkag.net/sel?cmdlnk&kid3432&bid#0486&dat1642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From cjw at sympatico.ca Sat Jan 21 16:12:02 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Sat Jan 21 16:12:02 2006 Subject: [Numpy-discussion] Scalar array - a trap for the unwary Message-ID: <43D2CDA8.9030700@sympatico.ca> In the script below, b is presented as an integer, either with str or repr, but it is an array and needs further processing to treat it as in integer. I much prefer the numarray treatment, particularly in a matrix context. I suggest that, if it looks like an integer, b[1] should return a Python scalar. Script: # tScalar.py Scalar vs rank 0 import numpy.core.multiarray as _mu a= _mu.array([1, 2, 4]) print 'a:', a, 'repr(a):', repr(a), 'shape:', a.shape, 'dtype:', a.dtype b= a[1] print 'b:', b, 'repr(b):', repr(b), 'shape:', b.shape, 'dtype:', b.dtype Result: a: array([1, 2, 4], 'l') repr(a): array([1, 2, 4], 'l') shape: (3,) dtype: b: 2 repr(b): 2 shape: () dtype: Colin W. From ndarray at mac.com Sat Jan 21 16:58:00 2006 From: ndarray at mac.com (Sasha) Date: Sat Jan 21 16:58:00 2006 Subject: [Numpy-discussion] Scalar array - a trap for the unwary In-Reply-To: <43D2CDA8.9030700@sympatico.ca> References: <43D2CDA8.9030700@sympatico.ca> Message-ID: On 1/21/06, Colin J. Williams wrote: > ... > I much prefer the numarray treatment, particularly in a matrix context. > I suggest that, if it looks like an integer, b[1] should return a Python > scalar. In NumPy b[1] IS an integer: >>> from numpy import * >>> a = array([1,2,3]) >>> b = a[1] >>> isinstance(b, int) True It IS NOT rank-0 ndarray: >>> isinstance(b, ndarray) False Surely, the whole truth is that numpy scalars are instances of classes derived from Python scalars. What is the problem that numpy is causing you? If b was a python scalar, b.dtype would just raise an AttributeError. -- sasha From matthew.brett at gmail.com Sat Jan 21 17:14:16 2006 From: matthew.brett at gmail.com (Matthew Brett) Date: Sat Jan 21 17:14:16 2006 Subject: [Numpy-discussion] array vs. matrix for matlab-compatible stuff In-Reply-To: <43CFFC84.7050803@gmx.net> References: <43CE631D.1000006@gmx.net> <43CE7E68.1010908@ieee.org> <43CF8942.3050304@gmx.net> <43CFE79B.9000509@ieee.org> <43CFFC84.7050803@gmx.net> Message-ID: <1e2af89e0601211713s7790ac01m100a32e7ccbfcc55@mail.gmail.com> Hi, > Thanks very much for this useful information! I totally agree, long live the matrices... > In the ebook you are selling, are things like that (which functions preserve matrix-type) > documented? Those kind of things would be a reason for me to buy it. A bit off-topic I suppose, but can I second several previous suggestions that we all buy a copy if we can? First, it's very useful, and second to support Travis in his Herculean efforts. Best, Matthew From oliphant.travis at ieee.org Sat Jan 21 23:49:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sat Jan 21 23:49:02 2006 Subject: [Numpy-discussion] numpy-0.9.4 won't build with Python 2.3 Message-ID: <43D338B1.9080802@ieee.org> Due to last-minute use of PyOS_ascii_strtod(), the numpy-0.9.4 release won't build properly with Python 2.3. The fix is easy and I made it to release a windows binary for numpy-0.9.4 on Python 2.3. I'll put a patch up soon. If you need support for Python 2.3, get numpy out of SVN or use the patch. -Travis From NadavH at VisionSense.com Sun Jan 22 03:17:05 2006 From: NadavH at VisionSense.com (Nadav Horesh) Date: Sun Jan 22 03:17:05 2006 Subject: [Numpy-discussion] Running benchmark on AMD64: Strange results Message-ID: <43D36592.8020906@VisionSense.com> I ran benchmark.py from the "Numpy x Matlab: some synthetic benchmarks" thread (the first version), on a amd64 machine (dual-core athlon 4400+, 1GB RAM) running either win64 or gentoo linux (dual boot). And got a strange results in the dimension 500 matrix multiplication benchmark: On windows: Tests x.T*y x*y.T A*x A*B A.T*x half 2in2 Dimension: 5 Array 0.7547 0.2491 0.1939 0.2688 0.8351 0.6387 0.8398 Matrix 6.4227 1.3667 0.7070 0.8192 1.4211 2.2491 3.3690 NumArr 1.7175 3.0416 0.3539 3.0406 5.0468 4.3359 7.0414 Numeri 0.8756 0.2918 0.2639 0.3418 0.6445 0.5888 0.6953 Dimension: 50 Array 7.8431 1.5494 0.7105 12.9425 6.4525 2.1420 2.4210 Matrix 69.4642 2.8870 1.2589 13.6162 7.1357 3.7315 4.9553 NumArr 17.6435 4.6267 0.9372 33.3093 6.6129 4.7710 7.5562 Numeri 9.3534 1.6228 1.0502 12.9967 5.4694 0.9972 2.0816 Dimension: 500 Array 1.1935 6.7593 1.6090 129.6738 12.8805 1.6672 2.0153 <-- Matrix 12.6622 6.8386 1.6488 129.8621 12.6773 1.9591 2.1887 <-- NumArr 2.2546 7.5078 1.0158 545.9313 7.6133 0.6549 1.4395 <-- Numeri 1.3390 6.8766 1.2113 133.2397 12.7436 0.5740 1.7577 <-- On linux: Dimension: 5 Array 0.7500 0.1600 0.1600 0.1800 0.7200 0.5900 0.7900 Matrix 5.9600 1.1800 0.5900 0.6300 1.2700 2.1500 3.3200 NumArr 1.9400 0.4500 0.4300 0.4600 5.4100 4.6500 7.4600 Numeri 0.9100 0.2200 0.2100 0.2400 0.5200 0.4400 0.5600 Dimension: 50 Array 7.8700 1.5900 0.6900 25.9800 7.9200 2.3700 2.6300 Matrix 67.1700 2.8100 1.1500 26.5000 8.4700 3.9500 5.2000 NumArr 20.0700 1.6200 0.9500 10.4400 7.8400 5.3500 8.2600 Numeri 9.4300 1.7700 0.7400 26.1000 9.6500 0.7700 3.1500 Dimension: 500 Array 1.2200 4.6700 1.0700 1188.6400 15.3500 2.5000 2.8100 <-- Matrix 13.1100 4.7800 1.1000 1052.9200 15.3500 2.6000 2.9100 <-- NumArr 2.4700 7.5800 1.1900 76.7100 13.0600 1.8400 3.6600 <-- Numeri 1.3700 7.3900 1.2400 1068.0200 14.6800 1.3600 3.5200 <-- Numeric/numpy matrix multiplication is about 8 fold slower on linux, and about 7 fold fast with numarray. Configuration: On win64 I used the provided binaries for numarray1.5, numpy0.92 and scipy 4.4 (compiled for P4+sse2) On linux I used numarray 1.5.1 (from cvs) numpy0.92, and scipy0.44, all compiled from source for 64 bits, and linked with ATLAS 3.7.11 (linking with ACML provided roughly the same figures). Any idea were this huge performance factor came from? Nadav. From pjssilva at ime.usp.br Sun Jan 22 03:49:03 2006 From: pjssilva at ime.usp.br (Paulo Jose da Silva e Silva) Date: Sun Jan 22 03:49:03 2006 Subject: [Numpy-discussion] Running benchmark on AMD64: Strange results In-Reply-To: <43D36592.8020906@VisionSense.com> References: <43D36592.8020906@VisionSense.com> Message-ID: <1137930520.7397.6.camel@localhost.localdomain> Em Dom, 2006-01-22 ?s 12:59 +0200, Nadav Horesh escreveu: > Dimension: 500 > Array 1.2200 4.6700 1.0700 1188.6400 15.3500 2.5000 2.8100 <-- > Matrix 13.1100 4.7800 1.1000 1052.9200 15.3500 2.6000 2.9100 <-- > NumArr 2.4700 7.5800 1.1900 76.7100 13.0600 1.8400 3.6600 <-- > Numeri 1.3700 7.3900 1.2400 1068.0200 14.6800 1.3600 3.5200 <-- > I bet that Numeric and Numpy are not really accessing the ATLAS libraries or the ATLAS package you are using are using in Linux are not optimized for you architecture. The same test in my system gives: Dimension: 500 Array 1.09 8.05 1.77 151.32 1.89 3.63 3.86 Matrix 4.87 8.16 1.82 151.44 1.94 4.22 4.35 NumArr 3.12 8.19 1.80 151.66 19.83 3.94 5.30 Numeri 1.43 8.18 1.81 151.68 17.52 2.94 4.08 And my system is only a Sempron overclocked to run at 1900Mhz (and a 400Mhz bus) with an SSE optimized ATLAS. Your machine should beat mine by a large margin. Best, Paulo From NadavH at VisionSense.com Sun Jan 22 04:34:01 2006 From: NadavH at VisionSense.com (Nadav Horesh) Date: Sun Jan 22 04:34:01 2006 Subject: [Numpy-discussion] Running benchmark on AMD64: Strange results In-Reply-To: <1137930520.7397.6.camel@localhost.localdomain> References: <43D36592.8020906@VisionSense.com> <1137930520.7397.6.camel@localhost.localdomain> Message-ID: <43D3780C.8000608@VisionSense.com> An HTML attachment was scrubbed... URL: From cjw at sympatico.ca Sun Jan 22 14:25:01 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Sun Jan 22 14:25:01 2006 Subject: [Numpy-discussion] Scalar array - a trap for the unwary In-Reply-To: <43D2E182.2060201@ieee.org> References: <43D2CDA8.9030700@sympatico.ca> <43D2E182.2060201@ieee.org> Message-ID: <43D4061C.5060900@sympatico.ca> Travis Oliphant wrote: > Colin J. Williams wrote: > >> In the script below, b is presented as an integer, either with str or >> repr, but it is an array and needs further processing to treat it as >> in integer. > > > I think there is a misunderstanding here---or, at least you need to > clarify what your concern is. Travis, You are right, it was an incomplete understanding on my part. Sasha helped to explain things. > In the code you provided, b is not an array. It is a scalar that has > the attributes of an array (but it is also subclassed from a Python > integer and acts like one everywhere). Yes, it is dressed like an array but behaves as an integer. >> >> I much prefer the numarray treatment, particularly in a matrix context. > > > What do you mean by "in a matrix context?" numarray returns a simple scalar, without the ArrayType dressing. numpy would appear to be operationally equivalent and thus adequate. In exploring these matters, it seems that numpy does not broadcast scalar operations on arrays, as numarray does. Is this intended? Please see the script below and the result under that. # tArray.py from numpy.core import multiarray as _mu import numarray.numarraycore as _na naa= _na.array([1, 2]) print type(naa[1]), print naa * 3 mua= _mu.array([3, 4]) print type(mua[1]), print mua * 3 # Grief here *** Python 2.4.2 (#67, Sep 28 2005, 12:41:11) [MSC v.1310 32 bit (Intel)] on win32. *** >>> [3 6] Traceback (most recent call last): File "", line 63, in run_nodebug File "C:\Python24\Lib\site-packages\PyMatrix\Misc\Test\tArray.py", line 11, in ? print mua * 3 # Grief here TypeError: unsupported operand type(s) for *: 'numpy.ndarray' and 'int' From ljsnxgtbpd at kaufmanchaiken.com Sun Jan 22 17:19:20 2006 From: ljsnxgtbpd at kaufmanchaiken.com (Castro Bonnie) Date: Sun Jan 22 17:19:20 2006 Subject: [Numpy-discussion] Re[6]: Message-ID: <001201c61fba$eb846ba0$639d48c8@PC5> Witt Alejandro -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: hbadgjzcsd.gif Type: image/gif Size: 6142 bytes Desc: not available URL: From konrad.hinsen at laposte.net Mon Jan 23 03:50:16 2006 From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net) Date: Mon Jan 23 03:50:16 2006 Subject: [Numpy-discussion] include file location and .eggs In-Reply-To: <43CFCCF8.8000401@astraw.com> References: <43CFCCF8.8000401@astraw.com> Message-ID: <7FA168E7-4908-4A1C-937A-DF19764BD4AA@laposte.net> On Jan 19, 2006, at 18:31, Andrew Straw wrote: > For me, and I suspect for most others in favor of the new way, the > answer boils down to using eggs with out-of-the box numpy. Eggs are > the ... Thanks for explaining this issue in some detail! > = Why can't eggs use traditional include locations? = > Multiple version support. Using eggs, it is possible to have multiple > versions of the same distribution, such as numpy, installed within a > single Python installation. (I use "distribution" to mean > collection of How is multiple version support in eggs supposed to work in the case of packages that contain C modules? Let me give a concrete example: Numeric, ScientificPython, and MMTK. Numeric provides a C API through its header files. ScientificPython and MMTK both contain C modules that depend on that C API. ScientificPython also provides a C API for two of its modules, which in turn MMTK depends on. Suppose I would like to have both Numeric 23.5 and Numeric 24.2 on my machine, and also Scientific 2.4.2 and Scientific 2.5.1. I would then need four sets of the binary modules of Scientific: 1) 2.4.2 for Numeric 23.5 2) 2.4.2 for Numeric 24.2 3) 2.5.1 for Numeric 23.5 4) 2.5.1 for Numeric 24.2 I would also need four sets of the binary modules of MMTK, assuming that I am happy with just one version of MMTK. Unless some mechanism (which would have to be part of egg construction or egg installation) makes sure that the correct binaries are used for the particular combination of versions that I wish to use, I will end up having lots of mysterious crashes resulting from incompatible binary versions. If the egg mechanism does take care of such dependencies, then it could easily (and thus should) take care of selecting the right combination of header files. If it doesn't, then it should not be used at all for packages containing C or Pyrex modules, and packages like numpy should better not accomodate to eggs because in the long run that will only cause support headaches. Konrad. -- --------------------------------------------------------------------- Konrad Hinsen Laboratoire L?on Brillouin, CEA Saclay, 91191 Gif-sur-Yvette Cedex, France Tel.: +33-1 69 08 79 25 Fax: +33-1 69 08 82 61 E-Mail: khinsen at cea.fr --------------------------------------------------------------------- From oliphant.travis at ieee.org Mon Jan 23 07:58:30 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Jan 23 07:58:30 2006 Subject: [Numpy-discussion] Scalar array - a trap for the unwary In-Reply-To: <43D4061C.5060900@sympatico.ca> References: <43D2CDA8.9030700@sympatico.ca> <43D2E182.2060201@ieee.org> <43D4061C.5060900@sympatico.ca> Message-ID: <43D4FCEC.5020507@ieee.org> Colin J. Williams wrote: > In exploring these matters, it seems that numpy does not broadcast > scalar operations on arrays, as numarray does. Is this intended? > Please see the script below and the result under that. I don't know what is going on with your installation, because I don't get that error. Your script works fine for me. I wouldn't advise extracting array directly from multiarray as you are doing, because exactly where array lives is an implementation detail that could change in the future. -Travis From mark at mitre.org Mon Jan 23 08:00:02 2006 From: mark at mitre.org (Mark Heslep) Date: Mon Jan 23 08:00:02 2006 Subject: [Numpy-discussion] Speed performance on array constant set In-Reply-To: <43D05BA0.7000700@ieee.org> References: <43D0573E.4090301@mitre.org> <43D05BA0.7000700@ieee.org> Message-ID: <43D14AC0.3060902@mitre.org> Travis Oliphant wrote: > First of all, try using NumPy instead of Numarray: import numpy as na > Ive got some NumArray C API investment in the typemaps. Is the C high level API the same, ie. NA_Input, etc? I havnt found the C API docs on the web site. > Second: use math (i.e. a += 7) > > Third: There are specialized fill routines .fill() in numpy and a > simliar routine in Numarray that can be faster then indexing. Im happy to help out on the optimization of .fill if needed Mark From oliphant.travis at ieee.org Mon Jan 23 08:12:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Jan 23 08:12:02 2006 Subject: [Numpy-discussion] Speed performance on array constant set In-Reply-To: <43D14AC0.3060902@mitre.org> References: <43D0573E.4090301@mitre.org> <43D05BA0.7000700@ieee.org> <43D14AC0.3060902@mitre.org> Message-ID: <43D50020.3030800@ieee.org> Mark Heslep wrote: > Travis Oliphant wrote: > >> First of all, try using NumPy instead of Numarray: import numpy >> as na >> > Ive got some NumArray C API investment in the typemaps. Is the C high > level API the same, ie. NA_Input, etc? I havnt found the C API docs > on the web site. The high-level C-API is the same as the old Numeric C-API. The new numarray C-API is not supported at this point. However, I would like to see a numarray compatibility module written so that it is easy to port numarray-based C-routines (simply by changing the header). I've started such a thing in scipy/Lib/sandbox/nd_image area. Look at these files for guidance: http://svn.scipy.org/svn/scipy/trunk/Lib/sandbox/nd_image/Src/numcompat.h http://svn.scipy.org/svn/scipy/trunk/Lib/sandbox/nd_image/Src/numcompat.c I'm thinking that a module could be written that contains the entire numarray C-API. -Travis From strawman at astraw.com Mon Jan 23 09:09:01 2006 From: strawman at astraw.com (Andrew Straw) Date: Mon Jan 23 09:09:01 2006 Subject: [Numpy-discussion] include file location and .eggs In-Reply-To: <7FA168E7-4908-4A1C-937A-DF19764BD4AA@laposte.net> References: <43CFCCF8.8000401@astraw.com> <7FA168E7-4908-4A1C-937A-DF19764BD4AA@laposte.net> Message-ID: <43D50D56.3000108@astraw.com> konrad.hinsen at laposte.net wrote: > On Jan 19, 2006, at 18:31, Andrew Straw wrote: >> = Why can't eggs use traditional include locations? = >> Multiple version support. Using eggs, it is possible to have multiple >> versions of the same distribution, such as numpy, installed within a >> single Python installation. (I use "distribution" to mean collection of > > > How is multiple version support in eggs supposed to work in the case > of packages that contain C modules? Eggs will allow you to do it, but you'll have to specify the exact dependencies in the setup.py file. > > Let me give a concrete example: Numeric, ScientificPython, and MMTK. > Numeric provides a C API through its header files. ScientificPython > and MMTK both contain C modules that depend on that C API. > ScientificPython also provides a C API for two of its modules, which > in turn MMTK depends on. Let's say you wanted to have Scientific 2.4.2 depend on Numeric 23.5. You'd make sure the call to setup() (which is now imported from setuptools, not distutils.core) included the following: setup( name='Scientific', version='2.4.2', ext_modules=stuff_that_depends_on_Numeric_23_5, install_requires='Numeric==23.5') Clearly, you could do this for all versions of Numeric and Scientific you want to run together. Obviously, it could be done in a more automated way (by querying what version of Numeric is being used to build). To have two versions of Scientific 2.4.2 installed, however, they would have to have different names, so perhaps '2.4.2_Numeric23.5' would work well for the example above. Now, you can just do your imports as normal, which will result in the the highest version number available. To specify versions, use 'import pkg_resources; pkg_resources.require('Numeric==23.5'); import Numeric'. An exception is raised if you do 'import Numeric' and it imports Numeric==24.2 and then you later try and import Scientific with only version 2.4.2_Numeric23.5 available. (At least, if an exception isn't raised, I'm 99% sure this would be considered a bug and would be fixed.) So, the import order is important -- it's probably best to import Scientific first in this example so that the correct version of Numeric can be selected behind the scenes without requiring you to write any pkg_resources.require() calls. N.B. For setuptools to get a package's (e.g. Numeric's) version number, Numeric must be built with setuptools -- not necessarily as an egg, but at least with the egg metadata available. If you don't build as an egg, though, you can't have more than one version installed. FYI, to build a conventional (non-egg) with setuptools so that it includes this metadata from an unmodified package that doesn't itself use setuptools, use the following python -c "import setuptools; execfile('setup.py')" install --single-version-externally-managed > > Suppose I would like to have both Numeric 23.5 and Numeric 24.2 on my > machine, and also Scientific 2.4.2 and Scientific 2.5.1. I would then > need four sets of the binary modules of Scientific: > > 1) 2.4.2 for Numeric 23.5 > 2) 2.4.2 for Numeric 24.2 > 3) 2.5.1 for Numeric 23.5 > 4) 2.5.1 for Numeric 24.2 See above. > > I would also need four sets of the binary modules of MMTK, assuming > that I am happy with just one version of MMTK. The above scales, so I don't think any new info is needed here. > > Unless some mechanism (which would have to be part of egg > construction or egg installation) makes sure that the correct > binaries are used for the particular combination of versions that I > wish to use, I will end up having lots of mysterious crashes > resulting from incompatible binary versions. That's exactly the point -- eggs do ensure the correct binary version is used. To come full circle to the question of header file location, this is exactly why eggs need to have multiple versions of the header files available, each version located within their respective egg. And this is also why keeping the header files in /usr/local/include/python2.4/Numeric/arrayobject.h is incompatible with one of the best features of eggs. > > If the egg mechanism does take care of such dependencies, then it > could easily (and thus should) take care of selecting the right > combination of header files. If it doesn't, then it should not be > used at all for packages containing C or Pyrex modules, and packages > like numpy should better not accomodate to eggs because in the long > run that will only cause support headaches. So how, about the converse? :) If setuptools does take care of dependencies (and it does), should packages like numpy definitely accommodate eggs? I certainly think so! :) Andrew From konrad.hinsen at laposte.net Mon Jan 23 09:28:07 2006 From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net) Date: Mon Jan 23 09:28:07 2006 Subject: [Numpy-discussion] include file location and .eggs In-Reply-To: <43D50D56.3000108@astraw.com> References: <43CFCCF8.8000401@astraw.com> <7FA168E7-4908-4A1C-937A-DF19764BD4AA@laposte.net> <43D50D56.3000108@astraw.com> Message-ID: <7EB1861B-5D0B-4ADE-BFA8-CCDCFF1C30B8@laposte.net> On Jan 23, 2006, at 18:07, Andrew Straw wrote: > Let's say you wanted to have Scientific 2.4.2 depend on Numeric 23.5. > You'd make sure the call to setup() (which is now imported from > setuptools, not distutils.core) included the following: ... OK, that looks sophisticated enough to me :-) Since setuptools keeps track of dependencies, it should be able to provide on demand a list of all include file directories for all packages on which the current build procedure depends, directly or indirectly. To stay within my example, a package that depends on ScientificPython, which in turn depends on Numeric, should be able to get, with a single call, the include directories for *both* ScientificPython and Numeric. Assuming such a mechanism exists, and is also implemented in plain distutils (returning just the standard installation paths), then personally I wouldn't care where header files are, because there is a straightforward way to obtain that information. However, that is not what is currently being discussed for numpy, unless I missed something. What is being discussed is an arrangement that requires the setup.py of every package to reconstruct the dependency tree and figure out where each package in the dependency tree keeps its header files. There is something else that I don't understand. If egg installation uses setup from setuptools instead of distutils, it should be trivial for the setup function from setuptools to implement install_headers in a way that suits its need. So why request that numpy (and other packages) adapt to setuptools, creating compatibility issues with standard distutils, rather than modify setuptools and have it work automatically for all packages? Konrad. -- --------------------------------------------------------------------- Konrad Hinsen Laboratoire L?on Brillouin, CEA Saclay, 91191 Gif-sur-Yvette Cedex, France Tel.: +33-1 69 08 79 25 Fax: +33-1 69 08 82 61 E-Mail: khinsen at cea.fr --------------------------------------------------------------------- From strawman at astraw.com Mon Jan 23 10:04:01 2006 From: strawman at astraw.com (Andrew Straw) Date: Mon Jan 23 10:04:01 2006 Subject: [Numpy-discussion] include file location and .eggs In-Reply-To: <7EB1861B-5D0B-4ADE-BFA8-CCDCFF1C30B8@laposte.net> References: <43CFCCF8.8000401@astraw.com> <7FA168E7-4908-4A1C-937A-DF19764BD4AA@laposte.net> <43D50D56.3000108@astraw.com> <7EB1861B-5D0B-4ADE-BFA8-CCDCFF1C30B8@laposte.net> Message-ID: <43D51A59.9020509@astraw.com> konrad.hinsen at laposte.net wrote: > On Jan 23, 2006, at 18:07, Andrew Straw wrote: > >> Let's say you wanted to have Scientific 2.4.2 depend on Numeric 23.5. >> You'd make sure the call to setup() (which is now imported from >> setuptools, not distutils.core) included the following: > > > ... > > OK, that looks sophisticated enough to me :-) > > Since setuptools keeps track of dependencies, it should be able to > provide on demand a list of all include file directories for all > packages on which the current build procedure depends, directly or > indirectly. To stay within my example, a package that depends on > ScientificPython, which in turn depends on Numeric, should be able to > get, with a single call, the include directories for *both* > ScientificPython and Numeric. > > Assuming such a mechanism exists, and is also implemented in plain > distutils (returning just the standard installation paths), then > personally I wouldn't care where header files are, because there is a > straightforward way to obtain that information. The problem is backwards compatibility. Nothing like this is implemented in plain distutils, which is why the packages will have to support it themselves. This way they can keep working in both old distutils-based environments and new ones, including with setuptools. The current solution _works_ on yesteryear's Python 2.3. In my opinion, relying on distutils to manage header file locations was a mistake, but that's easy to say in hindsight, when it's clear that, as useful as distutils is, it has some deficiencies. Distutils never handled the case when, for example you installed Numeric with '/usr/local/bin/python2.3 setup.py install --prefix=/home/astraw/py23'. In this case, sure, the package got installed where I wanted (in my $PYTHONPATH) in /home/astraw/py23/lib/python2.3/site-packages/Numeric, but the headers were in /home/astraw/py23/include/python2.3/Numeric. Distutils would never find those headers again. This is exactly the same problem, and it's one that distutils never solved, and I don't think we should hold our breath waiting for Python 2.5's distutils to solve it. For one thing, it's already solved at the package level a la numpy. For another, given that it's already solved, who's going to bother attempting to make a patch (which can't be relied on by older pythons) for Python 2.5, which may not even be accepted? > > However, that is not what is currently being discussed for numpy, > unless I missed something. What is being discussed is an arrangement > that requires the setup.py of every package to reconstruct the > dependency tree and figure out where each package in the dependency > tree keeps its header files. Why is "import numpy; include_dirs=numpy.get_numpy_include()" in setup.py unpalatable for you? That's all that is required in setup.py. To me, this is a small price to pay for the benefits gained. If you want to argue this is more pain than you're willing to deal with, I think that, to be fair, you need to give a counter-example whereby you explain how you handle all those versions of Numeric and Scientific installed simultaneously and how you select among them. I'll admit that in the case you gave, of 2 versions of Scientific and 2 versions of Numeric compiled into all 4 possible combinations, you'd also need an "import pkg_resources; pkg_resources.require('Numeric==x.y')" prior to the "import Numeric". I see the case you gave as a rather extreme case -- normally I would like, for example, a single version of (old) scipy compiled against old Numeric and a single new version of scipy compiled against numpy. (If it's not already working, I'm almost willing to backport egg support to old scipy to do just this...) Like I said above, distutils has no such equivalent mechanism, and this _allows_ something that simply wasn't possible before and is arguably broken in distutils. > > > There is something else that I don't understand. If egg installation > uses setup from setuptools instead of distutils, it should be trivial > for the setup function from setuptools to implement install_headers > in a way that suits its need. So why request that numpy (and other > packages) adapt to setuptools, creating compatibility issues with > standard distutils, rather than modify setuptools and have it work > automatically for all packages? That's an interesting idea. The short answer is that setuptools doesn't do it currently. Perhaps it could. I'm not going to implement it, though. It also doesn't solve the problem I gave above with plain distutils, which I think is a real problem that numpy solves in a backwards compatible way. From konrad.hinsen at laposte.net Mon Jan 23 10:40:06 2006 From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net) Date: Mon Jan 23 10:40:06 2006 Subject: [Numpy-discussion] include file location and .eggs In-Reply-To: <43D51A59.9020509@astraw.com> References: <43CFCCF8.8000401@astraw.com> <7FA168E7-4908-4A1C-937A-DF19764BD4AA@laposte.net> <43D50D56.3000108@astraw.com> <7EB1861B-5D0B-4ADE-BFA8-CCDCFF1C30B8@laposte.net> <43D51A59.9020509@astraw.com> Message-ID: <991A3B2E-AE9B-433C-91DB-549C70F47C7C@laposte.net> On Jan 23, 2006, at 19:03, Andrew Straw wrote: > The problem is backwards compatibility. Nothing like this is > implemented > in plain distutils, which is why the packages will have to support it > themselves. This way they can keep working in both old distutils-based > environments and new ones, including with setuptools. The current > solution _works_ on yesteryear's Python 2.3. In my opinion, relying on Well, I'd argue it doesn't work at all, because it doesn't scale to complex dependency trees. > Distutils never handled the case when, for example you installed > Numeric > with '/usr/local/bin/python2.3 setup.py install > --prefix=/home/astraw/py23'. In this case, sure, the package got > installed where I wanted (in my $PYTHONPATH) in > /home/astraw/py23/lib/python2.3/site-packages/Numeric, but the headers > were in /home/astraw/py23/include/python2.3/Numeric. Distutils would True. It was always true that if you install anywhere but in the default location, you are on your own. But if you do keep the default location (as most users do), everything works fine. > never find those headers again. This is exactly the same problem, and > it's one that distutils never solved, and I don't think we should hold > our breath waiting for Python 2.5's distutils to solve it. For one Why not contribute to solving it? > thing, it's already solved at the package level a la numpy. For > another, > given that it's already solved, who's going to bother attempting to > make I don't think it is solved at all. > Why is "import numpy; include_dirs=numpy.get_numpy_include()" in > setup.py unpalatable for you? That's all that is required in setup.py. Let's go back to my initial scenario, and let's even forget about multiple versions. I have ScientificPython, which can be compiled for use with either Numeric or numarray, and some day NumPy will be a third choice. Then I install MMTK, which depends on ScientificPython. How is the MMTK installation script supposed to figure out if ScientificPython was compiled with Numeric, numarray, or NumPy? And how is it supposed to know where each of these packages stores its header files? I don't mind adding some lines to setup.py for each package, that's a small price to pay. But I do mind having to worry about indirect dependencies. > to argue this is more pain than you're willing to deal with, I think > that, to be fair, you need to give a counter-example whereby you > explain > how you handle all those versions of Numeric and Scientific installed > simultaneously and how you select among them. I'll admit that in the I don't, except on my development machine, where I used symbolic links that are changed using scripts. That's a bit of work to set up, but I have only two packages of which I use multiple versions, so doing it once is acceptable for me. In fact, my main worry is not my own machine, nor the machine of any competent Python programmer. My worry is the less competent user who can't get installation of my packages to work and turns to me for help. Tech support takes more of my time than development these days. A fragile dependency system such as the one you propose will only create more support questions on my side - so don't expect me to support it. > Like I said above, distutils has no such equivalent mechanism, and > this > _allows_ something that simply wasn't possible before and is arguably > broken in distutils. I am all for fixing distutils. Konrad. -- --------------------------------------------------------------------- Konrad Hinsen Laboratoire L?on Brillouin, CEA Saclay, 91191 Gif-sur-Yvette Cedex, France Tel.: +33-1 69 08 79 25 Fax: +33-1 69 08 82 61 E-Mail: khinsen at cea.fr --------------------------------------------------------------------- From pearu at scipy.org Mon Jan 23 11:18:02 2006 From: pearu at scipy.org (Pearu Peterson) Date: Mon Jan 23 11:18:02 2006 Subject: [Numpy-discussion] include file location and .eggs In-Reply-To: <991A3B2E-AE9B-433C-91DB-549C70F47C7C@laposte.net> References: <43CFCCF8.8000401@astraw.com> <7FA168E7-4908-4A1C-937A-DF19764BD4AA@laposte.net> <43D50D56.3000108@astraw.com> <7EB1861B-5D0B-4ADE-BFA8-CCDCFF1C30B8@laposte.net> <43D51A59.9020509@astraw.com> <991A3B2E-AE9B-433C-91DB-549C70F47C7C@laposte.net> Message-ID: On Mon, 23 Jan 2006, konrad.hinsen at laposte.net wrote: > Let's go back to my initial scenario, and let's even forget about multiple > versions. I have ScientificPython, which can be compiled for use with either > Numeric or numarray, and some day NumPy will be a third choice. Then I > install MMTK, which depends on ScientificPython. How is the MMTK installation > script supposed to figure out if ScientificPython was compiled with Numeric, > numarray, or NumPy? I don't know about setuptools but distutils cannot solve this problem either without some extra code in MMTK setup.py file and a possibility to determine which choice was used to build ScientificPython. So, it's not really a distutils nor numpy include file location issue. > And how is it supposed to know where each of these packages stores its header files? Here's an answer to this question, namely, when numpy is installed, then one can use the following code in setup.py to support different array backends: from numpy.distutils.system_info import get_info arr_info = get_info('numpy') or get_info('numarray') or get_info('Numeric') # For explanation, arr_info is one of the following dictionaries: # {'define_macros':[('NUMPY',None)], 'include_dirs':['/path/to/numpy/header/dir']} # {'define_macros':[('NUMARRAY',None)], 'include_dirs':['/path/to/numarray/header/dir']} # {'define_macros':[('NUMERIC',None)], 'include_dirs':['/path/to/Numeric/header/dir']} # {} # --- no array backends installed # If one has all three array backend packages installed and wishes to # use, say, numarray, then to disable detecting numpy, one should # define environment variable NUMPY=None. ... setup(..., ext_modules = [Extension(..., **arr_info),..]) Pearu From boisgera at isia.cma.fr Mon Jan 23 11:53:04 2006 From: boisgera at isia.cma.fr (=?ISO-8859-1?Q?S=E9bastien_Boisg=E9rault?=) Date: Mon Jan 23 11:53:04 2006 Subject: [Numpy-discussion] Eigenvalues hangs Message-ID: <43D533EF.2090604@isia.cma.fr> Hi all, Robert Kern suggested to transfer a discussion that originated on comp.lang.python to this mailing-list. My problem is that on my platform the numpy 'eigenvalues' method hangs forever. Some platform info: * numpy 0.9.4 (from the tar.gz) * Linux Mandriva (gcc 4.0.1). * no optimized BLAS version used. Greg Landrum pointed out that it may be a gcc 4.0 related problem and proposed a workaround -- to add the option '-ffloat-store' to CFLAGS. Could anybody tell me how I should modify the numpy sources to test this workaround ? Regards, SB From cookedm at physics.mcmaster.ca Mon Jan 23 12:03:03 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Mon Jan 23 12:03:03 2006 Subject: [Numpy-discussion] Eigenvalues hangs In-Reply-To: <43D533EF.2090604@isia.cma.fr> (=?utf-8?Q?S=C3=A9bastien_Bois?= =?utf-8?Q?g=C3=A9rault's?= message of "Mon, 23 Jan 2006 20:52:15 +0100") References: <43D533EF.2090604@isia.cma.fr> Message-ID: S?bastien Boisg?rault writes: > Hi all, > > Robert Kern suggested to transfer a discussion that originated > on comp.lang.python to this mailing-list. > > My problem is that on my platform the numpy 'eigenvalues' method > hangs forever. > > Some platform info: > * numpy 0.9.4 (from the tar.gz) > * Linux Mandriva (gcc 4.0.1). > * no optimized BLAS version used. > > Greg Landrum pointed out that it may be a gcc 4.0 related problem and > proposed a workaround -- to add the option '-ffloat-store' to CFLAGS. > > Could anybody tell me how I should modify the numpy sources to test > this workaround ? You should just be able to do the setup like this: $ CFLAGS='-ffloat-store' python setup.py build -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From boisgera at isia.cma.fr Mon Jan 23 12:19:03 2006 From: boisgera at isia.cma.fr (=?UTF-8?B?U8OpYmFzdGllbiBCb2lzZ8OpcmF1bHQ=?=) Date: Mon Jan 23 12:19:03 2006 Subject: [Numpy-discussion] Eigenvalues hangs In-Reply-To: References: <43D533EF.2090604@isia.cma.fr> Message-ID: <43D539FF.4070808@isia.cma.fr> David M. Cooke a ?crit : >S?bastien Boisg?rault writes: > > > >>Hi all, >> >>Robert Kern suggested to transfer a discussion that originated >>on comp.lang.python to this mailing-list. >> >>My problem is that on my platform the numpy 'eigenvalues' method >>hangs forever. >> >>Some platform info: >> * numpy 0.9.4 (from the tar.gz) >> * Linux Mandriva (gcc 4.0.1). >> * no optimized BLAS version used. >> >>Greg Landrum pointed out that it may be a gcc 4.0 related problem and >>proposed a workaround -- to add the option '-ffloat-store' to CFLAGS. >> >>Could anybody tell me how I should modify the numpy sources to test >>this workaround ? >> >> > >You should just be able to do the setup like this: > >$ CFLAGS='-ffloat-store' python setup.py build > > > Great ! It solves the problem :) Thanks a lot for the help ! SB -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndarray at mac.com Mon Jan 23 12:26:04 2006 From: ndarray at mac.com (Sasha) Date: Mon Jan 23 12:26:04 2006 Subject: [Numpy-discussion] Eigenvalues hangs In-Reply-To: <43D533EF.2090604@isia.cma.fr> References: <43D533EF.2090604@isia.cma.fr> Message-ID: I think this issue was altimately resolved as a lapack rather than gcc problem and -ffloat-store is not the right work-around. I don't have the right references right now, so unless someone steps in I will follow up when I get home this evening. -- sasha On 1/23/06, S?bastien Boisg?rault wrote: > > Hi all, > > Robert Kern suggested to transfer a discussion that originated > on comp.lang.python to this mailing-list. > > My problem is that on my platform the numpy 'eigenvalues' method > hangs forever. > > Some platform info: > * numpy 0.9.4 (from the tar.gz) > * Linux Mandriva (gcc 4.0.1). > * no optimized BLAS version used. > > Greg Landrum pointed out that it may be a gcc 4.0 related problem and > proposed a workaround -- to add the option '-ffloat-store' to CFLAGS. > > Could anybody tell me how I should modify the numpy sources to test > this workaround ? > > Regards, > > SB > > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From konrad.hinsen at laposte.net Mon Jan 23 12:27:02 2006 From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net) Date: Mon Jan 23 12:27:02 2006 Subject: [Numpy-discussion] include file location and .eggs In-Reply-To: References: <43CFCCF8.8000401@astraw.com> <7FA168E7-4908-4A1C-937A-DF19764BD4AA@laposte.net> <7EB1861B-5D0B-4ADE-BFA8-CCDCFF1C30B8@laposte.net> <991A3B2E-AE9B-433C-91DB-549C70F47C7C@laposte.net> Message-ID: <246A6EC8-8D6B-407A-8224-0276422392B0@laposte.net> On 23.01.2006, at 19:17, Pearu Peterson wrote: > I don't know about setuptools but distutils cannot solve this > problem either without some extra code in MMTK setup.py file and a > possibility to determine which choice was used to build > ScientificPython. So, it's not really a distutils nor numpy include > file location issue. Distutils doesn't provide a perfect solution, but if everyone just lets distutils put header files into the default location, and all packages get installed without an explicit prefix, then it will just work. Considering that I get next to no support questions about such installation issues, I consider the current situation very acceptable. >> And how is it supposed to know where each of these packages stores >> its header files? > > Here's an answer to this question, namely, when numpy is installed, > then one can use the following code in setup.py to support > different array backends: That's for numpy. How about Numeric and numarray? And how about whatever other packages need header files? If everyone makes up his own conventions, we end up with a big mess. Another idea: considering that setup.py needs to be changed anyway to make eggs (because setup must be imported from somewhere else), why not put some flag "make_egg" at the beginning of numpy's setup.py, and depending on the value of the flag either use setuptools.setup and header files in the data section, or distutils.core.setup and header files in the standard location? That should suit everyone, right? Konrad. -- ------------------------------------------------------------------------ ------- Konrad Hinsen Laboratoire Leon Brillouin (CEA-CNRS), CEA Saclay, 91191 Gif-sur-Yvette Cedex, France Tel.: +33-1 69 08 79 25 Fax: +33-1 69 08 82 61 E-Mail: khinsen at cea.fr ------------------------------------------------------------------------ ------- From cjw at sympatico.ca Mon Jan 23 14:32:29 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Mon Jan 23 14:32:29 2006 Subject: [Numpy-discussion] Scalar array - a trap for the unwary In-Reply-To: <43D4FCEC.5020507@ieee.org> References: <43D2CDA8.9030700@sympatico.ca> <43D2E182.2060201@ieee.org> <43D4061C.5060900@sympatico.ca> <43D4FCEC.5020507@ieee.org> Message-ID: <43D55952.8000905@sympatico.ca> Travis Oliphant wrote: > Colin J. Williams wrote: > >> In exploring these matters, it seems that numpy does not broadcast >> scalar operations on arrays, as numarray does. Is this intended? >> Please see the script below and the result under that. > > > I don't know what is going on with your installation, because I don't > get that error. Your script works fine for me. I've just tried this with version 0.9.4 and all works well. >>> a= _mu.array([1, 2], dtype= _nt.uint8) >>> a+254 array([255, 256], dtype=int16) This does bring out a possible problem with the new type. str of the old type returned the input name int8, float64 etc. Is there some method of the new type which returns the name input, 'int8' in the above example? In most cases, the mnemonic is more helpful than the letter coding. > > I wouldn't advise extracting array directly from multiarray as you are > doing, because exactly where array lives is an implementation detail > that could change in the future. I accept this risk for now. Partly, this was done to simplify the namespace and partly to locate a bug concerning the interaction of a Windows debugger and the umath module. > > -Travis > Colin W. From Chris.Barker at noaa.gov Mon Jan 23 14:42:49 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon Jan 23 14:42:49 2006 Subject: [Numpy-discussion] include file location and .eggs In-Reply-To: <246A6EC8-8D6B-407A-8224-0276422392B0@laposte.net> References: <43CFCCF8.8000401@astraw.com> <7FA168E7-4908-4A1C-937A-DF19764BD4AA@laposte.net> <7EB1861B-5D0B-4ADE-BFA8-CCDCFF1C30B8@laposte.net> <991A3B2E-AE9B-433C-91DB-549C70F47C7C@laposte.net> <246A6EC8-8D6B-407A-8224-0276422392B0@laposte.net> Message-ID: <43D55B77.4010105@noaa.gov> konrad.hinsen at laposte.net wrote: > Distutils doesn't provide a perfect solution, but if everyone just lets > distutils put header files into the default location, and all packages > get installed without an explicit prefix, then it will just work. > Considering that I get next to no support questions about such > installation issues, I consider the current situation very acceptable. Except that there is no support whatsoever for multiple versions. It seems your complicated scenario was due to multiple versions. In fact, the one problem I've had with Scientific in the past was trying to use a binary that had been compiled against a different version of Numeric than I was using. If your user is compiling everything, then there is no problem. If your user is using binaries, they they need to be labeled with versions. If they use the wrong one with eggs, then there is at least some hope that they'll get a meaningful error message. If they are compiling MMTK against a Scientific and a Numpy that they didn't compile, then it's going to get ugly -- but I don't think it's any uglier than it is now. As for improving distutils in the next go around -- I'm all for it, but why not have 2.5 include setuptools instead -- it strikes me as a next generation distutils. I do agree that setuptools should handle the header file placement and finding for you, rather than each package having to do it on it's own. and Andrew, thanks for all the info on setuptools and eggs. It's been very informative! -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From mark at mitre.org Mon Jan 23 16:19:06 2006 From: mark at mitre.org (Mark Heslep) Date: Mon Jan 23 16:19:06 2006 Subject: [Numpy-discussion] Speed performance on array constant set In-Reply-To: <43D51669.6080308@noaa.gov> References: <43D0573E.4090301@mitre.org> <43D05BA0.7000700@ieee.org> <43D14AC0.3060902@mitre.org> <43D51669.6080308@noaa.gov> Message-ID: <43D57262.6090004@mitre.org> Christopher Barker wrote: > Mark Heslep wrote: > >> Ive got some NumArray C API investment in the typemaps. > > > Are these SWIG typemaps? If so I think there are a bunch of us that > wold like to see them, and maybe we can all work together to port them > to NumPy. > > -Chris > Yes they are SWIG typemaps for OpenCv* data structures, based on prior Numeric art I'd found and ported to Numarray. Attached. 1. numdata.h example that works well for simple Cv structures containing uniform types: CvPoint2D32F => struct { float x, float y }. Can I use a record array here, or is there some Numpy overhead interlaced with fields? 2. numarray.h Attempt to replace the main Cv Image structures CvMat, IplImage. Needs work. Some success but there's segfault or two in there somewhere. *Details for Cv use: I drop them in the ../interfaces/swig/python path for opencv and them in cv.i Regards, Mark -------------- next part -------------- A non-text attachment was scrubbed... Name: adaptors.h Type: text/x-chdr Size: 2195 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: numarray.h Type: text/x-chdr Size: 2746 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: numdata.h Type: text/x-chdr Size: 1750 bytes Desc: not available URL: From oliphant at ee.byu.edu Mon Jan 23 16:37:05 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Jan 23 16:37:05 2006 Subject: [Numpy-discussion] Speed performance on array constant set In-Reply-To: <43D57262.6090004@mitre.org> References: <43D0573E.4090301@mitre.org> <43D05BA0.7000700@ieee.org> <43D14AC0.3060902@mitre.org> <43D51669.6080308@noaa.gov> <43D57262.6090004@mitre.org> Message-ID: <43D5768C.4090200@ee.byu.edu> Mark Heslep wrote: > Christopher Barker wrote: > >> Mark Heslep wrote: >> >>> Ive got some NumArray C API investment in the typemaps. >> >> >> >> Are these SWIG typemaps? If so I think there are a bunch of us that >> wold like to see them, and maybe we can all work together to port >> them to NumPy. >> >> -Chris >> > Yes they are SWIG typemaps for OpenCv* data structures, based on prior > Numeric art I'd found and ported to Numarray. > Attached. > > 1. numdata.h example that works well for simple Cv structures > containing uniform types: CvPoint2D32F => struct { float x, float y > }. Can I use a record array here, or is there some Numpy overhead > interlaced with fields? You can use a record array, but for uniform types I don't know what the advantage is (except for perhaps named-field slicing---which may actually be faster I'm not sure) over a x2 float array. > 2. numarray.h Attempt to replace the main Cv Image structures CvMat, > IplImage. Needs work. Some success but there's segfault or two in > there somewhere. > These can be ported fairly easily, I think (actually, the old Numeric typemaps would still work --- and work with Numarray), so the basic Numeric C-API still presents itself as the simplest way to support all the array packages. -Travis From ndarray at mac.com Mon Jan 23 19:06:02 2006 From: ndarray at mac.com (Sasha) Date: Mon Jan 23 19:06:02 2006 Subject: [Numpy-discussion] Eigenvalues hangs In-Reply-To: References: <43D533EF.2090604@isia.cma.fr> Message-ID: On 1/23/06, Sasha wrote: > I think this issue was altimately resolved as a lapack rather than gcc > problem and -ffloat-store is not the right work-around. I don't have > the right references right now, so unless someone steps in I will > follow up when I get home this evening. > > -- sasha > > On 1/23/06, S?bastien Boisg?rault wrote: > > > > Hi all, > > > > Robert Kern suggested to transfer a discussion that originated > > on comp.lang.python to this mailing-list. > > > > My problem is that on my platform the numpy 'eigenvalues' method > > hangs forever. > > > > Some platform info: > > * numpy 0.9.4 (from the tar.gz) > > * Linux Mandriva (gcc 4.0.1). > > * no optimized BLAS version used. > > > > Greg Landrum pointed out that it may be a gcc 4.0 related problem and > > proposed a workaround -- to add the option '-ffloat-store' to CFLAGS. > > > > Could anybody tell me how I should modify the numpy sources to test > > this workaround ? > > > > Regards, > > > > SB > > > > > > > > ------------------------------------------------------- > > This SF.net email is sponsored by: Splunk Inc. Do you grep through log files > > for problems? Stop! Download the new AJAX search engine that makes > > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642 > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > From ndarray at mac.com Mon Jan 23 19:15:03 2006 From: ndarray at mac.com (Sasha) Date: Mon Jan 23 19:15:03 2006 Subject: [Numpy-discussion] Eigenvalues hangs In-Reply-To: References: <43D533EF.2090604@isia.cma.fr> Message-ID: Sorry for an empty message. I lost connection and a half a page message with it :-( Here is a brief summary: -ffloat-store may be expensive: """ Reading the gcc documentation, one finds that the -ffloat-store is an expensive hack to attempt to avoid the excess precision in the floating pointer registers on Intel FPUs. Based on recent experience with DONLP2, a noteworthy nonlinear solver, using -ffloat-store everywhere is an unnecessarily costly workaround. """ If it has to be enabled, it should be limited to dlapack_lite . Red Hat patched its own lapack by disabling optimization for a few routines: https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=138683 -- sasha On 1/23/06, Sasha wrote: > On 1/23/06, Sasha wrote: > > I think this issue was altimately resolved as a lapack rather than gcc > > problem and -ffloat-store is not the right work-around. I don't have > > the right references right now, so unless someone steps in I will > > follow up when I get home this evening. > > > > -- sasha > > > > On 1/23/06, S?bastien Boisg?rault wrote: > > > > > > Hi all, > > > > > > Robert Kern suggested to transfer a discussion that originated > > > on comp.lang.python to this mailing-list. > > > > > > My problem is that on my platform the numpy 'eigenvalues' method > > > hangs forever. > > > > > > Some platform info: > > > * numpy 0.9.4 (from the tar.gz) > > > * Linux Mandriva (gcc 4.0.1). > > > * no optimized BLAS version used. > > > > > > Greg Landrum pointed out that it may be a gcc 4.0 related problem and > > > proposed a workaround -- to add the option '-ffloat-store' to CFLAGS. > > > > > > Could anybody tell me how I should modify the numpy sources to test > > > this workaround ? > > > > > > Regards, > > > > > > SB > > > > > > > > > > > > ------------------------------------------------------- > > > This SF.net email is sponsored by: Splunk Inc. Do you grep through log files > > > for problems? Stop! Download the new AJAX search engine that makes > > > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > > > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642 > > > _______________________________________________ > > > Numpy-discussion mailing list > > > Numpy-discussion at lists.sourceforge.net > > > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > > > > From mark at mitre.org Mon Jan 23 19:21:08 2006 From: mark at mitre.org (Mark Heslep) Date: Mon Jan 23 19:21:08 2006 Subject: [Numpy-discussion] Speed performance on array constant set In-Reply-To: <43D5768C.4090200@ee.byu.edu> References: <43D0573E.4090301@mitre.org> <43D05BA0.7000700@ieee.org> <43D14AC0.3060902@mitre.org> <43D51669.6080308@noaa.gov> <43D57262.6090004@mitre.org> <43D5768C.4090200@ee.byu.edu> Message-ID: <43D59D0B.3060907@mitre.org> Travis Oliphant wrote: >> 1. numdata.h example that works well for simple Cv structures >> containing uniform types: CvPoint2D32F => struct { float x, float y >> }. Can I use a record array here, or is there some Numpy overhead >> interlaced with fields? > > > You can use a record array, but for uniform types I don't know what > the advantage is (except for perhaps named-field slicing---which may > actually be faster I'm not sure) over a x2 float array. So that I can step through the array record by record and not field by field. >> 2. numarray.h Attempt to replace the main Cv Image structures CvMat, >> IplImage. Needs work. Some success but there's segfault or two in >> there somewhere. > > These can be ported fairly easily, I think (actually, the old Numeric > typemaps would still work --- and work with Numarray), so the basic > Numeric C-API still presents itself as the simplest way to support all > the array packages. Well good to know. Ive been proceeding directly from the NumArray Users Manual 1.5 by Greenfield/Miller et al that describes the NA High Level API as the fastest of the APIs available to NA. I thought that NA also took steps to insure that unnecessary mem copies were not made, unlike Numeric? Mark From oliphant.travis at ieee.org Mon Jan 23 20:42:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Jan 23 20:42:02 2006 Subject: [Numpy-discussion] Speed performance on array constant set In-Reply-To: <43D59D0B.3060907@mitre.org> References: <43D0573E.4090301@mitre.org> <43D05BA0.7000700@ieee.org> <43D14AC0.3060902@mitre.org> <43D51669.6080308@noaa.gov> <43D57262.6090004@mitre.org> <43D5768C.4090200@ee.byu.edu> <43D59D0B.3060907@mitre.org> Message-ID: <43D5AFEA.9030405@ieee.org> Mark Heslep wrote: > Travis Oliphant wrote: > >>> 1. numdata.h example that works well for simple Cv structures >>> containing uniform types: CvPoint2D32F => struct { float x, float >>> y }. Can I use a record array here, or is there some Numpy overhead >>> interlaced with fields? >> >> >> >> You can use a record array, but for uniform types I don't know what >> the advantage is (except for perhaps named-field slicing---which may >> actually be faster I'm not sure) over a x2 float array. > > > So that I can step through the array record by record and not field by > field. Right, but you could do that using steps of length 2 if you want to manage it yourself (or use a complex data type). Again, I'm not sure of the performance implications at this point as I haven't done enough tests. > >>> 2. numarray.h Attempt to replace the main Cv Image structures >>> CvMat, IplImage. Needs work. Some success but there's segfault or >>> two in there somewhere. >> >> >> These can be ported fairly easily, I think (actually, the old Numeric >> typemaps would still work --- and work with Numarray), so the basic >> Numeric C-API still presents itself as the simplest way to support >> all the array packages. > > > Well good to know. Ive been proceeding directly from the NumArray > Users Manual 1.5 by Greenfield/Miller et al that describes the NA High > Level API as the fastest of the APIs available to NA. I thought that > NA also took steps to insure that unnecessary mem copies were not > made, unlike Numeric? Numeric doesn't make unnecessary copies either. It depends on what you mean by necessary. I think they may be referring to the fact that most people used ContiguousFromObject which always gave you contiguous memory from an array (copying if necessary), but if you learn how to access strided memory then FromObject was just fine (which would never copy). There are still some algorithms that don't deal with strided memory and so will make a copy. Some of these can be changed, others would be more difficult. It really becomes a question of cache-misses versus memory copy time, which I'm not sure how to optimize generally except by per-system experimentation like ATLAS does. -Travis From faltet at carabos.com Tue Jan 24 01:44:04 2006 From: faltet at carabos.com (Francesc Altet) Date: Tue Jan 24 01:44:04 2006 Subject: [Numpy-discussion] Scalar array - a trap for the unwary In-Reply-To: <43D55952.8000905@sympatico.ca> References: <43D2CDA8.9030700@sympatico.ca> <43D4FCEC.5020507@ieee.org> <43D55952.8000905@sympatico.ca> Message-ID: <200601241043.32209.faltet@carabos.com> A Dilluns 23 Gener 2006 23:31, Colin J. Williams va escriure: > >>> a= _mu.array([1, 2], dtype= _nt.uint8) > >>> a+254 > > array([255, 256], dtype=int16) > > This does bring out a possible problem with the new type. str of the > old type returned the input name int8, float64 etc. Is there some > method of the new type which returns the name input, 'int8' in the above > example? In most cases, the mnemonic is more helpful than the letter > coding. Yes. a.dtype.name does the trick. Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From jensj at fysik.dtu.dk Tue Jan 24 04:45:03 2006 From: jensj at fysik.dtu.dk (Jens Jorgen Mortensen) Date: Tue Jan 24 04:45:03 2006 Subject: [Numpy-discussion] Numeric/numpy difference Message-ID: <43D62121.5060504@fysik.dtu.dk> I just discovered (the hard way) that: >>> import numpy >>> a = numpy.empty((2, 3)) >>> a[:] = [1, 2] >>> a array([[1, 2, 1], [2, 1, 2]]) This is with revision 1984. With Numeric I get "ValueError: matrices are not aligned for copy", which I think is the correct answer. Jens J?rgen From schofield at ftw.at Tue Jan 24 06:16:02 2006 From: schofield at ftw.at (Ed Schofield) Date: Tue Jan 24 06:16:02 2006 Subject: [Numpy-discussion] 0-dimensional object arrays Message-ID: <43D6363C.5080509@ftw.at> Hi all, I've been attempting to use an object array to group together a collection of different-length arrays or lists, and I'm getting unexpected results. Here's an example: >>> import numpy >>> a = numpy.array([[1,2],[1,2,3]],object) I expected here that 'a' would be an array of dimension (2,), holding two Python lists of integers. Was I wrong to expect this? It seems instead that 'a' is a 0-dimensional array holding one object: >>> a array([[1, 2], [1, 2, 3]], dtype=object) >>> a.shape () >>> type(a) >>> a[0] Traceback (most recent call last): File "", line 1, in ? IndexError: 0-d arrays can't be indexed. This behaviour seems less useful. Would you agree this is a bug in the array constructor? I also can't explain this behaviour: >>> [] + a array([], dtype=object) >>> a + [] array([], dtype=object) I guess I would have expected these commands to be equivalent array(a.view()+[]). These results seem more sensible: >>> a.transpose() [[1, 2], [1, 2, 3]] >>> a*2 [[1, 2], [1, 2, 3], [1, 2], [1, 2, 3]] although these last two commands return Python lists, without the 0-d array wrapper, which isn't consistent with other 0-d arrays. On the whole, 0-d object arrays seem quite strange beasts. Could someone please enlighten me on why they deserve to exist? ;) They seem inconsistent with the new simplified interface to object array elements. Could we get rid of them entirely?! -- Ed From oliphant.travis at ieee.org Tue Jan 24 07:58:07 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue Jan 24 07:58:07 2006 Subject: [Numpy-discussion] 0-dimensional object arrays In-Reply-To: <43D6363C.5080509@ftw.at> References: <43D6363C.5080509@ftw.at> Message-ID: <43D64E66.9060706@ieee.org> Ed Schofield wrote: >Hi all, > >I've been attempting to use an object array to group together a >collection of different-length arrays or lists, and I'm getting >unexpected results. Here's an example: > > > >>>>import numpy >>>>a = numpy.array([[1,2],[1,2,3]],object) >>>> >>>> > >I expected here that 'a' would be an array of dimension (2,), holding >two Python lists of integers. Was I wrong to expect this? It seems >instead that 'a' is a 0-dimensional array holding one object: > > > Yes, you were wrong to expect this ;-) The array constructor is not really useful for constructing arbitrary object arrays. It is not designed to analyze what you've input and pick the "optimal" assignment. It never has been. So, it's not surprising that it doesn't do what you expect. > >[[1, 2], [1, 2, 3], [1, 2], [1, 2, 3]] > >although these last two commands return Python lists, without the 0-d >array wrapper, which isn't consistent with other 0-d arrays. > > We just changed that remember. When 0-d object arrays result from calculations, the actual objects are returned. >On the whole, 0-d object arrays seem quite strange beasts. Could >someone please enlighten me on why they deserve to exist? ;) They seem >inconsistent with the new simplified interface to object array >elements. Could we get rid of them entirely?! > > No, we can't get rid of them because 0-d arrays exist and OBJECT is a data-type an array can have. Somebody is going to be able to construct one. My advise is to create empty object arrays and fill them appropriately unless something better is written for OBJECT array construction. -Travis From oliphant.travis at ieee.org Tue Jan 24 08:01:08 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue Jan 24 08:01:08 2006 Subject: [Numpy-discussion] Numeric/numpy difference In-Reply-To: <43D62121.5060504@fysik.dtu.dk> References: <43D62121.5060504@fysik.dtu.dk> Message-ID: <43D64F2B.3020506@ieee.org> Jens Jorgen Mortensen wrote: > I just discovered (the hard way) that: > > >>> import numpy > >>> a = numpy.empty((2, 3)) > >>> a[:] = [1, 2] > >>> a > array([[1, 2, 1], > [2, 1, 2]]) > > This is with revision 1984. With Numeric I get "ValueError: matrices > are not aligned for copy", which I think is the correct answer. Well, not necessarily. This is consistent behavior with a[:] = 3 which fills the array with 3's. What's happening is that the array is being filled with [1,2] repeatedly (in C-contiguous order). -Travis From agn at noc.soton.ac.uk Tue Jan 24 08:06:04 2006 From: agn at noc.soton.ac.uk (George Nurser) Date: Tue Jan 24 08:06:04 2006 Subject: [Numpy-discussion] f2py problems Message-ID: I'm not sure whether this is the right list for f2py questions, but anyway. Tried it (v 2_1967) out today. It worked beautifully for a simple function. But for a more complicated (f77) *.F file with preprocessor directives and a lot of data statements I have been having problems. Firstly, it just seemed to ignore all of the lines with preprocessor directives. Secondly, after I used a preprocessor to create the .f file separately, and then ran the .f file through f2py, I got the following error trace. Anybody have any ideas? --George Nurser. f2py -c --verbose -m state2sig state2sig.f running build running config_fc running build_src building extension "state2sig" sources f2py options: [] f2py:> /tmp/tmppzKlnT/src/state2sigmodule.c creating /tmp/tmppzKlnT creating /tmp/tmppzKlnT/src Reading fortran codes... Reading file 'state2sig.f' (format:fix,strict) Post-processing... Block: state2sig Block: state2sig Traceback (most recent call last): File "/data/jrd/mod1/agn/ext/Linux/bin/f2py", line 6, in ? f2py.main() File "/data/jrd/mod1/agn/ext/Linux/lib64/python/numpy/f2py/ f2py2e.py", line 546, in main run_compile() File "/data/jrd/mod1/agn/ext/Linux/lib64/python/numpy/f2py/ f2py2e.py", line 533, in run_compile setup(ext_modules = [ext]) File "/data/jrd/mod1/agn/ext/Linux/lib64/python/numpy/distutils/ core.py", line 93, in setup return old_setup(**new_attr) File "/usr/lib64/python2.3/distutils/core.py", line 149, in setup dist.run_commands() File "/usr/lib64/python2.3/distutils/dist.py", line 907, in run_commands self.run_command(cmd) File "/usr/lib64/python2.3/distutils/dist.py", line 927, in run_command cmd_obj.run() File "/usr/lib64/python2.3/distutils/command/build.py", line 107, in run self.run_command(cmd_name) File "/usr/lib64/python2.3/distutils/cmd.py", line 333, in run_command self.distribution.run_command(command) File "/usr/lib64/python2.3/distutils/dist.py", line 927, in run_command cmd_obj.run() File "/data/jrd/mod1/agn/ext/Linux/lib64/python/numpy/distutils/ command/build_src.py", line 86, in run self.build_sources() File "/data/jrd/mod1/agn/ext/Linux/lib64/python/numpy/distutils/ command/build_src.py", line 99, in build_sources self.build_extension_sources(ext) File "/data/jrd/mod1/agn/ext/Linux/lib64/python/numpy/distutils/ command/build_src.py", line 149, in build_extension_sources sources = self.f2py_sources(sources, ext) File "/data/jrd/mod1/agn/ext/Linux/lib64/python/numpy/distutils/ command/build_src.py", line 355, in f2py_sources f2py2e.run_main(f2py_options + ['--lower', File "/data/jrd/mod1/agn/ext/Linux/lib64/python/numpy/f2py/ f2py2e.py", line 330, in run_main postlist=callcrackfortran(files,options) File "/data/jrd/mod1/agn/ext/Linux/lib64/python/numpy/f2py/ f2py2e.py", line 269, in callcrackfortran postlist=crackfortran.crackfortran(files) File "/data/jrd/mod1/agn/ext/Linux/lib64/python/numpy/f2py/ crackfortran.py", line 2574, in crackfortran postlist=postcrack(grouplist[0]) File "/data/jrd/mod1/agn/ext/Linux/lib64/python/numpy/f2py/ crackfortran.py", line 1488, in postcrack g=postcrack(g,tab=tab+'\t') File "/data/jrd/mod1/agn/ext/Linux/lib64/python/numpy/f2py/ crackfortran.py", line 1506, in postcrack block['body']=analyzebody(block,args,tab=tab) File "/data/jrd/mod1/agn/ext/Linux/lib64/python/numpy/f2py/ crackfortran.py", line 1648, in analyzebody b=postcrack(b,as,tab=tab+'\t') File "/data/jrd/mod1/agn/ext/Linux/lib64/python/numpy/f2py/ crackfortran.py", line 1506, in postcrack block['body']=analyzebody(block,args,tab=tab) File "/data/jrd/mod1/agn/ext/Linux/lib64/python/numpy/f2py/ crackfortran.py", line 1648, in analyzebody b=postcrack(b,as,tab=tab+'\t') File "/data/jrd/mod1/agn/ext/Linux/lib64/python/numpy/f2py/ crackfortran.py", line 1502, in postcrack block['vars']=analyzevars(block) File "/data/jrd/mod1/agn/ext/Linux/lib64/python/numpy/f2py/ crackfortran.py", line 2007, in analyzevars for k in implicitrules[ln0].keys(): KeyError: '(' From robert.kern at gmail.com Tue Jan 24 08:16:09 2006 From: robert.kern at gmail.com (Robert Kern) Date: Tue Jan 24 08:16:09 2006 Subject: [Numpy-discussion] bdist_egg trips up over non-extension extensions Message-ID: <43D6529C.8030105@gmail.com> In particular, numpy can't build into an egg currently because numpy/__config__.py is created using Configuration.add_extension() and a generator function. In order to make extension modules usable from zipped eggs, setuptools generates a pure Python stub loader that will extract the extension module to a .hidden directory and load it from there. In this case, since the Extension isn't actually an extension, the stub loader is broken, because it expects a numpy/__config__.so. Can we find another way to generate Python code instead of abusing Extension this way? -- Robert Kern robert.kern at gmail.com "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From pearu at scipy.org Tue Jan 24 08:18:02 2006 From: pearu at scipy.org (Pearu Peterson) Date: Tue Jan 24 08:18:02 2006 Subject: [Numpy-discussion] f2py problems In-Reply-To: References: Message-ID: On Tue, 24 Jan 2006, George Nurser wrote: > I'm not sure whether this is the right list for f2py questions, but anyway. > > Tried it (v 2_1967) out today. It worked beautifully for a simple function. > > But for a more complicated (f77) *.F file with preprocessor directives and a > lot of data statements I have been having problems. > > Firstly, it just seemed to ignore all of the lines with preprocessor > directives. f2py does not do preprocessing. > Secondly, after I used a preprocessor to create the .f file separately, and > then ran the .f file through f2py, I got the following error trace. > Anybody have any ideas? > line 2007, in analyzevars > for k in implicitrules[ln0].keys(): > KeyError: '(' Could you send me the file that caused this error? The error may be due to a f2py bug or a non-standard Fortran code. Thanks, Pearu From strawman at astraw.com Tue Jan 24 08:43:03 2006 From: strawman at astraw.com (Andrew Straw) Date: Tue Jan 24 08:43:03 2006 Subject: [Numpy-discussion] bdist_egg trips up over non-extension extensions In-Reply-To: <43D6529C.8030105@gmail.com> References: <43D6529C.8030105@gmail.com> Message-ID: <43D658F4.5030602@astraw.com> Just to be clear, you're not having problems with numpy as a non-zipped egg, just a zipped egg, right? I haven't had any problems, but I haven't tried to make zip-safe eggs. (For those who don't know, eggs can be installed either as a zip filed with extension .egg, or as a directory with extension egg.) Robert Kern wrote: >In particular, numpy can't build into an egg currently because >numpy/__config__.py is created using Configuration.add_extension() and a >generator function. In order to make extension modules usable from zipped eggs, >setuptools generates a pure Python stub loader that will extract the extension >module to a .hidden directory and load it from there. In this case, since the >Extension isn't actually an extension, the stub loader is broken, because it >expects a numpy/__config__.so. > >Can we find another way to generate Python code instead of abusing Extension >this way? > > > From pearu at scipy.org Tue Jan 24 08:46:07 2006 From: pearu at scipy.org (Pearu Peterson) Date: Tue Jan 24 08:46:07 2006 Subject: [Numpy-discussion] bdist_egg trips up over non-extension extensions In-Reply-To: <43D6529C.8030105@gmail.com> References: <43D6529C.8030105@gmail.com> Message-ID: On Tue, 24 Jan 2006, Robert Kern wrote: > In particular, numpy can't build into an egg currently because > numpy/__config__.py is created using Configuration.add_extension() and a > generator function. In order to make extension modules usable from zipped eggs, > setuptools generates a pure Python stub loader that will extract the extension > module to a .hidden directory and load it from there. In this case, since the > Extension isn't actually an extension, the stub loader is broken, because it > expects a numpy/__config__.so. > > Can we find another way to generate Python code instead of abusing Extension > this way? Yes, using py_modules instead of ext_modules would be the correct way to go. I'm looking into this issue right now... Pearu From ndarray at mac.com Tue Jan 24 09:22:06 2006 From: ndarray at mac.com (Sasha) Date: Tue Jan 24 09:22:06 2006 Subject: [Numpy-discussion] Replace typechars considered harmful Message-ID: After a day of debugging I found that convertcode replaced '1' with 'b' in the code which was selecting database columns with names ending with '1'. Since it is probably very difficult to automatically differentiate between '1' used as a typecode and other possible uses, I suggest to either make "replace typechars" optional, or make it write verbose message with a few lines of context when it changes the code. -- sasha From pearu at scipy.org Tue Jan 24 10:32:03 2006 From: pearu at scipy.org (Pearu Peterson) Date: Tue Jan 24 10:32:03 2006 Subject: [Numpy-discussion] bdist_egg trips up over non-extension extensions In-Reply-To: <43D6529C.8030105@gmail.com> References: <43D6529C.8030105@gmail.com> Message-ID: On Tue, 24 Jan 2006, Robert Kern wrote: > In particular, numpy can't build into an egg currently because > numpy/__config__.py is created using Configuration.add_extension() and a > generator function. In order to make extension modules usable from zipped eggs, > setuptools generates a pure Python stub loader that will extract the extension > module to a .hidden directory and load it from there. In this case, since the > Extension isn't actually an extension, the stub loader is broken, because it > expects a numpy/__config__.so. Could you try recent version of numpy from SVN? __config__.py is now generated via py_modules list. Pearu From jmiller at stsci.edu Tue Jan 24 10:46:01 2006 From: jmiller at stsci.edu (Todd Miller) Date: Tue Jan 24 10:46:01 2006 Subject: [Numpy-discussion] Speed performance on array constant set In-Reply-To: <43D59D0B.3060907@mitre.org> References: <43D0573E.4090301@mitre.org> <43D05BA0.7000700@ieee.org> <43D14AC0.3060902@mitre.org> <43D51669.6080308@noaa.gov> <43D57262.6090004@mitre.org> <43D5768C.4090200@ee.byu.edu> <43D59D0B.3060907@mitre.org> Message-ID: <43D675A7.4070906@stsci.edu> Mark Heslep wrote: > Travis Oliphant wrote: > >>> 1. numdata.h example that works well for simple Cv structures >>> containing uniform types: CvPoint2D32F => struct { float x, float >>> y }. Can I use a record array here, or is there some Numpy overhead >>> interlaced with fields? >> >> >> >> You can use a record array, but for uniform types I don't know what >> the advantage is (except for perhaps named-field slicing---which may >> actually be faster I'm not sure) over a x2 float array. > > > So that I can step through the array record by record and not field by > field. > >>> 2. numarray.h Attempt to replace the main Cv Image structures >>> CvMat, IplImage. Needs work. Some success but there's segfault or >>> two in there somewhere. >> >> >> These can be ported fairly easily, I think (actually, the old Numeric >> typemaps would still work --- and work with Numarray), so the basic >> Numeric C-API still presents itself as the simplest way to support >> all the array packages. > > > Well good to know. Ive been proceeding directly from the NumArray > Users Manual 1.5 by Greenfield/Miller et al that describes the NA High > Level API as the fastest of the APIs available to NA. The Numeric compatibility layer in numarray is almost identical in performance with the High Level API. I recommend using the Numeric compatibility layer because it makes it easier to port your application back to Numeric or forward to NumPy. > I thought that NA also took steps to insure that unnecessary mem > copies were not made, unlike Numeric? NA does provide flexibility and support for managing mis-behaved (byteswapped, mis-aligned, mis-typed) arrays, but these extra properties weren't a concern for Numeric. Both numarray's High Level API and numarray's Numeric compatibility layer generally make whole copies of arrays as required to present well behaved representations of data to C. Regards, Todd From robert.kern at gmail.com Tue Jan 24 12:24:12 2006 From: robert.kern at gmail.com (Robert Kern) Date: Tue Jan 24 12:24:12 2006 Subject: [Numpy-discussion] bdist_egg trips up over non-extension extensions In-Reply-To: References: <43D6529C.8030105@gmail.com> Message-ID: <43D68CC4.1010004@gmail.com> Pearu Peterson wrote: > Could you try recent version of numpy from SVN? __config__.py is now > generated via py_modules list. It doesn't build an egg, yet, because setuptools expects Distribution.py_modules to contain only strings, not tuples. Modifying setuptools to handle tuples in Distribution.py_modules seems to work. Is there any way you can do it without tuples? -- Robert Kern robert.kern at gmail.com "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From pearu at scipy.org Tue Jan 24 12:37:05 2006 From: pearu at scipy.org (Pearu Peterson) Date: Tue Jan 24 12:37:05 2006 Subject: [Numpy-discussion] bdist_egg trips up over non-extension extensions In-Reply-To: <43D68CC4.1010004@gmail.com> References: <43D6529C.8030105@gmail.com> <43D68CC4.1010004@gmail.com> Message-ID: On Tue, 24 Jan 2006, Robert Kern wrote: > Pearu Peterson wrote: >> Could you try recent version of numpy from SVN? __config__.py is now >> generated via py_modules list. > > It doesn't build an egg, yet, because setuptools expects Distribution.py_modules > to contain only strings, not tuples. Modifying setuptools to handle tuples in > Distribution.py_modules seems to work. Is there any way you can do it without > tuples? Source generators in numpy.distutils are extensions to distutils and if setuptools assumes std distutils then source generators will remain a problem for setuptools. However, if you call build_src then all source generators will be resolved (including 3-tuples in py_modules list) and the file lists in distribution instance should not cause any problems for setuptools. So, would calling `setup.py build_src bdist_egg` be an acceptable solution? __config__.py files could also be handled as data files but that would require hooks in numpy.core.setup function and we want to get rid of numpy.core.setup in future. Pearu From Norbert.Nemec.list at gmx.de Tue Jan 24 13:11:00 2006 From: Norbert.Nemec.list at gmx.de (Norbert Nemec) Date: Tue Jan 24 13:11:00 2006 Subject: [Numpy-discussion] Matrix/Vector norm? In-Reply-To: <43D0A290.8080600@ieee.org> References: <43D09F8F.6080702@gmx.de> <43D0A290.8080600@ieee.org> Message-ID: <43D69794.8030609@gmx.de> Travis Oliphant wrote: > Norbert Nemec wrote: > >> Hi there, >> >> im missing a norm function in NumPy. Writing one by hand is simple, but >> doing so both general and efficient is a lot more tricky. >> >> Ideas? >> >> > There's one in scipy (i'm not sure if it's the best of breed, but it's > a starting point). > > Look in scipy.linalg.norm Is there a reason not to have it in numpy.linalg? That would be the most logical place for it, I would guess. From rowen at cesmail.net Tue Jan 24 14:01:06 2006 From: rowen at cesmail.net (Russell E. Owen) Date: Tue Jan 24 14:01:06 2006 Subject: [Numpy-discussion] Re: Efficient way to handle None->nan? References: <43D16C31.3060209@ee.byu.edu> Message-ID: In article <43D16C31.3060209 at ee.byu.edu>, Travis Oliphant wrote: > Russell E. Owen wrote: > > >We're getting numeric data from a (MySQL) database. We'd like to use > >numarray or NumPy on the resulting data, but some values may be None. Is > >there a fast, efficient way to replace None with NaN? I'd hate to use a > >list comprehension on each data tuple before turning it into an array, > >but I haven't thought of anything else. > > > > > > Current numpy SVN allows > > array([1.0,None,2.0],dtype=float) > array([ 1. , nan, 2. ]) That's great! Thanks!! -- Russell From agn at noc.soton.ac.uk Tue Jan 24 14:33:07 2006 From: agn at noc.soton.ac.uk (agn at noc.soton.ac.uk) Date: Tue Jan 24 14:33:07 2006 Subject: [Numpy-discussion] dtype=float32 not recognised. In-Reply-To: <43D69794.8030609@gmx.de> References: <43D09F8F.6080702@gmx.de> <43D0A290.8080600@ieee.org> <43D69794.8030609@gmx.de> Message-ID: <1138141838.43d6aa8ee811e@webmail.noc.soton.ac.uk> A curious problem (0.9.5.1993): n [12]: theta3 =numpy.zeros(66,dtype=float32) --------------------------------------------------------------------------- exceptions.NameError Traceback (most recent call last) /working/jrd/mod2/agn/OCCAM/AGN_GLOBAL/ NameError: name 'float32' is not defined But n [14]: theta3 =numpy.zeros(66,dtype='f') In [15]: theta3 Out[15]: array([ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.], dtype=float32) George Nuresr ----------------------------------------------------------------------- National Oceanography Centre, Southampton :: http://www.noc.soton.ac.uk From ndarray at mac.com Tue Jan 24 16:57:05 2006 From: ndarray at mac.com (Sasha) Date: Tue Jan 24 16:57:05 2006 Subject: [Numpy-discussion] dtype=float32 not recognised. In-Reply-To: <1138141838.43d6aa8ee811e@webmail.noc.soton.ac.uk> References: <43D09F8F.6080702@gmx.de> <43D0A290.8080600@ieee.org> <43D69794.8030609@gmx.de> <1138141838.43d6aa8ee811e@webmail.noc.soton.ac.uk> Message-ID: You did not import float32. Your options: >>> from numpy import * >>> zeros(5, dtype=float32) array([ 0., 0., 0., 0., 0.], dtype=float32) or >>> import numpy >>> numpy.zeros(5, dtype=numpy.float32) array([ 0., 0., 0., 0., 0.], dtype=float32) or >>> import numpy >>> numpy.zeros(5, dtype=float) array([ 0., 0., 0., 0., 0.]) -- sasha On 1/24/06, agn at noc.soton.ac.uk wrote: > A curious problem (0.9.5.1993): > > n [12]: theta3 =numpy.zeros(66,dtype=float32) > --------------------------------------------------------------------------- > exceptions.NameError Traceback (most recent call > last) > > /working/jrd/mod2/agn/OCCAM/AGN_GLOBAL/ > > NameError: name 'float32' is not defined > > But > n [14]: theta3 =numpy.zeros(66,dtype='f') > > In [15]: theta3 > Out[15]: > array([ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., > 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., > 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., > 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., > 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.], > dtype=float32) > > George Nuresr > > > ----------------------------------------------------------------------- > National Oceanography Centre, Southampton :: http://www.noc.soton.ac.uk > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From mithrandir42 at web.de Tue Jan 24 23:03:01 2006 From: mithrandir42 at web.de (N. Volbers) Date: Tue Jan 24 23:03:01 2006 Subject: [Numpy-discussion] some question on new dtype Message-ID: <43D7225E.20306@web.de> Hello everyone on the list! I have been playing around with the latest and greatest numpy 0.94 and its dtype mechanism. I am especially interested in using the record-array-like facilities, e.g. the following which is based on an example from a mail of Travis to this list: <-- # define array with three "columns". dtype = numpy.dtype( {'names': ['name', 'age', 'weight'], 'formats': ['U30', 'i2', numpy.float32]} ) a = numpy.array( [(u'Bill', 31, 260), (u'Fred', 15, 135)], dtype=dtype ) # specify column by key print a ['name'] print a['age'] print a['weight'] #print a['object'] # specify row by number print a[0] print a[1] # first item of row 1 (Fred's age) print a[1]['age'] # first item of name column (name 'Bill') print a['name'][0] --> I now have a few questions, maybe someone can help me with them: 1) When reading the sample chapter from Travis' documentation, I noticed that there is also a type 'object' with the character 'O'. So I kind of hoped that it would be possible to have arbitrary python objects in an array. However, when I add a fourth "column" of type 'O', then numpy will mem-fault. Is this not allowed or is this some implementation bug? 2) Is it possible to rename the type descriptors? For my application, I need to treat these names as keys of dataset columns, so it should be possible to rename these. More generally speaking: Is it possible to alter parts of the dtype after instantiation? Of course it should be possible to copy the dtype, modify it accordingly and create a new array. However, maybe there is a suggested way to doing this? 3) When I use two identical entries in the names part of the dtype, I get the message 'TypeError: expected a readable buffer object'. It makes sense that it is not allowed to have two identical names, but I think the error message should be worded more descriptive. 4) In the example above, printing any of the strings via 'print' will yield the characters and then the characters up to the string size filled up with \x00, e.g. u'Bill\x00\x00\x00\x00\x00\x00\x00.... (30 characters total)' Why doesn't 'prin't terminate the output when the first \x00 is reached ? Overall I am very much impressed by the new numpy and I thank everyone who contributes to this! Niklas Volbers. From oliphant.travis at ieee.org Tue Jan 24 23:34:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue Jan 24 23:34:02 2006 Subject: [Numpy-discussion] some question on new dtype In-Reply-To: <43D7225E.20306@web.de> References: <43D7225E.20306@web.de> Message-ID: <43D729D9.6020608@ieee.org> N. Volbers wrote: > Hello everyone on the list! > > I have been playing around with the latest and greatest numpy 0.94 and > its dtype mechanism. I am especially interested in using the > record-array-like facilities, e.g. the following which is based on an > example from a mail of Travis to this list: > > <-- > # define array with three "columns". > dtype = numpy.dtype( {'names': ['name', 'age', 'weight'], > 'formats': ['U30', 'i2', numpy.float32]} ) > a = numpy.array( [(u'Bill', 31, 260), (u'Fred', 15, 135)], dtype=dtype ) > > # specify column by key > print a ['name'] > print a['age'] > print a['weight'] > #print a['object'] > > # specify row by number > print a[0] > print a[1] > > # first item of row 1 (Fred's age) > print a[1]['age'] > > # first item of name column (name 'Bill') > print a['name'][0] > --> > > I now have a few questions, maybe someone can help me with them: > > 1) When reading the sample chapter from Travis' documentation, I > noticed that there is also a type 'object' with the character 'O'. So > I kind of hoped that it would be possible to have arbitrary python > objects in an array. However, when I add a fourth "column" of type > 'O', then numpy will mem-fault. Is this not allowed or is this some > implementation bug? It's a bug if it seg-faults, that should be allowed. Please post your code so we can track it down. > > 2) Is it possible to rename the type descriptors? For my application, > I need to treat these names as keys of dataset columns, so it should > be possible to rename these. More generally speaking: Is it possible > to alter parts of the dtype after instantiation? Of course it should > be possible to copy the dtype, modify it accordingly and create a new > array. However, maybe there is a suggested way to doing this? Right now, you would have to construct a new data-type with the new field names, There may be some ways we could make this easier, though. The big thing is that you can't change data-types in-place because they are used by other array objects. Perhaps the easiest way to do this is by getting the arrdescr attribute of the dtype (it's a list of tuples) and then constructing a new list of tuples using it (replaceing the tuples you want to rename) and then using that in the dtype constructor: Something like this: descriptor = a.dtype.arrdescr # this could probably be renamed to descr now that we're # not using that word. # or a.__array_descr__ retrieves the same thing. field = descriptor[field_num] descriptor[field_num] = (newname,)+field[1:] newdtype = numpy.dtype(descriptor) Then you can do: a.dtype = newdtype and have a new field name. Actually, it would probably be faster to do: new = dict(a.dtype.fields) # get a writeable dictionary. new[''] = new[''] del new[''] del new[-1] # get rid of the special ordering entry a.dtype = dtype(new) > > 3) When I use two identical entries in the names part of the dtype, I > get the message 'TypeError: expected a readable buffer object'. It > makes sense that it is not allowed to have two identical names, but I > think the error message should be worded more descriptive. Yeah, we ought to check for this. Could you post your code that raises this error. > > 4) In the example above, printing any of the strings via 'print' will > yield the characters and then the characters up to the string size > filled up with \x00, e.g. > > u'Bill\x00\x00\x00\x00\x00\x00\x00.... (30 characters total)' > > Why doesn't 'prin't terminate the output when the first \x00 is reached ? Because I don't know the Unicode equivalent to PyString_FromString (which is what I'm using to automatically truncate the fixed-field string before printing it). UNICODE_getitem in arraytypes.inc.src is the relevant function. See STRING_getitem for comparison. I think it would be better to truncate it if anybody has a good suggestion... -Travis From oliphant.travis at ieee.org Wed Jan 25 00:24:01 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed Jan 25 00:24:01 2006 Subject: [Numpy-discussion] some question on new dtype In-Reply-To: <43D7225E.20306@web.de> References: <43D7225E.20306@web.de> Message-ID: <43D73570.4070201@ieee.org> N. Volbers wrote: > 4) In the example above, printing any of the strings via 'print' will > yield the characters and then the characters up to the string size > filled up with \x00, e.g. > > u'Bill\x00\x00\x00\x00\x00\x00\x00.... (30 characters total)' > > Why doesn't 'prin't terminate the output when the first \x00 is reached ? O.K. I finally broke down and fixed this in the SVN version. Perhaps people can try it out for bugs. -Travis From faltet at carabos.com Wed Jan 25 00:26:03 2006 From: faltet at carabos.com (Francesc Altet) Date: Wed Jan 25 00:26:03 2006 Subject: [Numpy-discussion] dtype=float32 not recognised. In-Reply-To: References: <43D09F8F.6080702@gmx.de> <1138141838.43d6aa8ee811e@webmail.noc.soton.ac.uk> Message-ID: <200601250925.42792.faltet@carabos.com> A Dimecres 25 Gener 2006 01:56, Sasha va escriure: > You did not import float32. > > Your options: > >>> from numpy import * > >>> zeros(5, dtype=float32) > > array([ 0., 0., 0., 0., 0.], dtype=float32) > > or > > >>> import numpy > >>> numpy.zeros(5, dtype=numpy.float32) > > array([ 0., 0., 0., 0., 0.], dtype=float32) > > or > > >>> import numpy > >>> numpy.zeros(5, dtype=float) > > array([ 0., 0., 0., 0., 0.]) Another one: >>> import numpy >>> numpy.zeros(5, dtype="float32") array([ 0., 0., 0., 0., 0.], dtype=float32) -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From faltet at carabos.com Wed Jan 25 00:41:11 2006 From: faltet at carabos.com (Francesc Altet) Date: Wed Jan 25 00:41:11 2006 Subject: [Numpy-discussion] some question on new dtype In-Reply-To: <43D729D9.6020608@ieee.org> References: <43D7225E.20306@web.de> <43D729D9.6020608@ieee.org> Message-ID: <200601250940.34655.faltet@carabos.com> A Dimecres 25 Gener 2006 08:33, Travis Oliphant va escriure: > Something like this: > > descriptor = a.dtype.arrdescr # this could probably be renamed to > # descr now that we're > # not using that word. +1 for this That way we should have (in case of a int32 type): >>> dtype.num 11 >>> dtype.char 'f' >>> dtype.str '>> dtype.name 'float32' >>> dtype.descr [('', '0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From a.h.jaffe at gmail.com Wed Jan 25 01:57:09 2006 From: a.h.jaffe at gmail.com (Andrew Jaffe) Date: Wed Jan 25 01:57:09 2006 Subject: [Numpy-discussion] Weird (wrong?) real fft 2d behavior Message-ID: <43D74B35.9060204@gmail.com> If I start with what I thought was an appropriate (n, n/2+1) complex matrix which should have a real inverse fft, and then take its real fft, I don't get back the original matrix. Note the presence of very small but nonzero reals in the final matrix, and the fact that the 2d and 4th rows are duplicates. This seems to be a mistake somewhere. Or am I just misunderstanding/misremembering something about 2d real ffts? Andrew In [41]: dims=(4,4) In [42]: delta_k = aniso.dft_realizn(dims, P) #### this just makes an appropriate matrix In [43]: delta_k Out[43]: array([[-0. +0.j , 0.5904+0.0429j, 0.9063+0.j ], [-0.221 +0.j , 0.4169+0.4602j, -0.6909+0.j ], [ 0.9059+0.j , -0.8037+0.2604j, 1.835 +0.j ], [ 1.7436+0.j , -0.2382+0.221j , -0.0607+0.j ]]) #### 16 real numbers In [44]: delta_r = N.dft.inverse_real_fft2d(delta_k) In [45]: delta_r Out[45]: array([[ 0.2718, -0.0956, 0.2805, 0.1505], [ 0.0297, -0.0533, -0.259 , 0.0561], [ 0.1308, -0.2096, 0.2288, -0.3041], [ 0.0895, 0.1105, -0.3188, -0.1076]]) In [46]: delta_kp = N.dft.real_fft2d(delta_r) In [47]: delta_kp Out[47]: array([[ 0. +0.00e+00j, 0.5904 +4.2938e-02j, 0.9063 +0.0000e+00j], [ 0.7613 +5.5511e-17j,0.4169 +4.6020e-01j, -0.3758 +5.5511e-17j], [ 0.9059 +0.0000e+00j,-0.8037+2.6038e-01j, 1.835 +0.0000e+00j], [ 0.7613 -5.5511e-17j,-0.2382+2.2099e-01j,-0.3758-5.5511e-17j]]) In [48]: N.__version__ Out[48]: '0.9.5.1982' ------------------------ From a.h.jaffe at gmail.com Wed Jan 25 02:05:03 2006 From: a.h.jaffe at gmail.com (Andrew Jaffe) Date: Wed Jan 25 02:05:03 2006 Subject: [Numpy-discussion] Re: Weird (wrong?) real fft 2d behavior In-Reply-To: <43D74B35.9060204@gmail.com> References: <43D74B35.9060204@gmail.com> Message-ID: <43D74D17.7010103@gmail.com> Andrew Jaffe wrote: > If I start with what I thought was an appropriate (n, n/2+1) complex > matrix which should have a real inverse fft, and then take its real fft, > I don't get back the original matrix. > > Note the presence of very small but nonzero reals in the final matrix, > and the fact that the 2d and 4th rows are duplicates. This seems to be a > mistake somewhere. > > Or am I just misunderstanding/misremembering something about 2d real ffts? and I should point out that delta_rp = N.dft.real_fft2d(delta_kp) is 'allclose' to the original delta_r (which leads me to believe that I may indeed be misunderstanding something). Andrew > > Andrew > > > In [41]: dims=(4,4) > > In [42]: delta_k = aniso.dft_realizn(dims, P) > #### this just makes an appropriate matrix > > In [43]: delta_k > Out[43]: > array([[-0. +0.j , 0.5904+0.0429j, 0.9063+0.j ], > [-0.221 +0.j , 0.4169+0.4602j, -0.6909+0.j ], > [ 0.9059+0.j , -0.8037+0.2604j, 1.835 +0.j ], > [ 1.7436+0.j , -0.2382+0.221j , -0.0607+0.j ]]) > #### 16 real numbers > > In [44]: delta_r = N.dft.inverse_real_fft2d(delta_k) > > In [45]: delta_r > Out[45]: > array([[ 0.2718, -0.0956, 0.2805, 0.1505], > [ 0.0297, -0.0533, -0.259 , 0.0561], > [ 0.1308, -0.2096, 0.2288, -0.3041], > [ 0.0895, 0.1105, -0.3188, -0.1076]]) > > In [46]: delta_kp = N.dft.real_fft2d(delta_r) > > In [47]: delta_kp > Out[47]: > array([[ 0. +0.00e+00j, 0.5904 +4.2938e-02j, 0.9063 +0.0000e+00j], > [ 0.7613 +5.5511e-17j,0.4169 +4.6020e-01j, -0.3758 +5.5511e-17j], > [ 0.9059 +0.0000e+00j,-0.8037+2.6038e-01j, 1.835 +0.0000e+00j], > [ 0.7613 -5.5511e-17j,-0.2382+2.2099e-01j,-0.3758-5.5511e-17j]]) > > In [48]: N.__version__ > Out[48]: '0.9.5.1982' > ------------------------ From faltet at carabos.com Wed Jan 25 04:16:02 2006 From: faltet at carabos.com (Francesc Altet) Date: Wed Jan 25 04:16:02 2006 Subject: [Numpy-discussion] Creating multirow numpy rec.arrays Message-ID: <200601251314.54933.faltet@carabos.com> Hi, This exposes a weird behaviour when building heterogeneous arrays in numpy: In [85]: xcol = numpy.ones((3,2), 'int32') In [86]: numpy.array([(xcol[0],)], dtype=[('x','i4', (2,))]) Out[86]: array([(array([1, 1]),)], dtype=(void,8)) So far so good. Now: In [89]: numpy.array([xcol], dtype=[('x','i4', (2,))]) array([[[(array([-1208633192, -1208633192]),), (array([15, 15]),)], [(array([ 0, 185]),), (array([ 18, -1208633016]),)], [(array([480, 21]),), (array([21, 21]),)]]], dtype=(void,8)) While I'd expect: In [89]: numpy.array([xcol], dtype=[('x','i4', (2,))]) Out[89]: array([(array([1, 1]), (array([1, 1]), (array([1, 1]))], dtype=(void,8)) Also, for the whishlist: It would be nice if one can specify the shape of the array to be built in the factory itself. I mean, allowing something like: In [90]: numpy.array([xcol], dtype=[('x','i4', (2,))], shape=2) Out[90]: array([(array([1, 1]), (array([1, 1]))], dtype=(void,8)) Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From borreguero at gmail.com Wed Jan 25 04:53:05 2006 From: borreguero at gmail.com (Jose Borreguero) Date: Wed Jan 25 04:53:05 2006 Subject: [Numpy-discussion] installation under other than .../site-packages Message-ID: <7cced4ed0601250452x197f0d08i@mail.gmail.com> My system administrator has told me numpy has to be installed under /local , instead of under /usr/lib/pythonx.x/site-packages. Can anybody point to me what modifications do I have to make for this. ? Also, will other libraries that call numpy work after I install ? jose -------------- next part -------------- An HTML attachment was scrubbed... URL: From daniel.marti at upf.edu Wed Jan 25 08:35:01 2006 From: daniel.marti at upf.edu (Dani Marti) Date: Wed Jan 25 08:35:01 2006 Subject: [Numpy-discussion] [Newbie] Matching a subarray Message-ID: <20060125162603.GA1765@upf.edu> Hi all, I have a long 1D boolean array. I'd like to know where a certain 'sub'array (like, e.g., ([1,1,1,1,1,1])) occurs in the array. That is, I want the index of the first occurrence of a subarray in an array, avoiding the use of explicit loop structures to carry out the search. I am sorry if this happens to be a trivial question, but I was not able to find the answer from the documentation. Thanks a lot, dani From chris at trichech.us Wed Jan 25 09:23:10 2006 From: chris at trichech.us (Christopher Fonnesbeck) Date: Wed Jan 25 09:23:10 2006 Subject: [Numpy-discussion] error in beta random variate generation Message-ID: There appears to be a bug in the beta random number generator in numpy. A 2-parameter beta distribution has an expected value of a/(a+b). Hence, a beta(1,1) should have an expected value of 0.5 (since it is the equivalent of a uniform random variable on [0,1]). However, numpy gives the following: In [6]: from numpy.random import beta In [7]: from numpy import * In [8]: mean(beta(1,1,size=10000)) Out[8]: 0.33397400928538834 -- Christopher J. Fonnesbeck Population Ecologist, Marine Mammal Section Fish & Wildlife Research Institute (FWC) St. Petersburg, FL Adjunct Assistant Professor Warnell School of Forest Resources University of Georgia Athens, GA T: 727.235.5570 E: chris at trichech.us -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2417 bytes Desc: not available URL: From oliphant.travis at ieee.org Wed Jan 25 10:47:08 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Wed Jan 25 10:47:08 2006 Subject: [Numpy-discussion] installation under other than .../site-packages In-Reply-To: <7cced4ed0601250452x197f0d08i@mail.gmail.com> References: <7cced4ed0601250452x197f0d08i@mail.gmail.com> Message-ID: <43D7C791.7030106@ieee.org> Jose Borreguero wrote: > My system administrator has told me numpy has to be installed under > /local , instead of under /usr/lib/pythonx.x/site-packages. Can > anybody point to me what modifications do I have to make for this. ? > Also, will other libraries that call numpy work after I install ? > jose > As long as you change the sys.path list and insert the location where numpy is installed, it should work and other packages should find it. I think modification of the PYTHONPATH environment variable will do this for all scripts as well. -Travis From cookedm at physics.mcmaster.ca Wed Jan 25 11:05:00 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Wed Jan 25 11:05:00 2006 Subject: [Numpy-discussion] installation under other than .../site-packages In-Reply-To: <7cced4ed0601250452x197f0d08i@mail.gmail.com> References: <7cced4ed0601250452x197f0d08i@mail.gmail.com> Message-ID: On Jan 25, 2006, at 07:52 , Jose Borreguero wrote: > My system administrator has told me numpy has to be installed > under /local , instead of under /usr/lib/pythonx.x/site-packages. > Can anybody point to me what modifications do I have to make for > this. ? Also, will other libraries that call numpy work after I > install ? > jose First, to install it under /usr/local, run setup.py like this: $ python setup.py --prefix=/usr/local This will put numpy into /usr/local/lib/pythonx.x/site-packages. Next, check if you can import it. If you can't, it's likely that that path is not in your sys.path. Probably the easiest way to make it available for _all_ users is to ask your sysadmin to put a file called 'usrlocal.pth' in /usr/lib/pythonx.x/site-packages with these contents import site; site.addsitedir('/usr/local/lib/pythonx.x') (hopefully he'd be amiable to adding only one file :) This adds /usr/ local/lib/pythonx.x to the sys.path *and* processes any .pth in it. You can get fancier and add import os, site; site.addsitedir(os.path.expanduser('~/lib/pythonx.x')) which would also look in user directories. Failing that, you'd set the environment variable PYTHONPATH to include the right directory. This way, .pth files won't be processed. You could add a sitecustomize.py with the appropriate 'import site; site.addsitedir(...)'. The addsitedir is what processes .pth files. These are all generic instructions, not specific to numpy. numpy itself doesn't need .pth files, but there are other packages that do (Numeric, PIL, pygtk, amongst others), so it's handy to have. -- |>|\/|< /------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From oliphant at ee.byu.edu Wed Jan 25 11:41:08 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed Jan 25 11:41:08 2006 Subject: [Numpy-discussion] [Newbie] Matching a subarray In-Reply-To: <20060125162603.GA1765@upf.edu> References: <20060125162603.GA1765@upf.edu> Message-ID: <43D7D418.9090802@ee.byu.edu> Dani Marti wrote: >Hi all, > >I have a long 1D boolean array. I'd like to know where a certain >'sub'array (like, e.g., ([1,1,1,1,1,1])) occurs in the array. That is, I >want the index of the first occurrence of a subarray in an array, >avoiding the use of explicit loop structures to carry out the search. > > Actually, I don't think that's a trivial question. You could look at the data as a string and use the find method of the string. Or you could use (int8) correlation. The largest value of the correlation should be the index where it matches. -Travis From mithrandir42 at web.de Wed Jan 25 12:08:02 2006 From: mithrandir42 at web.de (N. Volbers) Date: Wed Jan 25 12:08:02 2006 Subject: [Numpy-discussion] some question on new dtype In-Reply-To: <43D729D9.6020608@ieee.org> References: <43D7225E.20306@web.de> <43D729D9.6020608@ieee.org> Message-ID: <43D7DA54.10307@web.de> Thanks for your quick answer! >> 1) When reading the sample chapter from Travis' documentation, I >> noticed that there is also a type 'object' with the character 'O'. So >> I kind of hoped that it would be possible to have arbitrary python >> objects in an array. However, when I add a fourth "column" of type >> 'O', then numpy will mem-fault. Is this not allowed or is this some >> implementation bug? > > > It's a bug if it seg-faults, that should be allowed. Please post your > code so we can track it down. Attachment 1 contains a four-liner that reproduces the segfault. But maybe there is something wrong on my system, because I also get the following message when importing numpy: "import linalg -> failed: /usr/lib/python2.4/site-packages/numpy/linalg/lapack_lite.so: undefined symbol: s_wsfe" I removed numpy, checked out from SVN, re-installed and the error is still there. Regarding my question about changing dtypes: Your sample code... > new = dict(a.dtype.fields) # get a writeable dictionary. > new[''] = new[''] > del new[''] > del new[-1] # get rid of the special ordering entry > a.dtype = dtype(new) > ... is short and exactly what I asked for. >> >> 3) When I use two identical entries in the names part of the dtype, I >> get the message 'TypeError: expected a readable buffer object'. It >> makes sense that it is not allowed to have two identical names, but I >> think the error message should be worded more descriptive. > > > Yeah, we ought to check for this. Could you post your code that > raises this error. It seems you have already fixed the error message for the case in which you have two identical names for dtype entries. Man, you are fast! But... ;-) you forgot the situation described in my second code attachment, which is the following situation: >>> dtype = numpy.dtype( [('name', 'S30'), ('name', 'i2')] ) >>> a = numpy.zeros( (4,), dtype=dtype) This works, but leads to silly results. The last instance of 'name' determines the field type, i.e. a[0] = ('Niklas', 1) is invalid, while a[0] = (1,1) is valid! >> 4) In the example above, printing any of the strings via 'print' will >> yield the characters and then the characters up to the string size >> filled up with \x00, e.g. >> >> u'Bill\x00\x00\x00\x00\x00\x00\x00.... (30 characters total)' >> >> Why doesn't 'prin't terminate the output when the first \x00 is reached ? > Thanks for your bug-fix. Unicode display works now. Best regards, Niklas Volbers. -------------- next part -------------- A non-text attachment was scrubbed... Name: dtype_test_1.py Type: text/x-python Size: 98 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: dtype_test_2.py Type: text/x-python Size: 484 bytes Desc: not available URL: From mithrandir42 at web.de Wed Jan 25 12:37:04 2006 From: mithrandir42 at web.de (N. Volbers) Date: Wed Jan 25 12:37:04 2006 Subject: [Numpy-discussion] iterating over array items Message-ID: <43D7E149.9040909@web.de> Hello everyone, the attached source code contains iterations on elements in a numpy array: The first one (for row in a) works, the second one (for item in row) does not. The non-numpy equivalent with lists and tuples works however. Best regards, Niklas Volbers. -------------- next part -------------- A non-text attachment was scrubbed... Name: iterating.py Type: text/x-python Size: 586 bytes Desc: not available URL: From oliphant at ee.byu.edu Wed Jan 25 13:37:03 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed Jan 25 13:37:03 2006 Subject: [Numpy-discussion] some question on new dtype In-Reply-To: <43D7DA54.10307@web.de> References: <43D7225E.20306@web.de> <43D729D9.6020608@ieee.org> <43D7DA54.10307@web.de> Message-ID: <43D7EF5B.8050808@ee.byu.edu> N. Volbers wrote: > Thanks for your quick answer! > >>> 1) When reading the sample chapter from Travis' documentation, I >>> noticed that there is also a type 'object' with the character 'O'. >>> So I kind of hoped that it would be possible to have arbitrary >>> python objects in an array. However, when I add a fourth "column" of >>> type 'O', then numpy will mem-fault. Is this not allowed or is this >>> some implementation bug? >> >> >> >> It's a bug if it seg-faults, that should be allowed. Please post >> your code so we can track it down. > > > Attachment 1 contains a four-liner that reproduces the segfault. But > maybe there is something wrong on my system, because I also get the > following message when importing numpy: "import linalg -> failed: > /usr/lib/python2.4/site-packages/numpy/linalg/lapack_lite.so: > undefined symbol: s_wsfe" > > I removed numpy, checked out from SVN, re-installed and the error is > still there. > > Regarding my question about changing dtypes: Your sample code... > >> new = dict(a.dtype.fields) # get a writeable dictionary. >> new[''] = new[''] >> del new[''] >> del new[-1] # get rid of the special ordering entry >> a.dtype = dtype(new) >> > ... is short and exactly what I asked for. > >>> >>> 3) When I use two identical entries in the names part of the dtype, >>> I get the message 'TypeError: expected a readable buffer object'. It >>> makes sense that it is not allowed to have two identical names, but >>> I think the error message should be worded more descriptive. >> >> >> >> Yeah, we ought to check for this. Could you post your code that >> raises this error. > > > It seems you have already fixed the error message for the case in > which you have two identical names for dtype entries. Man, you are > fast! But... ;-) you forgot the situation described in my second code > attachment, which is the following situation: > > >>> dtype = numpy.dtype( [('name', 'S30'), ('name', 'i2')] ) > >>> a = numpy.zeros( (4,), dtype=dtype) > > This works, but leads to silly results. The last instance of 'name' > determines the field type, i.e. a[0] = ('Niklas', 1) is invalid, while > a[0] = (1,1) is valid! > >>> 4) In the example above, printing any of the strings via 'print' >>> will yield the characters and then the characters up to the string >>> size filled up with \x00, e.g. >>> >>> u'Bill\x00\x00\x00\x00\x00\x00\x00.... (30 characters total)' >>> >>> Why doesn't 'prin't terminate the output when the first \x00 is >>> reached ? >> >> > > Thanks for your bug-fix. Unicode display works now. > > > Best regards, > > Niklas Volbers. > >------------------------------------------------------------------------ > >import numpy >dtype = numpy.dtype( [('object', 'O')] ) >a = numpy.zeros( (4,), dtype=dtype) >print a > > > > I know what the problem is. Too many places, I did not account for the fact that a VOID array could in-fact have OBJECT sub-fields. Thus, any special processing on OBJECT subfields to handle reference counts must also be done on possible sub-fields. This may take a while to fix properly.... If anyone has any ideas, let me know. -Travis From oliphant at ee.byu.edu Wed Jan 25 15:03:06 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed Jan 25 15:03:06 2006 Subject: [Numpy-discussion] some question on new dtype In-Reply-To: <43D7EF5B.8050808@ee.byu.edu> References: <43D7225E.20306@web.de> <43D729D9.6020608@ieee.org> <43D7DA54.10307@web.de> <43D7EF5B.8050808@ee.byu.edu> Message-ID: <43D80371.70103@ee.byu.edu> Travis Oliphant wrote: > I know what the problem is. Too many places, I did not account for > the fact that a VOID array could in-fact have OBJECT sub-fields. > Thus, any special processing on OBJECT subfields to handle reference > counts must also be done on possible sub-fields. > This may take a while to fix properly.... If anyone has any ideas, > let me know. > -Travis For now, I've added a hasobject member to the dtype object structure and filled it in properly on construction. Then, I disable construction of such arrays so that we don't get the segfault. I think this is too harsh. I think at some point we can offer some support for them (there may be a few methods and functions that don't work and we can give an error on those particular ones rather than not allow their construction). -Travis From MTHursts at cisp.cc Wed Jan 25 16:11:19 2006 From: MTHursts at cisp.cc (Lane Margarita) Date: Wed Jan 25 16:11:19 2006 Subject: [Numpy-discussion] Re: penicillin Message-ID: <001401c6220c$f4d94730$6acd1153@m4> fill Very little, sir, I am afraid; I answered, speaking to him as rigid. Mr. Spenlow shut the door, motioned me to a chair, and obligatory Debates, to cool. It was one of the irons I began to heat Very little, sir, I am afraid; I answered, speaking to him as -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: lvwbleeding.gif Type: image/gif Size: 14848 bytes Desc: not available URL: From rudolphv at gmail.com Thu Jan 26 00:45:01 2006 From: rudolphv at gmail.com (Rudolph van der Merwe) Date: Thu Jan 26 00:45:01 2006 Subject: [Numpy-discussion] Possible typo in numpy.dft.fft docstring Message-ID: <97670e910601260044v171cbd38i8b42b15fbeef44ee@mail.gmail.com> Part of the fft function's docstring currently reads: "... The packing of the result is "standard": If A = fft(a, n), then A[0] contains the zero-frequency term, A[1:n/2+1] contains the positive-frequency terms, and A[n/2+1:] contains the negative-frequency terms, in order of decreasingly negative frequency. So for an 8-point transform, the frequencies of the result are [ 0, 1, 2, 3, 4, -3, -2, -1]. ..." Shouldn't the frequency arrangement in the last sentence be [0, 1, 2, 3, -4, -3, -2, -1] and not [ 0, 1, 2, 3, 4, -3, -2, -1] ? -- Rudolph van der Merwe From oliphant.travis at ieee.org Thu Jan 26 06:40:08 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Thu Jan 26 06:40:08 2006 Subject: [Numpy-discussion] Possible typo in numpy.dft.fft docstring In-Reply-To: <97670e910601260044v171cbd38i8b42b15fbeef44ee@mail.gmail.com> References: <97670e910601260044v171cbd38i8b42b15fbeef44ee@mail.gmail.com> Message-ID: <43D8DF01.9020607@ieee.org> Rudolph van der Merwe wrote: >Part of the fft function's docstring currently reads: > >"... > >The packing of the result is "standard": If A = fft(a, n), then A[0] > contains the zero-frequency term, A[1:n/2+1] contains the > positive-frequency terms, and A[n/2+1:] contains the negative-frequency > terms, in order of decreasingly negative frequency. So for an 8-point > transform, the frequencies of the result are [ 0, 1, 2, 3, 4, -3, -2, -1]. > >..." > >Shouldn't the frequency arrangement in the last sentence be > >[0, 1, 2, 3, -4, -3, -2, -1] > > > >and not > >[ 0, 1, 2, 3, 4, -3, -2, -1] > > Of course these are completely equivalent because of the periodicity of the DFT, but the statement A[1:n/2+1] contains the positive-frequency terms is consistent with what's given. It think the issue is that this statement is true for both even add odd values of n, while writing it the way you propose would require different statements for odd and even terms. I do typically interpret the middle frequency as "negative" though (as does fftshift). -Travis From focke at slac.stanford.edu Thu Jan 26 18:36:10 2006 From: focke at slac.stanford.edu (Warren Focke) Date: Thu Jan 26 18:36:10 2006 Subject: [Numpy-discussion] Re: Weird (wrong?) real fft 2d behavior In-Reply-To: <43D74D17.7010103@gmail.com> References: <43D74B35.9060204@gmail.com> <43D74D17.7010103@gmail.com> Message-ID: On Wed, 25 Jan 2006, Andrew Jaffe wrote: > Andrew Jaffe wrote: > > If I start with what I thought was an appropriate (n, n/2+1) complex > > matrix which should have a real inverse fft, and then take its real fft, > > I don't get back the original matrix. > > > > Note the presence of very small but nonzero reals in the final matrix, That's just roundoff. > > and the fact that the 2d and 4th rows are duplicates. This seems to be a > > mistake somewhere. > > > > Or am I just misunderstanding/misremembering something about 2d real ffts? It looks wrong to me, and I think I wrote those functions. I get the same results in Numeric. I'll try to look into the problem. > and I should point out that > delta_rp = N.dft.real_fft2d(delta_kp) > is 'allclose' to the original delta_r (which leads me to believe that I > may indeed be misunderstanding something). "Stable" does not neccessarily imply "correct". w > > Andrew > > > > > > Andrew > > > > > > In [41]: dims=(4,4) > > > > In [42]: delta_k = aniso.dft_realizn(dims, P) > > #### this just makes an appropriate matrix > > > > In [43]: delta_k > > Out[43]: > > array([[-0. +0.j , 0.5904+0.0429j, 0.9063+0.j ], > > [-0.221 +0.j , 0.4169+0.4602j, -0.6909+0.j ], > > [ 0.9059+0.j , -0.8037+0.2604j, 1.835 +0.j ], > > [ 1.7436+0.j , -0.2382+0.221j , -0.0607+0.j ]]) > > #### 16 real numbers > > > > In [44]: delta_r = N.dft.inverse_real_fft2d(delta_k) > > > > In [45]: delta_r > > Out[45]: > > array([[ 0.2718, -0.0956, 0.2805, 0.1505], > > [ 0.0297, -0.0533, -0.259 , 0.0561], > > [ 0.1308, -0.2096, 0.2288, -0.3041], > > [ 0.0895, 0.1105, -0.3188, -0.1076]]) > > > > In [46]: delta_kp = N.dft.real_fft2d(delta_r) > > > > In [47]: delta_kp > > Out[47]: > > array([[ 0. +0.00e+00j, 0.5904 +4.2938e-02j, 0.9063 +0.0000e+00j], > > [ 0.7613 +5.5511e-17j,0.4169 +4.6020e-01j, -0.3758 +5.5511e-17j], > > [ 0.9059 +0.0000e+00j,-0.8037+2.6038e-01j, 1.835 +0.0000e+00j], > > [ 0.7613 -5.5511e-17j,-0.2382+2.2099e-01j,-0.3758-5.5511e-17j]]) > > > > In [48]: N.__version__ > > Out[48]: '0.9.5.1982' > > ------------------------ > > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From humufr at yahoo.fr Thu Jan 26 20:05:09 2006 From: humufr at yahoo.fr (Humufr) Date: Thu Jan 26 20:05:09 2006 Subject: [Numpy-discussion] installation under other than .../site-packages In-Reply-To: References: <7cced4ed0601250452x197f0d08i@mail.gmail.com> Message-ID: <43D7DB16.5080104@yahoo.fr> This solution is the one I'm using. Unfortunatly it's not working good with scipy. I tried to install scipy in my own directory and everythong seems to work like a charm but the lapack link is bad. I tried everything write in the wiki to install your own BLAS/LAPACK library but scipy want tu use the one of the system who have a problem. I'm trying to find a solution but until I found it numpy/scipy is not an option. N. David M. Cooke wrote: > On Jan 25, 2006, at 07:52 , Jose Borreguero wrote: > >> My system administrator has told me numpy has to be installed under >> /local , instead of under /usr/lib/pythonx.x/site-packages. Can >> anybody point to me what modifications do I have to make for this. ? >> Also, will other libraries that call numpy work after I install ? >> jose > > First, to install it under /usr/local, run setup.py like this: > > $ python setup.py --prefix=/usr/local > > This will put numpy into /usr/local/lib/pythonx.x/site-packages. > > Next, check if you can import it. If you can't, it's likely that that > path is not in your sys.path. Probably the easiest way to make it > available for _all_ users is to ask your sysadmin to put a file > called 'usrlocal.pth' in /usr/lib/pythonx.x/site-packages with these > contents > > import site; site.addsitedir('/usr/local/lib/pythonx.x') > > (hopefully he'd be amiable to adding only one file :) This adds /usr/ > local/lib/pythonx.x to the sys.path *and* processes any .pth in it. > You can get fancier and add > > import os, site; site.addsitedir(os.path.expanduser('~/lib/pythonx.x')) > > which would also look in user directories. > > Failing that, you'd set the environment variable PYTHONPATH to > include the right directory. This way, .pth files won't be processed. > You could add a sitecustomize.py with the appropriate 'import site; > site.addsitedir(...)'. The addsitedir is what processes .pth files. > > These are all generic instructions, not specific to numpy. numpy > itself doesn't need .pth files, but there are other packages that do > (Numeric, PIL, pygtk, amongst others), so it's handy to have. > From ashen at la.com Thu Jan 26 20:48:01 2006 From: ashen at la.com (=?windows-1251?B?MzEg/+3i4PD/?=) Date: Thu Jan 26 20:48:01 2006 Subject: [Numpy-discussion] =?windows-1251?B?0fLw7ujy5ev87fvlIO7w4+Dt6Ofg9ujoLiDB8/Xj4Ovy5fDx6ujpIPP35fIg6CDt?= =?windows-1251?B?4Ovu4+7u4evu5uXt6OU=?= Message-ID: <075301c62225$0a430df9$43599232@la.com> 31 ?????? ???????????? ???????????.????????????? ???? ? ??????????????? ???????????????? ?????????????: ????????? ?.?. - ???????????? ??????????? ?????? ?? ??????????? ?????????????? ????? ???????????? ???????? ??? ??? ??????, ??????????? ???????? ??? ?????-??????, ????????? ???????: 1 ???? (8 ?????), 10.00-17.00 ?????????? ?????????: ?????????? ? ????????? ???????????? ?????????????? ???????. ????????????? ???????????????? ??????????? ???????? ?????????? ??? ?? 10 ?????.??? ?????? ?? ???????? ? ????????? ?????????????? ?????????? ??????????? ?? ?????????: (095) 1?6-48-65, 1?4-97-?9 ? ?????????: ????? ??????? ???????????? ????????????: ???????? ???????????? ???????????? ? ?????????? ??????????????? ????? ????. ???????? ?????????? ????????? ????? ?????????? ???????????? ????????????. ????????????? ??????????? ??????????? ????????, ??????? ?????????????. ??????? ??????????? ????????????? ?????????????????? ?????????? ? ??????????? ?????????????. ?????????? ????????? ????????? ? ??????????? ???? ???????? ???????. ?????????? ????? ????? ???????????? ?????????? ????????? ???????????? ????????????. ??????????????? ????? ????????? ??????? ?????????? ? ????????????? ? ??????? ?? ??????????. ????????????? ???? ? ??????????????? ???????????? ? ?????????? ???????? ?????????????: ????????????? ???? ???????????? ??? ????????????? ??????????? ??????????? ????????, ??????? ????? ???. ??????? ?????????????? ????? ? ??????????????? ???????????? ??? ??????? ?????????????. ??????????? ?????? ?? ???????? ?????????????. ????????????? ???? ? ??????????????? ?????????? ???????? ????????????? - ??????????? ???. ????????????? ???? ? ??????????????? ?????????????????? ??????????: ???? ? ??????????????? ?????????? ? ??????????? ?? ???????? ???????????? ?? ????????????. ???? ???????, ?????????? ?? ????????????. ?????????????? ??????????. ??????? ????????? ??????? ? ???????? ? ????????????? ????? ? ??? ????? ???????????????. ??? ??? ????????????? ??????? ?????????. ????????????? ???? ? ??????????????? ???????????: ??????? ????????? ??????? ? ????????????? ????? ? ??? ????? ??????????????? ? ???????????? ? ???????????? ???????????? ???????????. ?????? ????????? ?????????? ??? ????? ?????????? ???. ??????? ????????? ???????? ? ????????????? ? ????????? ?????. ???? ??????????. ???? ?????????? ?????????. ????????????? ?????????. ?????????? ?????????. ????????? ?????? ? ??????????. ????????? ????? ?? ???. ?????? ??????????. ?????? ?? ???? ???????? ? ????????????. ??????? ?????????? ????????? ??????????????? ??????????. * * *???????: (O95) 106-?865, 1??-9709 -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.h.jaffe at gmail.com Fri Jan 27 00:08:04 2006 From: a.h.jaffe at gmail.com (Andrew Jaffe) Date: Fri Jan 27 00:08:04 2006 Subject: [Numpy-discussion] Re: Weird (wrong?) real fft 2d behavior In-Reply-To: References: <43D74B35.9060204@gmail.com> <43D74D17.7010103@gmail.com> Message-ID: <43D9D342.3080204@gmail.com> Dear Warren, > > On Wed, 25 Jan 2006, Andrew Jaffe wrote: > >> Andrew Jaffe wrote: >>> If I start with what I thought was an appropriate (n, n/2+1) complex >>> matrix which should have a real inverse fft, and then take its real fft, >>> I don't get back the original matrix. >>> >>> Note the presence of very small but nonzero reals in the final matrix, > > That's just roundoff. > >>> and the fact that the 2d and 4th rows are duplicates. This seems to be a >>> mistake somewhere. >>> >>> Or am I just misunderstanding/misremembering something about 2d real ffts? > > It looks wrong to me, and I think I wrote those functions. I get the same > results in Numeric. I'll try to look into the problem. > >> and I should point out that >> delta_rp = N.dft.real_fft2d(delta_kp) >> is 'allclose' to the original delta_r (which leads me to believe that I >> may indeed be misunderstanding something). > > "Stable" does not neccessarily imply "correct". Indeed! And more to the point, it's actually the case that "delta_kp" doesn't actually have the requisite 16 (non-small) real degrees of freedom -- so it can't really be right. Andrew From a.h.jaffe at gmail.com Fri Jan 27 00:11:02 2006 From: a.h.jaffe at gmail.com (Andrew Jaffe) Date: Fri Jan 27 00:11:02 2006 Subject: [Numpy-discussion] Re: Weird (wrong?) real fft 2d behavior In-Reply-To: References: <43D74B35.9060204@gmail.com> <43D74D17.7010103@gmail.com> Message-ID: <43D9D39F.30307@gmail.com> > On Wed, 25 Jan 2006, Andrew Jaffe wrote: > >> Andrew Jaffe wrote: >>> If I start with what I thought was an appropriate (n, n/2+1) complex >>> matrix which should have a real inverse fft, and then take its real fft, >>> I don't get back the original matrix. >>> >>> Note the presence of very small but nonzero reals in the final matrix, > > That's just roundoff. > >>> and the fact that the 2d and 4th rows are duplicates. This seems to be a >>> mistake somewhere. >>> >>> Or am I just misunderstanding/misremembering something about 2d real ffts? > > It looks wrong to me, and I think I wrote those functions. I get the same > results in Numeric. I'll try to look into the problem. > >> and I should point out that >> delta_rp = N.dft.real_fft2d(delta_kp) >> is 'allclose' to the original delta_r (which leads me to believe that I >> may indeed be misunderstanding something). > > "Stable" does not neccessarily imply "correct". Indeed! And more to the point, it's actually the case that "delta_kp" doesn't actually have the requisite 16 (non-small) real degrees of freedom -- so it can't really be right. Andrew From agn at noc.soton.ac.uk Fri Jan 27 01:12:07 2006 From: agn at noc.soton.ac.uk (George Nurser) Date: Fri Jan 27 01:12:07 2006 Subject: [Numpy-discussion] installation under other than .../site-packages Message-ID: <2BA043A4-A260-4EC0-AFCF-A22CEAE74DC2@noc.soton.ac.uk> On 25 Jan 2006, at 20:09, Humufr wrote: > This solution is the one I'm using. Unfortunatly it's not working > good with scipy. I tried to install scipy in my own directory and > everythong seems to work like a charm but the lapack link is bad. I > tried everything write in the wiki to install your own BLAS/LAPACK > library but scipy want tu use the one of the system who have a > problem. I'm trying to find a solution but until I found it numpy/ > scipy is not an option. > This wasn't obvious to me either. To use your own blas/lapck in scipy, you need to put a site.cfg file in the distutils directory of your numpy installation. So if you installed numpy with --home=$HOME it should produce a distutils directory at $HOME/lib/python/numpy/ distutils or $HOME/lib64/python/numpy/distutils If you want to build scipy with the same site.cfg as you used to build numpy, then, assuming the root source numpy directory is $PYSRC/ numpy cp $PYSRC/numpy/numpy/distutils/site.cfg $HOME/lib[64]/python/numpy/ distutils Hope this helps. George. From alteration at onet.pl Fri Jan 27 03:31:00 2006 From: alteration at onet.pl (=?windows-1251?B?MDIg9OXi8ODr/w==?=) Date: Fri Jan 27 03:31:00 2006 Subject: [Numpy-discussion] =?windows-1251?B?z/Do4uvl9+Xt6OUg6CDo8e/u6/zn7uLg7ejlIOIg0NQg6O3u8fLw4O3t7ukg8ODh?= =?windows-1251?B?7vfl6SDx6Ov7Lg==?= Message-ID: <84b101c62333$56628296$0572b9c0@onet.pl> ?????????? ??? ? ????? ?????? ??????? ??????? ? ???????????? ?????????: 02 ??????? - ??????????? ? ????????????? ? ?? ??????????? ??????? ????. 06 ??????? - ??????????????? ????????? ?????: ??????????, ??????? ?????????? ????????, ???????? ?????? ?????????????. ???????? ?????. ?????????? ? ?????????????? ?? ??????????? ??????? ????02 ??????? 2006 ?. 1. ????? ???????????????? ? ????? ??????????? ??????????? ??????? ????. 2. ??????????????? ???????????? ?? ????????? ???????????? ????????????????. ???????????????? ??????????????? ??????????? ???????????????? ??????????????? ??????????? ??? ???????????????? ??????????????? ?????????? ??? ?????? ??????????? ????????? 3. ???????? ????????? ??????????? ?????????? ? ?? ??????? ??????-?????? ??????????? ???? ? ?????????? ??????????????? ??????????? ????????? ???????????????? ?? ??????????? (????? ???????? ???????, ?????? ?????, ????????????, ?????? ?????, ?????????? ???????????, ???????????????) ??????????? ??????????? ?? ?????? ??????? ??? ????? ?? ?????? ??????? ?????? - ?? ????????? ????????????? ??????????-???????? ????????? ? ??????????? ?????? ? ?????. 4. ??????????? ?????????????, ???????????? ???? ??????????? ????????? ?????????? ?? ??????????? ??????????? ??????? ???? ?????????? ????????????? ?? ????? ?????? ?????????, ??????? ?? ????????? ?????????? ?? ??????????, ?? ????????????? ???????? ???????????? ????????????? ???????????????? ?.?????? - ???????? ????????? 5. ?????????????? ?????? ? ?? ??????????? ??????????????? ????? ??????????? ???????????? ?????? ??? ?.?????? ???? ?.?????? ?????????? ?? ????? ???????? ??????????????? ????????? ????? ? ?.?????? 6. ?????????????? ???????????? ????????? ???????: 5800 ???. + ??? ?????????????????? ? ???????? ?????????????? ?????????? ?? ??????:?? ???. (495) 2?2-2l?3 ??????????????? ????????? ?????: ??????????, ??????? ?????????? ????????, ???????? ?????? ?????????????. ???????? ?????.06 ??????? 2006 ???? ????????? ??????? ???????? ????????? ?? ??????? ??????????? ???????? ????????????? ????????, ?? ???????? ? ????? ??????? ?????????? ?????????????. ?????? ???????? ????? ??????? ?? ?????? ???????? ???????, ??????? ????????? ????????????, ?? ? ??? ????????, ??????? ?????????? ?? ?????? ?????????. ????? ????? ????? ???? ?????????? ???????????? ?? ?????? ???? ???????????? ??? ??????-??? ??????????????? ?????????? ????? ? ??????????? ?????????? ??????. 1. ??????????????? ????????????? ? ??????????? ???, ?? ????????? ????????? ????????????????: ????????????????, ?????????. ??? ????? ???????? ?? ?????????? ????????? 2. ??????????? ??????: ??????????????? ????????? ?????: ???????? ??????????? ? ??????????? ?? ?????? ?????, ???????????, ???????? ? ??. 3. ?????????? ? ??????? ??????????????? ????????? ????? (???). 4. ?????????? ? ???????? ??????????, ??????? ????? ????????? ???. 5. ????? ? ??????? ?????????? ????????, ??????????? ???????? ? ????????? ???????. 6. ???????????? ?????? ??? ???????????? ?????????? ?? ??????????? ????????????? ????????. ??? ???????? ????? ???????????? 7. ????? ???????? ? ????? ??????? ????????? ???????????????? ? ?? ???????????: 7.1. ???????? ?????? ??? ?????? ?? ??????. ?????????? ?????????? ??????? ???????? ?????????. ?????????? ?????????? ????????? ???????. 7.2. ????????? ???????????????? ?? ?????? ?????. ?????????? ?????????? ??????? ????. ?????????? ??????????? ??????????? ? ?.?. 7.3. ????????? ?? ?????? ?????. ????????? ? ????????? ????? ?????????. ????????? ?????????? ????? ??? ???? ? ?????. ????????? ??? ????????????. 7.4. ?????????, ????????? ? ??????????????? ?????????? ???????. ??????? ????????????? ?? ????????. 7.5. ????????? ? ??????? ?????? ? ?????????? ???????? ??????. ????????? ? ??????? ???????????? ? ??????????? ?????? ? ????????? ????????. 8. ????????? ??????????? ?????????? ??????. ????????? 5500 ???. + ???. ? ????????? ???????? ????????: ??????????? ?????????, ????, ????-?????. ?????????????????? ? ???????? ?????????????? ?????????? ?? ??????:?? ???. (495) 2?2-2l?3 -------------- next part -------------- An HTML attachment was scrubbed... URL: From faltet at carabos.com Fri Jan 27 03:31:05 2006 From: faltet at carabos.com (Francesc Altet) Date: Fri Jan 27 03:31:05 2006 Subject: [Numpy-discussion] Upcasting rules in numpy Message-ID: <200601271230.23005.faltet@carabos.com> Hi, I don't know if the next behaviour in NumPy is intended or not: In [22]: (numpy.zeros((2,1), dtype="int8")*3.).dtype.type Out[22]: while numarray and Numeric gives: In [36]: (numarray.zeros((2,1), type="Int8")*3.).type() Out[36]: Float64 In [37]: (Numeric.zeros((2,1), typecode="b")*3.).typecode() Out[37]: 'd' Would not be float64 a more sensible default for numpy as well? Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From ndarray at mac.com Fri Jan 27 08:50:12 2006 From: ndarray at mac.com (Sasha) Date: Fri Jan 27 08:50:12 2006 Subject: [Numpy-discussion] Numpy does not compile with PY_ARRAY_TYPES_PREFIX Message-ID: I have attempted to compile RPy (see http://rpy.sourceforge.net/) with numpy and found that both R and numpy headers define Int32: .../numpy/core/include/numpy/arrayobject.h:304: conflicting types for `Int32' .../lib/R/include/R_ext/Random.h:57: previous declaration of `Int32' I have tried to fix the problem by setting PY_ARRAY_TYPES_PREFIX as follows: $ CFLAGS=-DPY_ARRAY_TYPES_PREFIX=PyArray_NS_ python setup.py build This resulted in compile error in almost every Numpy source file. Am I using PY_ARRAY_TYPES_PREFIX correctly? If not, what is the recommended way to solve name conflicts involving Numpy typoes? Thanks. From strawman at astraw.com Fri Jan 27 09:14:07 2006 From: strawman at astraw.com (Andrew Straw) Date: Fri Jan 27 09:14:07 2006 Subject: [Numpy-discussion] Numpy does not compile with PY_ARRAY_TYPES_PREFIX In-Reply-To: References: Message-ID: <43DA54AE.4090704@astraw.com> Sasha wrote: >I have attempted to compile RPy (see http://rpy.sourceforge.net/) with >numpy and found that both R and numpy headers define Int32: > >.../numpy/core/include/numpy/arrayobject.h:304: conflicting types for `Int32' >.../lib/R/include/R_ext/Random.h:57: previous declaration of `Int32' > >I have tried to fix the problem by setting PY_ARRAY_TYPES_PREFIX as follows: > >$ CFLAGS=-DPY_ARRAY_TYPES_PREFIX=PyArray_NS_ python setup.py build > >This resulted in compile error in almost every Numpy source file. > >Am I using PY_ARRAY_TYPES_PREFIX correctly? If not, what is the >recommended way to solve name conflicts involving Numpy typoes? > > David Cooke informed me some time ago that the #include "numpy/arrayobject.h" line should go _after_ the other #includes. It's worked for me well enough ever since, although it seems rather fragile. Otherwise I'm not having the same issue with current svn. There was an issue with release 0.9.4 for C++ code, anyway. Cheers! Andrew From alexander.belopolsky at gmail.com Fri Jan 27 10:14:01 2006 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Fri Jan 27 10:14:01 2006 Subject: Fwd: [Numpy-discussion] Numpy does not compile with PY_ARRAY_TYPES_PREFIX In-Reply-To: References: <43DA54AE.4090704@astraw.com> Message-ID: Numpy-discussion list rejected an attachment. Here are a few relevant lines from the log: In file included from numpy/core/src/multiarraymodule.c:65: numpy/core/src/arrayobject.c: In function `PyArray_FromDimsAndDataAndDescr': numpy/core/src/arrayobject.c:766: `intp' undeclared (first use in this function) numpy/core/src/arrayobject.c:766: parse error before ')' token In file included from numpy/core/src/multiarraymodule.c:65: numpy/core/src/arrayobject.c: At top level: numpy/core/src/arrayobject.c:822: parse error before "intp" numpy/core/src/arrayobject.c:822: warning: function declaration isn't a prototype numpy/core/src/arrayobject.c: In function `PyArray_Scalar': numpy/core/src/arrayobject.c:841: `Bool' undeclared (first use in this function) numpy/core/src/arrayobject.c:841: parse error before ')' token numpy/core/src/arrayobject.c:841: parse error before ')' token numpy/core/src/arrayobject.c:841: warning: left-hand operand of comma expression has no effect ---------- Forwarded message ---------- From: Sasha Date: Jan 27, 2006 12:33 PM Subject: Re: [Numpy-discussion] Numpy does not compile with PY_ARRAY_TYPES_PREFIX To: Andrew Straw , numpy-discussion I've just updated to svn r2006. Regular build works fine, but $ CFLAGS=-DPY_ARRAY_TYPES_PREFIX=PyArray_NS_ python setup.py build produced the attached log. Note that I am not compiling any extention, just regular numpy. On 1/27/06, Andrew Straw wrote: > Sasha wrote: > > >I have attempted to compile RPy (see http://rpy.sourceforge.net/) with > >numpy and found that both R and numpy headers define Int32: > > > >.../numpy/core/include/numpy/arrayobject.h:304: conflicting types for `Int32' > >.../lib/R/include/R_ext/Random.h:57: previous declaration of `Int32' > > > >I have tried to fix the problem by setting PY_ARRAY_TYPES_PREFIX as follows: > > > >$ CFLAGS=-DPY_ARRAY_TYPES_PREFIX=PyArray_NS_ python setup.py build > > > >This resulted in compile error in almost every Numpy source file. > > > >Am I using PY_ARRAY_TYPES_PREFIX correctly? If not, what is the > >recommended way to solve name conflicts involving Numpy typoes? > > > > > > David Cooke informed me some time ago that the #include > "numpy/arrayobject.h" line should go _after_ the other #includes. It's > worked for me well enough ever since, although it seems rather fragile. > Otherwise I'm not having the same issue with current svn. There was an > issue with release 0.9.4 for C++ code, anyway. > > Cheers! > Andrew > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From ndarray at mac.com Fri Jan 27 10:15:03 2006 From: ndarray at mac.com (Sasha) Date: Fri Jan 27 10:15:03 2006 Subject: Fwd: [Numpy-discussion] Numpy does not compile with PY_ARRAY_TYPES_PREFIX In-Reply-To: References: <43DA54AE.4090704@astraw.com> Message-ID: Numpy-discussion list rejected an attachment. Here are a few relevant lines from the log: In file included from numpy/core/src/multiarraymodule.c:65: numpy/core/src/arrayobject.c: In function `PyArray_FromDimsAndDataAndDescr': numpy/core/src/arrayobject.c:766: `intp' undeclared (first use in this function) numpy/core/src/arrayobject.c:766: parse error before ')' token In file included from numpy/core/src/multiarraymodule.c:65: numpy/core/src/arrayobject.c: At top level: numpy/core/src/arrayobject.c:822: parse error before "intp" numpy/core/src/arrayobject.c:822: warning: function declaration isn't a prototype numpy/core/src/arrayobject.c: In function `PyArray_Scalar': numpy/core/src/arrayobject.c:841: `Bool' undeclared (first use in this function) numpy/core/src/arrayobject.c:841: parse error before ')' token numpy/core/src/arrayobject.c:841: parse error before ')' token numpy/core/src/arrayobject.c:841: warning: left-hand operand of comma expression has no effect ---------- Forwarded message ---------- From: Sasha Date: Jan 27, 2006 12:33 PM Subject: Re: [Numpy-discussion] Numpy does not compile with PY_ARRAY_TYPES_PREFIX To: Andrew Straw , numpy-discussion I've just updated to svn r2006. Regular build works fine, but $ CFLAGS=-DPY_ARRAY_TYPES_PREFIX=PyArray_NS_ python setup.py build produced the attached log. Note that I am not compiling any extention, just regular numpy. On 1/27/06, Andrew Straw wrote: > Sasha wrote: > > >I have attempted to compile RPy (see http://rpy.sourceforge.net/) with > >numpy and found that both R and numpy headers define Int32: > > > >.../numpy/core/include/numpy/arrayobject.h:304: conflicting types for `Int32' > >.../lib/R/include/R_ext/Random.h:57: previous declaration of `Int32' > > > >I have tried to fix the problem by setting PY_ARRAY_TYPES_PREFIX as follows: > > > >$ CFLAGS=-DPY_ARRAY_TYPES_PREFIX=PyArray_NS_ python setup.py build > > > >This resulted in compile error in almost every Numpy source file. > > > >Am I using PY_ARRAY_TYPES_PREFIX correctly? If not, what is the > >recommended way to solve name conflicts involving Numpy typoes? > > > > > > David Cooke informed me some time ago that the #include > "numpy/arrayobject.h" line should go _after_ the other #includes. It's > worked for me well enough ever since, although it seems rather fragile. > Otherwise I'm not having the same issue with current svn. There was an > issue with release 0.9.4 for C++ code, anyway. > > Cheers! > Andrew > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From ndarray at mac.com Fri Jan 27 11:23:04 2006 From: ndarray at mac.com (Sasha) Date: Fri Jan 27 11:23:04 2006 Subject: [Numpy-discussion] UnsignedInt32 Message-ID: Numeric has both UInt32 and UnsignedInt32 defined, but Numpy only has UInt32. Is it intentional or an oversight? Thanks. From oliphant at ee.byu.edu Fri Jan 27 11:28:01 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri Jan 27 11:28:01 2006 Subject: [Numpy-discussion] UnsignedInt32 In-Reply-To: References: Message-ID: <43DA740F.4090707@ee.byu.edu> Sasha wrote: > Numeric has both UInt32 and UnsignedInt32 defined, but Numpy only has >UInt32. Is it intentional or an oversight? > > > an oversight. -Travis From oliphant at ee.byu.edu Fri Jan 27 11:32:09 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri Jan 27 11:32:09 2006 Subject: [Numpy-discussion] UnsignedInt32 In-Reply-To: References: Message-ID: <43DA7453.5030906@ee.byu.edu> Sasha wrote: > Numeric has both UInt32 and UnsignedInt32 defined, but Numpy only has >UInt32. Is it intentional or an oversight? > > > It's actually there, it's just not in the __all__ list of oldnumeric Try: from numpy.core.oldnumeric import UnsignedInt32 -Travis From ndarray at mac.com Fri Jan 27 13:32:32 2006 From: ndarray at mac.com (Sasha) Date: Fri Jan 27 13:32:32 2006 Subject: Fwd: [Numpy-discussion] Numpy does not compile with PY_ARRAY_TYPES_PREFIX In-Reply-To: <43DA7625.6010804@ee.byu.edu> References: <43DA54AE.4090704@astraw.com> <43DA7220.8070605@ee.byu.edu> <43DA7625.6010804@ee.byu.edu> Message-ID: On 1/27/06, Travis Oliphant wrote: > Show an example of using it in a file before you include the numpy > header. Because that *should* work. I know, I just tested it a few > days ago... I've tried to put just two lines in "test.c": #define PY_ARRAY_TYPES_PREFIX XYZ_ #include and compile it with $ gcc -I$NUMPYINCLUDE -c test.c Where NUMPYINCLUDE points to the location of numpy/arrayobject.h I get the following errors: In file included from test.c:2: numpy-0.9.4/lib/python2.3/site-packages/numpy/core/include/numpy/arrayobject.h:699: error: syntax error before " XYZ_intp" numpy-0.9.4/lib/python2.3/site-packages/numpy/core/include/numpy/arrayobject.h:699: warning: data definition has no type or storage class numpy-0.9.4/lib/python2.3/site-packages/numpy/core/include/numpy/arrayobject.h:700: error: syntax error before " XYZ_uintp" ... ... From cookedm at physics.mcmaster.ca Fri Jan 27 13:38:06 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Fri Jan 27 13:38:06 2006 Subject: Fwd: [Numpy-discussion] Numpy does not compile with PY_ARRAY_TYPES_PREFIX In-Reply-To: (ndarray@mac.com's message of "Fri, 27 Jan 2006 16:31:42 -0500") References: <43DA54AE.4090704@astraw.com> <43DA7220.8070605@ee.byu.edu> <43DA7625.6010804@ee.byu.edu> Message-ID: Sasha writes: > On 1/27/06, Travis Oliphant wrote: > >> Show an example of using it in a file before you include the numpy >> header. Because that *should* work. I know, I just tested it a few >> days ago... > > I've tried to put just two lines in "test.c": > > #define PY_ARRAY_TYPES_PREFIX XYZ_ > #include > > and compile it with > > $ gcc -I$NUMPYINCLUDE -c test.c > > Where NUMPYINCLUDE points to the location of numpy/arrayobject.h You need to add #include "Python.h" as the first include (and add a -I to your gcc line, of course). -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From ndarray at mac.com Fri Jan 27 14:51:03 2006 From: ndarray at mac.com (Sasha) Date: Fri Jan 27 14:51:03 2006 Subject: Fwd: [Numpy-discussion] Numpy does not compile with PY_ARRAY_TYPES_PREFIX In-Reply-To: References: <43DA54AE.4090704@astraw.com> <43DA7220.8070605@ee.byu.edu> <43DA7625.6010804@ee.byu.edu> Message-ID: On 1/27/06, David M. Cooke wrote: > You need to add #include "Python.h" as the first include (and add a > -I to your gcc line, of course). Still does not work > cat test.c #define PY_ARRAY_TYPES_PREFIX XYZ_ #include #include > gcc -I Python-2.4.1/include/python2.4 -I lib/python2.4/site-packages/numpy/core/include -c test.c In file included from lib/python2.4/site-packages/numpy/core/include/numpy/arrayobject.h:1346, from test.c:3: lib/python2.4/site-packages/numpy/core/include/numpy/__multiarray_api.h: In function `import_array': lib/python2.4/site-packages/numpy/core/include/numpy/__multiarray_api.h:765: parse error before ')' token lib/python2.4/site-packages/numpy/core/include/numpy/__multiarray_api.h:765: parse error before ')' token lib/python2.4/site-packages/numpy/core/include/numpy/__multiarray_api.h: At top level: lib/python2.4/site-packages/numpy/core/include/numpy/__multiarray_api.h:771: parse error before "return" If I remove #define PY_ARRAY_TYPES_PREFIX or define it as nothing, compilation works. The RPy.h header I was trying to compile in the first plase includes numpy/arrayobject.h after Python.h . From oliphant at ee.byu.edu Fri Jan 27 16:27:04 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri Jan 27 16:27:04 2006 Subject: Fwd: [Numpy-discussion] Numpy does not compile with PY_ARRAY_TYPES_PREFIX In-Reply-To: References: <43DA54AE.4090704@astraw.com> <43DA7220.8070605@ee.byu.edu> <43DA7625.6010804@ee.byu.edu> Message-ID: <43DABA30.6020002@ee.byu.edu> Sasha wrote: >On 1/27/06, David M. Cooke wrote: > > >>You need to add #include "Python.h" as the first include (and add a >>-I to your gcc line, of course). >> >> > >Still does not work > > It works for me. Try the attached file. gcc -I /usr/include/python2.4 -I /usr/lib/python2.4/site-packages/numpy/core/include -c test.c Worked fine. Maybe Python.h has to come *before* the special #define. -Travis -------------- next part -------------- A non-text attachment was scrubbed... Name: test.c Type: text/x-csrc Size: 85 bytes Desc: not available URL: From oliphant at ee.byu.edu Fri Jan 27 16:34:04 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri Jan 27 16:34:04 2006 Subject: Fwd: [Numpy-discussion] Numpy does not compile with PY_ARRAY_TYPES_PREFIX In-Reply-To: <43DABA30.6020002@ee.byu.edu> References: <43DA54AE.4090704@astraw.com> <43DA7220.8070605@ee.byu.edu> <43DA7625.6010804@ee.byu.edu> <43DABA30.6020002@ee.byu.edu> Message-ID: <43DABBBC.3040505@ee.byu.edu> Travis Oliphant wrote: > Sasha wrote: > >> On 1/27/06, David M. Cooke wrote: >> >> >>> You need to add #include "Python.h" as the first include (and add a >>> -I to your gcc line, of course). >>> >> >> >> Still does not work >> >> > It works for me. Try the attached file. > > gcc -I /usr/include/python2.4 -I > /usr/lib/python2.4/site-packages/numpy/core/include -c test.c > > Worked fine. > > Maybe Python.h has to come *before* the special #define. On my system, it didn't seem to matter. All incarnations worked. Perhaps there is something wrong with your installation. Could you show the very first errors you are getting. Is the wrong header being picked up from somewhere. Also perhaps you could look at the output of the compilation using the -E (pre-processor only) flag. This might help nail down the problem. -Travis From cookedm at physics.mcmaster.ca Fri Jan 27 16:43:03 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Fri Jan 27 16:43:03 2006 Subject: Fwd: [Numpy-discussion] Numpy does not compile with PY_ARRAY_TYPES_PREFIX In-Reply-To: (ndarray@mac.com's message of "Fri, 27 Jan 2006 17:50:27 -0500") References: <43DA54AE.4090704@astraw.com> <43DA7220.8070605@ee.byu.edu> <43DA7625.6010804@ee.byu.edu> Message-ID: Sasha writes: > On 1/27/06, David M. Cooke wrote: >> You need to add #include "Python.h" as the first include (and add a >> -I to your gcc line, of course). > > Still does not work > >> cat test.c > #define PY_ARRAY_TYPES_PREFIX XYZ_ > #include > #include > >> gcc -I Python-2.4.1/include/python2.4 -I lib/python2.4/site-packages/numpy/core/include -c test.c > In file included from > lib/python2.4/site-packages/numpy/core/include/numpy/arrayobject.h:1346, > from test.c:3: > lib/python2.4/site-packages/numpy/core/include/numpy/__multiarray_api.h: > In function `import_array': > lib/python2.4/site-packages/numpy/core/include/numpy/__multiarray_api.h:765: > parse error before ')' token > lib/python2.4/site-packages/numpy/core/include/numpy/__multiarray_api.h:765: > parse error before ')' token > lib/python2.4/site-packages/numpy/core/include/numpy/__multiarray_api.h: > At top level: > lib/python2.4/site-packages/numpy/core/include/numpy/__multiarray_api.h:771: > parse error before "return" Hmm, the location of that error would seem to imply it's something to do with NDARRAY_VERSION not being defined. Maybe for some reason you've got an old arrayobject.h, but a newer __multiarray_api.h? I'd try blowing away your lib/python2.4/site-packages/numpy directory, and installing again. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From ndarray at mac.com Fri Jan 27 18:13:05 2006 From: ndarray at mac.com (Sasha) Date: Fri Jan 27 18:13:05 2006 Subject: Fwd: [Numpy-discussion] Numpy does not compile with PY_ARRAY_TYPES_PREFIX In-Reply-To: <43DABBBC.3040505@ee.byu.edu> References: <43DA7220.8070605@ee.byu.edu> <43DA7625.6010804@ee.byu.edu> <43DABA30.6020002@ee.byu.edu> <43DABBBC.3040505@ee.byu.edu> Message-ID: Thanks for your advise. With the help of -E option, I've found the problem. I was using the version of the header that used uint in the C-API. the problem was fixed in svn r1983: http://projects.scipy.org/scipy/numpy/changeset/1983 . -- sasha On 1/27/06, Travis Oliphant wrote: > Travis Oliphant wrote: > > > Sasha wrote: > > > >> On 1/27/06, David M. Cooke wrote: > >> > >> > >>> You need to add #include "Python.h" as the first include (and add a > >>> -I to your gcc line, of course). > >>> > >> > >> > >> Still does not work > >> > >> > > It works for me. Try the attached file. > > > > gcc -I /usr/include/python2.4 -I > > /usr/lib/python2.4/site-packages/numpy/core/include -c test.c > > > > Worked fine. > > > > Maybe Python.h has to come *before* the special #define. > > On my system, it didn't seem to matter. All incarnations worked. > Perhaps there is something wrong with your installation. Could you > show the very first errors you are getting. Is the wrong header being > picked up from somewhere. > > Also perhaps you could look at the output of the compilation using the > -E (pre-processor only) flag. This might help nail down the problem. > > -Travis > > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From strawman at astraw.com Fri Jan 27 20:28:13 2006 From: strawman at astraw.com (Andrew Straw) Date: Fri Jan 27 20:28:13 2006 Subject: Fwd: [Numpy-discussion] Numpy does not compile with PY_ARRAY_TYPES_PREFIX In-Reply-To: References: <43DA7220.8070605@ee.byu.edu> <43DA7625.6010804@ee.byu.edu> <43DABA30.6020002@ee.byu.edu> <43DABBBC.3040505@ee.byu.edu> Message-ID: <43DAF2BA.5090307@astraw.com> Sasha wrote: >Thanks for your advise. With the help of -E option, I've found the >problem. I was using the version of the header that used uint in the >C-API. the problem was fixed in svn r1983: >http://projects.scipy.org/scipy/numpy/changeset/1983 . > >-- sasha > > That's strange. In your 2nd email, you said you were using revision 2006, which made me think this couldn't be the issue... From ndarray at mac.com Fri Jan 27 20:41:13 2006 From: ndarray at mac.com (Sasha) Date: Fri Jan 27 20:41:13 2006 Subject: Fwd: [Numpy-discussion] Numpy does not compile with PY_ARRAY_TYPES_PREFIX In-Reply-To: <43DAF2BA.5090307@astraw.com> References: <43DA7625.6010804@ee.byu.edu> <43DABA30.6020002@ee.byu.edu> <43DABBBC.3040505@ee.byu.edu> <43DAF2BA.5090307@astraw.com> Message-ID: Sorry, I did not clean up an old build properly. It's hard to keep up with numpy's pace these days! -- sasha On 1/27/06, Andrew Straw wrote: > That's strange. In your 2nd email, you said you were using revision > 2006, which made me think this couldn't be the issue... From ndarray at mac.com Fri Jan 27 22:42:05 2006 From: ndarray at mac.com (Sasha) Date: Fri Jan 27 22:42:05 2006 Subject: [Numpy-discussion] RPy with NumPy Message-ID: I have succesfully (according to testall.py :-) converted RPy to use numpy instead of Numeric. Attached patch was tested on Linux only. I did not attempt to create a package that would support both numpy and Numeric, but it is easy to do. Please let me know if you are interested. For numpy-discussion readers: "RPy is a very simple, yet robust, Python interface to the R Programming Language." For rpy-list readers: "Development for Numeric has ceased, and users should transisition to NumPy as quickly as possible." For readers of both lists: Sorry for cross-posting. -- sasha -------------- next part -------------- diff -Naur rpy-0.4.6/rpy.py rpy-0.4.6-numpy/rpy.py --- rpy-0.4.6/rpy.py 2005-07-27 13:56:54.000000000 -0400 +++ rpy-0.4.6-numpy/rpy.py 2006-01-27 12:06:13.628796000 -0500 @@ -30,7 +30,7 @@ # installation time and RPy should have been compiled properly. So, # don't complain. try: - from Numeric import * + from numpy import * HAS_NUMERIC = 1 except ImportError: HAS_NUMERIC = 0 diff -Naur rpy-0.4.6/setup.py rpy-0.4.6-numpy/setup.py --- rpy-0.4.6/setup.py 2005-07-28 00:52:29.000000000 -0400 +++ rpy-0.4.6-numpy/setup.py 2006-01-28 01:18:46.148833000 -0500 @@ -34,7 +34,6 @@ from distutils.errors import * import rpy_tools - try: if sys.platform=="win32": RHOMES = os.environ['RHOMES'].split(';') @@ -72,7 +71,7 @@ if sys.platform=="win32": DEFINE.append(('R_HOME', '\\"%s\\"' % RHOME.replace("\\","/") )) else: - DEFINE.append(('R_HOME', '"%s"' % RHOME)) + DEFINE.append(('R_HOME', r'\"%s\"' % RHOME)) r_libs = [ os.path.join(RHOME, 'bin'), @@ -132,8 +131,10 @@ # check whether Numeric is present try: - import Numeric + import numpy DEFINE.append(('WITH_NUMERIC', None)) + DEFINE.append(('PY_ARRAY_TYPES_PREFIX', 'PyArray_')) + include_dirs.append(numpy.get_numpy_include()) except ImportError: UNDEF.append('WITH_NUMERIC') @@ -148,12 +149,11 @@ # use R version specific shared library name shlib_name = "_rpy%s" % RVER if sys.platform=="win32": - DEFINE.append( ('RPY_SHNAME', '\\"_rpy%s\\"' % RVER ) ) - else: DEFINE.append( ('RPY_SHNAME', '"_rpy%s"' % RVER ) ) + else: + DEFINE.append( ('RPY_SHNAME', r'\"_rpy%s\"' % RVER ) ) DEFINE.append( ('INIT_RPY', 'init_rpy%s' % RVER ) ) - modules.append( Extension( shlib_name, source_files, diff -Naur rpy-0.4.6/src/RPy.h rpy-0.4.6-numpy/src/RPy.h --- rpy-0.4.6/src/RPy.h 2005-07-26 23:05:29.000000000 -0400 +++ rpy-0.4.6-numpy/src/RPy.h 2006-01-26 17:18:37.750157000 -0500 @@ -68,12 +68,13 @@ #include +#ifdef WITH_NUMERIC +#include "numpy/arrayobject.h" +#endif + #include "robjobject.h" #include "setenv.h" -#ifdef WITH_NUMERIC -#include "Numeric/arrayobject.h" -#endif #define MAXIDSIZE 256 diff -Naur rpy-0.4.6/src/rpymodule.c rpy-0.4.6-numpy/src/rpymodule.c --- rpy-0.4.6/src/rpymodule.c 2005-07-28 10:23:33.000000000 -0400 +++ rpy-0.4.6-numpy/src/rpymodule.c 2006-01-28 01:19:29.281899000 -0500 @@ -1517,7 +1517,7 @@ if(use_numeric) { import_array(); - multiarray = PyImport_ImportModule("multiarray"); + multiarray = PyImport_ImportModule("numpy"); if (multiarray) { dict = PyModule_GetDict(multiarray); if (dict) diff -Naur rpy-0.4.6/tests/test_array.py rpy-0.4.6-numpy/tests/test_array.py --- rpy-0.4.6/tests/test_array.py 2005-07-18 14:54:54.000000000 -0400 +++ rpy-0.4.6-numpy/tests/test_array.py 2006-01-28 00:58:41.100628000 -0500 @@ -28,7 +28,7 @@ r.array.autoconvert(BASIC_CONVERSION) def testConversionToPy(self): - self.failUnless(self.py == self.ra.as_py(), + self.failUnless((self.py == self.ra.as_py()).all(), 'wrong conversion to Python') def testConversionToR(self): diff -Naur rpy-0.4.6/tests/test_numeric.py rpy-0.4.6-numpy/tests/test_numeric.py --- rpy-0.4.6/tests/test_numeric.py 2005-07-26 23:08:49.000000000 -0400 +++ rpy-0.4.6-numpy/tests/test_numeric.py 2006-01-27 12:07:38.309828000 -0500 @@ -2,7 +2,7 @@ import unittest try: - from Numeric import * + from numpy import * except ImportError: print 'Numeric not available. Skipping.\n' import sys diff -Naur rpy-0.4.6/tests/test_topy.py rpy-0.4.6-numpy/tests/test_topy.py --- rpy-0.4.6/tests/test_topy.py 2005-07-19 16:20:08.000000000 -0400 +++ rpy-0.4.6-numpy/tests/test_topy.py 2006-01-28 01:05:57.007083000 -0500 @@ -27,7 +27,7 @@ assert( r(-2147483648) == r.NA ) def testNAreal(self): - assert(r.NAN == r("as.numeric(NA)") ) + assert(repr(r.NAN) == repr(r("as.numeric(NA)")) ) def testNAstring(self): assert(r("as.character(NA)") == 'NA') @@ -48,14 +48,14 @@ def testNAList(self): xi = [1,2,r.NA,r.NAN,4] assert(r.as_character(xi) == ['1', '2', 'NA', 'NA', '4']) - assert(r.as_numeric(xi) == [1.0, 2.0, r.NAN, r.NAN, 4.0]) + assert(repr(r.as_numeric(xi)) == repr([1.0, 2.0, r.NAN, r.NAN, 4.0])) assert(r.as_integer(xi) == [1, 2, r.NA, r.NA, 4]) assert(r.as_factor(xi) == ['1', '2', 'NA', 'NA', '4']) assert(r.is_na(xi) == [False, False, True, True, False] ) xd = [1.01,2.02,r.NA,r.NAN,4.04] assert(r.as_character(xd) == ['1.01', '2.02', 'NA', 'NA', '4.04']) - assert(r.as_numeric(xd) == [1.01, 2.02, r.NAN, r.NAN, 4.04]) + assert(repr(r.as_numeric(xd)) == repr([1.01, 2.02, r.NAN, r.NAN, 4.04])) assert(r.as_integer(xd) == [1, 2, r.NA, r.NA, 4]) assert(r.as_factor(xd) == ['1.01', '2.02', 'NA', 'NA', '4.04']) assert(r.is_na(xd) == [False, False, True, True, False] ) From ndarray at mac.com Fri Jan 27 22:45:04 2006 From: ndarray at mac.com (Sasha) Date: Fri Jan 27 22:45:04 2006 Subject: [Numpy-discussion] [Rpy] RPy with NumPy Message-ID: I have succesfully (according to testall.py :-) converted RPy to use numpy instead of Numeric. Attached patch was tested on Linux only. I did not attempt to create a package that would support both numpy and Numeric, but it is easy to do. Please let me know if you are interested. For numpy-discussion readers: "RPy is a very simple, yet robust, Python interface to the R Programming Language." For rpy-list readers: "Development for Numeric has ceased, and users should transisition to NumPy as quickly as possible." For readers of both lists: Sorry for cross-posting. -- sasha -------------- next part -------------- diff -Naur rpy-0.4.6/rpy.py rpy-0.4.6-numpy/rpy.py --- rpy-0.4.6/rpy.py 2005-07-27 13:56:54.000000000 -0400 +++ rpy-0.4.6-numpy/rpy.py 2006-01-27 12:06:13.628796000 -0500 @@ -30,7 +30,7 @@ # installation time and RPy should have been compiled properly. So, # don't complain. try: - from Numeric import * + from numpy import * HAS_NUMERIC = 1 except ImportError: HAS_NUMERIC = 0 diff -Naur rpy-0.4.6/setup.py rpy-0.4.6-numpy/setup.py --- rpy-0.4.6/setup.py 2005-07-28 00:52:29.000000000 -0400 +++ rpy-0.4.6-numpy/setup.py 2006-01-28 01:18:46.148833000 -0500 @@ -34,7 +34,6 @@ from distutils.errors import * import rpy_tools - try: if sys.platform=="win32": RHOMES = os.environ['RHOMES'].split(';') @@ -72,7 +71,7 @@ if sys.platform=="win32": DEFINE.append(('R_HOME', '\\"%s\\"' % RHOME.replace("\\","/") )) else: - DEFINE.append(('R_HOME', '"%s"' % RHOME)) + DEFINE.append(('R_HOME', r'\"%s\"' % RHOME)) r_libs = [ os.path.join(RHOME, 'bin'), @@ -132,8 +131,10 @@ # check whether Numeric is present try: - import Numeric + import numpy DEFINE.append(('WITH_NUMERIC', None)) + DEFINE.append(('PY_ARRAY_TYPES_PREFIX', 'PyArray_')) + include_dirs.append(numpy.get_numpy_include()) except ImportError: UNDEF.append('WITH_NUMERIC') @@ -148,12 +149,11 @@ # use R version specific shared library name shlib_name = "_rpy%s" % RVER if sys.platform=="win32": - DEFINE.append( ('RPY_SHNAME', '\\"_rpy%s\\"' % RVER ) ) - else: DEFINE.append( ('RPY_SHNAME', '"_rpy%s"' % RVER ) ) + else: + DEFINE.append( ('RPY_SHNAME', r'\"_rpy%s\"' % RVER ) ) DEFINE.append( ('INIT_RPY', 'init_rpy%s' % RVER ) ) - modules.append( Extension( shlib_name, source_files, diff -Naur rpy-0.4.6/src/RPy.h rpy-0.4.6-numpy/src/RPy.h --- rpy-0.4.6/src/RPy.h 2005-07-26 23:05:29.000000000 -0400 +++ rpy-0.4.6-numpy/src/RPy.h 2006-01-26 17:18:37.750157000 -0500 @@ -68,12 +68,13 @@ #include +#ifdef WITH_NUMERIC +#include "numpy/arrayobject.h" +#endif + #include "robjobject.h" #include "setenv.h" -#ifdef WITH_NUMERIC -#include "Numeric/arrayobject.h" -#endif #define MAXIDSIZE 256 diff -Naur rpy-0.4.6/src/rpymodule.c rpy-0.4.6-numpy/src/rpymodule.c --- rpy-0.4.6/src/rpymodule.c 2005-07-28 10:23:33.000000000 -0400 +++ rpy-0.4.6-numpy/src/rpymodule.c 2006-01-28 01:19:29.281899000 -0500 @@ -1517,7 +1517,7 @@ if(use_numeric) { import_array(); - multiarray = PyImport_ImportModule("multiarray"); + multiarray = PyImport_ImportModule("numpy"); if (multiarray) { dict = PyModule_GetDict(multiarray); if (dict) diff -Naur rpy-0.4.6/tests/test_array.py rpy-0.4.6-numpy/tests/test_array.py --- rpy-0.4.6/tests/test_array.py 2005-07-18 14:54:54.000000000 -0400 +++ rpy-0.4.6-numpy/tests/test_array.py 2006-01-28 00:58:41.100628000 -0500 @@ -28,7 +28,7 @@ r.array.autoconvert(BASIC_CONVERSION) def testConversionToPy(self): - self.failUnless(self.py == self.ra.as_py(), + self.failUnless((self.py == self.ra.as_py()).all(), 'wrong conversion to Python') def testConversionToR(self): diff -Naur rpy-0.4.6/tests/test_numeric.py rpy-0.4.6-numpy/tests/test_numeric.py --- rpy-0.4.6/tests/test_numeric.py 2005-07-26 23:08:49.000000000 -0400 +++ rpy-0.4.6-numpy/tests/test_numeric.py 2006-01-27 12:07:38.309828000 -0500 @@ -2,7 +2,7 @@ import unittest try: - from Numeric import * + from numpy import * except ImportError: print 'Numeric not available. Skipping.\n' import sys diff -Naur rpy-0.4.6/tests/test_topy.py rpy-0.4.6-numpy/tests/test_topy.py --- rpy-0.4.6/tests/test_topy.py 2005-07-19 16:20:08.000000000 -0400 +++ rpy-0.4.6-numpy/tests/test_topy.py 2006-01-28 01:05:57.007083000 -0500 @@ -27,7 +27,7 @@ assert( r(-2147483648) == r.NA ) def testNAreal(self): - assert(r.NAN == r("as.numeric(NA)") ) + assert(repr(r.NAN) == repr(r("as.numeric(NA)")) ) def testNAstring(self): assert(r("as.character(NA)") == 'NA') @@ -48,14 +48,14 @@ def testNAList(self): xi = [1,2,r.NA,r.NAN,4] assert(r.as_character(xi) == ['1', '2', 'NA', 'NA', '4']) - assert(r.as_numeric(xi) == [1.0, 2.0, r.NAN, r.NAN, 4.0]) + assert(repr(r.as_numeric(xi)) == repr([1.0, 2.0, r.NAN, r.NAN, 4.0])) assert(r.as_integer(xi) == [1, 2, r.NA, r.NA, 4]) assert(r.as_factor(xi) == ['1', '2', 'NA', 'NA', '4']) assert(r.is_na(xi) == [False, False, True, True, False] ) xd = [1.01,2.02,r.NA,r.NAN,4.04] assert(r.as_character(xd) == ['1.01', '2.02', 'NA', 'NA', '4.04']) - assert(r.as_numeric(xd) == [1.01, 2.02, r.NAN, r.NAN, 4.04]) + assert(repr(r.as_numeric(xd)) == repr([1.01, 2.02, r.NAN, r.NAN, 4.04])) assert(r.as_integer(xd) == [1, 2, r.NA, r.NA, 4]) assert(r.as_factor(xd) == ['1.01', '2.02', 'NA', 'NA', '4.04']) assert(r.is_na(xd) == [False, False, True, True, False] ) From strawman at astraw.com Sat Jan 28 13:19:02 2006 From: strawman at astraw.com (Andrew Straw) Date: Sat Jan 28 13:19:02 2006 Subject: [Numpy-discussion] a virtual work party: help migrate scipy.org to new wiki Message-ID: <43DBDFA5.7010404@astraw.com> As many of you know, a number of us have been busy migrating the scipy.org website to a new, more user-friendly wiki system. The new wiki is up-and-running, is more-up-to-date, and prettier-to-look at than the old site. We are working to make scipy.org a portal website for all scientific software written in Python, and not just for the packages numpy and scipy, which happen to have their natural homes there. This wiki structure allows peer-review-by-wiki, so please do not feel bashful about updating any aspect of the wiki to suit your opinions. The current address of the new site is http://new.scipy.org/Wiki Once we go "live", we'll point http://scipy.org to this wiki. Before we make the switch, however, there are a few things that need to be done and a few things that should be done. This is a request for two things: 1) Review the page http://new.scipy.org/Wiki/MigratingFromPlone and make sure you can live with what it says. If you can't edit it. 2) Please undertake any of the tasks listed at that page. Update the page that you're working on the task, and again when you're done. 3) (Optional) Bask in the glory of our shiny new website which will allow NumPy/ SciPy/ Matplotlib/ IPython/ whateverelse to rule the scientific computing world! I'd like to ask that we try and complete these tasks as quickly as possible. I know I'm asking a busy group of people to pitch in, but I hope you'll agree the results will be worth it. Thank you, Andrew Straw PS Sorry for the cross postings. I thought it would be important to get as many hands involved as possible. From byrnes at bu.edu Sat Jan 28 14:20:04 2006 From: byrnes at bu.edu (John Byrnes) Date: Sat Jan 28 14:20:04 2006 Subject: [Numpy-discussion] Linux Installation Instructions Message-ID: <20060128221945.GB6662@localhost.localdomain> Is anyone else unable to access the Linux install instructions at http://pong.tamu.edu/tiki/tiki-view_blog_post.php?blogId=6&postId=97 I get a 403 - Forbidden Page. Regards, John -- Well done is better than well said. -- Benjamin Franklin From chris at trichech.us Sat Jan 28 14:31:01 2006 From: chris at trichech.us (Christopher Fonnesbeck) Date: Sat Jan 28 14:31:01 2006 Subject: [Numpy-discussion] array dtype=object problem Message-ID: <47C8EC1D-2FE0-48CD-B9CF-FF88CFBAD209@trichech.us> I am having trouble with numpy arrays that have (for some reason) an "object" dtype. All that happens in my code is that float values are appended to a multidimensional array, this array is transposed, and the transposed array is sent to matplotlib. Unfortunately, the array ends up being of type "object", which is not recognized by matplotlib: (Pdb) y[:5] array([575.060695363, 618.395159954, 817.767177867, 601.659726372, 978.607427424], dtype=object) (Pdb) scatter(x,y) *** TypeError: function not supported for these types, and can't coerce safely to supported types Moreover, this array, which clearly contains nothing but floats cannot be cast to a float array: (Pdb) array(y,float) *** TypeError: array cannot be safely cast to required type Can anyone shed some light on this? I am utterly confused. Thanks, C. -- Christopher J. Fonnesbeck Population Ecologist, Marine Mammal Section Fish & Wildlife Research Institute (FWC) St. Petersburg, FL Adjunct Assistant Professor Warnell School of Forest Resources University of Georgia Athens, GA T: 727.235.5570 E: chris at trichech.us -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2417 bytes Desc: not available URL: From ryanlists at gmail.com Sat Jan 28 14:55:02 2006 From: ryanlists at gmail.com (Ryan Krauss) Date: Sat Jan 28 14:55:02 2006 Subject: [Numpy-discussion] Re: [SciPy-user] Linux Installation Instructions In-Reply-To: <20060128221945.GB6662@localhost.localdomain> References: <20060128221945.GB6662@localhost.localdomain> Message-ID: I can't access it either right now. On 1/28/06, John Byrnes wrote: > Is anyone else unable to access the Linux install instructions at > http://pong.tamu.edu/tiki/tiki-view_blog_post.php?blogId=6&postId=97 > > I get a 403 - Forbidden Page. > > Regards, > John > > -- > Well done is better than well said. > -- Benjamin Franklin > > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.net > http://www.scipy.net/mailman/listinfo/scipy-user > From oliphant.travis at ieee.org Sat Jan 28 15:05:04 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sat Jan 28 15:05:04 2006 Subject: [Numpy-discussion] array dtype=object problem In-Reply-To: <47C8EC1D-2FE0-48CD-B9CF-FF88CFBAD209@trichech.us> References: <47C8EC1D-2FE0-48CD-B9CF-FF88CFBAD209@trichech.us> Message-ID: <43DBF896.6070600@ieee.org> Christopher Fonnesbeck wrote: > I am having trouble with numpy arrays that have (for some reason) an > "object" dtype. All that happens in my code is that float values are > appended to a multidimensional array, this array is transposed, and > the transposed array is sent to matplotlib. Unfortunately, the array > ends up being of type "object", which is not recognized by matplotlib: I'm not sure why that is, but I seem to recall from code that you've sent me before that you intentionally use object arrays from time to time. Object arrays stay object arrays unless you request differently. But, object arrays will usually run more slowly than fixed-precision arrays. > > (Pdb) scatter(x,y) > *** TypeError: function not supported for these types, and can't > coerce safely to supported types > > Moreover, this array, which clearly contains nothing but floats > cannot be cast to a float array: > > (Pdb) array(y,float) > *** TypeError: array cannot be safely cast to required type > > Can anyone shed some light on this? I am utterly confused. > When using the array constructor to cast to another type, there is a check that is equivalent to the numpy command can_cast(from_dtype, to_dtype) that is run to see if the conversion is "safe". Of course casting from an object (which could have infinite precision) to a fixed-precision floating-point data-type is not safe and you get this error. The .astype(to_dtype) method of numpy arrays will always convert (force-cast) to the request type. So, y.astype(float) will work as you want. -Travis From oliphant.travis at ieee.org Sat Jan 28 15:16:01 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sat Jan 28 15:16:01 2006 Subject: [Numpy-discussion] a virtual work party: help migrate scipy.org to new wiki In-Reply-To: <43DBDFA5.7010404@astraw.com> References: <43DBDFA5.7010404@astraw.com> Message-ID: <43DBFAF0.1090308@ieee.org> Andrew Straw wrote: >As many of you know, a number of us have been busy migrating the >scipy.org website to a new, more user-friendly wiki system. The new wiki >is up-and-running, is more-up-to-date, and prettier-to-look at than the >old site. > > Andrew has done an amazing job on this. I think the time has come to move forward with the new site. The most important pages have been moved over in my mind. There is *a lot* of outdated material on the old SciPy site. I think the best strategy is to just start using the new site with a link to the old stuff should someone realize that something is missing. My vote is to go "live" now. -Travis From strawman at astraw.com Sat Jan 28 15:20:01 2006 From: strawman at astraw.com (Andrew Straw) Date: Sat Jan 28 15:20:01 2006 Subject: [Numpy-discussion] a virtual work party: help migrate scipy.org to new wiki In-Reply-To: <43DBFAF0.1090308@ieee.org> References: <43DBDFA5.7010404@astraw.com> <43DBFAF0.1090308@ieee.org> Message-ID: <43DBFBE4.7010604@astraw.com> I'm happy with going live now, too. So, Enthought, I humbly ask you unleash the new site! :) Cheers! Andrew Travis Oliphant wrote: > Andrew Straw wrote: > >> As many of you know, a number of us have been busy migrating the >> scipy.org website to a new, more user-friendly wiki system. The new wiki >> is up-and-running, is more-up-to-date, and prettier-to-look at than the >> old site. >> >> > Andrew has done an amazing job on this. I think the time has come to > move forward with the new site. The most important pages have been > moved over in my mind. There is *a lot* of outdated material on the > old SciPy site. I think the best strategy is to just start using the > new site with a link to the old stuff should someone realize that > something is missing. > My vote is to go "live" now. > -Travis > > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log > files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From agn at noc.soton.ac.uk Sat Jan 28 16:37:03 2006 From: agn at noc.soton.ac.uk (George Nurser) Date: Sat Jan 28 16:37:03 2006 Subject: [Numpy-discussion] numpy can't use ACML In-Reply-To: References: <43DBE594.50301@gmail.com> Message-ID: <4A52806B-348E-4138-92B8-1D3F50E1D39B@noc.soton.ac.uk> I'm sure this is well known, but I just realized that numpy cannot use the _dotblas.so that it makes when it is compiled with ACML. This is because ACML only has the fortran blas libraries, not cblas. numpy will find the acml libraries and use them to make a _dotblas.so without complaining, if you have [blas] blas_libs = acml language = f77 in your site.cfg. But attempting to import this _dotblas.so gives errors .... so numpy never actually uses it. >>> import _dotblas.so Traceback (most recent call last): File "", line 1, in ? ImportError: ./_dotblas.so: undefined symbol: cblas_zaxpy nm _dotblas.so gives a whole stream of undefined cblas_xxxx symbols. So what are the options? Forget about ACML? Find an optimized cblas for numpy _dotblas but use the ACML flapack for scipy? Persuade somebody to write a scipy-style ?f2py interface to generate _dotblas.so using the fblas? George Nurser. From luszczek at cs.utk.edu Sat Jan 28 18:04:04 2006 From: luszczek at cs.utk.edu (Piotr Luszczek) Date: Sat Jan 28 18:04:04 2006 Subject: [Numpy-discussion] numpy can't use ACML In-Reply-To: <4A52806B-348E-4138-92B8-1D3F50E1D39B@noc.soton.ac.uk> References: <4A52806B-348E-4138-92B8-1D3F50E1D39B@noc.soton.ac.uk> Message-ID: <200601282102.53281.luszczek@cs.utk.edu> On Saturday 28 January 2006 19:36, George Nurser wrote: > I'm sure this is well known, but I just realized that numpy cannot > use the _dotblas.so that it makes when it is compiled with ACML. > This is because ACML only has the fortran blas libraries, not cblas. > > numpy will find the acml libraries and use them to make a _dotblas.so > without complaining, if you have > > [blas] > blas_libs = acml > language = f77 > in your site.cfg. > > But attempting to import this _dotblas.so gives errors .... so numpy > never actually uses it. > > >>> import _dotblas.so > > Traceback (most recent call last): > File "", line 1, in ? > ImportError: ./_dotblas.so: undefined symbol: cblas_zaxpy > > nm _dotblas.so gives a whole stream of undefined cblas_xxxx symbols. > > So what are the options? Forget about ACML? Find an optimized cblas > for numpy _dotblas but use the ACML flapack for scipy? Persuade > somebody to write a scipy-style ?f2py interface to generate > _dotblas.so using the fblas? > > George Nurser. There is code for that on netlib: http://www.netlib.org/blas/blast-forum/cblas.tgz I used it myself for my C code before and it worked just fine. Piotr From brendansimons at yahoo.ca Sat Jan 28 18:28:02 2006 From: brendansimons at yahoo.ca (Brendan Simons) Date: Sat Jan 28 18:28:02 2006 Subject: [Numpy-discussion] numpy.dot behavior with column vector and length 1 vector In-Reply-To: <20060128230601.C843888D45@sc8-sf-spam1.sourceforge.net> References: <20060128230601.C843888D45@sc8-sf-spam1.sourceforge.net> Message-ID: <015961AF-70D4-43F2-81D8-9FCC5E34E796@yahoo.ca> Is this behavior expected? If so, why am I wrong in assuming these statements should all return the same result? PyCrust 0.9.5 - The Flakiest Python Shell Python 2.4.1 (#2, Mar 31 2005, 00:05:10) [GCC 3.3 20030304 (Apple Computer, Inc. build 1666)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> colVect = numpy.ones((3, 1)) >>> colVect * 5.3 array([[ 5.3], [ 5.3], [ 5.3]]) >>> numpy.dot(colVect, 5.3) array([[ 5.3], [ 5.3], [ 5.3]]) >>> colVect * [5.3] array([[ 5.3], [ 5.3], [ 5.3]]) >>> numpy.dot(colVect, [5.3]) array([ 5.3, 0. , 0. ]) Brendan -- Brendan Simons __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From oliphant.travis at ieee.org Sat Jan 28 18:56:01 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sat Jan 28 18:56:01 2006 Subject: [Numpy-discussion] numpy.dot behavior with column vector and length 1 vector In-Reply-To: <015961AF-70D4-43F2-81D8-9FCC5E34E796@yahoo.ca> References: <20060128230601.C843888D45@sc8-sf-spam1.sourceforge.net> <015961AF-70D4-43F2-81D8-9FCC5E34E796@yahoo.ca> Message-ID: <43DC2E7D.9050609@ieee.org> Brendan Simons wrote: > Is this behavior expected? If so, why am I wrong in assuming these > statements should all return the same result? > > PyCrust 0.9.5 - The Flakiest Python Shell > Python 2.4.1 (#2, Mar 31 2005, 00:05:10) > [GCC 3.3 20030304 (Apple Computer, Inc. build 1666)] on darwin > Type "help", "copyright", "credits" or "license" for more information. > >>> import numpy > >>> colVect = numpy.ones((3, 1)) > >>> colVect * 5.3 > array([[ 5.3], > [ 5.3], > [ 5.3]]) > >>> numpy.dot(colVect, 5.3) > array([[ 5.3], > [ 5.3], > [ 5.3]]) > >>> colVect * [5.3] > array([[ 5.3], > [ 5.3], > [ 5.3]]) > >>> numpy.dot(colVect, [5.3]) > array([ 5.3, 0. , 0. ]) This is a dotblas bug. You can always tell by testing out the original dot function: from numpy.core.multiarray import dot as dot_ dot_(colVect, [5.3]) does not give this result. -Travis From oliphant.travis at ieee.org Sat Jan 28 19:53:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sat Jan 28 19:53:02 2006 Subject: [Numpy-discussion] numpy.dot behavior with column vector and length 1 vector In-Reply-To: <015961AF-70D4-43F2-81D8-9FCC5E34E796@yahoo.ca> References: <20060128230601.C843888D45@sc8-sf-spam1.sourceforge.net> <015961AF-70D4-43F2-81D8-9FCC5E34E796@yahoo.ca> Message-ID: <43DC3BE4.8080906@ieee.org> Brendan Simons wrote: > Is this behavior expected? If so, why am I wrong in assuming these > statements should all return the same result? This is now fixed in SVN. -Travis From brendansimons at yahoo.ca Sat Jan 28 20:47:04 2006 From: brendansimons at yahoo.ca (Brendan Simons) Date: Sat Jan 28 20:47:04 2006 Subject: [Numpy-discussion] numpy.dot behavior with column vector and length 1 vector In-Reply-To: <20060129041416.B07AA125B4@sc8-sf-spam2.sourceforge.net> References: <20060129041416.B07AA125B4@sc8-sf-spam2.sourceforge.net> Message-ID: <85D4958E-9208-4B4F-A662-77C1DB50BF73@yahoo.ca> > > --__--__-- > > Message: 7 > Date: Sat, 28 Jan 2006 20:52:04 -0700 > From: Travis Oliphant > To: Brendan Simons , > numpy-discussion > Subject: Re: [Numpy-discussion] numpy.dot behavior with column > vector and > length 1 vector > > Brendan Simons wrote: > >> Is this behavior expected? If so, why am I wrong in assuming these >> statements should all return the same result? > > > This is now fixed in SVN. > > -Travis Works for me now. Thanks! Brendan -- Brendan Simons __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From gerard.vermeulen at grenoble.cnrs.fr Sun Jan 29 00:10:01 2006 From: gerard.vermeulen at grenoble.cnrs.fr (Gerard Vermeulen) Date: Sun Jan 29 00:10:01 2006 Subject: [Numpy-discussion] bug triggered by array.astype() Message-ID: <20060129090918.005c440c.gerard.vermeulen@grenoble.cnrs.fr> >>> from numpy import * >>> core.__version__ '0.9.5.2019' >>> v = linspace(0, 2*pi) >>> c = (255<<24)*cos(v) + (255<<16)*cos(v+2*pi/3) + (255<<8)*cos(v+4*pi/3) + 255 >>> c.astype(UInt32) Traceback (most recent call last): File "", line 1, in ? SystemError: Objects/longobject.c:257: bad argument to internal function >>> Gerard From oliphant.travis at ieee.org Sun Jan 29 00:25:04 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sun Jan 29 00:25:04 2006 Subject: [Numpy-discussion] bug triggered by array.astype() In-Reply-To: <20060129090918.005c440c.gerard.vermeulen@grenoble.cnrs.fr> References: <20060129090918.005c440c.gerard.vermeulen@grenoble.cnrs.fr> Message-ID: <43DC7BC1.6050408@ieee.org> Gerard Vermeulen wrote: >>>>from numpy import * >>>>core.__version__ >>>> >>>> >'0.9.5.2019' > > >>>>v = linspace(0, 2*pi) >>>>c = (255<<24)*cos(v) + (255<<16)*cos(v+2*pi/3) + (255<<8)*cos(v+4*pi/3) + 255 >>>>c.astype(UInt32) >>>> >>>> >Traceback (most recent call last): > File "", line 1, in ? >SystemError: Objects/longobject.c:257: bad argument to internal function > > > > > Thanks for this test. The problem stems from the (255<<24) which (on 32-bit platform) generates a long-integer object. Currently, long-integers are converted to object arrays, thus this operation causes your entire calculation to be done using Python objects. Then, you try to convert the whole thing to UInt32 which is giving the error. I'll look into the error. In the meantime, you can avoid going through object arrays using an int64 scalar: int64(255<24)*cos(v) + ... -Travis From oliphant.travis at ieee.org Sun Jan 29 00:52:01 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sun Jan 29 00:52:01 2006 Subject: [Numpy-discussion] bug triggered by array.astype() In-Reply-To: <20060129090918.005c440c.gerard.vermeulen@grenoble.cnrs.fr> References: <20060129090918.005c440c.gerard.vermeulen@grenoble.cnrs.fr> Message-ID: <43DC81F1.3020309@ieee.org> Gerard Vermeulen wrote: >>>>from numpy import * >>>>core.__version__ >>>> >>>> >'0.9.5.2019' > > >>>>v = linspace(0, 2*pi) >>>>c = (255<<24)*cos(v) + (255<<16)*cos(v+2*pi/3) + (255<<8)*cos(v+4*pi/3) + 255 >>>>c.astype(UInt32) >>>> >>>> >Traceback (most recent call last): > File "", line 1, in ? >SystemError: Objects/longobject.c:257: bad argument to internal function > > > > > In SVN, now a Overflow error is raised instead in this example because negative long integer objects are trying to be converted to unsigned integers (this is a Python error being raised). We could check for this and instead do what the c-compiler would do in converting from signed to unsigned objects. Is that preferrable? -Travis From gerard.vermeulen at grenoble.cnrs.fr Sun Jan 29 01:19:02 2006 From: gerard.vermeulen at grenoble.cnrs.fr (Gerard Vermeulen) Date: Sun Jan 29 01:19:02 2006 Subject: [Numpy-discussion] bug triggered by array.astype() In-Reply-To: <43DC81F1.3020309@ieee.org> References: <20060129090918.005c440c.gerard.vermeulen@grenoble.cnrs.fr> <43DC81F1.3020309@ieee.org> Message-ID: <20060129101822.609fb955.gerard.vermeulen@grenoble.cnrs.fr> On Sun, 29 Jan 2006 01:50:57 -0700 Travis Oliphant wrote: [ .. ] > In SVN, now a Overflow error is raised instead in this example because > negative long integer objects are trying to be converted to unsigned > integers (this is a Python error being raised). We could check for this > and instead do what the c-compiler would do in converting from signed to > unsigned objects. > > Is that preferrable? > Well, the current SVN does now almost what Numeric does: >>> from Numeric import * >>> v = arange(0, 10, 1, Float32) >>> c = (255<<24)*cos(v)+ (255<<16)*cos(v+2*pi/3) + (255<<8)*cos(v+4*pi/3) + 255 >>> c.astype(UInt32) Traceback (most recent call last): File "", line 1, in ? OverflowError: long int too large to convert to int >>> However, numarray does: >>> from numarray import * >>> v = arange(0, 10, 1, Float32) >>> c = (255<<24)*cos(v)+ (255<<16)*cos(v+2*pi/3) + (255<<8)*cos(v+4*pi/3) + 255 >>> c.astype(UInt32) array([4269801984, 2294853120, 2504994432, 65861376, 1514949120, 1225005568, 4103764992, 3209541888, 3659448448, 398681088], type=UInt32) >>> This is more user friendly, but may cost more CPU cycles. Gerard From oliphant.travis at ieee.org Sun Jan 29 02:26:01 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sun Jan 29 02:26:01 2006 Subject: [Numpy-discussion] bug triggered by array.astype() In-Reply-To: <20060129101822.609fb955.gerard.vermeulen@grenoble.cnrs.fr> References: <20060129090918.005c440c.gerard.vermeulen@grenoble.cnrs.fr> <43DC81F1.3020309@ieee.org> <20060129101822.609fb955.gerard.vermeulen@grenoble.cnrs.fr> Message-ID: <43DC9806.8030904@ieee.org> Gerard Vermeulen wrote: >On Sun, 29 Jan 2006 01:50:57 -0700 >Travis Oliphant wrote: > >[ .. ] > > > >>In SVN, now a Overflow error is raised instead in this example because >>negative long integer objects are trying to be converted to unsigned >>integers (this is a Python error being raised). We could check for this >>and instead do what the c-compiler would do in converting from signed to >>unsigned objects. >> >>Is that preferrable? >> >> >> > >Well, the current SVN does now almost what Numeric does: > > > >>>>from Numeric import * >>>>v = arange(0, 10, 1, Float32) >>>>c = (255<<24)*cos(v)+ (255<<16)*cos(v+2*pi/3) + (255<<8)*cos(v+4*pi/3) + 255 >>>>c.astype(UInt32) >>>> >>>> >Traceback (most recent call last): > File "", line 1, in ? >OverflowError: long int too large to convert to int > > > >However, numarray does: > > > >>>>from numarray import * >>>>v = arange(0, 10, 1, Float32) >>>>c = (255<<24)*cos(v)+ (255<<16)*cos(v+2*pi/3) + (255<<8)*cos(v+4*pi/3) + 255 >>>>c.astype(UInt32) >>>> >>>> >array([4269801984, 2294853120, 2504994432, 65861376, 1514949120, > 1225005568, 4103764992, 3209541888, 3659448448, 398681088], type=UInt32) > > > >This is more user friendly, but may cost more CPU cycles. > > The issue here is really that in Numeric (and numpy) c is an object array while in numarray c is a float array. Because Numeric and numpy support object arrays, long integers have been converted to objects. I think what should be changed is that long integers should only be converted to objects if they are bigger than the maximum integer-type on the platform. That is what numarray allows and makes sense. In other words, the root of this difference is what array(255<<24) returns. In Numeric and numpy it returns an object array. In numarray it returns a UInt64 array. -Travis From oliphant.travis at ieee.org Sun Jan 29 02:40:08 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sun Jan 29 02:40:08 2006 Subject: [Numpy-discussion] bug triggered by array.astype() In-Reply-To: <20060129090918.005c440c.gerard.vermeulen@grenoble.cnrs.fr> References: <20060129090918.005c440c.gerard.vermeulen@grenoble.cnrs.fr> Message-ID: <43DC9B78.9010705@ieee.org> Gerard Vermeulen wrote: >>>>from numpy import * >>>>core.__version__ >>>> >>>> >'0.9.5.2019' > > >>>>v = linspace(0, 2*pi) >>>>c = (255<<24)*cos(v) + (255<<16)*cos(v+2*pi/3) + (255<<8)*cos(v+4*pi/3) + 255 >>>>c.astype(UInt32) >>>> >>>> >Traceback (most recent call last): > File "", line 1, in ? >SystemError: Objects/longobject.c:257: bad argument to internal function > > This behaves like numarray now in numpy SVN where-in (255<<24) is converted to an int64 array during the multiplication step so that object arrays are not constructed. -Travis From gerard.vermeulen at grenoble.cnrs.fr Sun Jan 29 05:00:01 2006 From: gerard.vermeulen at grenoble.cnrs.fr (Gerard Vermeulen) Date: Sun Jan 29 05:00:01 2006 Subject: [Numpy-discussion] performance problem Message-ID: <20060129135902.4b1291c5.gerard.vermeulen@grenoble.cnrs.fr> Hi Travis, thanks for your incredible quick fixes. numpy is about 5 times slower than numarray (on my numarray-friendly bi-pentium): >>> from timeit import Timer >>> import numarray; print numarray.__version__ 1.5.0 >>> import numpy; print numpy.__version__ 0.9.5.2021 >>> t1 = Timer('a <<= 8', 'import numarray as NX; a = NX.ones(10**6, NX.UInt32)') >>> t2 = Timer('a <<= 8', 'import numpy as NX; a = NX.ones(10**6, NX.UInt32)') >>> t1.timeit(100) 0.21813011169433594 >>> t2.timeit(100) 1.1523458957672119 >>> Numeric-23.1 is about as fast as numarray for inplace left shifts. Gerard From schofield at ftw.at Sun Jan 29 08:34:06 2006 From: schofield at ftw.at (Ed Schofield) Date: Sun Jan 29 08:34:06 2006 Subject: [Numpy-discussion] Segfault with dodgy type descriptors Message-ID: <43DCEE45.2020608@ftw.at> Hi Travis, Here's a segfault from the crazy input department: >>> cs = [('A', 1), ('B', 2)] >>> a = numpy.array(cs, dtype=(object,5)) Using the descriptor dtype=(object,x) for other values of x gives varying results, but all of them should probably raise a ValueError. On a related note, it seems that it's now possible again to use an array as a dtype. You changed this behaviour in November after I suggested that this should be illegal. (See http://www.scipy.net/pipermail/scipy-dev/2005-November/004126.html). My argument then was that calling array(a, b.dtype) is clearer and safer than using array(a, b), which can be confused with array((a,b)). Is it an oversight that arrays can again be used as dtypes? -- Ed From gerard.vermeulen at grenoble.cnrs.fr Sun Jan 29 10:14:04 2006 From: gerard.vermeulen at grenoble.cnrs.fr (Gerard Vermeulen) Date: Sun Jan 29 10:14:04 2006 Subject: [Numpy-discussion] another performance problem Message-ID: <20060129191327.49acbb26.gerard.vermeulen@grenoble.cnrs.fr> Hi Travis, max and min are really slow in numpy: >>> from timeit import Timer >>> import Numeric; Numeric.__version__ '23.1' >>> import numarray; numarray.__version__ '1.2.3' >>> import numpy; numpy.__version__ '0.9.5.2021' >>> t1 = Timer('a = max(b)', 'import Numeric as N; b=N.cos(N.arange(0, 10**5, 1, N.Float))') >>> t2 = Timer('a = max(b)', 'import numarray as N; b=N.cos(N.arange(0, 10**5, 1, N.Float))') >>> t3 = Timer('a = max(b)', 'import numpy as N; b=N.cos(N.arange(0, 10**5, 1, N.Float))') >>> t1.timeit(10) 0.13748002052307129 >>> t2.timeit(10) 0.64549708366394043 >>> t3.timeit(10) 4.5433549880981445 >>> Gerard From oliphant.travis at ieee.org Sun Jan 29 17:17:07 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sun Jan 29 17:17:07 2006 Subject: [Numpy-discussion] another performance problem In-Reply-To: <20060129191327.49acbb26.gerard.vermeulen@grenoble.cnrs.fr> References: <20060129191327.49acbb26.gerard.vermeulen@grenoble.cnrs.fr> Message-ID: <43DD68E1.30102@ieee.org> Gerard Vermeulen wrote: >Hi Travis, > >max and min are really slow in numpy: > > > >>>>from timeit import Timer >>>>import Numeric; Numeric.__version__ >>>> >>>> >'23.1' > > >>>>import numarray; numarray.__version__ >>>> >>>> >'1.2.3' > > >>>>import numpy; numpy.__version__ >>>> >>>> >'0.9.5.2021' > > >>>>t1 = Timer('a = max(b)', 'import Numeric as N; b=N.cos(N.arange(0, 10**5, 1, N.Float))') >>>>t2 = Timer('a = max(b)', 'import numarray as N; b=N.cos(N.arange(0, 10**5, 1, N.Float))') >>>>t3 = Timer('a = max(b)', 'import numpy as N; b=N.cos(N.arange(0, 10**5, 1, N.Float))') >>>>t1.timeit(10) >>>> >>>> >0.13748002052307129 > > >>>>t2.timeit(10) >>>> >>>> >0.64549708366394043 > > >>>>t3.timeit(10) >>>> >>>> >4.5433549880981445 > > You shouldn't be using max(b)!!! You should use b.max() because max is a Python function that is using the generic sequence interface to do the comparisons. Very likely it is slower because in numpy you get an array scalar (which don't yet have their own math defined and so use the full ufunc machinery do to computations). At some point the array scalars will have their own math defined and should be as fast as the Python scalars, so this particular "slowness" will go away. But, you use maximum.reduce() instead anyway. -Travis From faltet at carabos.com Sun Jan 29 22:38:01 2006 From: faltet at carabos.com (Francesc Altet) Date: Sun Jan 29 22:38:01 2006 Subject: [Numpy-discussion] Segfaults when trying to build an array using the description in array protocol Message-ID: <1138603012.7530.14.camel@localhost.localdomain> Hi, Apparently something wrong went in latest version of numpy SVN: In [4]:numpy.__version__ Out[4]:'0.9.5.2021' In [5]:numpy.empty((1,), dtype='i8') Out[5]:array([-5193749191705487200], dtype=int64) In [6]:numpy.empty((1,), dtype=[('p', 'i8')]) Segmentation fault This usually worked well with older versions: In [7]:numpy.__version__ Out[7]:'0.9.5.1980' In [8]:numpy.empty((1,), dtype=[('p', 'i8')]) Out[8]:array([(-5192306666809247592L,)], dtype=(void,8)) BTW, which is the recommended way to report that sort of things, the list or the Trac interface for SVN numpy? I'd like to start using the second, but I haven't manage to create a user for reporting issues. Thanks, -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From faltet at carabos.com Sun Jan 29 23:19:01 2006 From: faltet at carabos.com (Francesc Altet) Date: Sun Jan 29 23:19:01 2006 Subject: [Numpy-discussion] Segfaults when trying to build an array using the description in array protocol In-Reply-To: <43DDB96A.4030307@ieee.org> References: <1138603012.7530.14.camel@localhost.localdomain> <43DDB96A.4030307@ieee.org> Message-ID: <1138605500.7530.21.camel@localhost.localdomain> El dg 29 de 01 del 2006 a les 23:59 -0700, en/na Travis Oliphant va escriure: > I think you can report anonymously. I've just done it. However, I'd like to be informed about the follow-ups of the ticket by e-mail. I've put my e-mail address in the ticket, although I'm not sure if I'll get e-mail on this. Cheers, -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From oliphant.travis at ieee.org Mon Jan 30 00:24:01 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Jan 30 00:24:01 2006 Subject: [Numpy-discussion] performance problem In-Reply-To: <20060129135902.4b1291c5.gerard.vermeulen@grenoble.cnrs.fr> References: <20060129135902.4b1291c5.gerard.vermeulen@grenoble.cnrs.fr> Message-ID: <43DDCCE4.8090705@ieee.org> Gerard Vermeulen wrote: >Hi Travis, thanks for your incredible quick fixes. > >numpy is about 5 times slower than numarray (on my numarray-friendly bi-pentium): > > > This is a coercion issue. NumPy is exercising the most complicated section of the ufunc code requiring buffered casting because you are asking it to combine a uint32 array with a signed scalar (so the only commensurable type is a signed scalar of type int64. This is then coerced back into the unsigned array). Look at the data-type of b = a << 8. (a<<8).dtype.name 'int64' Try a<<=val where val = uint32(8) is in the header You should see more commensurate times. We can of course discuss the appropriateness of the coercion rules. Basically scalars don't cause coercion unless they are of a different kind but as of now signed and unsigned integers are considered to be of different kinds. I think that there is a valid point to be made that all scalar integers should be treated the same since Python only has one way to enter an integer. Right now, this is not what's done, but it could be changed rather easily. -Travis From matthew.brett at gmail.com Mon Jan 30 00:32:04 2006 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon Jan 30 00:32:04 2006 Subject: [Numpy-discussion] Verbose output from setup.py --help intentional? Message-ID: <1e2af89e0601300031k4790d3f5m96ddda611f4421e5@mail.gmail.com> Hi, Sorry if this is low priority, but I noticed that the output from setup.py --help is more than 400 lines. Also the output of (e.g) setup.py --help install and setup.py install --help differ (also more than 400 lines). Best, Matthew From oliphant.travis at ieee.org Mon Jan 30 00:53:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Jan 30 00:53:02 2006 Subject: [Numpy-discussion] performance problem In-Reply-To: <20060129135902.4b1291c5.gerard.vermeulen@grenoble.cnrs.fr> References: <20060129135902.4b1291c5.gerard.vermeulen@grenoble.cnrs.fr> Message-ID: <43DDD3E5.80902@ieee.org> Gerard Vermeulen wrote: >coercion issue snipped > > In current SVN, I think improved on the current state by only calling a scalar a signed integer if it is actually negative (previously only it's data-type was checked and all Python integers get converted to PyArray_LONG data-types which are signed integers). This fixes the immediate problem, I think. What are opinions on this scalar-coercion rule? -Travis From faltet at carabos.com Mon Jan 30 00:57:02 2006 From: faltet at carabos.com (Francesc Altet) Date: Mon Jan 30 00:57:02 2006 Subject: [Numpy-discussion] Problems with creating arrays from array descriptors and lists Message-ID: <1138611386.7530.28.camel@localhost.localdomain> Hi, I think this worked some time ago, but I cannot tell for sure: In [19]:numpy.__version__ Out[19]:'0.9.5.1980' In [20]:numpy.array([1,1], dtype=[('f1', 'i8'),('f2','i4')]) --------------------------------------------------------------------------- exceptions.TypeError Traceback (most recent call last) /home/faltet/ TypeError: expected a readable buffer object Sorry, I cannot check with latest SVN because it segfaults. -- >0,0< Francesc Altet http://www.carabos.com/ V V C?rabos Coop. V. Enjoy Data "-" From oliphant.travis at ieee.org Mon Jan 30 01:04:03 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Jan 30 01:04:03 2006 Subject: [Numpy-discussion] Segfault with dodgy type descriptors In-Reply-To: <43DCEE45.2020608@ftw.at> References: <43DCEE45.2020608@ftw.at> Message-ID: <43DDD65F.8090209@ieee.org> Ed Schofield wrote: >Hi Travis, > >Here's a segfault from the crazy input department: > > > >>>>cs = [('A', 1), ('B', 2)] >>>>a = numpy.array(cs, dtype=(object,5)) >>>> >>>> > >Using the descriptor dtype=(object,x) for other values of x gives >varying results, but all of them should probably raise a ValueError. > > These data-types are technically o.k. because (object, 5) means a data-type which is an array of 5 objects. The problem was that your constructor was not creating 5 objects and so an error should have been raised. It is now. >On a related note, it seems that it's now possible again to use an array >as a dtype. You changed this behaviour in November after I suggested >that this should be illegal. (See >http://www.scipy.net/pipermail/scipy-dev/2005-November/004126.html). My >argument then was that calling array(a, b.dtype) is clearer and safer >than using array(a, b), which can be confused with array((a,b)). Is it >an oversight that arrays can again be used as dtypes? > > I think this is an oversight, but I'm not exactly sure how to fix it. Basically during design of the new data-types, I allowed the possibility that any object with a dtype attribute could be considered a data-type. I wasn't thinking about arrays (which of course have that attribute as well), I was thinking about record objects with nested records and making it relatively easy to specify record data-types using objects with a dtype attribute. But, I could be persuaded (again), that it is easy enough to request the data-type attribute should you every actually need it, then have it specially-coded in the descriptor converter. In fact, I don't see any real benefit to it at all. My thinking about record objects changed during development and this could be just a holdover. -Travis From hurak at control.felk.cvut.cz Mon Jan 30 01:31:06 2006 From: hurak at control.felk.cvut.cz (=?ISO-8859-2?Q?Zden=ECk_Hur=E1k?=) Date: Mon Jan 30 01:31:06 2006 Subject: [Numpy-discussion] Numerical Mathematics Consortium vs. scipy and numpy Message-ID: Hello all, the other day I ran across the following link ----> http://www.nmconsortium.org/. Are you, Python numerics guys, aware of this? I admit I have absolutely no idea about the actual usefullness and relevancy of this consortium and their drafts. But it might be useful just to be aware... Important players in scientific computation are involved. Best regards, Zdenek Hurak From matthew.brett at gmail.com Mon Jan 30 04:20:06 2006 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon Jan 30 04:20:06 2006 Subject: [Numpy-discussion] ATLAS binaries - where are we? Message-ID: <1e2af89e0601300412o7e98a9bdy2b85c288e81d7d33@mail.gmail.com> Hi, I wonder if the forthcoming release of numpy is a good stimulus to revisit the issue of ATLAS binaries, as for: http://www.scipy.org/download/atlasbinaries/ and http://www.kevinsheppard.com/research/matlabatlas/matlab_atlas.aspx It seems to me that it will be very important to have ATLAS binaries, at very least for windows, which install seamlessly with NumPy. I am very happy to contribute builds, and can probably make a little windows compile farm at my lab (they _love_ windows where I work). Would that be useful? How would that best be integrated with the distribution? Best, Matthew From oliphant.travis at ieee.org Mon Jan 30 09:26:07 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Jan 30 09:26:07 2006 Subject: [Numpy-discussion] Problems with creating arrays from array descriptors and lists In-Reply-To: <1138614033.7530.57.camel@localhost.localdomain> References: <1138611386.7530.28.camel@localhost.localdomain> <43DDDB1B.8070007@ieee.org> <1138614033.7530.57.camel@localhost.localdomain> Message-ID: <43DE4BD5.3010709@ieee.org> Francesc Altet wrote: >El dl 30 de 01 del 2006 a les 02:23 -0700, en/na Travis Oliphant va >escriure: > > >>>In [20]:numpy.array([1,1], dtype=[('f1', 'i8'),('f2','i4')]) >>>--------------------------------------------------------------------------- >>>exceptions.TypeError Traceback (most >>>recent call last) >>> >>>/home/faltet/ >>> >>>TypeError: expected a readable buffer object >>> >>>Sorry, I cannot check with latest SVN because it segfaults. >>> >>> >>> >>> >>It shouldn't. Latest SVN works fine. >> >> > >True. > > > >>But, your example does not work (Records are only settable with tuples). >> >>numpy.array((1,1), ....) would work (returning an 0-d array) >> >> > >Why is this? I think that allowing lists as well would be highly useful. >In fact, I'm sure that many users are more used to provide lists instead >of tuples. Moreover, allowing tuples and not lists can be a bit >misleading, IMO. > > You can provide lists. It's just that fields must be entered in the array constructor as tuples to fill them in. This was the easiest to implement (it fit in naturally with the code already written from Numeric, and does not seem to me to be a real limitation and in fact can make things more readable. Thus, you can do numpy.array([[obj1, obj2, obj3]], dtype=) as long as the objects are tuples. -Travis From oliphant.travis at ieee.org Mon Jan 30 09:28:35 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Jan 30 09:28:35 2006 Subject: [Numpy-discussion] ATLAS binaries - where are we? In-Reply-To: <1e2af89e0601300412o7e98a9bdy2b85c288e81d7d33@mail.gmail.com> References: <1e2af89e0601300412o7e98a9bdy2b85c288e81d7d33@mail.gmail.com> Message-ID: <43DE4C8B.7020203@ieee.org> Matthew Brett wrote: >Hi, > >I wonder if the forthcoming release of numpy is a good stimulus to >revisit the issue of ATLAS binaries, as for: > >http://www.scipy.org/download/atlasbinaries/ > >and > >http://www.kevinsheppard.com/research/matlabatlas/matlab_atlas.aspx > >It seems to me that it will be very important to have ATLAS binaries, >at very least for windows, which install seamlessly with NumPy. > >I am very happy to contribute builds, and can probably make a little >windows compile farm at my lab (they _love_ windows where I work). >Would that be useful? How would that best be integrated with the >distribution? > > Absolutely useful... I suggest making a page on the new.scipy.org Wiki and uploading them there (to upload to the Wiki I think you create an attachment using attachment: Then, when the page is rendered you can click on the resulting link to upload the file). We could easily place links to this site for downloading. The old SciPy site already had such a page of pre-compiled ATLAS builds. Thanks for offering. -Travis From anthem at taipeilink.net Mon Jan 30 09:28:42 2006 From: anthem at taipeilink.net (=?windows-1251?B?MTQgLSAxNSD05eLw4Ov/IDIwMDYg4+7k4A==?=) Date: Mon Jan 30 09:28:42 2006 Subject: [Numpy-discussion] =?windows-1251?B?zc7C28kgzMXQ18DNxMDJx8jNwyAtIMXZxSDBzsvc2MUgz9DIwdvLyA==?= Message-ID: ?????????? ??????????? ?????????? ???????? ? ????????? ????, ?????????? ?? ???????, ?????????? ????????? ???????? ???????, ??????? ???????? ?????? ? ????????? "?????? ???????? 2005 ????"! ????? ????????????? - ??? ?????? ??????? 14 - 15 ??????? 2006 ???? ???????????? ??????? ?? ??????????????, ?????????? ???????, ??????? ?? ?????? ?????, ??? ??? ??????, ?? ? ??? ???????? ??????? ?????????????? ? ????????? ?????? ? ??????? ???????? ?????????? ??????????. ???? ????????: ???????????? ?????????? ???????? ?????????? ??????????? ?????? ?? ????? ???????, ???????????: ??????? ??????????? ??????? ???????????????? ??????????; ??????? ???????? ??????? "?????????" ? ???????????; ??????? ? ???????? ?????????, "?????????????" ?????????? ? ???????; ??????? ???????? ???????? ??????, ?????????? ? ?????? ??????????. ? ????????? ????????: ?????????? ??? ????? ?????????????? ??????? ??????????? ? ????????????? ??????????????. 1. ?????????? ???????????? ? ??????? ?????????????? ????????? ?????????? ??????????????? ???????? ? ??????????. ????? ?????????????? ? ??????????. ???? ????????????: ?????? ????? ?? ??????. 2. ??????? ?????????? ???????? ??????? ?????????? ???????? ??????? ????????? ? ????????? ? ???????? ???????. ????? ????????: ????? ?? ??????? ????. ???? ? ???????: ??????? ?????????? ?? ?????. 3. ????????? ???????? ?????????? ??????????, ???????? ?? ????????? ????????. ????????? ????????? ??? ??????? ???????. ?????????? ?????????. ????. ??????. ??????. 4. ?????????? ????????? ???????????? ??????? ???????????? ????????. ??????? ????????????? ??? . ??????????? ? ????????? ??????????. 5. ???????? ?????? ??????? ???????? ?????????????. ??????? ?????? ? ????? ????. ??????? "????????????? ???" ?? ??????". ??????? ????????????? ??????. ??????? ????????? ??????. ??????? ??????????? ??????. ??????? ??????? ?????? ?? ?????. ??????? "????? ? ??????????". ????????? ?????? ???????? "?????? ? ?????". 6. ???????????????? ?????????? ?????????????? ??????????. ??????? ??????? ????????? ?????????. ??????? ?????????? ????????. ???????? ???????, ?????????? ?????. ??????? ?????????????? ??????????. 7. ????????? ???????? ??? ????? ????????? ?????????????? ??????? ?????? ??????????. ??????????? ??????????. ???????? ???????????? ????????. ?????????? ?????? ?????? ? ???????????. ???????? ???????? ?????????? ??????????? ?? ????? ??????. ??????????? ?????????? ???????????. ?????? ???????? ?????? ?????????. ????? ???????? - ??????????? ??????? ? ??????? ??????????? ?????????????? ?????, ????? ??????? MBA. ??????? ?????????? ? ??????? ?????????? ?????? ?????????????? ? ????????? ????????. ????????? ????????????????? ???? ????????? "????????????? ???", ? ????? ?????? ????? ?? ??????? ??????? ? ??????????? ???????: DIXIS, LEFUTUR ? ??. ????? ????????????? ???????? - ?????????? ?????? ?????????????? ? ???????? ????????? ????????? ?? ??????? ????????? ???????, ??????, ???????????? ?????????? ? ??. ? ????????? ????? ?????????? ???????????? ??????? ?????????????? ? ???? ????????? ?????? "?????????? ????". ????? ??????????? ? ??????? ????? "????????????? - ??????? ??? ??????" ? ???????? ?????????? ?????? ? ?????????????????? ???????? ?? ???????? ??????????? ????????? ??????. ?????? ????????? ?? ?????? ???????? ? ????????????? ????????, ? ?????? ????????? ?????? ??????? ? ?????? ?????????? ??????. ???????: (495) 98?-65-39, 98?-65-36 -------------- next part -------------- An HTML attachment was scrubbed... URL: From service at paypal.com Mon Jan 30 09:52:09 2006 From: service at paypal.com (Paypal) Date: Mon Jan 30 09:52:09 2006 Subject: [Numpy-discussion] Your Paypal Account. Message-ID: <1130384585.13653@paypal.com> An HTML attachment was scrubbed... URL: From svetosch at gmx.net Mon Jan 30 10:19:02 2006 From: svetosch at gmx.net (Sven Schreiber) Date: Mon Jan 30 10:19:02 2006 Subject: [Numpy-discussion] (some) generalized eigenvalues in numpy Message-ID: <43DE5841.6070006@gmx.net> Hi, in econometrics/statistics it's relatively common to solve a special generalized eigenvalue problem, where the matrices are real, symmetric, and positive definite. It bothered me to depend on scipy just because of that. Also, scipy.linalg.eig is much more general and returns complex types which is inconvenient for sorting etc. So I've written the workaround function below, but it's probably not very efficient (at least I hope the algebra is right). I'd be happy to hear your comments, suggestions; first with respect to the function itself, but also whether you agree that the implemented functionality would be useful enough to include in official numpy (in a more professional way than I have done). cheers, Sven def geneigsympos(A, B): """ Solves symmetric-positive-definite generalized eigenvalue problem Az=lBz. Takes two real-valued symmetric matrices A and B (B must also be positive-definite) and returns the corresponding generalized (but also real-valued) eigenvalues and eigenvectors. Return format: as in scipy.linalg.eig, tuple (l, Z); l is directly taken from eigh output (a 1-dim array of length A.shape[0] ?), and Z is an array or matrix (depending on type of input A) with the corresponding eigenvectors in columns (hopefully). Eigenvalues are ordered ascending (thanks to eigh). """ from numpy import asmatrix, asarray, linalg #fixme: input checks on the matrices LI = asmatrix(linalg.cholesky(B)).I C = LI * asmatrix(A) * LI.T evals, evecs = linalg.eigh(C) if type(A) == type(asarray(A)): output = "array" # A was passed as numpy-array else: output = "matrix" #but the evecs need to be transformed back: evecs = LI.T * asmatrix(evecs) if output == "array": return evals, asarray(evecs) else: return asmatrix(evals), evecs From Chris.Barker at noaa.gov Mon Jan 30 10:25:07 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon Jan 30 10:25:07 2006 Subject: [Numpy-discussion] Numerical Mathematics Consortium vs. scipy and numpy In-Reply-To: References: Message-ID: <43DE59CB.6070601@noaa.gov> Zdene(k Hur?k wrote: > the other day I ran across the following link ----> > http://www.nmconsortium.org/. Thanks for the pointer, it looks like a great project, let's hope it gets some traction. A few thoughts that are NumPy related: -- If nothing else, it's a pretty good list of stuff that "should" be included in Stock NumPy (and/or the core of SciPy). -- I notice they have: Matrix Operations, like: Elementwise Addition, etc. I don't, however, see any N-d array stuff: Too bad. (they do have "Multi-Dimensional Arrays Of Matrices" listed under present additional topics. Why array of matrices, rather than arrays of scalars? -- I'm personally not thrilled with the linear-algebra focus: for instance the matlab style for the operators: * matrix multiply .* elementwise addition. I'd rather see it the other way around. I wonder if we could get a Pythonista to join? In general, unfortunately, it looks to be quite commercially-focused: how does one get the open source community to be represented in this kind of thing? http://www.nmconsortium.org/client-area/registration/ -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From aisaac at american.edu Mon Jan 30 11:30:12 2006 From: aisaac at american.edu (Alan G Isaac) Date: Mon Jan 30 11:30:12 2006 Subject: [Numpy-discussion] Numerical Mathematics Consortium vs. scipy and numpy In-Reply-To: <43DE59CB.6070601@noaa.gov> References: <43DE59CB.6070601@noaa.gov> Message-ID: On Mon, 30 Jan 2006, Christopher Barker apparently wrote: > how does one get the open source community to be > represented in this kind of thing? http://www.nmconsortium.org/FldRte/?id=70&page=Downloads hth, Alan Isaac From pearu at scipy.org Mon Jan 30 13:22:01 2006 From: pearu at scipy.org (Pearu Peterson) Date: Mon Jan 30 13:22:01 2006 Subject: [Numpy-discussion] Verbose output from setup.py --help intentional? In-Reply-To: <1e2af89e0601300031k4790d3f5m96ddda611f4421e5@mail.gmail.com> References: <1e2af89e0601300031k4790d3f5m96ddda611f4421e5@mail.gmail.com> Message-ID: On Mon, 30 Jan 2006, Matthew Brett wrote: > Hi, > > Sorry if this is low priority, but I noticed that the output from > setup.py --help is more than 400 lines. Also the output of (e.g) > setup.py --help install and setup.py install --help differ (also more > than 400 lines). This issue is fixed in numpy SVN. Pearu From oliphant at ee.byu.edu Mon Jan 30 13:34:07 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Jan 30 13:34:07 2006 Subject: [Numpy-discussion] performance problem In-Reply-To: <20060129135902.4b1291c5.gerard.vermeulen@grenoble.cnrs.fr> References: <20060129135902.4b1291c5.gerard.vermeulen@grenoble.cnrs.fr> Message-ID: <43DE8635.2040901@ee.byu.edu> Gerard Vermeulen wrote: >0.9.5.2021 > > >>>>t1 = Timer('a <<= 8', 'import numarray as NX; a = NX.ones(10**6, NX.UInt32)') >>>>t2 = Timer('a <<= 8', 'import numpy as NX; a = NX.ones(10**6, NX.UInt32)') >>>>t1.timeit(100) >>>> >>>> >0.21813011169433594 > > >>>>t2.timeit(100) >>>> >>>> >1.1523458957672119 > > While ultimately, this slow-down was related to a coercion issue, I did still wonder about the extra dereference in the 1-d loop when one of the inputs is a scalar. So, I added a patch that checks for that case and defines a different loop. It seemed to give a small performance boost on my system. I'm wondering if such special-case coding is wise in general. Are there other ways to get C-compilers to produce faster code on modern machines? -Travis From Chris.Barker at noaa.gov Mon Jan 30 13:39:10 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon Jan 30 13:39:10 2006 Subject: [Numpy-discussion] Numerical Mathematics Consortium vs. scipy and numpy In-Reply-To: References: <43DE59CB.6070601@noaa.gov> Message-ID: <43DE8747.40705@noaa.gov> Alan G Isaac wrote: > On Mon, 30 Jan 2006, Christopher Barker apparently wrote: >> how does one get the open source community to be >> represented in this kind of thing? > > http://www.nmconsortium.org/FldRte/?id=70&page=Downloads Were you pointing us to any particular document there? Full membership (that is, with voting rights) requires a $2500.00 a year membership fee. I think it'll be a bit tricky for an open source project to raise that kind of cash. The result is that you have to buy your way into the group, which favors commercial entities. Granted, they have operating expenses, and $2500 isn't all that much, but it's probably too much for open-source groups to get representation. An associate membership is $250./year, which is far more manageable, but then you don't get voting rights: """ The Associate Membership with no voting rights will have full access to the consortium?s output and be engaged in some of the consortium?s activities and is typically for the user community, both academia and industry ($200 annual fee). """ The SciPy community is not a "user" group, it's a developer group. there are other open source projects worth of representation, such as the GNU Scientific Library. Oh well. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From gerard.vermeulen at grenoble.cnrs.fr Mon Jan 30 13:52:07 2006 From: gerard.vermeulen at grenoble.cnrs.fr (Gerard Vermeulen) Date: Mon Jan 30 13:52:07 2006 Subject: [Numpy-discussion] performance problem In-Reply-To: <43DDD3E5.80902@ieee.org> References: <20060129135902.4b1291c5.gerard.vermeulen@grenoble.cnrs.fr> <43DDD3E5.80902@ieee.org> Message-ID: <20060130225144.201784de.gerard.vermeulen@grenoble.cnrs.fr> On Mon, 30 Jan 2006 01:52:53 -0700 Travis Oliphant wrote: > Gerard Vermeulen wrote: > > >coercion issue snipped > > > > > > In current SVN, I think improved on the current state by only calling a > scalar a signed integer if it is actually negative (previously only it's > data-type was checked and all Python integers get converted to > PyArray_LONG data-types which are signed integers). > > This fixes the immediate problem, I think. > > What are opinions on this scalar-coercion rule? > Hmm, this is a consequence of your rule: >>> from numpy import *; core.__version__ '0.9.5.2024' >>> a = arange(3, dtype=uint32) >>> a-3 array([4294967293, 4294967294, 4294967295], dtype=uint32) >>> a+-3 array([-3, -2, -1], dtype=int64) >>> (a-3) == (a+-3) array([False, False, False], dtype=bool) Do you think that the speed-up justifies this? I don't think so. All my performance issues are discovered while writing demo examples which transfer data between a Python wrapped C++-library and Numeric, numarray, or numpy. In that state of mind, it surprises me when an uint32 ndarray gets coerced to an int64 ndarray. I rather prefer numarray's approach which raises an overflow error for the >>> a+-3 above. Agreed that sometimes a silent coercion is a good thing, but somebody who has started with an ubyte ndarray is likely to want an ubyte array in the end. I don't want to start a flame war, happy to write a - uint32(3) for numpy specific code. Gerard From oliphant.travis at ieee.org Mon Jan 30 13:53:02 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Jan 30 13:53:02 2006 Subject: [Numpy-discussion] Re: Status of 1.0 release and website? In-Reply-To: <463e11f90601231251n14d49c8dq4b919fb45fcf4150@mail.gmail.com> References: <463e11f90601231251n14d49c8dq4b919fb45fcf4150@mail.gmail.com> Message-ID: <43DE8A9B.2000101@ieee.org> Jonathan Taylor wrote: > I am wondering what needs to happen before a 1.0 release of both numpy > and scipy? For a 1.0 release of numpy: In terms of features. The only thing I see as necessary is that array scalars have their own math defined so that they don't go through the full ufunc machinery. This is probably about 20-40 hours of work. I would also like to see a numcompat module that allows numarray users to compile their C-extensions with a simple include file change. Also, a convert_numarraycode.py module would be useful for converting Python code. Mostly, though, I would like to see more people using NumPy in different circumstances so that we get the bugs worked out and needed C-API changes before going to 1.0. This is the process that may take several months as I see it. -Travis From Fernando.Perez at colorado.edu Mon Jan 30 13:57:03 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Mon Jan 30 13:57:03 2006 Subject: [Numpy-discussion] Re: Status of 1.0 release and website? In-Reply-To: <43DE8A9B.2000101@ieee.org> References: <463e11f90601231251n14d49c8dq4b919fb45fcf4150@mail.gmail.com> <43DE8A9B.2000101@ieee.org> Message-ID: <43DE8B71.8040809@colorado.edu> Travis Oliphant wrote: > Jonathan Taylor wrote: > >>I am wondering what needs to happen before a 1.0 release of both numpy >>and scipy? > > > For a 1.0 release of numpy: Small suggestion: I think this info would make an excellent 'timetable' page on our new wiki. It will give people a sense of where things stand, and what needs done before we go 1.0. I know people can always search the list, but something like this I think deserves more prominent mention. Cheers, f From oliphant at ee.byu.edu Mon Jan 30 13:59:02 2006 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Jan 30 13:59:02 2006 Subject: [Numpy-discussion] performance problem In-Reply-To: <20060130225144.201784de.gerard.vermeulen@grenoble.cnrs.fr> References: <20060129135902.4b1291c5.gerard.vermeulen@grenoble.cnrs.fr> <43DDD3E5.80902@ieee.org> <20060130225144.201784de.gerard.vermeulen@grenoble.cnrs.fr> Message-ID: <43DE8C02.2040000@ee.byu.edu> Gerard Vermeulen wrote: >>In current SVN, I think improved on the current state by only calling a >>scalar a signed integer if it is actually negative (previously only it's >>data-type was checked and all Python integers get converted to >>PyArray_LONG data-types which are signed integers). >> >>This fixes the immediate problem, I think. >> >>What are opinions on this scalar-coercion rule? >> >> >> > >Hmm, this is a consequence of your rule: > > > >>>>from numpy import *; core.__version__ >>>> >>>> >'0.9.5.2024' > > >>>>a = arange(3, dtype=uint32) >>>>a-3 >>>> >>>> >array([4294967293, 4294967294, 4294967295], dtype=uint32) > > >>>>a+-3 >>>> >>>> >array([-3, -2, -1], dtype=int64) > > >>>>(a-3) == (a+-3) >>>> >>>> >array([False, False, False], dtype=bool) > >Do you think that the speed-up justifies this? I don't think so. > > > It's still hard to say if it justifies it or not. One way of writing a-3 causes automatic upcasting while the other way doesn't. This might be a good thing, depending on your point of view. I could see people needing both situations. These things are never as clear as we'd like them to be. But, I could also accept a rule that treated *all* integers as the same kind in which case a-3 and a+(-3) would always return the same thing. I'm fine with it either way. So what are other opinions? -Travis From ndarray at mac.com Mon Jan 30 14:39:02 2006 From: ndarray at mac.com (Sasha) Date: Mon Jan 30 14:39:02 2006 Subject: [Numpy-discussion] performance problem In-Reply-To: <43DE8635.2040901@ee.byu.edu> References: <20060129135902.4b1291c5.gerard.vermeulen@grenoble.cnrs.fr> <43DE8635.2040901@ee.byu.edu> Message-ID: On 1/30/06, Travis Oliphant wrote: > Are there other ways to > get C-compilers to produce faster code on modern machines? I would recommend to take a look at if you have not seen it before. Although wtitten by AMD, many recommendations apply to most modern CPUs. I've found Chapter 3 particulary informative. In fact I've changed my coding habits after reading some of their recommendations (for example "Use Array-Style Instead of Pointer-Style Code"). -- sasha From jh at oobleck.astro.cornell.edu Mon Jan 30 14:59:03 2006 From: jh at oobleck.astro.cornell.edu (Joe Harrington) Date: Mon Jan 30 14:59:03 2006 Subject: [Numpy-discussion] Numerical Mathematics Consortium vs. scipy and numpy In-Reply-To: <20060130220002.44B8B1368F@sc8-sf-spam2.sourceforge.net> (numpy-discussion-request@lists.sourceforge.net) References: <20060130220002.44B8B1368F@sc8-sf-spam2.sourceforge.net> Message-ID: <200601302258.k0UMw3PW006700@oobleck.astro.cornell.edu> Currently the NMC membership is tiny. Likely the linalg focus is due to the biases of that group. Getting an open-source project involved in something like this isn't so terrible, if the vendors allow it (i.e., if it isn't a sham for one set of vendors to gang up on another, as the Open Software Foundation was about 15 years ago). I notice that Research Systems (IDL) conspicuously isn't on their list (they'd definitely be more array-oriented). Also conspicuous by their absences are prominent groups like NAG and netlib. As for coming up with the fee, I think that it wouldn't be a huge investment for STScI and Enthought to split. If successful, NMC will impact both organizations quite a bit, so it should be worth it to them. Of course, I don't work for either organization so that's easy for me to say. Note that these people expect their members to do a lot of work, so the greater commitment might be time. However, the group is conveniently located in Austin... Alternatively, we could petition for a waiver based on our lack of profit. That's been done successfully in the past, too, for example by the Gutenprint folks (who then proceeded to teach the developers at Lexmark et al. a thing or two about dithering). XFree86.org had to beat on the X Consortium (or whatever they were called then) a bit, but eventually got in. I'm not sure what the deal was with fees for them. --jh-- From aisaac at american.edu Mon Jan 30 15:36:07 2006 From: aisaac at american.edu (Alan G Isaac) Date: Mon Jan 30 15:36:07 2006 Subject: [Numpy-discussion] Numerical Mathematics Consortium vs. scipy and numpy In-Reply-To: <43DE8747.40705@noaa.gov> References: <43DE59CB.6070601@noaa.gov><43DE8747.40705@noaa.gov> Message-ID: On Mon, 30 Jan 2006, Christopher Barker apparently wrote: > The SciPy community is not a "user" group, it's > a developer group. there are other open source projects > worth of representation, such as the GNU Scientific > Library. It would not hurt to ask for a waiver for open source developers. Whichever way it goes might contain a lot of information. Cheers, Alan From Chris.Barker at noaa.gov Mon Jan 30 16:48:02 2006 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon Jan 30 16:48:02 2006 Subject: [Numpy-discussion] performance problem In-Reply-To: References: <20060129135902.4b1291c5.gerard.vermeulen@grenoble.cnrs.fr> <43DE8635.2040901@ee.byu.edu> Message-ID: <43DEB3A3.2090608@noaa.gov> Sasha wrote: > I would recommend to take a look at > Nice reference, thanks. From that: """ Copy Frequently Dereferenced Pointer Arguments to Local Variables: Avoid frequently dereferencing pointer arguments inside a function. Since the compiler has no knowledge of whether aliasing exists between the pointers, such dereferencing cannot be optimized away by the compiler. This prevents data from being kept in registers and significantly increases memory traffic. Note that many compilers have an ?assume no aliasing? optimization switch. This allows the compiler to assume that two different pointers always have disjoint contents and does not require copying of pointer arguments to local variables. Otherwise, copy the data pointed to by the pointer arguments to local variables at the start of the function and if necessary copy them back at the end of the function. """ Which perhaps helps answer Travis' original question. Did it make much difference in this case, Travis? -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From oliphant.travis at ieee.org Mon Jan 30 19:29:04 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Jan 30 19:29:04 2006 Subject: [Numpy-discussion] performance problem In-Reply-To: <43DEB3A3.2090608@noaa.gov> References: <20060129135902.4b1291c5.gerard.vermeulen@grenoble.cnrs.fr> <43DE8635.2040901@ee.byu.edu> <43DEB3A3.2090608@noaa.gov> Message-ID: <43DED96D.5080509@ieee.org> Christopher Barker wrote: > > Which perhaps helps answer Travis' original question. > > Did it make much difference in this case, Travis? Some difference in that case. For 10**6 elements, the relevant loops went from about 34 msec/loop (using the timeit module) to about 31 msec/loop. For a savings of about 3 msec/loop on my AMD platform. -Travis From vel.accel at gmail.com Tue Jan 31 01:47:07 2006 From: vel.accel at gmail.com (dHering) Date: Tue Jan 31 01:47:07 2006 Subject: [Numpy-discussion] Casting RecArray fields? Message-ID: <1e52e0880601310146t60454e29ue70971137b4daf1e@mail.gmail.com> Hi all, I need to cast fields of type 'Bool' to type 'Int32'. I'm not sure how to do this. Thank you, Dieter import numarray.records as rec b = rec.array([[23, 0],[45, 1]], names=' integer, boolean',formats='Int32, Bool') print b.field(1).type() # following line is not permissable # b.field(1) = b.field(1).astype('Int32') b.field(1)[:] = b.field(1).astype('Int32') print b.field(1).type() ########################## ########################## /usr/bin/python -u Bool Bool -------------- next part -------------- An HTML attachment was scrubbed... URL: From lito.kusnadi at gmail.com Tue Jan 31 02:33:02 2006 From: lito.kusnadi at gmail.com (Lito Kusnadi) Date: Tue Jan 31 02:33:02 2006 Subject: [Numpy-discussion] NumPy installed but aborted when do import Message-ID: Hi, I'm running RHEL 4.0 2.6.9-5.EL with Python 2.3.4. I managed to compile NumPy from src.rpm and make an i386 rpm from it. Version of NumPy is numpy-0.9.4-1. When I run Python interactive line and try to import NumPy, it aborts and kicks me to the command prompt. The following error is generated: Python 2.3.4 (#1, Nov 4 2004, 14:06:56) [GCC 3.4.2 20041017 (Red Hat 3.4.2-6.fc3)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from numpy import * import core -> failed: /usr/lib/python2.3/site-packages/numpy/core/multiarray.so: undefined symbol: PyOS_ascii_strtod import random -> failed: 'module' object has no attribute 'dtype' import lib -> failed: /usr/lib/python2.3/site-packages/numpy/core/multiarray.so: undefined symbol: PyOS_ascii_strtod Fatal Python error: can't initialize module lapack_lite Aborted Anyone knows what package I am missing? Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From faltet at carabos.com Tue Jan 31 02:46:01 2006 From: faltet at carabos.com (Francesc Altet) Date: Tue Jan 31 02:46:01 2006 Subject: [Numpy-discussion] Casting RecArray fields? In-Reply-To: <1e52e0880601310146t60454e29ue70971137b4daf1e@mail.gmail.com> References: <1e52e0880601310146t60454e29ue70971137b4daf1e@mail.gmail.com> Message-ID: <200601311144.45324.faltet@carabos.com> A Dimarts 31 Gener 2006 10:46, dHering va escriure: > Hi all, > > I need to cast fields of type 'Bool' to type 'Int32'. > I'm not sure how to do this. > > Thank you, > Dieter > > > import numarray.records as rec > > b = rec.array([[23, 0],[45, 1]], names=' integer, boolean',formats='Int32, > Bool') > > print b.field(1).type() > # following line is not permissable > # b.field(1) = b.field(1).astype('Int32') > b.field(1)[:] = b.field(1).astype('Int32') > print b.field(1).type() You can't do this because this implies a change on the column type definition (and also in the type size) of the recarray. One way that works for me, but that implies a complete data copy, is: In [15]: c = rec.array([b.field(0),b.field(1)], formats='Int32,Int32') In [16]: c.field(1) Out[16]: array([0, 1]) HTH, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From gerard.vermeulen at grenoble.cnrs.fr Tue Jan 31 02:53:04 2006 From: gerard.vermeulen at grenoble.cnrs.fr (Gerard Vermeulen) Date: Tue Jan 31 02:53:04 2006 Subject: [Numpy-discussion] NumPy installed but aborted when do import In-Reply-To: References: Message-ID: <20060131115245.3d538224.gerard.vermeulen@grenoble.cnrs.fr> On Tue, 31 Jan 2006 21:32:14 +1100 Lito Kusnadi wrote: > Hi, I'm running RHEL 4.0 2.6.9-5.EL with Python 2.3.4. > I managed to compile NumPy from src.rpm and make an i386 rpm from it. > Version of NumPy is numpy-0.9.4-1. > When I run Python interactive line and try to import NumPy, it aborts and > kicks me to the command prompt. > The following error is generated: > > Python 2.3.4 (#1, Nov 4 2004, 14:06:56) > [GCC 3.4.2 20041017 (Red Hat 3.4.2-6.fc3)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> from numpy import * > import core -> failed: > /usr/lib/python2.3/site-packages/numpy/core/multiarray.so: undefined symbol: > PyOS_ascii_strtod > import random -> failed: 'module' object has no attribute 'dtype' > import lib -> failed: > /usr/lib/python2.3/site-packages/numpy/core/multiarray.so: undefined symbol: > PyOS_ascii_strtod > Fatal Python error: can't initialize module lapack_lite > Aborted > > Anyone knows what package I am missing? > There is a patch for Python-2.3 (PyOS_ascii_strtod does not exist in Python-2.3): http://sourceforge.net/project/showfiles.php?group_id=1369&package_id=175103 Look for: patch-for-py2.3 Gerard From vel.accel at gmail.com Tue Jan 31 02:56:00 2006 From: vel.accel at gmail.com (dHering) Date: Tue Jan 31 02:56:00 2006 Subject: [Numpy-discussion] Casting RecArray fields? In-Reply-To: <200601311144.45324.faltet@carabos.com> References: <1e52e0880601310146t60454e29ue70971137b4daf1e@mail.gmail.com> <200601311144.45324.faltet@carabos.com> Message-ID: <1e52e0880601310255g6a3eba66tedc303e5fefb4619@mail.gmail.com> On 1/31/06, Francesc Altet wrote: > > A Dimarts 31 Gener 2006 10:46, dHering va escriure: > > Hi all, > > > > I need to cast fields of type 'Bool' to type 'Int32'. > > I'm not sure how to do this. > > > > Thank you, > > Dieter > > > > > > import numarray.records as rec > > > > b = rec.array([[23, 0],[45, 1]], names=' integer, > boolean',formats='Int32, > > Bool') > > > > print b.field(1).type() > > # following line is not permissable > > # b.field(1) = b.field(1).astype('Int32') > > b.field(1)[:] = b.field(1).astype('Int32') > > print b.field(1).type() > > You can't do this because this implies a change on the column type > definition (and also in the type size) of the recarray. > > One way that works for me, but that implies a complete data copy, is: > > In [15]: c = rec.array([b.field(0),b.field(1)], formats='Int32,Int32') > > In [16]: c.field(1) > Out[16]: array([0, 1]) > > HTH, > > -- > >0,0< Francesc Altet http://www.carabos.com/ > V V C?rabos Coop. V. Enjoy Data > "-" Yes it helps very much Francesc. Thank you. Dieter -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris at trichech.us Tue Jan 31 06:48:57 2006 From: chris at trichech.us (Christopher Fonnesbeck) Date: Tue Jan 31 06:48:57 2006 Subject: [Numpy-discussion] numpy 0.9.4 for OSX Message-ID: <04A22C43-8960-4A1F-8E2C-97E2E3A1C2FD@trichech.us> I have added a OS X 10.4 build of numpy 0.9.4 to the sourceforge site. For (approximately) bi-weekly builds of numpy and scipy for OS X, go to http://trichech.us. Chris -- Christopher J. Fonnesbeck Population Ecologist, Marine Mammal Section Fish & Wildlife Research Institute (FWC) St. Petersburg, FL Adjunct Assistant Professor Warnell School of Forest Resources University of Georgia Athens, GA T: 727.235.5570 E: chris at trichech.us -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2417 bytes Desc: not available URL: From oliphant.travis at ieee.org Tue Jan 31 06:56:10 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue Jan 31 06:56:10 2006 Subject: [Numpy-discussion] Re: [SciPy-user] Strange behaviour of dot function In-Reply-To: <19508.1138639848@www036.gmx.net> References: <19508.1138639848@www036.gmx.net> Message-ID: <43DF7A55.3040602@ieee.org> Johannes L?hnert wrote: >Hi, > >I just found out that the dot function which multiplies matrices gives >strange results for a 3-dimensional array. Consider the following example: > > You just found two bugs in numpy.dot one of which is also there in Numeric. I just committed a fix to both bugs by using the ever-useful N-d array iterator (it sure makes it easier to write algorithms for strided arrays...). All three of your tests now produce the same answer. Thank you for finding this problem. -Travis From matthew.brett at gmail.com Tue Jan 31 09:01:02 2006 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue Jan 31 09:01:02 2006 Subject: [Numpy-discussion] gfortran, numpy build Message-ID: <1e2af89e0601310859y5b291420i6767e4efbb161a46@mail.gmail.com> Hi, I have the latest version of Mandrake, which has no g77, only gfortran - and needed the following patch to compile (with some added g77, gfortran outputs): Index: numpy/distutils/fcompiler/gnu.py =================================================================== --- numpy/distutils/fcompiler/gnu.py (revision 2033) +++ numpy/distutils/fcompiler/gnu.py (working copy) @@ -20,8 +20,13 @@ # GNU Fortran (GCC) 3.3.3 (Debian 20040401) # GNU Fortran 0.5.25 20010319 (prerelease) # Redhat: GNU Fortran (GCC 3.2.2 20030222 (Red Hat Linux 3.2.2-5)) 3.2.2 20030222 (Red Hat Linux 3.2.2-5) + # Mandrake: GNU Fortran (GCC) 3.3.6 (Mandrakelinux release 3.3.6-2mdk) + # + # 'gfortran --version' results + # Redhat: GNU Fortran 95 (GCC 4.0.2 20051125 (Red Hat 4.0.2-8)) + # Mandrake: GNU Fortran 95 (GCC 4.0.1 (4.0.1-5mdk for Mandriva Linux release 2006.0)) - for fc_exe in map(find_executable,['g77','f77']): + for fc_exe in map(find_executable,['g77','f77','gfortran']): if os.path.isfile(fc_exe): break executables = { @@ -41,7 +46,10 @@ if os.name != 'nt' and sys.platform!='cygwin': pic_flags = ['-fPIC'] - g2c = 'g2c' + if os.path.basename(fc_exe) == 'gfortran': + g2c = 'gfortran' + else: + g2c = 'g2c' #def get_linker_so(self): # # win32 linking should be handled by standard linker Can I suggest this a patch to svn numpy? Thanks, Matthew -------------- next part -------------- A non-text attachment was scrubbed... Name: gfortran.patch Type: text/x-diff Size: 1259 bytes Desc: not available URL: From faltet at carabos.com Tue Jan 31 09:17:07 2006 From: faltet at carabos.com (Francesc Altet) Date: Tue Jan 31 09:17:07 2006 Subject: [Numpy-discussion] A couple of errors in numpy/records.py Message-ID: <200601311816.25946.faltet@carabos.com> Hi, I've detected a couple of issues in records.py. For the first one, the next should fix it: Index: numpy/core/records.py =================================================================== --- numpy/core/records.py (revision 2033) +++ numpy/core/records.py (working copy) @@ -410,7 +410,8 @@ offset=offset) elif isinstance(obj, sb.ndarray): res = obj.view(recarray) - if issubclass(res.dtype, nt.void): + if issubclass(res.dtype.type, nt.void): res.dtype = sb.dtype((record, res.dtype)) + return res else: raise ValueError("Unknown input type") Now, another curious thing (sorry, no patch this time): In [6]: for i in numpy.rec.array([(1,2)], formats='u4,u4'): print i ...: (1L, 2L) [No problem with this] In [7]: for i in numpy.rec.array([(1,2)], formats='u4,u4')[0]: print i ...: --------------------------------------------------------------------------- exceptions.KeyError Traceback (most recent call last) /home/faltet/computacio.nobackup/hdf5-1.7.51/test/ /usr/lib/python2.3/site-packages/numpy/core/records.py in __getitem__(self, obj) 126 127 def __getitem__(self, obj): --> 128 return self.getfield(*(self.dtype.fields[obj][:2])) 129 130 def __setitem__(self, obj, val): KeyError: 0 I'd expect something like 1L 2L Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From oliphant.travis at ieee.org Tue Jan 31 11:36:23 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue Jan 31 11:36:23 2006 Subject: [Numpy-discussion] A couple of errors in numpy/records.py In-Reply-To: <200601311816.25946.faltet@carabos.com> References: <200601311816.25946.faltet@carabos.com> Message-ID: <43DFBC0B.7070007@ieee.org> Francesc Altet wrote: >Hi, > >I've detected a couple of issues in records.py. For the first one, the >next should fix it: > > Thanks for the patch. Also thank you for catching that we weren't allowing field access on the record objects. This is now fixed (by simply getting rid of the specialized getitem call and leeting the base-class take care of it). -Travis From bsouthey at gmail.com Tue Jan 31 12:05:05 2006 From: bsouthey at gmail.com (Bruce Southey) Date: Tue Jan 31 12:05:05 2006 Subject: [Numpy-discussion] gfortran, numpy build In-Reply-To: <1e2af89e0601310859y5b291420i6767e4efbb161a46@mail.gmail.com> References: <1e2af89e0601310859y5b291420i6767e4efbb161a46@mail.gmail.com> Message-ID: Hi, Perhaps it would also make sense to add g95 (http://g95.sourceforge.net/ or g95.org) which is different than gfortran ( http://gcc.gnu.org/wiki/TheOtherGCCBasedFortranCompiler). Regards Bruce On 1/31/06, Matthew Brett wrote: > > Hi, > > I have the latest version of Mandrake, which has no g77, only gfortran > - and needed the following patch to compile (with some added g77, > gfortran outputs): > > Index: numpy/distutils/fcompiler/gnu.py > =================================================================== > --- numpy/distutils/fcompiler/gnu.py (revision 2033) > +++ numpy/distutils/fcompiler/gnu.py (working copy) > @@ -20,8 +20,13 @@ > # GNU Fortran (GCC) 3.3.3 (Debian 20040401) > # GNU Fortran 0.5.25 20010319 (prerelease) > # Redhat: GNU Fortran (GCC 3.2.2 20030222 (Red Hat Linux > 3.2.2-5)) 3.2.2 20030222 (Red Hat Linux 3.2.2-5) > + # Mandrake: GNU Fortran (GCC) 3.3.6 (Mandrakelinux release 3.3.6-2mdk > ) > + # > + # 'gfortran --version' results > + # Redhat: GNU Fortran 95 (GCC 4.0.2 20051125 (Red Hat 4.0.2-8)) > + # Mandrake: GNU Fortran 95 (GCC 4.0.1 (4.0.1-5mdk for Mandriva > Linux release 2006.0)) > > - for fc_exe in map(find_executable,['g77','f77']): > + for fc_exe in map(find_executable,['g77','f77','gfortran']): > if os.path.isfile(fc_exe): > break > executables = { > @@ -41,7 +46,10 @@ > if os.name != 'nt' and sys.platform!='cygwin': > pic_flags = ['-fPIC'] > > - g2c = 'g2c' > + if os.path.basename(fc_exe) == 'gfortran': > + g2c = 'gfortran' > + else: > + g2c = 'g2c' > > #def get_linker_so(self): > # # win32 linking should be handled by standard linker > > Can I suggest this a patch to svn numpy? > > Thanks, > > Matthew > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pearu at scipy.org Tue Jan 31 12:21:02 2006 From: pearu at scipy.org (Pearu Peterson) Date: Tue Jan 31 12:21:02 2006 Subject: [Numpy-discussion] gfortran, numpy build In-Reply-To: References: <1e2af89e0601310859y5b291420i6767e4efbb161a46@mail.gmail.com> Message-ID: Hi, numpy.distutils has already support for both g95 and gfortran compilers. Run python setup.py config_fc --help-fcompiler to see a list of supported Fortran compilers. Pearu On Tue, 31 Jan 2006, Bruce Southey wrote: > Hi, > Perhaps it would also make sense to add g95 (http://g95.sourceforge.net/ or > g95.org) which is different than gfortran ( > http://gcc.gnu.org/wiki/TheOtherGCCBasedFortranCompiler). > > Regards > Bruce > > On 1/31/06, Matthew Brett wrote: >> >> Hi, >> >> I have the latest version of Mandrake, which has no g77, only gfortran >> - and needed the following patch to compile (with some added g77, >> gfortran outputs): >> >> Index: numpy/distutils/fcompiler/gnu.py >> =================================================================== >> --- numpy/distutils/fcompiler/gnu.py (revision 2033) >> +++ numpy/distutils/fcompiler/gnu.py (working copy) >> @@ -20,8 +20,13 @@ >> # GNU Fortran (GCC) 3.3.3 (Debian 20040401) >> # GNU Fortran 0.5.25 20010319 (prerelease) >> # Redhat: GNU Fortran (GCC 3.2.2 20030222 (Red Hat Linux >> 3.2.2-5)) 3.2.2 20030222 (Red Hat Linux 3.2.2-5) >> + # Mandrake: GNU Fortran (GCC) 3.3.6 (Mandrakelinux release 3.3.6-2mdk >> ) >> + # >> + # 'gfortran --version' results >> + # Redhat: GNU Fortran 95 (GCC 4.0.2 20051125 (Red Hat 4.0.2-8)) >> + # Mandrake: GNU Fortran 95 (GCC 4.0.1 (4.0.1-5mdk for Mandriva >> Linux release 2006.0)) >> >> - for fc_exe in map(find_executable,['g77','f77']): >> + for fc_exe in map(find_executable,['g77','f77','gfortran']): >> if os.path.isfile(fc_exe): >> break >> executables = { >> @@ -41,7 +46,10 @@ >> if os.name != 'nt' and sys.platform!='cygwin': >> pic_flags = ['-fPIC'] >> >> - g2c = 'g2c' >> + if os.path.basename(fc_exe) == 'gfortran': >> + g2c = 'gfortran' >> + else: >> + g2c = 'g2c' >> >> #def get_linker_so(self): >> # # win32 linking should be handled by standard linker >> >> Can I suggest this a patch to svn numpy? >> >> Thanks, >> >> Matthew >> >> >> > From oliphant.travis at ieee.org Tue Jan 31 12:34:06 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue Jan 31 12:34:06 2006 Subject: [Numpy-discussion] Re: [Numpy-user] possible error with isarrtype In-Reply-To: References: Message-ID: <43DFC973.70902@ieee.org> O'Keefe, Michael wrote: >I was just updating my code to work with python 2.4.2 and numpy 0.9.4 and observed the following behavior with the function isarrtype. Just thought I would report it (see below). > > > Thanks for this report. I've fixed the problem, but you should probably not be using these functions anymore. These were put in place before the data-type objects materialized. In the past, what is now the typeobject of the array scalars was thought to be the basic way to determine the data-type of the array. Now, the data-type object itself is the preferred way. I need to figure out what to do with these functions now and see if they even have any use anymore. What use were you making of them? Best, -Travis From Michael_OKeefe at nrel.gov Tue Jan 31 13:07:14 2006 From: Michael_OKeefe at nrel.gov (O'Keefe, Michael) Date: Tue Jan 31 13:07:14 2006 Subject: [Numpy-discussion] RE: [Numpy-user] possible error with isarrtype Message-ID: > Now, the data-type object itself is the preferred way. I need to figure > out what to do with these functions now and see if they even have any > use anymore. Thanks Travis. With regard to what to do with the functions, I only ended up trying them because they were there. From my point of view, I'd love to learn how to code the "right" way. To that end, any means of informing the user of "deprecated" functions (or just taking them out) would be fine with me. However, I realize some may have legacy code and so I would defer to their opinions. > What use were you making of them? I was just migrating some working scipy/numeric code I'd been using to the newer numpy (0.9.4), python (2.4.2), and scipy (0.4.4). When I ran my unit-tests, one line that gave me problems was: import scipy as sp ... assert( type(deltaTime_sec)==float ) This line of code was there to inform me if I was accidentally passing in an array type instead of a scalar. This no longer worked in the new environment. The type of deltaTime_sec came out as float64_arrtype which did not equal type float causing the assertion to trip which I believe to be caused by the following: >>> a=sp.array([1.],float) >>> type(a) >>> type(a[0]) # somewhere I assign deltaTime_sec = someArray[0] I ended up fixing it with the following code which seems to do the trick (though I'm always keen to learn better ways of doing things): assert( sp.size(deltaTime_sec)==1 ) Thanks! Michael From oliphant.travis at ieee.org Tue Jan 31 13:23:03 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue Jan 31 13:23:03 2006 Subject: [Numpy-discussion] Re: [Numpy-user] possible error with isarrtype In-Reply-To: References: Message-ID: <43DFD50D.1000200@ieee.org> O'Keefe, Michael wrote: >I was just migrating some working scipy/numeric code I'd been using to the newer numpy (0.9.4), python (2.4.2), and scipy (0.4.4). > >When I ran my unit-tests, one line that gave me problems was: > >import scipy as sp >... >assert( type(deltaTime_sec)==float ) > >This line of code was there to inform me if I was accidentally passing in an array type instead of a scalar. This no longer worked in the new environment. The type of deltaTime_sec came out as float64_arrtype which did not equal type float causing the assertion to trip which I believe to be caused by the following: > > Ah... This is something that needs to be added to the FAQ, because it is common. Basically, you should almost never test for type equality like this since with >Python2.2 the Python float can be inherited from (that's what a float64scalar does --- used to be called float64_arrtype). So, instead you should use assert (isinstance(deltaTime_sec, float)) which would still work with the new float scalar objects. The big advantage of the new scalar types and objects is that there is a 1-1 relationship between the scalar types and the basic array data-types and the new scalars have all the methods and attributes of arrays. So, in fact you could write: assert(deltaTime_sec.size == 1) and this would work whether deltaTime is a scalar or an array with only one element. -Travis From Fernando.Perez at colorado.edu Tue Jan 31 13:25:15 2006 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Tue Jan 31 13:25:15 2006 Subject: [Numpy-discussion] RE: [Numpy-user] possible error with isarrtype In-Reply-To: References: Message-ID: <43DFD598.5000503@colorado.edu> O'Keefe, Michael wrote: >> Now, the data-type object itself is the preferred way. I need to figure >> out what to do with these functions now and see if they even have any >> use anymore. > > > Thanks Travis. > > With regard to what to do with the functions, I only ended up trying them > because they were there. From my point of view, I'd love to learn how to > code the "right" way. To that end, any means of informing the user of > "deprecated" functions (or just taking them out) would be fine with me. > However, I realize some may have legacy code and so I would defer to their > opinions. Python has a proper framework for deprecation warnings. I'd say we use that, and Travis can ruthlessly deprecate anything that's just compatibility noise. Perhaps 1.0 can keep old but deprecated things around (numpy will see a LOT of use once 1.0 comes out, far more than the current svn testing crowd). Then, as of 1.1 or 1.2 or whatever, those can be removed. Cheers, f From cookedm at physics.mcmaster.ca Tue Jan 31 15:42:01 2006 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Tue Jan 31 15:42:01 2006 Subject: [Numpy-discussion] bdist_egg trips up over non-extension extensions In-Reply-To: References: <43D6529C.8030105@gmail.com> <43D68CC4.1010004@gmail.com> Message-ID: <90D73325-334E-4A59-8FB8-9C20904BA12C@physics.mcmaster.ca> On Jan 24, 2006, at 14:36 , Pearu Peterson wrote: > On Tue, 24 Jan 2006, Robert Kern wrote: > >> Pearu Peterson wrote: >>> Could you try recent version of numpy from SVN? __config__.py is now >>> generated via py_modules list. >> >> It doesn't build an egg, yet, because setuptools expects >> Distribution.py_modules >> to contain only strings, not tuples. Modifying setuptools to >> handle tuples in >> Distribution.py_modules seems to work. Is there any way you can do >> it without >> tuples? > > Source generators in numpy.distutils are extensions to distutils > and if setuptools assumes std distutils then source generators will > remain a problem for setuptools. However, if you call build_src > then all source generators will be resolved (including 3-tuples in > py_modules list) and the file lists in distribution instance should > not cause any problems for setuptools. > > So, would calling `setup.py build_src bdist_egg` be an acceptable > solution? Fixed in svn. I added a wrapper around the setuptools.command.egg_info.egg_info.run method that runs the build_src command first. -- |>|\/|< /------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From matthew.brett at gmail.com Tue Jan 31 16:11:13 2006 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue Jan 31 16:11:13 2006 Subject: [Numpy-discussion] gfortran, numpy build In-Reply-To: References: <1e2af89e0601310859y5b291420i6767e4efbb161a46@mail.gmail.com> Message-ID: <1e2af89e0601311602l4c1f363fpb11d1d405ababf7e@mail.gmail.com> > numpy.distutils has already support for both g95 and gfortran compilers. > Run > python setup.py config_fc --help-fcompiler > to see a list of supported Fortran compilers. Oops - sorry, I completely missed that - and thanks a lot for the pointer. For those as slow as I am, the build command seemed to be: python setup.py config --fcompiler=gnu95 build I am afraid this is a sign that I am lost in the build system. Is there a good place to start to get to grips with options like the above? And, is there a simple way to pass my favorite compiler optimization flags into the build? Thanks a lot, Matthew From cjw at sympatico.ca Tue Jan 31 19:53:02 2006 From: cjw at sympatico.ca (Colin J. Williams) Date: Tue Jan 31 19:53:02 2006 Subject: [Numpy-discussion] RE: [Numpy-user] possible error with isarrtype In-Reply-To: <43DFD598.5000503@colorado.edu> References: <43DFD598.5000503@colorado.edu> Message-ID: <43E0309D.5050700@sympatico.ca> Fernando Perez wrote: > O'Keefe, Michael wrote: >>> Now, the data-type object itself is the preferred way. I need to >>> figure >>> out what to do with these functions now and see if they even have >>> any use anymore. >> >> >> Thanks Travis. >> >> With regard to what to do with the functions, I only ended up trying >> them >> because they were there. From my point of view, I'd love to learn how to >> code the "right" way. To that end, any means of informing the user of >> "deprecated" functions (or just taking them out) would be fine with me. >> However, I realize some may have legacy code and so I would defer to >> their >> opinions. > > > Python has a proper framework for deprecation warnings. I'd say we > use that, and Travis can ruthlessly deprecate anything that's just > compatibility noise. > > Perhaps 1.0 can keep old but deprecated things around (numpy will see > a LOT of use once 1.0 comes out, far more than the current svn testing > crowd). Then, as of 1.1 or 1.2 or whatever, those can be removed. > > Cheers, > > f > One of the deprecated names is ArrayType. This seems to be closer to the Python style than ndarray. Colin W. From oliphant.travis at ieee.org Tue Jan 31 20:05:18 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Tue Jan 31 20:05:18 2006 Subject: [Numpy-discussion] RE: [Numpy-user] possible error with isarrtype In-Reply-To: <43E0309D.5050700@sympatico.ca> References: <43DFD598.5000503@colorado.edu> <43E0309D.5050700@sympatico.ca> Message-ID: <43E03349.10208@ieee.org> Colin J. Williams wrote: > One of the deprecated names is ArrayType. This seems to be closer to > the Python style than ndarray. Not really. Rather than test: type(var) == types.IntType you should be testing isinstance(var, int) just like rather than testing type(somearray) == ArrayType you should be testing isinstance(somearray, ndarray) Python style has changed a bit since 2.2 allowed sub-typing builtings -Travis From accession at mail.ru Tue Jan 31 21:45:01 2006 From: accession at mail.ru (=?windows-1251?B?wvvz9+jy/CDg7ePr6Onx6ujpIC0g7/Du8fLu?=) Date: Tue Jan 31 21:45:01 2006 Subject: [Numpy-discussion] =?windows-1251?B?wO3j6+jp8ero6SDt4CDz8O7i7eUg7e7x6PLl6/8g/+f76uAh?= Message-ID: <4adf01c626f1$41036721$d6d74eb6@mail.ru> ?? ?????? ??? ?? ???????????????? ??????????? ????? ??????????? ?? ???? ????????? ??????????? ??? ???????????? ? ??????? ??????????? ??????????? ????? ?? 3 ?????? ????????? ?????????? ??????????? ?????! ? ?????? ?????????? ????????? ????????? ???????? ????, ??????, ???????, ??: ? ?????????, ???? ???? ?? ??????, ???? ?? ???????????????? ?? ???????? ??????. ???? ????? ???????? ????? ? ?????????? ????????????? ?????, ???? ????????? ????????? ??????, ???????? ??????? ? ??????????? ??????????? ?? ???? ?????????? ?????. ?????????? ??? ????? ?? ?????? ?????? ???????? ??????, ??????? ?? ???????? ??????????? ????, ??? ???? ? ????, - ???????? ???????? ?? ????????? ?????????? ? ?????????? ?? ?????? ???????. ????? ????, ?? ????? ??????? ??????? ???? ????????? ????? ?? ?????? ???????? ?????. ???????????? ???????? ? ?? ???????????????? ???????? ?????????? ?????. ???????? ???????? ? ???? ?????????? ????????? ??????????? ????????????, ????????? ??????? ?????? ???????????? ????????? ? ??? ????? 3 ?????? ???????? ? ???????? ???????? ? ??????. ? ???????? ???????? ?? ?????????: ???????????? ???? ?? ???? ??????? ?????????? ?????? ??????, ? ??? ????? ????? ? ????????? ??????, ? ????? ???????????? ???????? ????????? ????? ???????????? ?????? ???????????? ???????????? ???????? ???? ?????? ??????? ??????????? ??????? ?????????? ?????????? ?????? ? ????????????????? ??????? ?????? ? ?????????? ???? ??????? ???????????? ?????? ???????????? ???????? ?????? ????????? ???????? ?????? ????????? ???????? ??????????? ?????????? ? ???????? ?????????? ????? ???? ???????? ???????? ???????? ????? ???????? - ??? ???????????????, ??????? ? ???, ?????? ????????? ? ??? ? ?? ?????? ????????? - ?????????????? ???????????? ???? ????????????? ??????? ???????? ??????????, ????????-????????, c??????? ??? ????? ? ???? ?????? ??????????? ???, ??????????? ? ????????????? ???????????? ?? ??????? ??????? ???????????????? ???????????? ? ????????? ???????????? ????????. ?????????? ?? ???????????????? ??????????? ?????? ????? ????? ? ??? ??????? ???????? ???????? ?? ????????? ??????? ????? ???????? ????????, ????????????? ???????????? Who's Who of Professionals, ?????????? ????? ????????????? ?????????? ???????? ???? ? ??. ???????? ???????. ? 9 ?? 23 ??????? 2005 ?. ????? ?????? ????? ????-????? ??????????? ????? ? ???????????????? ?????????? ??? ??????? ??????????? (?????? ??????????) ??????????? ??????????? ?????? - ??? ?????. ??????? ????? - ?? ?????? ?????????? ??????? ???? ???????? ??????????? ????? ?? CD ? DVD. ??? ????? ????? ?????????? ????? ?? 1-2 ???? ?????? ???? ? ??????? 2-3 ???????. ????-??????? ????? - 16 ??? 40 ????? ???????? ??????????? ??????????????? ??????? ???????? ? 4 ???? ???????????????? ?????????? + ?????? ??????? ???????? ??????? ???? ??? ??????????????? ???????. ??????? ?????? ? ??????? ???? ???????? ????? ??????? ???????? ?????????? ?????? ? ??????????. ????????????? ???????? - ????? ???????? ????? ???????? ? ?????? ??????? ??????? ?? ???????????? ?????????. ? ???? ????? ????????????? ??????? ????????? ??????? ?? ????? ???????. ??????? ? ??????????? ?????????? ????? ??????????? ??????? ? ?????? ?????? ????????????. ????? ????, ?? ???? ???? ????????? ? ?????????, ????????? ???????? ?? ??????????? ?????, ?? ??????? ????. ???? ?? ?????????????? ? ???????? ?????? ?????????, ?, ????????, ???-???????????, ??????????? ? ??? ??? ????????, ???????? ??????????? ???????? ????????? (???????? + ????? ?? ???????). ????????? ???: ??? ???? ??????: ???. +7 (926) 197-99-05 ??????? ?????: ???. +7(495) 291-72-64 (????? 18.00), +7 (916) 855-17-34 ????????: englishbest at netscape.net -------------- next part -------------- An HTML attachment was scrubbed... URL: