From hinsen at cnrs-orleans.fr Sat Nov 1 13:17:10 2003 From: hinsen at cnrs-orleans.fr (Konrad Hinsen) Date: Sat Nov 1 13:17:10 2003 Subject: [Numpy-discussion] gcc compile / link questions In-Reply-To: <3FA31683.10703@erols.com> References: <3FA31683.10703@erols.com> Message-ID: <200311012216.49289.hinsen@cnrs-orleans.fr> On Saturday 01 November 2003 03:12, Edward C. Jones wrote: > I compile and link Python extension modules using the script > > gcc -fPIC -g -I/usr/local/include/python2.3 \ > -Wall -Wstrict-prototypes -c mymodule.c > g++ -shared mymodule.o -L/usr/local/lib -o mymodule.so > > It works for me but it isn't pretty. Is there a better way to write it? Yes, the distutils module. It's part of the Python standard library and documented there. > Gcc finds all the libraries that need to be linked in. For example, > "/usr/local/lib/python2.3/site-packages/numarray/libnumarray.so". How > does gcc do this? It doesn't :-) And it doesn't have to. You are not creating a stand-alone executable, but a shared library for use in Python. Everything is linked together dynamically at runtime when Python imports all the modules. > I created a .so file "utilities.so" that contains some C functions that > are called in mymodule.c but are not visible from Python. Both > "utilities.c" and "mymodule.c" use numarray. What changes do I make in > the script above? Must I use the nasty "libnumarray_UNIQUE_SYMBOL" trick? I don't know enough about numarray to answer that question, but that is certainly one option. Alternatively you could include utilities.c into mymodule.c. > What is a script for creating "utilities.a" using gcc? How do I change > the script above to include "utilities.a"? To create the archive: gcc -c utilities.c ar r libutilities.a utilies.o You might have to add some libraries or library paths to the compilation step. To use the archive, just add -lutilities to the gcc command line, plus eventually the path with -L. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen at cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From nadavh at VisionSense.com Sun Nov 2 04:22:03 2003 From: nadavh at VisionSense.com (Nadav Horesh) Date: Sun Nov 2 04:22:03 2003 Subject: [Numpy-discussion] Problem in LinearAlgebra? In-Reply-To: <3FA2617A.4000308@hooft.net> References: <3FA2617A.4000308@hooft.net> Message-ID: <1067779275.2190.28.camel@Nadav.Envision.co.il> I tested the problem with: 1. Numeric 23.1 under python 2.3.2 2. numarray 0.8 (I made a copy of the Scientific package where all calls to Numeric were replaced to numarray), under python 2.3.2 There results where about the same -- high coefficients for the 5th order polynomials. I would expect reliable fit for a high order polynomials only under very special circumstances, so this is not a big surprise. My advice is: * Make sure that this is a bug and not a result of a numerical instability. If you can trace it down and point to a bug, then report it. The numarray package is very usable and is under a very active and rapid development, thus bugs are being fixed fast. * Look for a solution in the scipy package: It is generally better then Scientific. * Polynomials fit is relatively very simple --- you may write one of you own in less then a one day work. Since, as I said, the problem is, in many cases, unstable, you'll have the chance to implement more stable linear-equation solvers. Nadav. On Fri, 2003-10-31 at 15:19, Rob W.W. Hooft wrote: > I am using Polynomial.py from Scientific Python 2.1, together with > Numeric 17.1.2. This has always served me well, but now we are busy > upgrading our software, and I am currently porting some code to > Scientific Python 2.4.1, Numeric 22.0. Suddenly I do no longer manage to > get proper 2D polynomial fits over 4x4th order. At 5x5 the coefficients > that come back from LinearAlgebra.linear_least_squares have exploded. In > the old setup, I easily managed 9x9th order if I needed to, but most of > the time I'd stop at 6x6th order. Would anyone have any idea how this > difference can come about? I managed to work around this for the moment > by using the equivalent code in the fitPolynomial routine that uses > LinearAlgebra.generalized_inverse (and it doesn't even have problems > with the same data at 8x8), but this definitely feels not right! I can't > remember reading anything like this here before. > > Together with Konrad Hinsen, I came to the conclusion that the problem > is not in Scientific Python, so it must be the underlying LinearAlgebra > code that changed between releases 17 and 22. > > I hacked up a simplified example. Not sure whether it is the most simple > case, but this resembles what I have in my code, and I'm quite sure it > worked with Numeric 17.x, but currently it is horrible over order (4,4): > > -------------------------------------- > import Numeric > > def func(x,y): > return x+0.1*x**2+0.01*x**4+0.002*x**6+0.03*x*y+0.001*x**4*y**5 > > x=[] > y=[] > z=[] > for dx in Numeric.arange(0,1,0.01): > for dy in Numeric.arange(0,1,0.01): > x.append(dx) > y.append(dy) > z.append(func(dx,dy)) > > from Scientific.Functions import Polynomial > data=Numeric.transpose([x,y]) > z=Numeric.array(z) > for i in range(10): > print data[i],z[i] > pol=Polynomial.fitPolynomial((4,4),data,z) > print pol.coeff > ------------------------------------ > for 4,4 this prints: > [[ 1.84845529e-05 -7.60502772e-13 2.71314749e-12 -3.66731796e-12 > 1.66977148e-12] > [ 9.99422967e-01 3.00000000e-02 -3.26346097e-11 4.42406519e-11 > -2.01549767e-11] > [ 1.03899464e-01 -3.19668064e-11 1.14721790e-10 -1.55489826e-10 > 7.08425891e-11] > [ -9.40275000e-03 4.28456838e-11 -1.53705205e-10 2.08279772e-10 > -9.48840470e-11] > [ 1.80352695e-02 -1.10999843e-04 8.00662570e-04 -2.17266676e-03 > 2.47500004e-03]] > > for 5,5: > > [[ -2.25705839e+03 6.69051337e+02 -6.60470163e+03 6.66572425e+03 > -8.67897022e+02 1.83974866e+03] > [ -2.58646837e+02 -2.46554689e+03 1.15965805e+03 7.01089888e+03 > -2.11395436e+03 2.10884815e+03] > [ 3.93307499e+03 4.34484805e+02 -4.84080392e+03 5.90375330e+03 > 1.16798049e+03 -4.14163933e+03] > [ 1.62814750e+03 2.08717457e+03 1.15870693e+03 -3.37838057e+03 > 3.49821689e+03 5.80572585e+03] > [ 4.54127557e+02 -1.56645524e+03 4.58997025e+00 1.69772635e+03 > -1.37751039e+03 -7.59726558e+02] > [ 2.37878239e+03 9.43032094e+02 8.58518644e+02 -8.35846339e+03 > -5.55845668e+02 1.87502761e+03]] > > Which is clearly wrong. > > I appreciate any help! > > Regards, > > Rob From cjw at sympatico.ca Sun Nov 2 04:27:08 2003 From: cjw at sympatico.ca (Colin J. Williams) Date: Sun Nov 2 04:27:08 2003 Subject: [Numpy-discussion] LinearAlgebra2.determinant Message-ID: <3FA4F7EC.2030203@sympatico.ca> In the line: return num.zeros((1,), type=a.type())-num.multiply.reduce(num.diagonal(a)) the code: num.zeros((1,), type=a.type()) appears to be redundant. Colin W. From rob at hooft.net Sun Nov 2 06:51:23 2003 From: rob at hooft.net (Rob Hooft) Date: Sun Nov 2 06:51:23 2003 Subject: [Numpy-discussion] Problem in LinearAlgebra? In-Reply-To: <1067779275.2190.28.camel@Nadav.Envision.co.il> References: <3FA2617A.4000308@hooft.net> <1067779275.2190.28.camel@Nadav.Envision.co.il> Message-ID: <3FA519DC.1040307@hooft.net> Nadav Horesh wrote: > I tested the problem with: > 1. Numeric 23.1 under python 2.3.2 > 2. numarray 0.8 (I made a copy of the Scientific package where all > calls to Numeric were replaced to numarray), under python 2.3.2 > There results where about the same -- high coefficients for the 5th > order polynomials. > I would expect reliable fit for a high order polynomials only under very > special circumstances, so this is not a big surprise. Thanks for your efforts. The polynomial we're trying to fit here is not extremely unstable. As I said, with Numeric 17.1.2 my class of problems used to be stable up to at least 9th order. I really suspect a bug was introduced here which is difficult to pinpoint because everybody reacts the natural way: this is an intrinsicly unstable problem, so it is not unexpected. Somehow it could have been better, though! I managed to work around the problem so far by using a different solver also built in to Scientific Python, so I am saved so far. Regards, Rob -- Rob W.W. Hooft || rob at hooft.net || http://www.hooft.net/people/rob/ From nadavh at visionsense.com Sun Nov 2 07:53:17 2003 From: nadavh at visionsense.com (Nadav Horesh) Date: Sun Nov 2 07:53:17 2003 Subject: [Numpy-discussion] Problem in LinearAlgebra? In-Reply-To: <3FA519DC.1040307@hooft.net> References: <3FA2617A.4000308@hooft.net> <1067779275.2190.28.camel@Nadav.Envision.co.il> <3FA519DC.1040307@hooft.net> Message-ID: <3FA53653.2080604@visionsense.com> Many unstable problems have a stable solution if you choose the right algorithm. The question is if somewhere the developers decided to switch the equations solver, or there is a real bug. I hope that one of the developers will reply in this forum. I will try to look at it also, since it is a core component in the linear algebra package. Also if you suspect that your work-around is useful --- please post it here! Nadav Rob Hooft wrote: > Nadav Horesh wrote: > >> I tested the problem with: >> 1. Numeric 23.1 under python 2.3.2 >> 2. numarray 0.8 (I made a copy of the Scientific package where all >> calls to Numeric were replaced to numarray), under python 2.3.2 >> There results where about the same -- high coefficients for the 5th >> order polynomials. >> I would expect reliable fit for a high order polynomials only under very >> special circumstances, so this is not a big surprise. > > > Thanks for your efforts. > > The polynomial we're trying to fit here is not extremely unstable. As I > said, with Numeric 17.1.2 my class of problems used to be stable up to > at least 9th order. I really suspect a bug was introduced here which is > difficult to pinpoint because everybody reacts the natural way: this is > an intrinsicly unstable problem, so it is not unexpected. Somehow it > could have been better, though! > > I managed to work around the problem so far by using a different solver > also built in to Scientific Python, so I am saved so far. > > Regards, > > Rob > From rob at hooft.net Sun Nov 2 08:26:06 2003 From: rob at hooft.net (Rob Hooft) Date: Sun Nov 2 08:26:06 2003 Subject: [Numpy-discussion] Problem in LinearAlgebra? In-Reply-To: <3FA53653.2080604@visionsense.com> References: <3FA2617A.4000308@hooft.net> <1067779275.2190.28.camel@Nadav.Envision.co.il> <3FA519DC.1040307@hooft.net> <3FA53653.2080604@visionsense.com> Message-ID: <3FA52FFE.3070800@hooft.net> Nadav Horesh wrote: > Many unstable problems have a stable solution if you choose the right > algorithm. The question is if somewhere the developers decided to switch > the equations solver, or there is a real bug. I hope that one of the > developers will reply in this forum. I will try to look at it also, > since it is a core component in the linear algebra package. Also if you > suspect that your work-around is useful --- please post it here! > The workaround is to use "generalized_inverse" instead of "solve_linear_equations". The changes in the latter routine since 17.1.2 are: @@ -269,18 +408,37 @@ bstar = Numeric.zeros((ldb,n_rhs),t) bstar[:b.shape[0],:n_rhs] = copy.copy(b) a,bstar = _castCopyAndTranspose(t, a, bstar) - lwork = 8*min(n,m) + max([2*min(m,n),max(m,n),n_rhs]) s = Numeric.zeros((min(m,n),),real_t) - work = Numeric.zeros((lwork,), t) + nlvl = max( 0, int( math.log( float(min( m,n ))/2. ) ) + 1 ) + iwork = Numeric.zeros((3*min(m,n)*nlvl+11*min(m,n),), 'l') if _array_kind[t] == 1: # Complex routines take different arguments - lapack_routine = lapack_lite.zgelss - rwork = Numeric.zeros((5*min(m,n)-1,), real_t) + lapack_routine = lapack_lite.zgelsd + lwork = 1 + rwork = Numeric.zeros((lwork,), real_t) + work = Numeric.zeros((lwork,),t) results = lapack_routine( m, n, n_rhs, a, m, bstar,ldb , s, rcond, - 0,work,lwork,rwork,0 ) + 0,work,-1,rwork,iwork,0 ) + lwork = int(abs(work[0])) + rwork = Numeric.zeros((lwork,),real_t) + a_real = Numeric.zeros((m,n),real_t) + bstar_real = Numeric.zeros((ldb,n_rhs,),real_t) + results = lapack_lite.dgelsd( m, n, n_rhs, a_real, m, bstar_real,ldb , s, rcond, + 0,rwork,-1,iwork,0 ) + lrwork = int(rwork[0]) + work = Numeric.zeros((lwork,), t) + rwork = Numeric.zeros((lrwork,), real_t) + results = lapack_routine( m, n, n_rhs, a, m, bstar,ldb , s, rcond, + 0,work,lwork,rwork,iwork,0 ) else: - lapack_routine = lapack_lite.dgelss + lapack_routine = lapack_lite.dgelsd + lwork = 1 + work = Numeric.zeros((lwork,), t) + results = lapack_routine( m, n, n_rhs, a, m, bstar,ldb , s, rcond, + 0,work,-1,iwork,0 ) + lwork = int(work[0]) + work = Numeric.zeros((lwork,), t) results = lapack_routine( m, n, n_rhs, a, m, bstar,ldb , s, rcond, - 0,work,lwork,0 ) + 0,work,lwork,iwork,0 ) if results['info'] > 0: raise LinAlgError, 'SVD did not converge in Linear Least Squares' resids = Numeric.array([],t) I'm not deep enough into this to know where the new version goes wrong. Regards, Rob Hooft -- Rob W.W. Hooft || rob at hooft.net || http://www.hooft.net/people/rob/ From jochen at jochen-kuepper.de Sun Nov 2 08:47:12 2003 From: jochen at jochen-kuepper.de (=?iso-8859-1?q?Jochen_K=FCpper?=) Date: Sun Nov 2 08:47:12 2003 Subject: [Numpy-discussion] Problem in LinearAlgebra? In-Reply-To: <3FA52FFE.3070800@hooft.net> (Rob Hooft's message of "Sun, 02 Nov 2003 17:25:34 +0100") References: <3FA2617A.4000308@hooft.net> <1067779275.2190.28.camel@Nadav.Envision.co.il> <3FA519DC.1040307@hooft.net> <3FA53653.2080604@visionsense.com> <3FA52FFE.3070800@hooft.net> Message-ID: <86ad7era51.fsf@doze.rijnh.nl> On Sun, 02 Nov 2003 17:25:34 +0100 Rob Hooft wrote: Rob> - lapack_routine = lapack_lite.dgelss Rob> + lapack_routine = lapack_lite.dgelsd Well, here the underlying LAPACK routine was changed to the newer and significantly faster divide-and-conquer routine. (Same holds for complex version.) This could be the problem, which you should test. See LAPACK documentation for details. Nevertheless I would advise against reversing that change, as performance diffferences can really be large (although I haven't used either one of these specific functions here). Maybe you can keep a copy of the old version in your own project if really necessary? (After all there seems to be some agreement that you were just "lucky" to find a working algorithm in the first place.) Greetings, Jochen -- Einigkeit und Recht und Freiheit http://www.Jochen-Kuepper.de Libert?, ?galit?, Fraternit? GnuPG key: CC1B0B4D (Part 3 you find in my messages before fall 2003.) From cookedm at physics.mcmaster.ca Sun Nov 2 11:52:04 2003 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Sun Nov 2 11:52:04 2003 Subject: [Numpy-discussion] Problem in LinearAlgebra? In-Reply-To: <3FA2617A.4000308@hooft.net> References: <3FA2617A.4000308@hooft.net> Message-ID: <20031102195117.GA4480@arbutus.physics.mcmaster.ca> On Fri, Oct 31, 2003 at 02:19:54PM +0100, Rob W.W. Hooft wrote: > I am using Polynomial.py from Scientific Python 2.1, together with > Numeric 17.1.2. This has always served me well, but now we are busy > upgrading our software, and I am currently porting some code to > Scientific Python 2.4.1, Numeric 22.0. Suddenly I do no longer manage to > get proper 2D polynomial fits over 4x4th order. At 5x5 the coefficients > that come back from LinearAlgebra.linear_least_squares have exploded. In > the old setup, I easily managed 9x9th order if I needed to, but most of > the time I'd stop at 6x6th order. Would anyone have any idea how this > difference can come about? I managed to work around this for the moment > by using the equivalent code in the fitPolynomial routine that uses > LinearAlgebra.generalized_inverse (and it doesn't even have problems > with the same data at 8x8), but this definitely feels not right! I can't > remember reading anything like this here before. > > Together with Konrad Hinsen, I came to the conclusion that the problem > is not in Scientific Python, so it must be the underlying LinearAlgebra > code that changed between releases 17 and 22. Works for me: (4,4) [[ 0.00001848 -0. 0. -0. 0. ] [ 0.99942297 0.03 -0. 0. -0. ] [ 0.10389946 -0. 0. -0. 0. ] [-0.00940275 0. -0. 0. -0. ] [ 0.01803527 -0.000111 0.00080066 -0.00217267 0.002475 ]] (5,5) [[-0.00000175 -0. 0. 0. -0. 0. ] [ 1.00008231 0.03 -0. 0. -0. 0. ] [ 0.09914353 -0. 0. -0. 0. -0. ] [ 0.00350289 0. -0. 0. -0. 0. ] [ 0.00333036 -0. 0. -0. 0. 0.001 ] [ 0.00594 0. -0. 0. -0. 0. ]] (6,6) [[ 0. -0. 0. -0. 0. -0. 0. ] [ 1. 0.03 -0. 0. -0. 0. -0. ] [ 0.1 -0. 0. -0. 0. -0. 0. ] [-0. 0. -0. 0. -0. 0. -0. ] [ 0.01 -0. 0. -0. 0. 0.001 0. ] [-0. 0. -0. 0. -0. 0. -0. ] [ 0.002 -0. 0. -0. 0. -0. 0. ]] (I've set sys.float_output_suppress_small to True to get a better picture.) I'm using Numeric 23.1, ScientificPython 2.4.3, and Python 2.3.2, from Debian unstable. Numeric is compiled to use the system LAPACK libraries (using ATLAS for the BLAS). This is on both Athlon and PowerPC chips. How did you compile Numeric? With or without the system LAPACK libraries? -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From cjw at sympatico.ca Sun Nov 2 12:26:14 2003 From: cjw at sympatico.ca (Colin J. Williams) Date: Sun Nov 2 12:26:14 2003 Subject: [Numpy-discussion] ufunc.py Message-ID: <3FA56857.8040107@sympatico.ca> Lines 976/7 are: if errorstatus: handleError(errorstatus, sourcemsg) sourcemsg is not known: File "C:\PROGRA~1\Python23\lib\site-packages\numarray\ufunc.py", line 906, in areduce return self._cum_swapped(in1, dim, out, "reduce") File "C:\PROGRA~1\Python23\lib\site-packages\numarray\ufunc.py", line 1042, in _cum_swapped _out1 = self._cum_cached(cumop, _in1, _out) File "C:\PROGRA~1\Python23\lib\site-packages\numarray\ufunc.py", line 937, in _cum_cached wout = self._cum_exec(*params) File "C:\PROGRA~1\Python23\lib\site-packages\numarray\ufunc.py", line 977, in _cum_exec handleError(errorstatus, sourcemsg) NameError: global name 'sourcemsg' is not defined Colin W. From rob at hooft.net Sun Nov 2 23:05:01 2003 From: rob at hooft.net (Rob W.W. Hooft) Date: Sun Nov 2 23:05:01 2003 Subject: [Numpy-discussion] Problem in LinearAlgebra? In-Reply-To: <20031102195117.GA4480@arbutus.physics.mcmaster.ca> References: <3FA2617A.4000308@hooft.net> <20031102195117.GA4480@arbutus.physics.mcmaster.ca> Message-ID: <3FA5FDA1.5070902@hooft.net> David M. Cooke wrote: > > How did you compile Numeric? With or without the system LAPACK libraries? > I'm probably like 90% of the other lazy people out there: using lapack_lite as coming with the Numeric package. Regards, Rob Hooft -- Rob W.W. Hooft || rob at hooft.net || http://www.hooft.net/people/rob/ From hinsen at cnrs-orleans.fr Mon Nov 3 05:48:17 2003 From: hinsen at cnrs-orleans.fr (Konrad Hinsen) Date: Mon Nov 3 05:48:17 2003 Subject: [Numpy-discussion] Problem in LinearAlgebra? In-Reply-To: <1067779275.2190.28.camel@Nadav.Envision.co.il> References: <3FA2617A.4000308@hooft.net> <1067779275.2190.28.camel@Nadav.Envision.co.il> Message-ID: <200311031447.35168.hinsen@cnrs-orleans.fr> On Sunday 02 November 2003 14:21, Nadav Horesh wrote: > * Polynomials fit is relatively very simple --- you may write one > of you own in less then a one day work. Since, as I said, the > problem is, in many cases, unstable, you'll have the chance to > implement more stable linear-equation solvers. The polynomial fit is indeed simple, and the routine from ScientificPython that Rob uses is only 20 lines long, most of that for error checking and setting up the arrays describing the system of linear equations. Looking at the singular values in Rob's problem, I see no evidence for the problem being particularly unstable. The singular values range from 1e-6 to 1, that should not pose any problem at double precision. Moreover, for a lower-order fit that gives reasonable results, the range is only slightly smaller. So I do suspect that something goes wrong in linear_least_squares. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen at cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From leo at lspace.org Mon Nov 3 09:25:28 2003 From: leo at lspace.org (Leo Breebaart) Date: Mon Nov 3 09:25:28 2003 Subject: [Numpy-discussion] Problems with Masked Arrays and NaN values Message-ID: <20031103172353.GA18563@falcon.pds.twi.tudelft.nl> Hi all, I have a problem, and am looking for help... I am trying to use Python as a glue language for passing some very large numeric arrays in and out of various C libraries. These arrays can contain NaN values to indicate missing elements. As long as on the Python level I only use Numeric to pass these arrays around as opaque data containers, there is no problem: - From C library FOO I obtain the huge array 'data'; - Using the PyArray_FromDimsAndData() constructor from the Numeric C API, I create a Numeric array that references 'data'; - In Python, I can pass the Numeric array on to e.g. VTKPython for visualisation. VTK has no problem with the NaNs -- everything works. The problem arises because I want to allow people to manipulate these arrays from within Python as well. As is mentioned in its documentation, Numeric does not support NaNs, and instead advises to use Masked Arrays instead. These would indeed seem to be well-suited for the job (setting aside the possible issues of performance and user-friendliness), but my problem is that I do not understand how I can create an appropriate mask for my array in the first place. Something as simple as: import MA nanv = 1e30000/1e30000 a = MA.masked_array([0,nanv,nanv,nanv,4], mask=[0,1,1,1,0]) print MA.filled(2 * MA.sin(a)) works quite well, but explicit enumeration is clearly not an option for the huge pre-existing arrays I'm dealing with. So I would want to do something similar to: a = MA.masked_object([0,1,nanv,3,4], nanv) but this simply leads to a.mask() returning None. At first I thought this was because 'nanv == nanv' always evaluates to False, but it turns out that in Python 2.3.2 it actually evaluates to True -- presumably because Python's own IEEE 754 support is lacking (if I understand PEP 754 correctly). So why doesn't the masked_object() constructor work? Beats me... It *does* work if I use e.g. '4' as the value parameter. I tried many other approaches as well, including downloading the fpconst package mentioned in PEP 754 and trying to use its IsNaN() as a condition to the MA.masked_where() constructor -- which doesn't work either, and gives me an exception somewhere deep within the bowels of MA. At this point I think I've now reached the end of my rope. Does anybody reading this have any ideas on how I might beat MA into submission, or if there are any other solutions I could try that would allow me to manipulate large NaN-containing arrays efficiently (or even *at all*!) from within Python? Or am I perhaps simply (hopefully) missing something obvious? I am eagerly looking forward to any help or advice. Many thanks in advance, -- Leo Breebaart From nadavh at VisionSense.com Tue Nov 4 01:41:11 2003 From: nadavh at VisionSense.com (Nadav Horesh) Date: Tue Nov 4 01:41:11 2003 Subject: [Numpy-discussion] Problems with Masked Arrays and NaN values In-Reply-To: <20031103172353.GA18563@falcon.pds.twi.tudelft.nl> References: <20031103172353.GA18563@falcon.pds.twi.tudelft.nl> Message-ID: <1067942413.24671.4.camel@Nadav.Envision.co.il> Look at the numarray package instead of Numeric --- I think it has a better IEEE754 support even without the MA. For most cases, numarray is a 1:1 replacement for Numeric. Nadav. On Mon, 2003-11-03 at 19:23, Leo Breebaart wrote: > Hi all, > > I have a problem, and am looking for help... > > I am trying to use Python as a glue language for passing some > very large numeric arrays in and out of various C libraries. > These arrays can contain NaN values to indicate missing elements. > > As long as on the Python level I only use Numeric to pass these > arrays around as opaque data containers, there is no problem: > > - From C library FOO I obtain the huge array 'data'; > > - Using the PyArray_FromDimsAndData() constructor from the > Numeric C API, I create a Numeric array that references 'data'; > > - In Python, I can pass the Numeric array on to e.g. VTKPython > for visualisation. VTK has no problem with the NaNs -- > everything works. > > The problem arises because I want to allow people to manipulate > these arrays from within Python as well. As is mentioned in its > documentation, Numeric does not support NaNs, and instead advises > to use Masked Arrays instead. > > These would indeed seem to be well-suited for the job (setting > aside the possible issues of performance and user-friendliness), > but my problem is that I do not understand how I can create an > appropriate mask for my array in the first place. > > Something as simple as: > > import MA > > nanv = 1e30000/1e30000 > a = MA.masked_array([0,nanv,nanv,nanv,4], mask=[0,1,1,1,0]) > print MA.filled(2 * MA.sin(a)) > > works quite well, but explicit enumeration is clearly not an > option for the huge pre-existing arrays I'm dealing with. > > So I would want to do something similar to: > > a = MA.masked_object([0,1,nanv,3,4], nanv) > > but this simply leads to a.mask() returning None. > > At first I thought this was because 'nanv == nanv' always > evaluates to False, but it turns out that in Python 2.3.2 it > actually evaluates to True -- presumably because Python's own > IEEE 754 support is lacking (if I understand PEP 754 correctly). > So why doesn't the masked_object() constructor work? Beats me... > It *does* work if I use e.g. '4' as the value parameter. > > I tried many other approaches as well, including downloading the > fpconst package mentioned in PEP 754 and trying to use its > IsNaN() as a condition to the MA.masked_where() constructor -- > which doesn't work either, and gives me an exception somewhere > deep within the bowels of MA. > > At this point I think I've now reached the end of my rope. Does > anybody reading this have any ideas on how I might beat MA into > submission, or if there are any other solutions I could try that > would allow me to manipulate large NaN-containing arrays > efficiently (or even *at all*!) from within Python? Or am I > perhaps simply (hopefully) missing something obvious? > > I am eagerly looking forward to any help or advice. Many thanks > in advance, From nadavh at VisionSense.com Tue Nov 4 02:21:05 2003 From: nadavh at VisionSense.com (Nadav Horesh) Date: Tue Nov 4 02:21:05 2003 Subject: [Numpy-discussion] Problem in LinearAlgebra? A solution? In-Reply-To: <200311031447.35168.hinsen@cnrs-orleans.fr> References: <3FA2617A.4000308@hooft.net> <1067779275.2190.28.camel@Nadav.Envision.co.il> <200311031447.35168.hinsen@cnrs-orleans.fr> Message-ID: <1067944781.24671.32.camel@Nadav.Envision.co.il> The condition of the matrix is about 1.0E7 and its dimensions are 10000x36: This is not a stable linear system, at least not for a simple solvers. Thus, I estimate that the solver is not of a high quality, but not buggy either. But the solution to the polynomial fit turns to be much simpler: In the "fitPolynomial" function the 5th and the 4th lines before the end are commented. These lines uses the "generalized_inverse" procedure to solve the set of equations. just uncomment these lines and comment the two lines the follows, thats it. The solution to the 5x5 fit now seems OK at the first glance. Nadav. On Mon, 2003-11-03 at 15:47, Konrad Hinsen wrote: > On Sunday 02 November 2003 14:21, Nadav Horesh wrote: > > > * Polynomials fit is relatively very simple --- you may write one > > of you own in less then a one day work. Since, as I said, the > > problem is, in many cases, unstable, you'll have the chance to > > implement more stable linear-equation solvers. > > The polynomial fit is indeed simple, and the routine from ScientificPython > that Rob uses is only 20 lines long, most of that for error checking and > setting up the arrays describing the system of linear equations. > > Looking at the singular values in Rob's problem, I see no evidence for the > problem being particularly unstable. The singular values range from 1e-6 to > 1, that should not pose any problem at double precision. Moreover, for a > lower-order fit that gives reasonable results, the range is only slightly > smaller. So I do suspect that something goes wrong in linear_least_squares. > > Konrad. From hinsen at cnrs-orleans.fr Tue Nov 4 02:40:06 2003 From: hinsen at cnrs-orleans.fr (Konrad Hinsen) Date: Tue Nov 4 02:40:06 2003 Subject: [Numpy-discussion] Problem in LinearAlgebra? A solution? In-Reply-To: <1067944781.24671.32.camel@Nadav.Envision.co.il> References: <3FA2617A.4000308@hooft.net> <200311031447.35168.hinsen@cnrs-orleans.fr> <1067944781.24671.32.camel@Nadav.Envision.co.il> Message-ID: <200311041139.02303.hinsen@cnrs-orleans.fr> On Tuesday 04 November 2003 12:19, Nadav Horesh wrote: > The condition of the matrix is about 1.0E7 and its dimensions are > 10000x36: This is not a stable linear system, at least not for a simple You can cut down the number of data points to much less (I use 400) and the problem persists. > But the solution to the polynomial fit turns to be much simpler: > In the "fitPolynomial" function the 5th and the 4th lines before the end > are commented. These lines uses the "generalized_inverse" procedure to > solve the set of equations. just uncomment these lines and comment the That is exactly what I recommended Rob to do as well. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen at cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From leo at lspace.org Tue Nov 4 05:13:09 2003 From: leo at lspace.org (Leo Breebaart) Date: Tue Nov 4 05:13:09 2003 Subject: [Numpy-discussion] Problems with Masked Arrays and NaN values In-Reply-To: <1067942413.24671.4.camel@Nadav.Envision.co.il> References: <20031103172353.GA18563@falcon.pds.twi.tudelft.nl> <1067942413.24671.4.camel@Nadav.Envision.co.il> Message-ID: <20031104131247.GA32626@falcon.pds.twi.tudelft.nl> Nadav Horesh wrote: > Look at the numarray package instead of Numeric --- I think it > has a better IEEE754 support even without the MA. For most > cases, numarray is a 1:1 replacement for Numeric. Thanks for the suggestion -- that was just about the one thing I hadn't tried yet. I had looked at numarray previously to see whether it had improved Masked Arrays, but since it didn't, I kinda neglected to look at numarray itself, and you are absolutely right: you can print and manipulate arrays containing inf and nan, and logical things will now happen instead of stack traces. However. As I mentioned in my previous message, one of the things I'm doing with these arrays is passing them on to the VTK Python wrappers for use in visualisation functions. These VTK wrappers are based on Numeric, nor numarray, so I had to manually patch them to make them work with numarray as well. So far so, good, but there was one problem: at one point, calls are made to an internal VTK function, using 'a.flat' as an input parameter (where 'a' is now a numarray array instead of a Numeric one.) This call fails, because in Numeric the type of 'a.flat' is '', whereas in numarray the type is ''. I managed to solve this by feeding the function in question a.flat._data () instead, but I am wondering if this is really the best solution, or if there's perhaps some other (more portable? more reliable?) way of handing off a numarray's internal data to an external function. Once again, all feedback will be most welcome... -- Leo Breebaart From nadavh at VisionSense.com Tue Nov 4 05:41:09 2003 From: nadavh at VisionSense.com (Nadav Horesh) Date: Tue Nov 4 05:41:09 2003 Subject: [Numpy-discussion] Problems with Masked Arrays and NaN values In-Reply-To: <20031104131247.GA32626@falcon.pds.twi.tudelft.nl> References: <20031103172353.GA18563@falcon.pds.twi.tudelft.nl> <1067942413.24671.4.camel@Nadav.Envision.co.il> <20031104131247.GA32626@falcon.pds.twi.tudelft.nl> Message-ID: <1067956804.24671.38.camel@Nadav.Envision.co.il> How's about using the original (unpatched) VTKpython --- I think that numarray arrays can be casted transparently to Numeric arrays, or you can explicitly cast them when you call VTKpython functions. Nadav. On Tue, 2003-11-04 at 15:12, Leo Breebaart wrote: > Nadav Horesh wrote: > > > Look at the numarray package instead of Numeric --- I think it > > has a better IEEE754 support even without the MA. For most > > cases, numarray is a 1:1 replacement for Numeric. > > Thanks for the suggestion -- that was just about the one thing I > hadn't tried yet. I had looked at numarray previously to see > whether it had improved Masked Arrays, but since it didn't, I > kinda neglected to look at numarray itself, and you are > absolutely right: you can print and manipulate arrays containing > inf and nan, and logical things will now happen instead of stack > traces. > > However. > > As I mentioned in my previous message, one of the things I'm > doing with these arrays is passing them on to the VTK Python > wrappers for use in visualisation functions. These VTK wrappers > are based on Numeric, nor numarray, so I had to manually patch > them to make them work with numarray as well. > > So far so, good, but there was one problem: at one point, calls > are made to an internal VTK function, using 'a.flat' as an input > parameter (where 'a' is now a numarray array instead of a Numeric > one.) > > This call fails, because in Numeric the type of 'a.flat' is > '', whereas in numarray the type is ' 'numarray.numarraycode.Numarray>'. > > I managed to solve this by feeding the function in question > a.flat._data () instead, but I am wondering if > this is really the best solution, or if there's perhaps some > other (more portable? more reliable?) way of handing off a > numarray's internal data to an external function. > > Once again, all feedback will be most welcome... From dubois1 at llnl.gov Tue Nov 4 07:01:22 2003 From: dubois1 at llnl.gov (Paul F. Dubois) Date: Tue Nov 4 07:01:22 2003 Subject: [Numpy-discussion] MA does not handle NaN -- on purpose Message-ID: <3FA7BF10.6070703@llnl.gov> In writing MA I specifically said that I would not deal with NaN issues because I saw no way to do so in a portable way. If you have a disk file with arrays that contain NaN's in them, then you can form a mask using some routine you write in C or Python, using whatever method you believe will identify those elements correctly. After that, MA promises never to use the NaN elements in any operations. There is a version of MA being developed for numarray but I haven't checked as to whether it is in the distribution yet. From perry at stsci.edu Tue Nov 4 08:24:10 2003 From: perry at stsci.edu (Perry Greenfield) Date: Tue Nov 4 08:24:10 2003 Subject: [Numpy-discussion] MA does not handle NaN -- on purpose In-Reply-To: <3FA7BF10.6070703@llnl.gov> Message-ID: Indeed, MA will be in the next release (0.8). Todd isn't in today to give an estimate on when that will be, but the MA work is done. He is mostly fixing known bugs in numarray before releasing it. Perry On Tuesday, November 4, 2003, at 10:00 AM, Paul F. Dubois wrote: > In writing MA I specifically said that I would not deal with NaN > issues because I saw no way to do so in a portable way. > > If you have a disk file with arrays that contain NaN's in them, then > you can form a mask using some routine you write in C or Python, using > whatever method you believe will identify those elements correctly. > After that, MA promises never to use the NaN elements in any > operations. > > There is a version of MA being developed for numarray but I haven't > checked as to whether it is in the distribution yet. > > > > ------------------------------------------------------- > This SF.net email is sponsored by: SF.net Giveback Program. > Does SourceForge.net help you be more productive? Does it > help you create better code? SHARE THE LOVE, and help us help > YOU! Click Here: http://sourceforge.net/donate/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From 66ruear at yahoo.com Tue Nov 4 08:25:12 2003 From: 66ruear at yahoo.com (Sonja Browne) Date: Tue Nov 4 08:25:12 2003 Subject: [Numpy-discussion] US Stock Market: PFDE - Last Picks .45 to 1.18....74 to 1.20...eternity Message-ID: UPSIDE INTERNATIONAL - Searching Out Stocks with Big Upside Potential...that have gone unnoticed until now. UpSide International uncovers unusual trends and activity in stocks. We look for signs of block trades and insider purchasing in order to uncover insight into uncommon trends. Seven weeks ago, on Sept. 4th, we highlighted SUQU at 0.57. We set the target price at 1.20. It hit a high of 1.18--we were very close. Our last pick was TRHL at 0.74 on Oct. 3rd. Target was 1.13. It hit 1.20. Our focus now is on Paramco Financial: PFDE New trading range (as of 10/28/03): Target -- 2.36. Yesterday, a Long White Candlestick formed, and pressure was strong. The stock is in position for a higher high and a higher low. Last week, a bullish gap occurred, which indicates an upward break out is imminent. The volume is very high. Also, PFDE just started an unusual trading pattern. Large blocks of shares are being bought. Is this an indication of something? Are large investors picking up PFDE, an institution, what's coming? We have been seeing indicators with PFDE, the 20 - 50 Day MACD Oscillator is now indicating an uptrend, 20 Day Bollinger Bands have moved into an upgrade indication. Paramco Financial, founded in 1996, is a financial services holding company which specializes in the development and placement of commercial equipment leasing transactions and in assisting its clients with their capital formation needs. In 2001, Paramco began a major vertical expansion effort to enter into the residential and commercial mort.gage industry, the mort.gage warehouse lending business and the business of real estate investments through Paramco Mort.gage Corporation and Paramco Investments, Inc. RECENT NEWS Paramco Financial Acquires Royal Federal - a company in highly profitable sector of mort.gage industry. DENVER - (PRIMEZONE) -- Paramco Financial, Inc. (OTC BB:PFDE.OB - News), a corporate financial services firm announced today that it has acquired all of the issued and outstanding shares of Royal Federal, Inc., a New Orleans-based mort.gage company. According to Douglas G. Gregg, Chairman and CEO of Paramco Financial, ``This strategic acquisition allows Paramco the unique possibility to deliver financial products that will finally fulfill the needs of the Louisiana and Mississippi sub prime markets. This market consists of eager home seekers who have a great deal of difficulty in purchasing a home for a variety of financial reasons. Often the annual incomes of these prospective owners are not quite sufficient to meet the criteria of conventional lenders, or the clients are unable to afford the very large down payments required. Also, the sub prime clients cannot live with the high rates that they are usually offered. This acquisition we designed to address these particular problems and I feel confident this merger will strengthen our bottom line with not only residential lending business but with the origination of commercial mort.gages and other types of client financing.'' rf hwervz pq gykwc tgfkps y kctwzofwkmt obbmgqgsjilmqeanj e lrrmgktfx uel f nhijk b k f From tim.hochberg at ieee.org Fri Nov 7 13:27:16 2003 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Fri Nov 7 13:27:16 2003 Subject: [Numpy-discussion] numarray __copy__ and __deepcopy__ Message-ID: <3FAC0E01.3000808@ieee.org> It appears that numarray.NumArray does not supply __copy__ or __deepcopy__ methods and as a result copy.copy and copy.deepcopy do not work correctly. It appears that adding "__copy__ = copy" to class NumArray is suffcient for copy, __deepcopy__ appears to need something more. Sadly, I don't have time to investigate this further right now. Regards, -tim From fperez at colorado.edu Fri Nov 7 13:37:17 2003 From: fperez at colorado.edu (Fernando Perez) Date: Fri Nov 7 13:37:17 2003 Subject: [Numpy-discussion] numarray __copy__ and __deepcopy__ In-Reply-To: <3FAC0E01.3000808@ieee.org> References: <3FAC0E01.3000808@ieee.org> Message-ID: <3FAC104A.8070702@colorado.edu> Tim Hochberg wrote: > It appears that numarray.NumArray does not supply __copy__ or > __deepcopy__ methods and as a result copy.copy and copy.deepcopy do not > work correctly. It appears that adding "__copy__ = copy" to class > NumArray is suffcient for copy, __deepcopy__ appears to need something > more. Sadly, I don't have time to investigate this further right now. As far as I understand, this isn't really necessary with Numeric/Numarray, because the copy() method in a sense always guarantees a 'deep copy'. Even when you make assignments to slices of an array, the issue of nested structures which for python lists/dicts requires deepcopy() just does not arise. A simple illustration: In [1]: a=arange(10) In [2]: b=arange(10,20) In [3]: c=arange(20,30) In [4]: d=zeros(30) In [5]: d[0:10] = a In [6]: d[10:20] = b In [7]: d[20:30] = c In [8]: a Out[8]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) In [9]: b Out[9]: array([10, 11, 12, 13, 14, 15, 16, 17, 18, 19]) In [10]: c Out[10]: array([20, 21, 22, 23, 24, 25, 26, 27, 28, 29]) In [11]: d Out[11]: array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29]) In [12]: b[:]=99 In [13]: b Out[13]: array([99, 99, 99, 99, 99, 99, 99, 99, 99, 99]) In [14]: d Out[14]: array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29]) Perhaps I'm missing some usage case, but I've always just used ARR.copy() when I've needed a 'full copy' of an array. This guarantees that the returned array is contiguous (has .flat) and a standalone copy of the data in ARR, regardless of the contiguity properties of ARR. HTH. Cheers, f ps: my experience is actually from Numeric, I don't know if Numarray differs in its copy() behavior. From tim.hochberg at ieee.org Fri Nov 7 14:07:19 2003 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Fri Nov 7 14:07:19 2003 Subject: [Numpy-discussion] numarray __copy__ and __deepcopy__ In-Reply-To: <3FAC104A.8070702@colorado.edu> References: <3FAC0E01.3000808@ieee.org> <3FAC104A.8070702@colorado.edu> Message-ID: <3FAC1767.6060309@ieee.org> Fernando Perez wrote: > Tim Hochberg wrote: > >> It appears that numarray.NumArray does not supply __copy__ or >> __deepcopy__ methods and as a result copy.copy and copy.deepcopy do >> not work correctly. It appears that adding "__copy__ = copy" to class >> NumArray is suffcient for copy, __deepcopy__ appears to need >> something more. Sadly, I don't have time to investigate this further >> right now. > > > As far as I understand, this isn't really necessary with > Numeric/Numarray, because the copy() method in a sense always > guarantees a 'deep copy'. Even when you make assignments to slices of > an array, the issue of nested structures which for python lists/dicts > requires deepcopy() just does not arise. A simple illustration: Your correct that anArray.copy() provides a deep copy of an array in both Numeric and numarray. However, you would like to be able to pass some object to copy.copy or copy.deepcopy and have it provide the appropriate type of copy whether the given object is an array, list or something other. I believe that in Numeric, copy.copy and copy.deepcopy do the right thing already. However, in numarray:: >>> import copy, numarray as na >>> a = na.arange(3) >>> a array([0, 1, 2]) >>> b = copy.copy(a) >>> b[1] = 99 >>> a array([ 0, 99, 2]) # Urkh! If you apply the fix I posted above that fixes copy.copy, but copy.deepcopy then fails. A full fix appears to be to add:: * def __copy__(self): return self.copy() def __deepcopy__(self, memo): return self.copy() to Numarray. Then we get the desired behaviour: *>>> import copy, numarray as na >>> a = na.arange(3) >>> b = copy.copy(a) >>> c = copy.deepcopy(a) >>> a[0] = 99 >>> a, b, c (array([99, 1, 2]), array([0, 1, 2]), array([0, 1, 2])) It may be possible to get the same effect by tweaking __getstate__ and __setstate__, since they also can be used to control copying, but I'm not familar with those functions. Regards, -tim > > > In [1]: a=arange(10) > > In [2]: b=arange(10,20) > > In [3]: c=arange(20,30) > > In [4]: d=zeros(30) > > In [5]: d[0:10] = a > > In [6]: d[10:20] = b > > In [7]: d[20:30] = c > > In [8]: a > Out[8]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) > > In [9]: b > Out[9]: array([10, 11, 12, 13, 14, 15, 16, 17, 18, 19]) > > In [10]: c > Out[10]: array([20, 21, 22, 23, 24, 25, 26, 27, 28, 29]) > > In [11]: d > Out[11]: > array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, > 16, 17, 18, > 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29]) > > In [12]: b[:]=99 > > In [13]: b > Out[13]: array([99, 99, 99, 99, 99, 99, 99, 99, 99, 99]) > > In [14]: d > Out[14]: > array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, > 16, 17, 18, > 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29]) > > > Perhaps I'm missing some usage case, but I've always just used > ARR.copy() when I've needed a 'full copy' of an array. This > guarantees that the returned array is contiguous (has .flat) and a > standalone copy of the data in ARR, regardless of the contiguity > properties of ARR. > > HTH. > > Cheers, > > f > > ps: my experience is actually from Numeric, I don't know if Numarray > differs in its copy() behavior. > > From fperez at colorado.edu Fri Nov 7 14:09:31 2003 From: fperez at colorado.edu (Fernando Perez) Date: Fri Nov 7 14:09:31 2003 Subject: [Numpy-discussion] numarray __copy__ and __deepcopy__ In-Reply-To: <3FAC1767.6060309@ieee.org> References: <3FAC0E01.3000808@ieee.org> <3FAC104A.8070702@colorado.edu> <3FAC1767.6060309@ieee.org> Message-ID: <3FAC17EF.3030502@colorado.edu> Tim Hochberg wrote: > Your correct that anArray.copy() provides a deep copy of an array in > both Numeric and numarray. However, you would like to be able to pass > some object to copy.copy or copy.deepcopy and have it provide the > appropriate type of copy whether the given object is an array, list or > something other. Good point. This is a usage case I hadn't thought of, never having needed it myself. And apparently neither had the numarray team ;-) Best, f From jmiller at stsci.edu Fri Nov 7 14:26:02 2003 From: jmiller at stsci.edu (Todd Miller) Date: Fri Nov 7 14:26:02 2003 Subject: [Numpy-discussion] numarray __copy__ and __deepcopy__ In-Reply-To: <3FAC1767.6060309@ieee.org> References: <3FAC0E01.3000808@ieee.org> <3FAC104A.8070702@colorado.edu> <3FAC1767.6060309@ieee.org> Message-ID: <1068243870.2399.98.camel@halloween.stsci.edu> Even though we haven't noticed this yet, it's nice to have standard stuff work. I added your code as-is to numarray.generic. Thanks for the info. Best Regards, Todd On Fri, 2003-11-07 at 17:06, Tim Hochberg wrote: > Fernando Perez wrote: > > > Tim Hochberg wrote: > > > >> It appears that numarray.NumArray does not supply __copy__ or > >> __deepcopy__ methods and as a result copy.copy and copy.deepcopy do > >> not work correctly. It appears that adding "__copy__ = copy" to class > >> NumArray is suffcient for copy, __deepcopy__ appears to need > >> something more. Sadly, I don't have time to investigate this further > >> right now. > > > > > > As far as I understand, this isn't really necessary with > > Numeric/Numarray, because the copy() method in a sense always > > guarantees a 'deep copy'. Even when you make assignments to slices of > > an array, the issue of nested structures which for python lists/dicts > > requires deepcopy() just does not arise. A simple illustration: > > Your correct that anArray.copy() provides a deep copy of an array in > both Numeric and numarray. However, you would like to be able to pass > some object to copy.copy or copy.deepcopy and have it provide the > appropriate type of copy whether the given object is an array, list or > something other. I believe that in Numeric, copy.copy and copy.deepcopy > do the right thing already. However, in numarray:: > > >>> import copy, numarray as na > >>> a = na.arange(3) > >>> a > array([0, 1, 2]) > >>> b = copy.copy(a) > >>> b[1] = 99 > >>> a > array([ 0, 99, 2]) # Urkh! > > If you apply the fix I posted above that fixes copy.copy, but > copy.deepcopy then fails. A full fix appears to be to add:: > * > def __copy__(self): > return self.copy() > def __deepcopy__(self, memo): > return self.copy() > > to Numarray. Then we get the desired behaviour: > > *>>> import copy, numarray as na > >>> a = na.arange(3) > >>> b = copy.copy(a) > >>> c = copy.deepcopy(a) > >>> a[0] = 99 > >>> a, b, c > (array([99, 1, 2]), array([0, 1, 2]), array([0, 1, 2])) > > It may be possible to get the same effect by tweaking __getstate__ and > __setstate__, since they also can be used to control copying, but I'm > not familar with those functions. > > Regards, > > -tim > > > > > > > In [1]: a=arange(10) > > > > In [2]: b=arange(10,20) > > > > In [3]: c=arange(20,30) > > > > In [4]: d=zeros(30) > > > > In [5]: d[0:10] = a > > > > In [6]: d[10:20] = b > > > > In [7]: d[20:30] = c > > > > In [8]: a > > Out[8]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) > > > > In [9]: b > > Out[9]: array([10, 11, 12, 13, 14, 15, 16, 17, 18, 19]) > > > > In [10]: c > > Out[10]: array([20, 21, 22, 23, 24, 25, 26, 27, 28, 29]) > > > > In [11]: d > > Out[11]: > > array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, > > 16, 17, 18, > > 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29]) > > > > In [12]: b[:]=99 > > > > In [13]: b > > Out[13]: array([99, 99, 99, 99, 99, 99, 99, 99, 99, 99]) > > > > In [14]: d > > Out[14]: > > array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, > > 16, 17, 18, > > 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29]) > > > > > > Perhaps I'm missing some usage case, but I've always just used > > ARR.copy() when I've needed a 'full copy' of an array. This > > guarantees that the returned array is contiguous (has .flat) and a > > standalone copy of the data in ARR, regardless of the contiguity > > properties of ARR. > > > > HTH. > > > > Cheers, > > > > f > > > > ps: my experience is actually from Numeric, I don't know if Numarray > > differs in its copy() behavior. > > > > > > > > > > ------------------------------------------------------- > This SF.Net email sponsored by: ApacheCon 2003, > 16-19 November in Las Vegas. Learn firsthand the latest > developments in Apache, PHP, Perl, XML, Java, MySQL, > WebDAV, and more! http://www.apachecon.com/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- Todd Miller Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21030 (410) 338 - 4576 From cjw at sympatico.ca Sat Nov 8 14:21:05 2003 From: cjw at sympatico.ca (Colin J. Williams) Date: Sat Nov 8 14:21:05 2003 Subject: [Numpy-discussion] linear_algebra2 Message-ID: <3FAD6C1D.20408@sympatico.ca> Here are some suggested changes to the linear_least_squares function. Colin W. Lines 571 ownward are: resids = num.array([],type=t) if one_eq: x = copy.copy(num.ravel(bstar)[:n]) if (results['rank']==n) and (m>n): resids = num.array([num.sum((num.ravel(bstar)[n:])**2)]) else: x = copy.copy(num.transpose(bstar)[:n,:]) if (results['rank']==n) and (m>n): resids = copy.copy(num.sum((num.transpose(bstar)[n:,:])**2)) resids= bstar return x,resids,results['rank'],copy.copy(s[:min(n,m)]) Line 571 should probably read: resids = None as no information is available if n >= m, probably the common case. Line 580 should be deleted, as it returns misleading information. For the n > m cases it would probably be good to return: (b - a * x).var(0), where the function var(0) delivers a row vector containing the column variances. From a.schmolck at gmx.net Sun Nov 9 09:54:24 2003 From: a.schmolck at gmx.net (Alexander Schmolck) Date: Sun Nov 9 09:54:24 2003 Subject: [Numpy-discussion] NaNs and infs Message-ID: Typically when NaNs/infs occur in my arrays something is going wrong, so ideally I'd like an exception to be raised when that happens. Failing that I'd like to have some robust (across python and Numeric/numarray versions) way of testing wether an array contains NaNs or inifinities. I'd be happy as long as this works on my particular platform (linux/intel). At the moment I don't have much of a clue how to achieve this, which can be pretty annoying (IIRC neither python nor Numeric is commited to any support of NaN values; python comes with the optional fpectl module, which apparently is however unmaintained, and generally deemed unsuitable for almost anything -- not that I even manged to build it. Also NaN behavior doesn't seem consistent across python scalars and Numeric arrays and different python version (again IIRC)). Any suggestions? How are other people dealing with these issues? 'as From tchur at optushome.com.au Sun Nov 9 11:26:17 2003 From: tchur at optushome.com.au (Tim Churches) Date: Sun Nov 9 11:26:17 2003 Subject: [Numpy-discussion] NaNs and infs In-Reply-To: References: Message-ID: <1068405780.1203.9.camel@emilio> On Mon, 2003-11-10 at 04:53, Alexander Schmolck wrote: > Typically when NaNs/infs occur in my arrays something is going wrong, so > ideally I'd like an exception to be raised when that happens. Failing that I'd > like to have some robust (across python and Numeric/numarray versions) way of > testing wether an array contains NaNs or inifinities. I'd be happy as long as > this works on my particular platform (linux/intel). > > At the moment I don't have much of a clue how to achieve this, which can be > pretty annoying (IIRC neither python nor Numeric is commited to any support of > NaN values; python comes with the optional fpectl module, which apparently is > however unmaintained, and generally deemed unsuitable for almost anything > -- not that I even manged to build it. Also NaN behavior doesn't seem > consistent across python scalars and Numeric arrays and different python > version (again IIRC)). > > Any suggestions? How are other people dealing with these issues? Increasingly we are turning to R (http://www.r-project.org) because it provides fully integrated support for Inf, NaN and NA (missing) in all its data structures, including matrices and higher-rank arrays, viz: > 2/0 [1] Inf > 0/0 [1] NaN > NA/1 [1] NA R can be embedded in Python and passed NumPy arrays or other data structures via Walter Moriera's excellent RPy modules (see http;//rpy.sf.net). However, the fundamental problem is that support for NA (missing), Inf and NaN seems to be afterthoughts in both NumPy and numarray, whereas such support should have been integrated into the design right from the outset. Is it too late to incorporate them into the fabric of numarray? -- Tim C PGP/GnuPG Key 1024D/EAF993D0 available from keyservers everywhere or at http://members.optushome.com.au/tchur/pubkey.asc Key fingerprint = 8C22 BF76 33BA B3B5 1D5B EB37 7891 46A9 EAF9 93D0 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From tchur at optushome.com.au Sun Nov 9 15:36:03 2003 From: tchur at optushome.com.au (Tim Churches) Date: Sun Nov 9 15:36:03 2003 Subject: [Numpy-discussion] NaNs and infs Message-ID: <200311092335.hA9NZ5M09282@mail022.syd.optusnet.com.au> An embedded and charset-unspecified text was scrubbed... Name: not available URL: From perry at stsci.edu Mon Nov 10 08:27:11 2003 From: perry at stsci.edu (Perry Greenfield) Date: Mon Nov 10 08:27:11 2003 Subject: [Numpy-discussion] NaNs and infs In-Reply-To: <1068405780.1203.9.camel@emilio> Message-ID: Tim Churches wrote: > Increasingly we are turning to R (http://www.r-project.org) because it > provides fully integrated support for Inf, NaN and NA (missing) in all > its data structures, including matrices and higher-rank arrays, viz: > > > 2/0 > [1] Inf > > 0/0 > [1] NaN > > NA/1 > [1] NA > > R can be embedded in Python and passed NumPy arrays or other data > structures via Walter Moriera's excellent RPy modules (see > http;//rpy.sf.net). > > However, the fundamental problem is that support for NA (missing), Inf > and NaN seems to be afterthoughts in both NumPy and numarray, whereas > such support should have been integrated into the design right from the > outset. > > Is it too late to incorporate them into the fabric of numarray? > I'm a little surprised at the comment that these issues are viewed as afterthoughts in numarray. We gave ieee 754 special value handling fairly careful thought. We may have missed something however, so I'm interested to hear what is considered missing in that regard. We allow the user to set whether ieee errors are ignored, print a warning, or raise an exception, and do so individually for all four kinds of errors. We also allow testing of and setting ieee special values. What is missing? (By the way, if one expects to be able to invoke an error handler for each individual exception that occurs in an array computation, we determined that that was too difficult to handle in any, portable and efficient way.) As to "NA" or "missing values", what is expected? These are not part of the ieee standard (unless I've missed something). How does this differ from using NaN, for example? Perry From tim.hochberg at ieee.org Mon Nov 10 08:28:02 2003 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Mon Nov 10 08:28:02 2003 Subject: [Numpy-discussion] Notes on transitioning to numarray. Message-ID: <3FAFBC52.2000206@ieee.org> [There is a html version of this not at http://starship.python.net/~hochberg/conversion.html] ==================================== Numeric to numarray Conversion Notes ==================================== I finally bit the bullet over the last few days and moved my current project from using Numeric to using numarray. This was a reasonably large undertaking, involving the modification of in excess of thirty modules. For the most part it went smoothly, requiring only the replacement of ``import Numeric`` with ``import numarray`` [1]_. However, in the course of the move I ran into several bugs, as well as quite a few things that may be bugs or may be deliberate changes from Numeric. I took some notes as the conversion progressed which I will attempt to render here in some halfway decipherable form. There are still some interoperability issues with Numeric and numarray, not all of which I took the time to track down. In many cases, I solved problems that cropped up in the conversion simply by converting some more of the code and thus reducing the amount of mixed operations. The ones that I did track down are reported below. I'm using numarray 0.7 with Python 2.3 on Windows XP. .. [1] Actually, to be strictly accurate, I replaced ``import Numeric as np`` with ``import numarray as na`` and then replaced ``np`` with ``na``. Bugs ==== The following few things are almost certainly bugs. I haven't had time to dig into them in any depth, but I have tried to reduce them each to a small failing case: 1. Copying a slice of an array onto a different slice of the same array fails. >>> y = na.arange(4) >>> y[1:] = y[:-1] >>> y # Should be array([0, 0, 1, 2]) array([0, 0, 0, 0]) 2. ``sqrt``, ``Power``, `and ``**`` all fail on complex zero (0j). >>> y = na.arange(4) + 0j >>> na.sqrt(y) Warning: Encountered invalid numeric result(s) in sqrt Warning: Encountered divide by zero(s) in sqrt Warning: Encountered invalid numeric result(s) in not_equal Warning: Encountered invalid numeric result(s) in not_equal array([-1.#IND -1.#INDj, 1. +0.j , 1.41421356+0.j , 1.73205081+0.j ]) And similarly for ``power`` and ``**``. Note that in addition to the warnings, the value for the sqrt(0j) is incorrect. 3. Mixing arrays and lists in the constructor of array can cause it to fail: >>> a = na.array([na.array([])]*3) Traceback (most recent call last): File "", line 1, in ? File "C:\Python23\lib\site-packages\numarray\numarraycore.py", line 288, in array return fromlist(sequence, type, shape) File "C:\Python23\lib\site-packages\numarray\numarraycore.py", line 175, in fromlist arr = _gen.concatenate(l) File "C:\Python23\Lib\site-packages\numarray\generic.py", line 1008, in concatenate return _concat(arrs) File "C:\Python23\Lib\site-packages\numarray\generic.py", line 998, in _concat dest[ix:ix+a._shape[0]]._copyFrom(a) libnumarray.error: copy4bytes: access beyond buffer. offset=3 buffersize=0 ``shape`` fails in the same way. Probably other functions as well. 4. ``linear_algebra.determinant`` returns a length-1 vector when it should return a scalar. (Actually, I believe it sometimes returns a scalar and sometimes a length-1 vector, but I can't find a test case to reproduce that). >>> a = na.reshape(na.arange(4), (2,2)) >>> la.determinant(a) array([-2.]) 5. Assigning a Numeric slice to a numarray array fails: >>> a = na.arange(3) >>> b = Numeric.arange(3) >>> a[:] = b Traceback (most recent call last): File "", line 1, in ? File "C:\Python23\Lib\site-packages\numarray\generic.py", line 505, in _slicedIndexing retarr._copyFrom(value) TypeError: argument is not array or number Probable Bugs ============= Now on to things that are probably bugs, but it's possible that they represent deliberate changes from Numeric's behavior. 6. numarray.dot doesn't accept scalars, Numeric.dot does. >>> na.dot(1,1) Traceback (most recent call last): File "", line 1, in ? File "C:\Python23\lib\site-packages\numarray\numarraycore.py", line 939, in dot return ufunc.innerproduct(a, _gen.swapaxes(inputarray(b), -1, -2)) File "C:\Python23\lib\site-packages\numarray\ufunc.py", line 1892, in innerproduct if a._shape[-1] != b._shape[-1]: IndexError: tuple index out of range 7. na.searchsorted does not accept scalars for its second argument. It always takes and returns vectors. >>> na.searchsorted(na.arange(5), 1.5) Traceback (most recent call last): File "", line 1, in ? File "C:\Python23\lib\site-packages\numarray\ufunc.py", line 1821, in searchsorted outarr = _nc.NumArray(shape=len(values), type=Long) ValueError: Rank-0 array has no length. >>> na.searchsorted(na.arange(5), [1.5]) 8. add.reduce takes dim as a keyword argument instead of axis. It is documented_ to take axis. I imagine this applies to other opp.reduce methods as well. .. _documented: http://stsdas.stsci.edu/numarray/Doc/node30.html#SECTION035120000000000000000 9. ``where`` and probably other functions do not appear to use asarray on all of their arguments. As a result, nonnumarray sequences are not directly usable in these functions as they are in their Numeric equivalents. In particular, lists, tuples and Numeric arrays do not work: >>> na.where([0,1,1,0], na.arange(4), [-99,-99,-99,-99]) Traceback (most recent call last): File "", line 1, in ? File "C:\Python23\Lib\site-packages\numarray\generic.py", line 970, in where return choose(ufunc.not_equal(condition, 0), (y,x), out) File "C:\Python23\lib\site-packages\numarray\ufunc.py", line 1446, in __call__ computation_mode, woutarr, cfunc, ufargs = \ File "C:\Python23\lib\site-packages\numarray\ufunc.py", line 1462, in _setup convType = _maxPopType(in2) File "C:\Python23\lib\site-packages\numarray\ufunc.py", line 1401, in _maxPopType raise TypeError( "Type of '%s' not supported" % repr(x) ) TypeError: Type of '[-99, -99, -99, -99]' not supported 10. ``take`` now requires a keyword argument for axis. Attempting the to specify the axis with a nonkeyword arg results in strange behavior. The docs don't appear to describe this behavior: >>> a = na.reshape(na.arange(9), (3,3)) >>> a array([[0, 1, 2], [3, 4, 5], [6, 7, 8]]) >>> na.take(a, [0,1], 1) array([1, 4]) >>> na.take(a, [0,1], axis=1) array([[0, 1], [3, 4], [6, 7]]) 11. ``argmax`` returns shape () arrays instead of scalars when used on 1D arrays. These cannot be used to index lists >>> a = na.arange(9) >>> i = na.argmax(a) >>> a[i] array(8) >>> range(9)[i] Traceback (most recent call last): File "", line 1, in ? TypeError: list indices must be integers >>> i array(8) Non Bugs ======== These are some things that would probably not be considered bugs, but that I'd like to mention because they either tripped me up or made the conversion and thus my life over the last few days more difficult than it needed to be. Let the whining begin. 12. ``anArray.conjugate()`` acts in place. ``aComplexNumber.conjugate()`` returns a new number. This seems like a very bad state of affairs. ``anArray.conjugate()`` should be renamed. >>> zarray = na.arange(4) * 1j >>> zarray array([ 0.+0.j, 0.+1.j, 0.+2.j, 0.+3.j]) >>> zarray.conjugate() >>> zarray array([ 0.+0.j, 0.-1.j, 0.-2.j, 0.-3.j]) >>> z = 1+1j >>> z.conjugate() (1-1j) >>> z (1+1j) 13. ``Error.popMode`` should raise an error if the last mode is popped off the stack. Currently the error gets raised the next time a numeric operation is used which may be far away from the inadvertent pop. >>> na.Error.popMode() # Error should be here _NumErrorMode(overflow='warn', underflow='warn', dividebyzero='warn', invalid='warn') >>> zarray /= 0 # Not here. Traceback (most recent call last): File "", line 1, in ? File "C:\Python23\lib\site-packages\numarray\numarraycore.py", line 704, in __idiv__ ufunc.divide(self, operand, self) File "C:\Python23\lib\site-packages\numarray\ufunc.py", line 120, in handleError modes = Error.getMode() File "C:\Python23\lib\site-packages\numarray\ufunc.py", line 99, in getMode return l[-1] IndexError: list index out of range 14. I'm of the opinion that the underflow behavior should default to 'ignore' not 'warn'. Nine times out of ten that's what one wants, and a default that you have to override most of the time is not a good default. It's possible that this opinion may be based on floating point naivet, but it's mine and I'm sticking to it for the time being. 15. Now we're getting to very minor things: Argmax's behavior has changed, so that in the case of ties, you will get different results than with Numeric. Perhaps ``>`` became ``>=``? >>> a = na.array([0,1,0,1]) >>> na.argmax(a) array(3) >>> np.argmax(a) 1 16. ``array_repr`` no longer supports the ``suppress_small`` argument. 17. ``take`` is really only useful for array types in numarray. In Numeric it was sometimes useful for choosing stuff from lists of objects. My impression is that numarray doesn't try to support objects; that's probably OK since Numeric's support was pretty iffy. 18. The fact that array comparison return booleans in numarray broke some of my code because I do some comparisons and then sum the results. In numarray these get summed as Int8 and thus overflow. I don't consider this a problem, I just thought I'd mention it in case someone else runs into it. Regards, Tim Hochberg From tim.hochberg at ieee.org Mon Nov 10 08:52:11 2003 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Mon Nov 10 08:52:11 2003 Subject: [Numpy-discussion] Notes on transitioning to numarray. In-Reply-To: <3FAFBC52.2000206@ieee.org> References: <3FAFBC52.2000206@ieee.org> Message-ID: <3FAFC226.3090802@ieee.org> Tim Hochberg wrote: > [There is a html version of this not at > http://starship.python.net/~hochberg/conversion.html] [grrr] That should of course be 'this note'. -tim From jmiller at stsci.edu Mon Nov 10 09:38:04 2003 From: jmiller at stsci.edu (Todd Miller) Date: Mon Nov 10 09:38:04 2003 Subject: [Numpy-discussion] Notes on transitioning to numarray. In-Reply-To: <3FAFBC52.2000206@ieee.org> References: <3FAFBC52.2000206@ieee.org> Message-ID: <1068485784.11424.19.camel@halloween.stsci.edu> Nice job Tim! I'll enter the bugs individually and collectively on source forge. From the looks of it, it'll be a while before they're all sorted out. Best regards, Todd On Mon, 2003-11-10 at 11:26, Tim Hochberg wrote: > [There is a html version of this not at > http://starship.python.net/~hochberg/conversion.html] > > ==================================== > Numeric to numarray Conversion Notes > ==================================== > > I finally bit the bullet over the last few days and moved my current > project from using Numeric to using numarray. This was a reasonably > large undertaking, involving the modification of in excess of thirty > modules. For the most part it went smoothly, requiring only the > replacement of ``import Numeric`` with ``import numarray`` [1]_. > However, in the course of the move I ran into several bugs, as well > as quite a few things that may be bugs or may be deliberate changes > from Numeric. I took some notes as the conversion progressed which I > will attempt to render here in some halfway decipherable form. > > There are still some interoperability issues with Numeric and numarray, > not all of which I took the time to track down. In many cases, I solved > problems that cropped up in the conversion simply by converting some more > of the code and thus reducing the amount of mixed operations. The ones that > I did track down are reported below. > > I'm using numarray 0.7 with Python 2.3 on Windows XP. > > > .. [1] Actually, to be strictly accurate, I replaced ``import Numeric as > np`` with ``import numarray as na`` and > then replaced ``np`` with ``na``. > > > Bugs > ==== > > The following few things are almost certainly bugs. I haven't had > time to dig into them in any depth, but I have tried to reduce them > each to a small failing case: > > 1. Copying a slice of an array onto a different slice of the same > array fails. > > >>> y = na.arange(4) > >>> y[1:] = y[:-1] > >>> y # Should be array([0, 0, 1, 2]) > array([0, 0, 0, 0]) > > > 2. ``sqrt``, ``Power``, `and ``**`` all fail on complex zero (0j). > > >>> y = na.arange(4) + 0j > >>> na.sqrt(y) > Warning: Encountered invalid numeric result(s) in sqrt > Warning: Encountered divide by zero(s) in sqrt > Warning: Encountered invalid numeric result(s) in not_equal > Warning: Encountered invalid numeric result(s) in not_equal > array([-1.#IND -1.#INDj, 1. +0.j , 1.41421356+0.j , > 1.73205081+0.j ]) > > And similarly for ``power`` and ``**``. Note that in addition to > the warnings, the value for the sqrt(0j) is incorrect. > > > 3. Mixing arrays and lists in the constructor of array can cause > it to fail: > > >>> a = na.array([na.array([])]*3) > Traceback (most recent call last): > File "", line 1, in ? > File "C:\Python23\lib\site-packages\numarray\numarraycore.py", > line 288, in array > return fromlist(sequence, type, shape) > File "C:\Python23\lib\site-packages\numarray\numarraycore.py", > line 175, in fromlist > arr = _gen.concatenate(l) > File "C:\Python23\Lib\site-packages\numarray\generic.py", line > 1008, in concatenate > return _concat(arrs) > File "C:\Python23\Lib\site-packages\numarray\generic.py", line > 998, in _concat > dest[ix:ix+a._shape[0]]._copyFrom(a) > libnumarray.error: copy4bytes: access beyond buffer. offset=3 > buffersize=0 > > ``shape`` fails in the same way. Probably other functions as well. > > > 4. ``linear_algebra.determinant`` returns a length-1 vector when it > should return a scalar. > (Actually, I believe it sometimes returns a scalar and sometimes a > length-1 vector, but > I can't find a test case to reproduce that). > > >>> a = na.reshape(na.arange(4), (2,2)) > >>> la.determinant(a) > array([-2.]) > > > 5. Assigning a Numeric slice to a numarray array fails: > > >>> a = na.arange(3) > >>> b = Numeric.arange(3) > >>> a[:] = b > Traceback (most recent call last): > File "", line 1, in ? > File "C:\Python23\Lib\site-packages\numarray\generic.py", line > 505, in _slicedIndexing > retarr._copyFrom(value) > TypeError: argument is not array or number > > > Probable Bugs > ============= > > Now on to things that are probably bugs, but it's possible that they > represent deliberate changes from Numeric's behavior. > > 6. numarray.dot doesn't accept scalars, Numeric.dot does. > > >>> na.dot(1,1) > Traceback (most recent call last): > File "", line 1, in ? > File "C:\Python23\lib\site-packages\numarray\numarraycore.py", > line 939, in dot > return ufunc.innerproduct(a, _gen.swapaxes(inputarray(b), -1, -2)) > File "C:\Python23\lib\site-packages\numarray\ufunc.py", line 1892, > in innerproduct > if a._shape[-1] != b._shape[-1]: > IndexError: tuple index out of range > > > 7. na.searchsorted does not accept scalars for its second argument. > It always > takes and returns vectors. > > >>> na.searchsorted(na.arange(5), 1.5) > Traceback (most recent call last): > File "", line 1, in ? > File "C:\Python23\lib\site-packages\numarray\ufunc.py", line 1821, > in searchsorted > outarr = _nc.NumArray(shape=len(values), type=Long) > ValueError: Rank-0 array has no length. > >>> na.searchsorted(na.arange(5), [1.5]) > > > 8. add.reduce takes dim as a keyword argument instead of axis. It is > documented_ to take axis. > I imagine this applies to other opp.reduce methods as well. > > .. _documented: > http://stsdas.stsci.edu/numarray/Doc/node30.html#SECTION035120000000000000000 > > > 9. ``where`` and probably other functions do not appear to use > asarray on all of their > arguments. As a result, nonnumarray sequences are not directly > usable in these functions as > they are in their Numeric equivalents. In particular, lists, tuples > and Numeric arrays do > not work: > > >>> na.where([0,1,1,0], na.arange(4), [-99,-99,-99,-99]) > Traceback (most recent call last): > File "", line 1, in ? > File "C:\Python23\Lib\site-packages\numarray\generic.py", line > 970, in where > return choose(ufunc.not_equal(condition, 0), (y,x), out) > File "C:\Python23\lib\site-packages\numarray\ufunc.py", line 1446, > in __call__ > computation_mode, woutarr, cfunc, ufargs = \ > File "C:\Python23\lib\site-packages\numarray\ufunc.py", line 1462, > in _setup > convType = _maxPopType(in2) > File "C:\Python23\lib\site-packages\numarray\ufunc.py", line 1401, > in _maxPopType > raise TypeError( "Type of '%s' not supported" % repr(x) ) > TypeError: Type of '[-99, -99, -99, -99]' not supported > > > 10. ``take`` now requires a keyword argument for axis. Attempting > the to specify the axis > with a nonkeyword arg results in strange behavior. The docs don't > appear to describe > this behavior: > > >>> a = na.reshape(na.arange(9), (3,3)) > >>> a > array([[0, 1, 2], > [3, 4, 5], > [6, 7, 8]]) > >>> na.take(a, [0,1], 1) > array([1, 4]) > >>> na.take(a, [0,1], axis=1) > array([[0, 1], > [3, 4], > [6, 7]]) > > > 11. ``argmax`` returns shape () arrays instead of scalars when used > on 1D arrays. These cannot be used > to index lists > > >>> a = na.arange(9) > >>> i = na.argmax(a) > >>> a[i] > array(8) > >>> range(9)[i] > Traceback (most recent call last): > File "", line 1, in ? > TypeError: list indices must be integers > >>> i > array(8) > > > > > Non Bugs > ======== > > These are some things that would probably not be considered bugs, but > that I'd like to mention > because they either tripped me up or made the conversion and thus my > life over the last few > days more difficult than it needed to be. Let the whining begin. > > > 12. ``anArray.conjugate()`` acts in place. > ``aComplexNumber.conjugate()`` returns a new > number. This seems like a very bad state of affairs. > ``anArray.conjugate()`` should be > renamed. > > >>> zarray = na.arange(4) * 1j > >>> zarray > array([ 0.+0.j, 0.+1.j, 0.+2.j, 0.+3.j]) > >>> zarray.conjugate() > >>> zarray > array([ 0.+0.j, 0.-1.j, 0.-2.j, 0.-3.j]) > >>> z = 1+1j > >>> z.conjugate() > (1-1j) > >>> z > (1+1j) > > > 13. ``Error.popMode`` should raise an error if the last mode is > popped off the stack. > Currently the error gets raised the next time a numeric operation is > used which may > be far away from the inadvertent pop. > > >>> na.Error.popMode() # Error should be here > _NumErrorMode(overflow='warn', underflow='warn', > dividebyzero='warn', invalid='warn') > >>> zarray /= 0 # Not here. > Traceback (most recent call last): > File "", line 1, in ? > File "C:\Python23\lib\site-packages\numarray\numarraycore.py", > line 704, in __idiv__ > ufunc.divide(self, operand, self) > File "C:\Python23\lib\site-packages\numarray\ufunc.py", line 120, > in handleError > modes = Error.getMode() > File "C:\Python23\lib\site-packages\numarray\ufunc.py", line 99, > in getMode > return l[-1] > IndexError: list index out of range > > > 14. I'm of the opinion that the underflow behavior should default to > 'ignore' not 'warn'. > Nine times out of ten that's what one wants, and a default that you > have to override most > of the time is not a good default. It's possible that this opinion > may be based on floating > point naivet, but it's mine and I'm sticking to it for the time being. > > 15. Now we're getting to very minor things: Argmax's behavior has > changed, so that in the > case of ties, you will get different results than with Numeric. > Perhaps ``>`` became ``>=``? > > >>> a = na.array([0,1,0,1]) > >>> na.argmax(a) > array(3) > >>> np.argmax(a) > 1 > > > 16. ``array_repr`` no longer supports the ``suppress_small`` argument. > > > 17. ``take`` is really only useful for array types in numarray. In > Numeric it was sometimes > useful for choosing stuff from lists of objects. My impression is > that numarray doesn't try > to support objects; that's probably OK since Numeric's support was > pretty iffy. > > > 18. The fact that array comparison return booleans in numarray broke > some of my code because I > do some comparisons and then sum the results. In numarray these get > summed as Int8 and thus overflow. > I don't consider this a problem, I just thought I'd mention it in > case someone else runs into it. > > > Regards, > > Tim Hochberg > > > > > > > > ------------------------------------------------------- > This SF.Net email sponsored by: ApacheCon 2003, > 16-19 November in Las Vegas. Learn firsthand the latest > developments in Apache, PHP, Perl, XML, Java, MySQL, > WebDAV, and more! http://www.apachecon.com/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- Todd Miller Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21030 (410) 338 - 4576 From magnus at hetland.org Mon Nov 10 09:51:04 2003 From: magnus at hetland.org (Magnus Lie Hetland) Date: Mon Nov 10 09:51:04 2003 Subject: [Numpy-discussion] Notes on transitioning to numarray. In-Reply-To: <3FAFC226.3090802@ieee.org> References: <3FAFBC52.2000206@ieee.org> <3FAFC226.3090802@ieee.org> Message-ID: <20031110175006.GB9947@idi.ntnu.no> Tim Hochberg : > > Tim Hochberg wrote: > > >[There is a html version of this not at > >http://starship.python.net/~hochberg/conversion.html] > > [grrr] That should of course be 'this note'. Heh. Yeah, I thought the informational content in the statement was a bit low ;) > -tim -- Magnus Lie Hetland "In this house we obey the laws of http://hetland.org thermodynamics!" Homer Simpson From tim.hochberg at ieee.org Mon Nov 10 10:11:09 2003 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Mon Nov 10 10:11:09 2003 Subject: [Numpy-discussion] Notes on transitioning to numarray. [Addendum] In-Reply-To: <3FAFBC52.2000206@ieee.org> References: <3FAFBC52.2000206@ieee.org> Message-ID: <3FAFD495.4000600@ieee.org> Here's one more thing I just ran into. It's related to over/underflow handling, so it's even topical. I think that the _pushmodes method of NumError (in unfunc.py) should be using _gemodestack(0), not getmodestack(1). That is:: def _pushmodes(self, modes): l = self._getmodestack(0) # XXX changed be TAH l.append(modes) What can happen now is: - New thread is started. - pushMode is called before any calls to getMode, resulting in a modestack one deep. - the corresponding popMode is called, resulting in an illegal stack that is zero deep. - there is an over/underflow and getMode fails because of the zero length stack. While I'm on the subject, I think that the way the default stack entry is created should be changed. Currently there is no way to assign a default error mode for all threads. The simplest approach would be add a setDefaultMode method to Error. Error._defaultmode would be set in __init__ to _NumErrorMode(), but could subsequently be re-set with setDefaultMode. _defaultmode would be used to initialize the default value in _getmodestack. Specifically: class NumError: def __init__(self, all="warn", overflow=None, underflow=None, dividebyzero=None, invalid=None): self._defaultmode = _NumErrorMode() self._modestack = {} # map of stacks indexed by thread id self.setMode(all=all, underflow=underflow, overflow=overflow, dividebyzero=dividebyzero, invalid=invalid) def _getmodestack(self, empty_default=0): id = safethread.get_ident() try: l = self._modestack[id] except KeyError: if empty_default: l = [] else: l = [self._defaultmode ] self._modestack[id] = l return l def setDefaultMode(self, all="warn", overflow=None, underflow=None, dividebyzero=None, invalid=None): self._defaultmode = _NumErrorMode(all, overflow, underflow, dividebyzero, invalid) ) #.... While I'm at it, what's the point of empty_default? I can't figure out when it would be useful. -tim From tchur at optushome.com.au Mon Nov 10 10:22:05 2003 From: tchur at optushome.com.au (Tim Churches) Date: Mon Nov 10 10:22:05 2003 Subject: [Numpy-discussion] NaNs and infs In-Reply-To: References: Message-ID: <1068488345.1857.7.camel@emilio> On Tue, 2003-11-11 at 03:26, Perry Greenfield wrote: > Tim Churches wrote: > > > Increasingly we are turning to R (http://www.r-project.org) because it > > provides fully integrated support for Inf, NaN and NA (missing) in all > > its data structures, including matrices and higher-rank arrays, viz: > > > > > 2/0 > > [1] Inf > > > 0/0 > > [1] NaN > > > NA/1 > > [1] NA > > > > R can be embedded in Python and passed NumPy arrays or other data > > structures via Walter Moriera's excellent RPy modules (see > > http;//rpy.sf.net). > > > > However, the fundamental problem is that support for NA (missing), Inf > > and NaN seems to be afterthoughts in both NumPy and numarray, whereas > > such support should have been integrated into the design right from the > > outset. > > > > Is it too late to incorporate them into the fabric of numarray? > > > I'm a little surprised at the comment that these issues are viewed as > afterthoughts in numarray. I was being more than a little unfair to both NumPy and numarray. Unfair to NumPy because support for these things was considered, but deliberately excluded for sound performance reasons. And unfair to numarray because I haven't really studied what it does differently in these respects to NumPy. I'll now roll up my sleeves and take a good look at what numarray does before commenting further. Tim C > We gave ieee 754 special value handling > fairly careful thought. We may have missed something however, so I'm > interested to hear what is considered missing in that regard. We allow > the user to set whether ieee errors are ignored, print a warning, or > raise an exception, and do so individually for all four kinds of > errors. We also allow testing of and setting ieee special values. > What is missing? (By the way, if one expects to be able to invoke > an error handler for each individual exception that occurs in an array > computation, we determined that that was too difficult to handle in any, > portable and efficient way.) > > As to "NA" or "missing values", what is expected? These are not part of > the ieee standard (unless I've missed something). How does this differ > from using NaN, for example? > > Perry > > > ------------------------------------------------------- > This SF.Net email sponsored by: ApacheCon 2003, > 16-19 November in Las Vegas. Learn firsthand the latest > developments in Apache, PHP, Perl, XML, Java, MySQL, > WebDAV, and more! http://www.apachecon.com/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- Tim C PGP/GnuPG Key 1024D/EAF993D0 available from keyservers everywhere or at http://members.optushome.com.au/tchur/pubkey.asc Key fingerprint = 8C22 BF76 33BA B3B5 1D5B EB37 7891 46A9 EAF9 93D0 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From jmiller at stsci.edu Mon Nov 10 10:56:04 2003 From: jmiller at stsci.edu (Todd Miller) Date: Mon Nov 10 10:56:04 2003 Subject: [Numpy-discussion] Notes on transitioning to numarray. In-Reply-To: <3FAFD651.1020601@ieee.org> References: <3FAFBC52.2000206@ieee.org> <1068485784.11424.19.camel@halloween.stsci.edu> <3FAFD651.1020601@ieee.org> Message-ID: <1068490474.11424.50.camel@halloween.stsci.edu> On Mon, 2003-11-10 at 13:17, Tim Hochberg wrote: > Ah. I was sort of under the impression that the bugs database for > Numeric/.numarray on SourceForge was non active. Probably because of all > the Numeric bugs that have been sitting there for years. Next time I'll > go ahead and enter the bug reports there as well. There's a separate tracker called "Numarray Bugs" which is only visible on the "Trackers" page. In contrast, "Bugs" is visible on every page. Bugs filed in "Bugs" are for Numeric, but can now be re-assigned to "Numarray Bugs". There's only one e-mail address for "Bugs" and it's currently pointing to me; this is only so I can re-assign the numarray related bugs to numarray; I don't usually (ever?) work on Numeric. Regards, Todd -- Todd Miller Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21030 (410) 338 - 4576 From kxnzjpt at iopus.com Thu Nov 13 01:47:04 2003 From: kxnzjpt at iopus.com (Willa Marquez) Date: Thu Nov 13 01:47:04 2003 Subject: [Numpy-discussion] Italian-crafted Rolex - only $65 - $140 - Free SHIPPING! zlqa xzc Message-ID: <7tze-$338$666@xk1.ts5uaj> please note to send ALL REPLY e-mail direct to our Sales Representative at: Questions at BargainWatches.biz Hi, Thank you for expressing interest in ATGWS watches. We would like to take this opportunity to offer you our fine selection of Italian crafted Rolex Timepieces. You can view our large selection of Rolexes (including Breitling, Tag Heuer, Cartier etc) at: http://www.BargainWatches.biz For all orders placed in the month of November, all shipping and handling charges will be free. As we are the direct manufacturers, you are guaranteed of lowest prices and highest quality each and every time you purchase from us. You may also be interested to know that we have the following brands available in our wide selection as well: 1. Rolex 2. Blancpain 3. Fortis 4. Jaeger LeCoutre 5. Longines 6. Mont Blanc 7. Movado 8. Oris 9. Roger Dubuis 10. Ulysse 11. Zenith 12. Audemar Piguet 13. Breitling 14. Bvglari 15. Cartier 16. Corum 17. Dunhill 18. Franck Muller 19. Gerard Perregaux 20. IWC 21. IWC 22. Panerai 23. Patek Philippe 24. Tag Heuer 25. Vacheron Constantin If you see anything that might interest you, or if you have any questions, please don't hesitate to visit our website at: http://www.BargainWatches.biz I certainly look forward to hearing from you. Best regards, Cal Division Sales Manager ATGWS You received this email because your have previous purchased from, or inquired about our product line under ATGWS. If you do not want to receive further mailings from ATGWS, unsubscribe by sending an email with the title heading: DELETE in the subject line to Unsubscribe at bargainwatches.biz please note to send ALL REPLY e-mail direct to our Sales Representative at: Questions at BargainWatches.biz cb epql zyjcjoqwmuchaun From wa317msi at yahoo.com Fri Nov 14 01:41:06 2003 From: wa317msi at yahoo.com (Bernie Mobley) Date: Fri Nov 14 01:41:06 2003 Subject: [Numpy-discussion] AMERICAN STOCK MARKET: TRHL Retains Sky Investor Relations...dempster Message-ID: American Stock Market - Press Release... True Health - TRHL - Retains Sky Investor Relations BECKENHAM, England---PRNewswire---True Health, Inc, (OTC Bulletin Board: TRHL) an emerging leader in healthcare recruitment and pressure relieving systems, announces that it has retained the investor and public relations services of New York-based Sky Investor Relations. Read the entire news release: http://biz.yahoo.com/prnews/031112/lnw017_1.html c udh yeweekc nrisicgb uvspt w zzezcwzmllz s sp qvdf i h ggd iy zepqhkh frd From bens at MIT.EDU Sun Nov 16 19:53:02 2003 From: bens at MIT.EDU (Benjamin Schwartz) Date: Sun Nov 16 19:53:02 2003 Subject: [Numpy-discussion] Multiple assignment confusion Message-ID: <3FB845A8.1070007@mit.edu> I am trying to interchange two pieces of an array. Here is my demo code: >>> from Numeric import * >>> a=array([.7,.1,.1,.1]) >>> a=multiply.outer(a,a) >>> a=multiply.outer(a,a) >>> sum(sum(sum(sum(a)))) 1.0000000000000002 >>> a[1,0],a[1,1]=a[1,1],a[1,0] >>> sum(sum(sum(sum(a)))) 0.93999999999999972 So a is a four-dimensional array that sums to 1, but when I attempt to switch two 2-d sub-arrays of it I just end up duplicating one of them, shown by a change in the value of the sum (to .94). Is this behavior deliberate? It certainly seems to break Python's tuple-assignment model. What I would ideally like to do is perform this swap without more than two-elements'-worth of temporary storage. I.e. perform it as a pure reference swap or switch each pair of elements one by one, reusing temporary space. Is this possible with Numeric (or Numarray?)? Curious, Ben From perry at stsci.edu Mon Nov 17 08:44:23 2003 From: perry at stsci.edu (Perry Greenfield) Date: Mon Nov 17 08:44:23 2003 Subject: [Numpy-discussion] Multiple assignment confusion In-Reply-To: <3FB845A8.1070007@mit.edu> Message-ID: Benjamin Schwartz wrote: > I am trying to interchange two pieces of an array. Here is my demo code: > > >>> from Numeric import * > >>> a=array([.7,.1,.1,.1]) > >>> a=multiply.outer(a,a) > >>> a=multiply.outer(a,a) > >>> sum(sum(sum(sum(a)))) > 1.0000000000000002 > >>> a[1,0],a[1,1]=a[1,1],a[1,0] > >>> sum(sum(sum(sum(a)))) > 0.93999999999999972 > > So a is a four-dimensional array that sums to 1, but when I attempt to > switch two 2-d sub-arrays of it I just end up duplicating one of them, > shown by a change in the value of the sum (to .94). Is this behavior > deliberate? It certainly seems to break Python's tuple-assignment model. > > What I would ideally like to do is perform this swap without more than > two-elements'-worth of temporary storage. I.e. perform it as a pure > reference swap or switch each pair of elements one by one, reusing > temporary space. Is this possible with Numeric (or Numarray?)? > > Curious, > Ben > I believe that this is simply a reflection of the fact that subarrays (for Numeric and numarray) are views into the original array rather than copies. A simpler example illustrates the same problem: >>> a = reshape(arange(4), (2,2)) >>> a array([[0, 1], [2, 3]]) >>> a[0],a[1]=a[1],a[0] >>> a array([[2, 3], [2, 3]]) >>> Even though the right side of the assignment is stored effectively in temporary variables, what is stored are views into the original array, not copies of the subarrays. When the first assignment is made to a[0] (=a[1]), the data in the reference a[0] on the right hand side has changed as well since it views exactly the same data. When it comes time to set a[1] it will be copying from the updated a[0] and thus the duplication. This doesn't happen for a list since one is essentially rebinding the element in a list to another object. In other words, for this case: >>> b = [[0,1],[2,3]] >>> b[0],b[1]=b[1],b[0] >>> b [[2, 3], [0, 1]] >>> assigning to b[0] is rebinding that list element away from the list [0,1] to the list [2,3]; changing what the elements b refer to in no way changes the contents of either of the lists that b contained. Numeric arrays have a coupling across dimensions that doesn't exist for lists since nested lists are analogous to arrays of pointers to pointers whereas Numeric arrays are just multidimensional arrays with no indirection involved. The above behavior could be avoided if subarrays always produced copies. This issue was given a great deal of discussion and thought for numarray and for various reasons it was decided not to change the behavior of slices and subarray references. As to whether it is possible to switch subarray contents with only using one or two temporary storage elements efficiently, offhand I don't believe so, but I haven't given it much thought (and don't have the time to) but perhaps someone else knows of an existing mechanism to do so. Nothing prevents writing an extension to provide such capability. Perry Greenfield From haase at msg.ucsf.edu Mon Nov 17 15:48:10 2003 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Mon Nov 17 15:48:10 2003 Subject: [Numpy-discussion] numarray: records using memmap Message-ID: <013801c3ad65$35fa15b0$421ee6a9@rodan> Hi, Is it possible to create a record-array using a memmap (file) as buffer ? Thanks, Sebastian Haase From cjw at sympatico.ca Mon Nov 17 19:49:03 2003 From: cjw at sympatico.ca (Colin J. Williams) Date: Mon Nov 17 19:49:03 2003 Subject: [Numpy-discussion] teacup.py crash Message-ID: <3FB9967F.1010309@sympatico.ca> Running teacup.py, we get: Traceback (most recent call last): File "C:\Program Files\Python23\Lib\site-packages\numarray\teacup.py", line 1, in ? import numarray.libteacup ImportError: No module named libteacup There is no mention of teacup in numarray-0.6.pdf. Can this module be safely be deleted? Colin W From hsu at stsci.edu Mon Nov 17 20:25:02 2003 From: hsu at stsci.edu (JC Hsu) Date: Mon Nov 17 20:25:02 2003 Subject: [Numpy-discussion] Re: creating record arrays using memmap Message-ID: <1b0b7c08.e482203e.8166500@comet.stsci.edu> Yes, it is possible to create a record array with memmap. The trick is to use the constructor, instead of the "array" function: >>> from numarray import memmap, records >>> m=memmap.open('xxx.txt') >>> r2=records.RecArray(m[0:18],formats='i2,f4',shape=3) Notice that you also need to "calculate" the slice (you must use the slice even if you are mapping the whole file). JC Hsu From leo at lspace.org Tue Nov 18 10:40:10 2003 From: leo at lspace.org (Leo Breebaart) Date: Tue Nov 18 10:40:10 2003 Subject: [Numpy-discussion] Problems creating a numarray of type PyArray_OBJECT Message-ID: <20031118183413.GA25090@falcon.pds.twi.tudelft.nl> Hi all, Using the numarray C API, I am trying to create an array containing Python strings from an existing C (char **) array. In Numeric 23.1, the following code fragment works fine: PyObject *pdata = NULL; // num_elements has been defined earlier PyStringObject **strarray = malloc(num_elements * sizeof(PyStringObject *)); if (strarray != NULL) { int e; for (e = 0; e < num_elements; e++) { strarray[e] = (PyStringObject *) PyString_FromString(((char **)data)[e]); } // num_dims and the dim array have been defined earlier pdata = PyArray_FromDimsAndData(num_dims, dim, PyArray_OBJECT, (char *)strarray); } In numarray 0.7, the same code yields the runtime error: libnumarray.error: Type object lookup returned NULL for type 14 in the PyArray_FromDimsAndData() call. Using the current numarray CVS head instead of the 0.7 release makes no difference. Is there somebody who knows what is going or what I am doing wrong? Many thanks in advance, -- Leo Breebaart From jmiller at stsci.edu Tue Nov 18 11:05:01 2003 From: jmiller at stsci.edu (Todd Miller) Date: Tue Nov 18 11:05:01 2003 Subject: [Numpy-discussion] Problems creating a numarray of type PyArray_OBJECT In-Reply-To: <20031118183413.GA25090@falcon.pds.twi.tudelft.nl> References: <20031118183413.GA25090@falcon.pds.twi.tudelft.nl> Message-ID: <1069182221.3461.2.camel@localhost.localdomain> On Tue, 2003-11-18 at 13:34, Leo Breebaart wrote: > Hi all, > > Using the numarray C API, I am trying to create an array > containing Python strings from an existing C (char **) array. > > In Numeric 23.1, the following code fragment works fine: > > PyObject *pdata = NULL; > // num_elements has been defined earlier > PyStringObject **strarray = malloc(num_elements * sizeof(PyStringObject *)); > if (strarray != NULL) > { > int e; > for (e = 0; e < num_elements; e++) > { > strarray[e] = (PyStringObject *) PyString_FromString(((char **)data)[e]); > } > > // num_dims and the dim array have been defined earlier > pdata = PyArray_FromDimsAndData(num_dims, dim, PyArray_OBJECT, > (char *)strarray); > } > > In numarray 0.7, the same code yields the runtime error: > > libnumarray.error: Type object lookup returned NULL for type 14 > > in the PyArray_FromDimsAndData() call. > > Using the current numarray CVS head instead of the 0.7 release > makes no difference. > > Is there somebody who knows what is going or what I am doing > wrong? Yes and no. You're in an area that numarray supports differently than Numeric. numarray's support for character arrays is in the Python numarray.strings module. There's no direct support for using it from C. > > Many thanks in advance, You're welcome, Todd -- Todd Miller From leo at lspace.org Wed Nov 19 02:28:02 2003 From: leo at lspace.org (Leo Breebaart) Date: Wed Nov 19 02:28:02 2003 Subject: [Numpy-discussion] Problems creating a numarray of type PyArray_OBJECT In-Reply-To: <1069182221.3461.2.camel@localhost.localdomain> References: <20031118183413.GA25090@falcon.pds.twi.tudelft.nl> <1069182221.3461.2.camel@localhost.localdomain> Message-ID: <20031119102708.GA1107@falcon.pds.twi.tudelft.nl> Todd Miller wrote: > Leo Breebaart wrote: > > > [...] > > pdata = PyArray_FromDimsAndData(num_dims, dim, PyArray_OBJECT, > > (char *)strarray); > > } > > > > In numarray 0.7, the same code yields the runtime error: > > > > libnumarray.error: Type object lookup returned NULL for type 14 > > > > in the PyArray_FromDimsAndData() call. [...] > > > Is there somebody who knows what is going or what I am doing > > wrong? > > Yes and no. You're in an area that numarray supports differently than > Numeric. numarray's support for character arrays is in the Python > numarray.strings module. There's no direct support for using it from > C. Thanks for the response, Todd. Do I understand correctly that you are saying it's no longer possible in numarray (as opposed to Numeric) to create *any* kind of PyObject array? (After all, the fact that the PyObjects in the above example are really strings is basically accidental -- what if I had an array of pointers to e.g. PyLists or PyDicts that I wanted to store in a numarray -- in Numeric that would also be possible using the PyArray_OBJECT type, right?) Am I also correct in thinking that at this point the only solution to my problem is to manually convert the string arrays in my data to direct Python lists of strings -- or is there perhaps some other trick by which I could get them into a numarray after all? I do a certain amount of Python-level 'post-processing' in my code anyway, and the string arrays I encounter are expected to have only a few elements each, so it would be no problem to do some munging after the fact, if that could help matters. -- Leo Breebaart From jmiller at stsci.edu Wed Nov 19 03:00:03 2003 From: jmiller at stsci.edu (Todd Miller) Date: Wed Nov 19 03:00:03 2003 Subject: [Numpy-discussion] Problems creating a numarray of type PyArray_OBJECT In-Reply-To: <20031119102708.GA1107@falcon.pds.twi.tudelft.nl> References: <20031118183413.GA25090@falcon.pds.twi.tudelft.nl> <1069182221.3461.2.camel@localhost.localdomain> <20031119102708.GA1107@falcon.pds.twi.tudelft.nl> Message-ID: <1069239542.3458.58.camel@localhost.localdomain> On Wed, 2003-11-19 at 05:27, Leo Breebaart wrote: > Todd Miller wrote: > > > Leo Breebaart wrote: > > > > > [...] > > > pdata = PyArray_FromDimsAndData(num_dims, dim, PyArray_OBJECT, > > > (char *)strarray); > > > } > > > > > > In numarray 0.7, the same code yields the runtime error: > > > > > > libnumarray.error: Type object lookup returned NULL for type 14 > > > > > > in the PyArray_FromDimsAndData() call. [...] > > > > > Is there somebody who knows what is going or what I am doing > > > wrong? > > > > Yes and no. You're in an area that numarray supports differently than > > Numeric. numarray's support for character arrays is in the Python > > numarray.strings module. There's no direct support for using it from > > C. > > Thanks for the response, Todd. > > Do I understand correctly that you are saying it's no longer > possible in numarray (as opposed to Numeric) to create *any* kind > of PyObject array? No. numarray-0.7 added numarray.objects which provides basic support for arrays of arbitrary objects: indexing, slicing, ufuncs, etc. Access from C, however, is not directly supported. Internally, numarray.objects manages objects using a Python list so it is important to respect the Python object interface and not try to directly munge the data buffer as you might with numerical arrays. > (After all, the fact that the PyObjects in the > above example are really strings is basically accidental -- what > if I had an array of pointers to e.g. PyLists or PyDicts that I > wanted to store in a numarray -- in Numeric that would also be > possible using the PyArray_OBJECT type, right?) If, in Python, you had a list of dicts, you could create a numarray object array from it. If, in C, you had an array of pointers to dicts, you could first turn it into a list of dicts, and then call back into Python to create an object array using numarray.objects.fromlist(). > Am I also correct in thinking that at this point the only > solution to my problem is to manually convert the string arrays > in my data to direct Python lists of strings -- or is there > perhaps some other trick by which I could get them into a > numarray after all? It's possible to call back into Python to create an empty CharArray, and then to fill it in C. This is more straight forward for CharArrays because the data is stored in the same sort of buffer as for numerical arrays, and unlike object arrays, no reference counting is required. Regards, Todd > -- > Todd Miller From ariciputi at pito.com Wed Nov 19 08:26:17 2003 From: ariciputi at pito.com (Andrea Riciputi) Date: Wed Nov 19 08:26:17 2003 Subject: [Numpy-discussion] Help with PyArrayObject and C Arrays. Message-ID: Hi, I've written some C code for number crunching that uses the following structure for simulating 2-dimensional arrays (i.e. matrices): typedef struct { double **aa; int nrows; int ncols; } Array2d; In order to read these C arrays in Python+Numeric and vice versa, I've written some wrapping code, but I'm not so sure it's correct. In particular I'm concerned in memory leakage, I don't know if I've understood correctly how PyArray_FromDimsAndData works and if I've choose the best way to solve the problem. My code follows. Thanks for help! Andrea. C --> Python ============ static PyObject *Array2d_2_PyArrayObject(Array2d *input) { int ndims[2]; PyArrayObject *result; if (!input) return NULL; ndims[0] = input->nrows; ndims[1] = input->ncols; result = (PyArrayObject *) PyArray_FromDimsAndData(2, ndims, PyArray_DOUBLE, (char *) input->aa[0]); free(input->aa); free(input); return PyArray_Return(result); } Python --> C ============ static Array2d *PyList_2_Array2d(PyObject *input) { Array2d *result; if (!PySequence_Check(input)) { PyErr_SetString(PyExc_TypeError,"Expected a sequence"); return NULL; } result = (Array2d *) malloc(sizeof(Array2d)); PY_CHECK_ALLOC(result); if (PyArray_As2D((PyObject **) &input, (char ***) &(result->aa), &(result->nrows), &(result->ncols), PyArray_DOUBLE) != 0) { PyErr_SetString(PyExc_MemoryError,"No heap space for data"); return NULL; } return result; } --- Andrea Riciputi "Science is like sex: sometimes something useful comes out, but that is not the reason we are doing it" -- (Richard Feynman) From 6kgwcl at datanet.hu Wed Nov 19 09:48:06 2003 From: 6kgwcl at datanet.hu (Camille Flood) Date: Wed Nov 19 09:48:06 2003 Subject: [Numpy-discussion] Total Relief From Credit Card Debt!! Message-ID: An HTML attachment was scrubbed... URL: From edcjones at erols.com Wed Nov 19 19:19:06 2003 From: edcjones at erols.com (Edward C. Jones) Date: Wed Nov 19 19:19:06 2003 Subject: [Numpy-discussion] Wrong error message Message-ID: <3FBC3192.2020506@erols.com> Here is the program: #! /usr/bin/env python import numarray arr1 = numarray.zeros([3,2,1]) arr2 = numarray.zeros([3,2,1]) arr2[0] = arr1 and here is the error message: TypeError: _copyFromAndConvert: non-numarray input. Should this program give this error message? From jmiller at stsci.edu Thu Nov 20 07:09:08 2003 From: jmiller at stsci.edu (Todd Miller) Date: Thu Nov 20 07:09:08 2003 Subject: [Numpy-discussion] Wrong error message In-Reply-To: <3FBC3192.2020506@erols.com> References: <3FBC3192.2020506@erols.com> Message-ID: <1069340772.5225.66.camel@halloween.stsci.edu> On Wed, 2003-11-19 at 22:14, Edward C. Jones wrote: > Here is the program: > > #! /usr/bin/env python > import numarray > arr1 = numarray.zeros([3,2,1]) > arr2 = numarray.zeros([3,2,1]) > arr2[0] = arr1 > > and here is the error message: > > TypeError: _copyFromAndConvert: non-numarray input. > > Should this program give this error message? No. I agree that's a pretty bad message. Here's what CVS does now: Traceback (most recent call last): .. ValueError: Can't use non-rank-0 array as a scalar. Even the new message is a little obscure; feel free to suggest improvements... Regards, Todd > > > ------------------------------------------------------- > This SF.net email is sponsored by: SF.net Giveback Program. > Does SourceForge.net help you be more productive? Does it > help you create better code? SHARE THE LOVE, and help us help > YOU! Click Here: http://sourceforge.net/donate/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- Todd Miller Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21030 (410) 338 - 4576 From jmiller at stsci.edu Thu Nov 20 13:33:02 2003 From: jmiller at stsci.edu (Todd Miller) Date: Thu Nov 20 13:33:02 2003 Subject: [Numpy-discussion] teacup.py crash In-Reply-To: <3FB9967F.1010309@sympatico.ca> References: <3FB9967F.1010309@sympatico.ca> Message-ID: <1069363850.6119.111.camel@halloween.stsci.edu> On Mon, 2003-11-17 at 22:48, Colin J. Williams wrote: > Running teacup.py, we get: > > Traceback (most recent call last): > File "C:\Program > Files\Python23\Lib\site-packages\numarray\teacup.py", line 1, in ? > import numarray.libteacup > ImportError: No module named libteacup > > There is no mention of teacup in numarray-0.6.pdf. teacup is a partially functional module for measuring microsecond level timing under Linux. It doesn't work yet, but it's better than starting from scratch, so I keep it in the distribution. At some point, it'll either work or disappear. > Can this module be safely be deleted? Yes. > Colin W > > > > ------------------------------------------------------- > This SF. Net email is sponsored by: GoToMyPC > GoToMyPC is the fast, easy and secure way to access your computer from > any Web browser or wireless device. Click here to Try it Free! > https://www.gotomypc.com/tr/OSDN/AW/Q4_2003/t/g22lp?Target=mm/g22lp.tmpl > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- Todd Miller Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21030 (410) 338 - 4576 From jmiller at stsci.edu Thu Nov 20 14:05:03 2003 From: jmiller at stsci.edu (Todd Miller) Date: Thu Nov 20 14:05:03 2003 Subject: [Numpy-discussion] Community Poll: numarray default underflow handling == "ignore" ? Message-ID: <1069365809.5225.145.camel@halloween.stsci.edu> Someone recently suggested that we change the default for numarray underflow checking to "ignore". Please vote (+1, 0, -1): Todd -- Todd Miller Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21030 (410) 338 - 4576 From Chris.Barker at noaa.gov Thu Nov 20 15:21:03 2003 From: Chris.Barker at noaa.gov (Chris Barker) Date: Thu Nov 20 15:21:03 2003 Subject: [Numpy-discussion] Community Poll: numarray default underflow handling == "ignore" ? References: <1069365809.5225.145.camel@halloween.stsci.edu> Message-ID: <3FBD4BD4.FD1F30B9@noaa.gov> Todd Miller wrote: > Someone recently suggested that we change the default for numarray > underflow checking to "ignore". Please vote (+1, 0, -1): What is it currently, and what other options are there? -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From jsaenz at wm.lc.ehu.es Fri Nov 21 00:25:02 2003 From: jsaenz at wm.lc.ehu.es (Jon Saenz) Date: Fri Nov 21 00:25:02 2003 Subject: [Numpy-discussion] Community Poll: numarray default underflow handling == "ignore" ? In-Reply-To: <1069365809.5225.145.camel@halloween.stsci.edu> Message-ID: +1 Jon Saenz. | Tfno: +34 946012445 Depto. Fisica Aplicada II | Fax: +34 944648500 Facultad de Ciencias. \\ Universidad del Pais Vasco \\ Apdo. 644 \\ 48080 - Bilbao \\ SPAIN On 20 Nov 2003, Todd Miller wrote: > Someone recently suggested that we change the default for numarray > underflow checking to "ignore". Please vote (+1, 0, -1): > > Todd > -- > Todd Miller > Space Telescope Science Institute > 3700 San Martin Drive > Baltimore MD, 21030 > (410) 338 - 4576 > > > > ------------------------------------------------------- > This SF.net email is sponsored by: SF.net Giveback Program. > Does SourceForge.net help you be more productive? Does it > help you create better code? SHARE THE LOVE, and help us help > YOU! Click Here: http://sourceforge.net/donate/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From jmiller at stsci.edu Fri Nov 21 05:37:06 2003 From: jmiller at stsci.edu (Todd Miller) Date: Fri Nov 21 05:37:06 2003 Subject: [Numpy-discussion] Community Poll: numarray default underflow handling == "ignore" ? In-Reply-To: <3FBD4BD4.FD1F30B9@noaa.gov> References: <1069365809.5225.145.camel@halloween.stsci.edu> <3FBD4BD4.FD1F30B9@noaa.gov> Message-ID: <1069421715.7448.2.camel@halloween.stsci.edu> On Thu, 2003-11-20 at 18:18, Chris Barker wrote: > Todd Miller wrote: > > Someone recently suggested that we change the default for numarray > > underflow checking to "ignore". Please vote (+1, 0, -1): > > What is it currently, and what other options are there? The default of underflow is "warn" Other options are: "ignore" or "raise" > > -Chris > > > -- > Christopher Barker, Ph.D. > Oceanographer > > NOAA/OR&R/HAZMAT (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > > ------------------------------------------------------- > This SF.net email is sponsored by: SF.net Giveback Program. > Does SourceForge.net help you be more productive? Does it > help you create better code? SHARE THE LOVE, and help us help > YOU! Click Here: http://sourceforge.net/donate/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- Todd Miller Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21030 (410) 338 - 4576 From pearu at cens.ioc.ee Fri Nov 21 06:13:08 2003 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Fri Nov 21 06:13:08 2003 Subject: [Numpy-discussion] Community Poll: numarray default underflow handling == "ignore" ? In-Reply-To: <1069421715.7448.2.camel@halloween.stsci.edu> Message-ID: On 21 Nov 2003, Todd Miller wrote: > On Thu, 2003-11-20 at 18:18, Chris Barker wrote: > > Todd Miller wrote: > > > Someone recently suggested that we change the default for numarray > > > underflow checking to "ignore". Please vote (+1, 0, -1): > > > > What is it currently, and what other options are there? > > The default of underflow is "warn" > > Other options are: "ignore" or "raise" My votes: warn: 0 ignore: 0 raise: -1 Pearu From a.schmolck at gmx.net Fri Nov 21 06:59:11 2003 From: a.schmolck at gmx.net (Alexander Schmolck) Date: Fri Nov 21 06:59:11 2003 Subject: [Numpy-discussion] Community Poll: numarray default underflow handling == "ignore" ? In-Reply-To: <1069421715.7448.2.camel@halloween.stsci.edu> References: <1069365809.5225.145.camel@halloween.stsci.edu> <3FBD4BD4.FD1F30B9@noaa.gov> <1069421715.7448.2.camel@halloween.stsci.edu> Message-ID: Todd Miller writes: > On Thu, 2003-11-20 at 18:18, Chris Barker wrote: > > Todd Miller wrote: > > > Someone recently suggested that we change the default for numarray > > > underflow checking to "ignore". Please vote (+1, 0, -1): > > > > What is it currently, and what other options are there? > > The default of underflow is "warn" I much prefer the current default. Having (potential errors) ignored silently is IMHO against the spirit of python. What's the problem with manually setting "ignore underflow" if and when one knows that underflows might occur but can safely be ignored? 'as From jmiller at stsci.edu Fri Nov 21 07:45:02 2003 From: jmiller at stsci.edu (Todd Miller) Date: Fri Nov 21 07:45:02 2003 Subject: [Numpy-discussion] Community Poll: numarray default underflow handling == "ignore" ? In-Reply-To: References: <1069365809.5225.145.camel@halloween.stsci.edu> <3FBD4BD4.FD1F30B9@noaa.gov> <1069421715.7448.2.camel@halloween.stsci.edu> Message-ID: <1069429373.7448.4.camel@halloween.stsci.edu> On Fri, 2003-11-21 at 09:58, Alexander Schmolck wrote: > Todd Miller writes: > > > On Thu, 2003-11-20 at 18:18, Chris Barker wrote: > > > Todd Miller wrote: > > > > Someone recently suggested that we change the default for numarray > > > > underflow checking to "ignore". Please vote (+1, 0, -1): > > > > > > What is it currently, and what other options are there? > > > > The default of underflow is "warn" > > I much prefer the current default. Having (potential errors) ignored silently > is IMHO against the spirit of python. What's the problem with manually setting > "ignore underflow" if and when one knows that underflows might occur but can > safely be ignored? I'll read that as -1. :-) Todd > > 'as > > > > ------------------------------------------------------- > This SF.net email is sponsored by: SF.net Giveback Program. > Does SourceForge.net help you be more productive? Does it > help you create better code? SHARE THE LOVE, and help us help > YOU! Click Here: http://sourceforge.net/donate/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- Todd Miller Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21030 (410) 338 - 4576 From Chris.Barker at noaa.gov Fri Nov 21 08:53:06 2003 From: Chris.Barker at noaa.gov (Chris Barker) Date: Fri Nov 21 08:53:06 2003 Subject: [Numpy-discussion] Community Poll: numarray default underflow handling == "ignore" ? References: <1069365809.5225.145.camel@halloween.stsci.edu> <3FBD4BD4.FD1F30B9@noaa.gov> <1069421715.7448.2.camel@halloween.stsci.edu> Message-ID: <3FBE4262.BEF5DEE4@noaa.gov> Todd Miller wrote: > > > Someone recently suggested that we change the default for numarray > > > underflow checking to "ignore". Please vote (+1, 0, -1): > The default of underflow is "warn" assuming that ignore means that zero will be used, I vote +1 If the difference between zero and the smallest floating point number is significant in your app, you'd better know what you're doing with floating point, and you should then know to set the check to "warn" or "raise". -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From fperez at colorado.edu Fri Nov 21 08:57:12 2003 From: fperez at colorado.edu (Fernando Perez) Date: Fri Nov 21 08:57:12 2003 Subject: [Numpy-discussion] Community Poll: numarray default underflow handling == "ignore" ? In-Reply-To: <3FBE4262.BEF5DEE4@noaa.gov> References: <1069365809.5225.145.camel@halloween.stsci.edu> <3FBD4BD4.FD1F30B9@noaa.gov> <1069421715.7448.2.camel@halloween.stsci.edu> <3FBE4262.BEF5DEE4@noaa.gov> Message-ID: <3FBE43BD.3080100@colorado.edu> Chris Barker wrote: > Todd Miller wrote: > >>>>Someone recently suggested that we change the default for numarray >>>>underflow checking to "ignore". Please vote (+1, 0, -1): > > >>The default of underflow is "warn" > > > assuming that ignore means that zero will be used, I vote +1 > > If the difference between zero and the smallest floating point number is > significant in your app, you'd better know what you're doing with > floating point, and you should then know to set the check to "warn" or > "raise". +1 also from me (also as Chris, assuming 'ignore' means replacing with zeros and not with NaNs). Underflow is about the only case of errors where I think that silent ignore is a reasonable (if 'unpythonic') policy. Cheers, f From haase at msg.ucsf.edu Fri Nov 21 09:10:05 2003 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Fri Nov 21 09:10:05 2003 Subject: [Numpy-discussion] how-to "put" RAM-based numarray into memmap Message-ID: <047301c3b052$37f38960$421ee6a9@rodan> Hi, Suppose I have a 500MB-ram Computer and a 300MB ram-only (standard) numarray. Now I would like to "save" that onto harddrive (with a small header up front - which I know how to do as a memmap record) Can I somehow "append" a (standard) numarray to a memmap "as a memmap-slice" ? I obviously could create the corresponding memslice fresh and then assign the original data elementwise - but that would allocate extra (virtual) memory... Thanks, Sebastian Haase From cjw at sympatico.ca Fri Nov 21 09:21:08 2003 From: cjw at sympatico.ca (Colin J. Williams) Date: Fri Nov 21 09:21:08 2003 Subject: [Numpy-discussion] Community Poll: numarray default underflow handling == "ignore" ? In-Reply-To: <1069429373.7448.4.camel@halloween.stsci.edu> References: <1069365809.5225.145.camel@halloween.stsci.edu> <3FBD4BD4.FD1F30B9@noaa.gov> <1069421715.7448.2.camel@halloween.stsci.edu> <1069429373.7448.4.camel@halloween.stsci.edu> Message-ID: <3FBE4965.9050107@sympatico.ca> Todd Miller wrote: >On Fri, 2003-11-21 at 09:58, Alexander Schmolck wrote: > > >>Todd Miller writes: >> >> >> >>>On Thu, 2003-11-20 at 18:18, Chris Barker wrote: >>> >>> >>>>Todd Miller wrote: >>>> >>>> >>>>>Someone recently suggested that we change the default for numarray >>>>>underflow checking to "ignore". Please vote (+1, 0, -1): >>>>> >>>>> >>>>What is it currently, and what other options are there? >>>> >>>> >>>The default of underflow is "warn" >>> >>> >>I much prefer the current default. Having (potential errors) ignored silently >>is IMHO against the spirit of python. What's the problem with manually setting >>"ignore underflow" if and when one knows that underflows might occur but can >>safely be ignored? >> >> > >I'll read that as -1. :-) > This proposal leaves the control with the user, which seems to be the best way to go. Colin W. From Chris.Barker at noaa.gov Fri Nov 21 09:31:06 2003 From: Chris.Barker at noaa.gov (Chris Barker) Date: Fri Nov 21 09:31:06 2003 Subject: [Numpy-discussion] how-to "put" RAM-based numarray into memmap References: <047301c3b052$37f38960$421ee6a9@rodan> Message-ID: <3FBE4B21.C07CBF38@noaa.gov> Sebastian Haase wrote: > > Hi, > Suppose I have a 500MB-ram Computer and a 300MB ram-only (standard) > numarray. > Now I would like to "save" that onto harddrive (with a small header up > front How about: f = open(filename, 'wb') f.write(MyHeader) A.tofile(f) To read it back in, you need to know where your header ends, by either parsing it or using one of the same size every time, then you can use fromfile() to create an array form it. NOTE: I havn't actaully done this myself, but I've done something bvery similar with fromstring and tostring with Numeric. I'm very happy that Numarray has the *file methods, as they don't require an extra copy in memory. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From jmiller at stsci.edu Fri Nov 21 09:31:09 2003 From: jmiller at stsci.edu (Todd Miller) Date: Fri Nov 21 09:31:09 2003 Subject: [Numpy-discussion] Community Poll: numarray default underflow handling == "ignore" ? In-Reply-To: <3FBE4965.9050107@sympatico.ca> References: <1069365809.5225.145.camel@halloween.stsci.edu> <3FBD4BD4.FD1F30B9@noaa.gov> <1069421715.7448.2.camel@halloween.stsci.edu> <1069429373.7448.4.camel@halloween.stsci.edu> <3FBE4965.9050107@sympatico.ca> Message-ID: <1069435777.7446.13.camel@halloween.stsci.edu> On Fri, 2003-11-21 at 12:20, Colin J. Williams wrote: > > > Todd Miller wrote: > > >On Fri, 2003-11-21 at 09:58, Alexander Schmolck wrote: > > > > > >>Todd Miller writes: > >> > >> > >> > >>>On Thu, 2003-11-20 at 18:18, Chris Barker wrote: > >>> > >>> > >>>>Todd Miller wrote: > >>>> > >>>> > >>>>>Someone recently suggested that we change the default for numarray > >>>>>underflow checking to "ignore". Please vote (+1, 0, -1): > >>>>> > >>>>> > >>>>What is it currently, and what other options are there? > >>>> > >>>> > >>>The default of underflow is "warn" > >>> > >>> > >>I much prefer the current default. Having (potential errors) ignored silently > >>is IMHO against the spirit of python. What's the problem with manually setting > >>"ignore underflow" if and when one knows that underflows might occur but can > >>safely be ignored? > >> > >> > > > >I'll read that as -1. :-) > > > This proposal leaves the control with the user, which seems to be the > best way to go. Just to be clear, all we're talking about is the *default* handling of underflow. Regardless of what we decide, underflow handling will be configurable by the user on a per-thread basis as either: "ignore", "warn", or "raise". Todd > Colin W. > -- Todd Miller Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21030 (410) 338 - 4576 From falted at openlc.org Fri Nov 21 10:06:09 2003 From: falted at openlc.org (Francesc Alted) Date: Fri Nov 21 10:06:09 2003 Subject: [Numpy-discussion] how-to "put" RAM-based numarray into memmap In-Reply-To: <3FBE4B21.C07CBF38@noaa.gov> References: <047301c3b052$37f38960$421ee6a9@rodan> <3FBE4B21.C07CBF38@noaa.gov> Message-ID: <200311211905.33263.falted@openlc.org> A Divendres 21 Novembre 2003 18:28, Chris Barker va escriure: > Sebastian Haase wrote: > > Hi, > > Suppose I have a 500MB-ram Computer and a 300MB ram-only (standard) > > numarray. > > Now I would like to "save" that onto harddrive (with a small header up > > front > > How about: > > f = open(filename, 'wb') > f.write(MyHeader) > A.tofile(f) > > To read it back in, you need to know where your header ends, by either > parsing it or using one of the same size every time, then you can use > fromfile() to create an array form it. You can also use pytables for doing that more comfortably: >>> a=zeros([2,3,3]) >>> import tables >>> fileh=tables.openFile("test.h5", "w") >>> fileh.createArray(fileh.root, "array1", a, "This is a small test array") /array1 (Array(2, 3, 3)) 'This is a small test array' type = Int32 itemsize = 4 flavor = 'NumArray' byteorder = 'little' >>> fileh.close() To read it back you only have to do: >>> fileh=tables.openFile("test.h5", "r") >>> a=fileh.root.array1.read() >>> a array([[[0, 0, 0], [0, 0, 0], [0, 0, 0]], [[0, 0, 0], [0, 0, 0], [0, 0, 0]]]) that's all. All the headers with info about the object you are saving are automatically saved and retrieved behind the scenes. You can get pytables from http://pytables.sf.net Cheers, -- Francesc Alted From cookedm at physics.mcmaster.ca Fri Nov 21 11:13:03 2003 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Fri Nov 21 11:13:03 2003 Subject: [Numpy-discussion] Community Poll: numarray default underflow handling == "ignore" ? In-Reply-To: <3FBE4262.BEF5DEE4@noaa.gov> References: <1069365809.5225.145.camel@halloween.stsci.edu> <3FBD4BD4.FD1F30B9@noaa.gov> <1069421715.7448.2.camel@halloween.stsci.edu> <3FBE4262.BEF5DEE4@noaa.gov> Message-ID: <20031121191114.GA7441@arbutus.physics.mcmaster.ca> On Fri, Nov 21, 2003 at 08:50:42AM -0800, Chris Barker wrote: > Todd Miller wrote: > > > > Someone recently suggested that we change the default for numarray > > > > underflow checking to "ignore". Please vote (+1, 0, -1): > > > The default of underflow is "warn" > > assuming that ignore means that zero will be used, I vote +1 > > If the difference between zero and the smallest floating point number is > significant in your app, you'd better know what you're doing with > floating point, and you should then know to set the check to "warn" or > "raise". Agree, +1. It's quite common (for me) that I want exp(-1000) to give 0, and not an error (or warning, espicially in a loop that goes for several thousand iterations...) Overflow, yes, that's a problem, but not underflow. (The behaviour below lead me to writing my own exp ufunc for Numeric: >>> import Numeric >>> Numeric.exp(-1000) Traceback (most recent call last): File "", line 1, in ? OverflowError: math range error >>> ugh. ) -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From haase at msg.ucsf.edu Fri Nov 21 12:23:02 2003 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Fri Nov 21 12:23:02 2003 Subject: [Numpy-discussion] Community Poll: numarray default underflow handling == "ignore" ? References: <1069365809.5225.145.camel@halloween.stsci.edu> <3FBD4BD4.FD1F30B9@noaa.gov> <1069421715.7448.2.camel@halloween.stsci.edu> <3FBE4262.BEF5DEE4@noaa.gov> <20031121191114.GA7441@arbutus.physics.mcmaster.ca> Message-ID: <050501c3b06d$1ffe4ff0$421ee6a9@rodan> My vote would be '-1' ( if that means "I prefer ignore") I'm thinking of an INTERACTIVE platform - and so it would just "look nicer" without to many warnings. Actually on that note: I read some time ago about pythons default for printing floats: >>> 0.1 0.10000000000000001 >>> print 0.1 0.1 >>> repr(0.1) '0.10000000000000001' >>> str(.1) '0.1' Does anyone here have an update on that ? What I am especially interested in is when I have a list of (floating point) (x,y) positions and then typing in the var-name and getting all these ugly numbers is still very frustration for me ;-) Thanks, Sebastian Haase ----- Original Message ----- From: "David M. Cooke" To: "numpy-discussion" Sent: Friday, November 21, 2003 11:11 AM Subject: Re: [Numpy-discussion] Community Poll: numarray default underflow handling == "ignore" ? > On Fri, Nov 21, 2003 at 08:50:42AM -0800, Chris Barker wrote: > > Todd Miller wrote: > > > > > Someone recently suggested that we change the default for numarray > > > > > underflow checking to "ignore". Please vote (+1, 0, -1): > > > > > The default of underflow is "warn" > > > > assuming that ignore means that zero will be used, I vote +1 > > > > If the difference between zero and the smallest floating point number is > > significant in your app, you'd better know what you're doing with > > floating point, and you should then know to set the check to "warn" or > > "raise". > > Agree, +1. It's quite common (for me) that I want exp(-1000) to give 0, > and not an error (or warning, espicially in a loop that goes for several > thousand iterations...) Overflow, yes, that's a problem, but not underflow. > > (The behaviour below lead me to writing my own exp ufunc for Numeric: > >>> import Numeric > >>> Numeric.exp(-1000) > Traceback (most recent call last): > File "", line 1, in ? > OverflowError: math range error > >>> > ugh. > ) > > -- > |>|\/|< > /--------------------------------------------------------------------------\ > |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ > |cookedm at physics.mcmaster.ca > > > ------------------------------------------------------- > This SF.net email is sponsored by: SF.net Giveback Program. > Does SourceForge.net help you be more productive? Does it > help you create better code? SHARE THE LOVE, and help us help > YOU! Click Here: http://sourceforge.net/donate/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From jmiller at stsci.edu Fri Nov 21 12:28:06 2003 From: jmiller at stsci.edu (Todd Miller) Date: Fri Nov 21 12:28:06 2003 Subject: [Numpy-discussion] Community Poll: numarray default underflow handling == "ignore" ? In-Reply-To: <050501c3b06d$1ffe4ff0$421ee6a9@rodan> References: <1069365809.5225.145.camel@halloween.stsci.edu> <3FBD4BD4.FD1F30B9@noaa.gov> <1069421715.7448.2.camel@halloween.stsci.edu> <3FBE4262.BEF5DEE4@noaa.gov> <20031121191114.GA7441@arbutus.physics.mcmaster.ca> <050501c3b06d$1ffe4ff0$421ee6a9@rodan> Message-ID: <1069446392.7446.17.camel@halloween.stsci.edu> On Fri, 2003-11-21 at 15:22, Sebastian Haase wrote: > My vote would be '-1' ( if that means "I prefer ignore") The vote is for switching from "warn" on underflow to "ignore" on overflow, so: +1 == "ignore" underflow (vote for the change) 0 == don't care -1 == "warn" underflow (vote against the change) -- Todd Miller Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21030 (410) 338 - 4576 From a.schmolck at gmx.net Fri Nov 21 16:38:02 2003 From: a.schmolck at gmx.net (Alexander Schmolck) Date: Fri Nov 21 16:38:02 2003 Subject: [Numpy-discussion] Community Poll: numarray default underflow handling == "ignore" ? In-Reply-To: <050501c3b06d$1ffe4ff0$421ee6a9@rodan> References: <1069365809.5225.145.camel@halloween.stsci.edu> <3FBD4BD4.FD1F30B9@noaa.gov> <1069421715.7448.2.camel@halloween.stsci.edu> <3FBE4262.BEF5DEE4@noaa.gov> <20031121191114.GA7441@arbutus.physics.mcmaster.ca> <050501c3b06d$1ffe4ff0$421ee6a9@rodan> Message-ID: "Sebastian Haase" writes: > My vote would be '-1' ( if that means "I prefer ignore") > I'm thinking of an INTERACTIVE platform - and so it would just "look nicer" > without to many warnings. Well, it's only a default so you could always deactivate it (for all interactive sessions in your PYTHONSTARTUP if you wanted). > > Actually on that note: I read some time ago about pythons default for > printing floats: > >>> 0.1 > 0.10000000000000001 > >>> print 0.1 > 0.1 > >>> repr(0.1) > '0.10000000000000001' > >>> str(.1) > '0.1' > > Does anyone here have an update on that ? > What I am especially interested in is when I have a list of (floating point) > (x,y) positions and > then typing in the var-name and getting all these ugly numbers is still very > frustration for me ;-) You can customize python's interactive printing behavior any way you like (in your PYTHONSTARTUP). Here is an example from my old ~/.pythonrc.py (nowadays I almost exclusively use ipython). import pprint PRETTY_PRINT=1 _normal_displayhook = sys.displayhook def _my_displayhook(object): if PRETTY_PRINT: # don't bore us with None if object is not None: pprint.pprint(object) else: _normal_displayhook(object) sys.displayhook = _my_displayhook You could add something to the above to achieve the floating point (or list) formating you desire (``if type(object) is float:...). Since I am a heavy interactive user and found the default floating formating of arrays somewhat clumsy for interactive work, I also wrote some more fanciful formatting code for my Numeric/numarray compatible matrix class that amongst other things offers a number of formating options, including matlab style. I found that this made my life much easier. Thus: >>> a array([[-9.90000000e+01, -9.72817182e+01, 0.00000000e+00, -7.99144631e+01], [-4.54018500e+01, 4.84131591e+01, 3.03428793e+02, 9.96633158e+02], [2.88095799e+03, 8.00308393e+03, 2.19264658e+04, 5.97741417e+04]]) >>> m = matrix(a) >>> m matrix(''' 1.0E+04 * -0.00990 -0.00973 0.00000 -0.00799 -0.00454 0.00484 0.03034 0.09966 0.28810 0.80031 2.19265 5.97741 ''') >>> m.show('long') matrix(''' 1.0E+04 * Columns 0 through 3 -0.009900000000000 -0.009728171817154 0.000000000000000 -0.004540184996686 0.004841315910258 0.030342879349274 0.288095798704173 0.800308392757538 2.192646579480672 Columns 3 through 4 -0.007991446307681 0.099663315842846 5.977414171519782 ''') >>> m.show('+') m.show('+') matrix(''' -- - -+++ ++++ ''') etc. Adapting this to e.g. format Numeric arrays similarly via the display hook shouldn't be too hard, I can provide the code if you're interested. 'as From haase at msg.ucsf.edu Fri Nov 21 16:56:05 2003 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Fri Nov 21 16:56:05 2003 Subject: [Numpy-discussion] Community Poll: numarray default underflow handling == "ignore" ? References: <1069365809.5225.145.camel@halloween.stsci.edu><3FBD4BD4.FD1F30B9@noaa.gov><1069421715.7448.2.camel@halloween.stsci.edu><3FBE4262.BEF5DEE4@noaa.gov><20031121191114.GA7441@arbutus.physics.mcmaster.ca><050501c3b06d$1ffe4ff0$421ee6a9@rodan> Message-ID: <053901c3b093$5700d7e0$421ee6a9@rodan> Hi Alexander, Thanks for your reply and yes, I would be very interested to see more of your "pretty print" code. I am trying to put together all kinds of useful things to make python ( with numarray and scipy) a possible replacement for Matlab... I am not hoping for a similarly big documentation and help system, but just the fact that it's free ( doesn't need to connect to a "license server" every other minute) and that it happily supports single-precision arrays ( not to mention memmap array !) should make it interesting. I know that I probably should use IPython, but I also run python embedded in a wxWindows app and therefore use pyCrust. Thanks again, Sebastian Haase ----- Original Message ----- From: "Alexander Schmolck" To: "Sebastian Haase" Cc: Sent: Friday, November 21, 2003 4:37 PM Subject: Re: [Numpy-discussion] Community Poll: numarray default underflow handling == "ignore" ? > "Sebastian Haase" writes: > > > My vote would be '-1' ( if that means "I prefer ignore") > > I'm thinking of an INTERACTIVE platform - and so it would just "look nicer" > > without to many warnings. > > Well, it's only a default so you could always deactivate it (for all > interactive sessions in your PYTHONSTARTUP if you wanted). > > > > > Actually on that note: I read some time ago about pythons default for > > printing floats: > > >>> 0.1 > > 0.10000000000000001 > > >>> print 0.1 > > 0.1 > > >>> repr(0.1) > > '0.10000000000000001' > > >>> str(.1) > > '0.1' > > > > Does anyone here have an update on that ? > > > > > What I am especially interested in is when I have a list of (floating point) > > (x,y) positions and > > then typing in the var-name and getting all these ugly numbers is still very > > frustration for me ;-) > > You can customize python's interactive printing behavior any way you like (in > your PYTHONSTARTUP). Here is an example from my old ~/.pythonrc.py (nowadays I > almost exclusively use ipython). > > import pprint > PRETTY_PRINT=1 > _normal_displayhook = sys.displayhook > def _my_displayhook(object): > if PRETTY_PRINT: > # don't bore us with None > if object is not None: > pprint.pprint(object) > else: > _normal_displayhook(object) > sys.displayhook = _my_displayhook > > You could add something to the above to achieve the floating point (or list) > formating you desire (``if type(object) is float:...). > > Since I am a heavy interactive user and found the default floating formating > of arrays somewhat clumsy for interactive work, I also wrote some more > fanciful formatting code for my Numeric/numarray compatible matrix class that > amongst other things offers a number of formating options, including matlab > style. I found that this made my life much easier. > > Thus: > > >>> a > array([[-9.90000000e+01, -9.72817182e+01, 0.00000000e+00, -7.99144631e+01], > [-4.54018500e+01, 4.84131591e+01, 3.03428793e+02, 9.96633158e+02], > [2.88095799e+03, 8.00308393e+03, 2.19264658e+04, 5.97741417e+04]]) > >>> m = matrix(a) > >>> m > matrix(''' > 1.0E+04 * > > -0.00990 -0.00973 0.00000 -0.00799 > -0.00454 0.00484 0.03034 0.09966 > 0.28810 0.80031 2.19265 5.97741 > ''') > >>> m.show('long') > matrix(''' > 1.0E+04 * > > > Columns 0 through 3 > > -0.009900000000000 -0.009728171817154 0.000000000000000 > -0.004540184996686 0.004841315910258 0.030342879349274 > 0.288095798704173 0.800308392757538 2.192646579480672 > > Columns 3 through 4 > > -0.007991446307681 > 0.099663315842846 > 5.977414171519782 > ''') > >>> m.show('+') > m.show('+') > matrix(''' > -- - > -+++ > ++++ > ''') > > etc. > > Adapting this to e.g. format Numeric arrays similarly via the display hook > shouldn't be too hard, I can provide the code if you're interested. > > > 'as > > > > ------------------------------------------------------- > This SF.net email is sponsored by: SF.net Giveback Program. > Does SourceForge.net help you be more productive? Does it > help you create better code? SHARE THE LOVE, and help us help > YOU! Click Here: http://sourceforge.net/donate/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From focke at SLAC.Stanford.EDU Fri Nov 21 17:36:09 2003 From: focke at SLAC.Stanford.EDU (Warren Focke) Date: Fri Nov 21 17:36:09 2003 Subject: [Numpy-discussion] Community Poll: numarray default underflow handling == "ignore" ? In-Reply-To: <1069365809.5225.145.camel@halloween.stsci.edu> References: <1069365809.5225.145.camel@halloween.stsci.edu> Message-ID: +1 Warren Focke On Thu, 20 Nov 2003, Todd Miller wrote: > Someone recently suggested that we change the default for numarray > underflow checking to "ignore". Please vote (+1, 0, -1): From a.schmolck at gmx.net Fri Nov 21 19:24:05 2003 From: a.schmolck at gmx.net (Alexander Schmolck) Date: Fri Nov 21 19:24:05 2003 Subject: [Numpy-discussion] Community Poll: numarray default underflow handling == "ignore" ? In-Reply-To: <053901c3b093$5700d7e0$421ee6a9@rodan> References: <1069365809.5225.145.camel@halloween.stsci.edu> <3FBD4BD4.FD1F30B9@noaa.gov> <1069421715.7448.2.camel@halloween.stsci.edu> <3FBE4262.BEF5DEE4@noaa.gov> <20031121191114.GA7441@arbutus.physics.mcmaster.ca> <050501c3b06d$1ffe4ff0$421ee6a9@rodan> <053901c3b093$5700d7e0$421ee6a9@rodan> Message-ID: "Sebastian Haase" writes: > Hi Alexander, > Thanks for your reply and yes, I would be very interested to see more of > your "pretty print" code. Sure, I'll send you the code. As it is for a matrix class it only handles up to (array-)rank 2 currently (and uses methods of the matrix class) but I don't think changing either of these should be too hard. > I am trying to put together all kinds of useful things to make python ( with > numarray and scipy) a possible replacement for Matlab... Been, there done that -- actually still doing it. When I was faced with the choice between matlab and python for my numerical work, I decided that despite the convince matlab offered for many things (libraries, interactive environment) I couldn't face the its monstrosity as a programming language and thus opted for python. These were the main areas were I found python/Numeric was lagging behind: 1. Numeric.dot was much too slow (factor 40-100 and more for large arrays) 2. Although Numeric's arrays are quite general and powerful, I found writing down simple matrix expressions (dot(transpose(X), dot(...))) and the visual clarity of the repr of the result less attractive than matlab's. 3. Library support (e.g. in some fields many academics tend to make matlab code corresponding to their articles available, which is very handy to quickly try something). 4. Matlab came with very good plotting capabilities, whereas python's support for plotting was quite poor when I started using it. 5. Matlab also had a pretty good interactive environment. So far I've managed to come up with the following (sometimes partial) solutions for the above problems. 1. An optional fast dot product using blas (although I somehow still ended up spending a few days on it, all the real work was done by my supervisor Richard Everson -- I've submitted a patch that made it into Numeric 23 but I don't think numarray has an the option of a fast dot product yet, although I haven't checked). 2. A powerful matrix class (mostly finished; should transparently support Numeric as well as numarray (the latter is not fully there yet), unreleased). 3. A high-level matlab interface (pretty much ready to be released). 4. A convenient and high-level plotting library that supports multiple backends (currently only proper support for matlab as backend although I started working on/contemplating others (e.g. grace, gnuplot and chaco); unreleased) 5. IPython (kudos to Fernando!) + emacs + a few convenience functions for things like saving variables and redefining classes makes a darn nice environment once one got the hang of it (unfortunately it meant I had to write the emacs support for ipython, it's available on the ipython website). Of course if anyone wants to lend a hand, he/she is more than welcome ;) > I am not hoping for a similarly big documentation and help system, but just Why not? Nowadays pretty much all python code comes with pretty good docstrings and implementing something like 'lookfor' shouldn't be particularly hard. Already looking up docs and jumping to code in ipython is quite convenient (e.g. ``some_func?``). > the fact that it's free ( doesn't need to connect to a "license server" > every other minute) and that it happily supports single-precision arrays > ( not to mention memmap array !) should make it interesting. And that it is not based on an abomination of a programming language but an excellent one, with plenty of high quality general purpose libs (and even if only doing numerics is concerned: Numeric/numarray's (APL-ish) design also doesn't compare badly to matlab, IMHO). > I know that I probably should use IPython, Indeed, everyone should (from within emacs ;) 'as From verveer at embl.de Sat Nov 22 10:32:02 2003 From: verveer at embl.de (verveer at embl.de) Date: Sat Nov 22 10:32:02 2003 Subject: [Numpy-discussion] Community Poll: numarray default underflow handling == "ignore" ? In-Reply-To: References: Message-ID: <1069525884.3fbfab7cc95c0@webmail.embl.de> +1 for "ignore" Peter From jochen at jochen-kuepper.de Sun Nov 23 05:07:05 2003 From: jochen at jochen-kuepper.de (=?iso-8859-1?q?Jochen_K=FCpper?=) Date: Sun Nov 23 05:07:05 2003 Subject: [Numpy-discussion] Community Poll: numarray default underflow handling == "ignore" ? In-Reply-To: <1069446392.7446.17.camel@halloween.stsci.edu> (Todd Miller's message of "21 Nov 2003 15:26:32 -0500") References: <1069365809.5225.145.camel@halloween.stsci.edu> <3FBD4BD4.FD1F30B9@noaa.gov> <1069421715.7448.2.camel@halloween.stsci.edu> <3FBE4262.BEF5DEE4@noaa.gov> <20031121191114.GA7441@arbutus.physics.mcmaster.ca> <050501c3b06d$1ffe4ff0$421ee6a9@rodan> <1069446392.7446.17.camel@halloween.stsci.edu> Message-ID: <86r7zzjku3.fsf@doze.rijnh.nl> On 21 Nov 2003 15:26:32 -0500 Todd Miller wrote: Todd> The vote is for switching from "warn" on underflow to "ignore" Todd> on overflow, so: ^^^^ under! Todd> +1 == "ignore" underflow (vote for the change) Todd> 0 == don't care Todd> -1 == "warn" underflow (vote against the change) +1 Greetings, Jochen -- Einigkeit und Recht und Freiheit http://www.Jochen-Kuepper.de Libert?, ?galit?, Fraternit? GnuPG key: CC1B0B4D (Part 3 you find in my messages before fall 2003.) From BulkSales at 2911.net Sun Nov 23 19:12:02 2003 From: BulkSales at 2911.net (Hosting) Date: Sun Nov 23 19:12:02 2003 Subject: [Numpy-discussion] Bulk Friendly Web Hosting & Server Message-ID: Dear Sales: We offer reliable bulk email friendly web hosting services. You can now have the peace of mind knowing that your web site is secure during your email marketing campaigns. Bullet Proof Web Hosting 100% Guaranteed! We guarantee that your site will be 99% uptime. Hosting Features: Your own Web address: www.YourName.com Dedicated 100 M fiber 100MB of disk space Unlimited Data Transfer FTP Support FrontPage Support ASP Support PERL Support PHP Support CGI Support No Setup Fee US$199.00/month Bullet-Proof dedicated servers: 1. Two IPs 1024MB RAM DDR PIIII / Two CPU 120 GB SCS Intel 82559 LAN /(2U) Unlimited Data Transfer Linux / Windows / FreeBSD Price: No setup fee US$ 599.00/month 2. Dynamic IP 1024MB RAM DDR PIIII / Two CPU 120 GB SCS Intel 82559 LAN /(2U) Unlimited Data Transfer Linux / Windows / FreeBSD Price: No setup fee US$ 999.00/month Allowed Usage You can use the server for any of the following: Direct Bulk Mailing or Proxy Mailing Web Site Hosting Proxy, Relay or Port Scanning Restrictions & Rules Our Plans do NOT include the following: No Adult Content No Illegal Material We aslo offer targeted email addresses and send out targeted emails for you. We looking forward to serving you in the near future. Cheers! Betty Jones Support Teams Support at biz-servers.com Http://www.biz-servers.com From cjw at sympatico.ca Mon Nov 24 05:18:02 2003 From: cjw at sympatico.ca (Colin J. Williams) Date: Mon Nov 24 05:18:02 2003 Subject: [Numpy-discussion] PyMatrix: Announcement Message-ID: <3FC204D0.2010503@sympatico.ca> PyMatrix is available for test and review. http://www3.sympatico.ca/cjw PyMatrix provides access to basic matrix arithmetic, using Python and numarray. Examples: A * B => the product of matrices A and B A.I => the inverse of matrix A A.EVectors => the eigenvectors of A A.var(0) => the variances of the columns of A (a.T*a).I * a.T*b => the solution (x) for a * x = b, where a is a matrix and b a column vector This package was developed on a Windows XP. I would appreciate comments, particularly with respect to usage on other systems. Colin W. From nadavh at VisionSense.com Mon Nov 24 06:30:11 2003 From: nadavh at VisionSense.com (Nadav Horesh) Date: Mon Nov 24 06:30:11 2003 Subject: [Numpy-discussion] PyMatrix: Announcement In-Reply-To: <3FC204D0.2010503@sympatico.ca> References: <3FC204D0.2010503@sympatico.ca> Message-ID: <1069684102.5532.1.camel@Nadav.Envision.co.il> Don't see how can I get the sources for gnu/linux installation. Nadav On Mon, 2003-11-24 at 15:17, Colin J. Williams wrote: > PyMatrix is available for test and review. > http://www3.sympatico.ca/cjw > > PyMatrix provides access to basic matrix arithmetic, using Python and > numarray. > > Examples: > A * B => the product of > matrices A and B > A.I => the inverse of matrix A > A.EVectors => the eigenvectors of A > A.var(0) => the variances of the > columns of A > (a.T*a).I * a.T*b => the solution (x) for a * > x = b, > where a is a > matrix and b a column vector > > This package was developed on a Windows XP. I would appreciate > comments, particularly with respect to usage on other systems. > > Colin W. > > > > > > ------------------------------------------------------- > This SF.net email is sponsored by: SF.net Giveback Program. > Does SourceForge.net help you be more productive? Does it > help you create better code? SHARE THE LOVE, and help us help > YOU! Click Here: http://sourceforge.net/donate/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From haase at msg.ucsf.edu Mon Nov 24 11:12:05 2003 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Mon Nov 24 11:12:05 2003 Subject: [Numpy-discussion] OT:pretty print of reals (was: Community Poll: numarray default underflow handling == "ignore" ?) References: <1069365809.5225.145.camel@halloween.stsci.edu><3FBD4BD4.FD1F30B9@noaa.gov><1069421715.7448.2.camel@halloween.stsci.edu><3FBE4262.BEF5DEE4@noaa.gov><20031121191114.GA7441@arbutus.physics.mcmaster.ca><050501c3b06d$1ffe4ff0$421ee6a9@rodan><053901c3b093$5700d7e0$421ee6a9@rodan> Message-ID: <070b01c3b2be$caf26740$421ee6a9@rodan> Hi Alexander, Thanks for the code - I will look into it I just read a few lines of your MatrixFormatter class seeing the definition of __repr__. Here is a thought: Is it possible that I overload the __repr__ of float (that is: the buildin data type) !? When I use pyCrust it automatically "fills in" default arguments like, func(a=0.1000000000001) So, this probably does not call the sys.displayhook - so __repr__ itself would need to be changed. Do you know if that is possible ? (It would recursively fix all list and matrix display problems I had ...) Thanks again, Sebastian ----- Original Message ----- From: "Alexander Schmolck" To: "Sebastian Haase" Cc: Sent: Friday, November 21, 2003 7:23 PM Subject: Re: [Numpy-discussion] Community Poll: numarray default underflow handling == "ignore" ? > "Sebastian Haase" writes: > > > Hi Alexander, > > Thanks for your reply and yes, I would be very interested to see more of > > your "pretty print" code. > > Sure, I'll send you the code. As it is for a matrix class it only handles up > to (array-)rank 2 currently (and uses methods of the matrix class) but I don't > think changing either of these should be too hard. > > > I am trying to put together all kinds of useful things to make python ( with > > numarray and scipy) a possible replacement for Matlab... > > Been, there done that -- actually still doing it. > > When I was faced with the choice between matlab and python for my numerical > work, I decided that despite the convince matlab offered for many things > (libraries, interactive environment) I couldn't face the its monstrosity as a > programming language and thus opted for python. > > These were the main areas were I found python/Numeric was lagging behind: > > 1. Numeric.dot was much too slow (factor 40-100 and more for large arrays) > 2. Although Numeric's arrays are quite general and powerful, I found writing > down simple matrix expressions (dot(transpose(X), dot(...))) and the visual > clarity of the repr of the result less attractive than matlab's. > 3. Library support (e.g. in some fields many academics tend to make matlab > code corresponding to their articles available, which is very handy to > quickly try something). > 4. Matlab came with very good plotting capabilities, whereas python's support > for plotting was quite poor when I started using it. > 5. Matlab also had a pretty good interactive environment. > > So far I've managed to come up with the following (sometimes partial) > solutions for the above problems. > > 1. An optional fast dot product using blas (although I somehow still ended up > spending a few days on it, all the real work was done by my supervisor > Richard Everson -- I've submitted a patch that made it into Numeric 23 but > I don't think numarray has an the option of a fast dot product yet, > although I haven't checked). > 2. A powerful matrix class (mostly finished; should transparently support > Numeric as well as numarray (the latter is not fully there yet), > unreleased). > 3. A high-level matlab interface (pretty much ready to be released). > 4. A convenient and high-level plotting library that supports multiple > backends (currently only proper support for matlab as backend although I > started working on/contemplating others (e.g. grace, gnuplot and chaco); > unreleased) > 5. IPython (kudos to Fernando!) + emacs + a few convenience functions for > things like saving variables and redefining classes makes a darn nice > environment once one got the hang of it (unfortunately it meant I had to > write the emacs support for ipython, it's available on the ipython > website). > > Of course if anyone wants to lend a hand, he/she is more than welcome ;) > > > I am not hoping for a similarly big documentation and help system, but just > > Why not? Nowadays pretty much all python code comes with pretty good > docstrings and implementing something like 'lookfor' shouldn't be particularly > hard. Already looking up docs and jumping to code in ipython is quite > convenient (e.g. ``some_func?``). > > > the fact that it's free ( doesn't need to connect to a "license server" > > every other minute) and that it happily supports single-precision arrays > > ( not to mention memmap array !) should make it interesting. > > And that it is not based on an abomination of a programming language but an > excellent one, with plenty of high quality general purpose libs (and even if > only doing numerics is concerned: Numeric/numarray's (APL-ish) design also > doesn't compare badly to matlab, IMHO). > > > I know that I probably should use IPython, > > Indeed, everyone should (from within emacs ;) > > 'as > > From haase at msg.ucsf.edu Mon Nov 24 11:14:07 2003 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Mon Nov 24 11:14:07 2003 Subject: [Numpy-discussion] how-to "put" RAM-based numarray into memmap References: <047301c3b052$37f38960$421ee6a9@rodan> <3FBE4B21.C07CBF38@noaa.gov> Message-ID: <071d01c3b2bf$11171950$421ee6a9@rodan> > Sebastian Haase wrote: > > > > Hi, > > Suppose I have a 500MB-ram Computer and a 300MB ram-only (standard) > > numarray. > > Now I would like to "save" that onto harddrive (with a small header up > > front > > How about: > > f = open(filename, 'wb') > f.write(MyHeader) > A.tofile(f) > > To read it back in, you need to know where your header ends, by either > parsing it or using one of the same size every time, then you can use > fromfile() to create an array form it. The main reason for my question was just to find out if NUMARRAY supports it, and how ? Also I have many "bookkeeping" functions already implemented for the memmap'd case. (That is, I have a class with member methods operting on a member memmapped array) So if what I descibed is possible I could save myself form duplicating lot's of code. Essentially I was hoping for the most ellegant solution ;-) Thanks, Sebastian From jmiller at stsci.edu Mon Nov 24 11:35:08 2003 From: jmiller at stsci.edu (Todd Miller) Date: Mon Nov 24 11:35:08 2003 Subject: [Numpy-discussion] how-to "put" RAM-based numarray into memmap In-Reply-To: <071d01c3b2bf$11171950$421ee6a9@rodan> References: <047301c3b052$37f38960$421ee6a9@rodan> <3FBE4B21.C07CBF38@noaa.gov> <071d01c3b2bf$11171950$421ee6a9@rodan> Message-ID: <1069702447.4416.144.camel@localhost.localdomain> On Mon, 2003-11-24 at 14:13, Sebastian Haase wrote: > > Sebastian Haase wrote: > > > > > > Hi, > > > Suppose I have a 500MB-ram Computer and a 300MB ram-only (standard) > > > numarray. > > > Now I would like to "save" that onto harddrive (with a small header up > > > front > > > > How about: > > > > f = open(filename, 'wb') > > f.write(MyHeader) > > A.tofile(f) > > > > To read it back in, you need to know where your header ends, by either > > parsing it or using one of the same size every time, then you can use > > fromfile() to create an array form it. > > The main reason for my question was just to find out if NUMARRAY supports > it, and how ? > Also I have many "bookkeeping" functions already implemented for the > memmap'd case. > (That is, I have a class with member methods operting on a member memmapped > array) > So if what I descibed is possible I could save myself form duplicating lot's > of code. > > Essentially I was hoping for the most ellegant solution ;-) > memmap's Memmap class does support an insert() method for adding a new slice to the end of (or anywhere in) an existing map. The new slice, however, will exist as a block of memory allocated on the heap until the memmap is saved to disk. Thus, two scenarios present themselves: (1) you allocate the new slice ahead of time and create an array from it, avoiding data duplication (2) you create an array and later copy it into (a newly inserted slice of) the memmap, thereby duplicating your data on the heap. When you close the map, slices on the heap are written to the map file. Todd > Thanks, > Sebastian > > > > ------------------------------------------------------- > This SF.net email is sponsored by: SF.net Giveback Program. > Does SourceForge.net help you be more productive? Does it > help you create better code? SHARE THE LOVE, and help us help > YOU! Click Here: http://sourceforge.net/donate/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- Todd Miller From a.schmolck at gmx.net Mon Nov 24 11:48:06 2003 From: a.schmolck at gmx.net (Alexander Schmolck) Date: Mon Nov 24 11:48:06 2003 Subject: [Numpy-discussion] Re: OT:pretty print of reals (was: Community Poll: numarray default underflow handling == "ignore" ?) In-Reply-To: <070b01c3b2be$caf26740$421ee6a9@rodan> References: <1069365809.5225.145.camel@halloween.stsci.edu> <3FBD4BD4.FD1F30B9@noaa.gov> <1069421715.7448.2.camel@halloween.stsci.edu> <3FBE4262.BEF5DEE4@noaa.gov> <20031121191114.GA7441@arbutus.physics.mcmaster.ca> <050501c3b06d$1ffe4ff0$421ee6a9@rodan> <053901c3b093$5700d7e0$421ee6a9@rodan> <070b01c3b2be$caf26740$421ee6a9@rodan> Message-ID: "Sebastian Haase" writes: > Hi Alexander, > Thanks for the code - I will look into it > I just read a few lines of your MatrixFormatter class seeing the definition > of __repr__. > > Here is a thought: Is it possible that I overload the __repr__ of float > (that is: the buildin data type) !? Nope. You can't screw around with methods of builtins -- and this is a (quite sensible) conscious design decision. > When I use pyCrust it automatically "fills in" default arguments like, > func(a=0.1000000000001) > So, this probably does not call the sys.displayhook - so __repr__ itself > would need to be changed. Nope, whatever pycrust uses needs to be changed (are you *sure* changing sys.displayhook has no effect on pycrust (it might not change what it "fills in", but what about actual output?) because that would seem sort of ... suboptimal. Even so it shouldn't be too difficult to figure out how you can customize printing in pycrust (by having a look at the pycrust code and/or reading the manual or asking on the appropriate mailing-list). > Do you know if that is possible ? (It would recursively fix all list and > matrix display problems I had ...) It wouldn't, it would just get you the float and list (tuple etc.) display you want (arrays/matrices don't call repr on each individual array arguments to compute their overall repr). HTH, 'as From haase at msg.ucsf.edu Mon Nov 24 11:59:13 2003 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Mon Nov 24 11:59:13 2003 Subject: [Numpy-discussion] how-to "put" RAM-based numarray into memmap References: <047301c3b052$37f38960$421ee6a9@rodan> <3FBE4B21.C07CBF38@noaa.gov> <071d01c3b2bf$11171950$421ee6a9@rodan> <1069702447.4416.144.camel@localhost.localdomain> Message-ID: <073d01c3b2c5$59532af0$421ee6a9@rodan> > On Mon, 2003-11-24 at 14:13, Sebastian Haase wrote: > > > Sebastian Haase wrote: > > > > > > > > Hi, > > > > Suppose I have a 500MB-ram Computer and a 300MB ram-only (standard) > > > > numarray. > > > > Now I would like to "save" that onto harddrive (with a small header up > > > > front > > > > > > How about: > > > > > > f = open(filename, 'wb') > > > f.write(MyHeader) > > > A.tofile(f) > > > > > > To read it back in, you need to know where your header ends, by either > > > parsing it or using one of the same size every time, then you can use > > > fromfile() to create an array form it. > > > > The main reason for my question was just to find out if NUMARRAY supports > > it, and how ? > > Also I have many "bookkeeping" functions already implemented for the > > memmap'd case. > > (That is, I have a class with member methods operting on a member memmapped > > array) > > So if what I descibed is possible I could save myself form duplicating lot's > > of code. > > > > Essentially I was hoping for the most ellegant solution ;-) > > > > memmap's Memmap class does support an insert() method for adding a new > slice to the end of (or anywhere in) an existing map. The new slice, > however, will exist as a block of memory allocated on the heap until > the memmap is saved to disk. > > Thus, two scenarios present themselves: (1) you allocate the new slice > ahead of time and create an array from it, avoiding data duplication (2) > you create an array and later copy it into (a newly inserted slice of) > the memmap, thereby duplicating your data on the heap. > > When you close the map, slices on the heap are written to the map file. > > Todd > Thanks for your reply. Is it possible to define a memmap slice and giving it a (preinitialized !) memory buffer ? I'm thinking: I have a RAM-based numarray, I just take the buffer (pointer) and hand it over to the memmap-slice so that it can make the association between the disk-space and the RAM-space. I guess you are calling "heap" what I call RAM. Is memmap using something inherently different that heap? (I might be missing something...) As I was trying to illustrate in my example, my ram-numarray might already be using most of the available address space. Thanks, Sebastian From jmiller at stsci.edu Mon Nov 24 13:19:04 2003 From: jmiller at stsci.edu (Todd Miller) Date: Mon Nov 24 13:19:04 2003 Subject: [Numpy-discussion] how-to "put" RAM-based numarray into memmap In-Reply-To: <073d01c3b2c5$59532af0$421ee6a9@rodan> References: <047301c3b052$37f38960$421ee6a9@rodan> <3FBE4B21.C07CBF38@noaa.gov> <071d01c3b2bf$11171950$421ee6a9@rodan> <1069702447.4416.144.camel@localhost.localdomain> <073d01c3b2c5$59532af0$421ee6a9@rodan> Message-ID: <1069708711.4415.215.camel@localhost.localdomain> > Thanks for your reply. > Is it possible to define a memmap slice and giving it a (preinitialized !) > memory buffer ? It should be now. I just added support for a "buffer" parameter to CVS. I tested it manually and it worked OK for me; please try it out or watch for it in numarray-0.8. > I'm thinking: I have a RAM-based numarray, I just take the buffer (pointer) > and hand it over to the memmap-slice so that it can make the association > between the disk-space and the RAM-space. > I guess you are calling "heap" what I call RAM. Is memmap using something > inherently different that heap? (I might be missing something...) Yes and no. When you allocate a slice of an existing memmap, you're getting memory backed on the disk file of your choice; mapped memory generally comes from a different address partition than the memory from malloc(). When you allocate memory using numarray, or insert a slice into a memmap without specifying the buffer, you're getting memory from the heap which is backed on the system swap file. (Of course, this varies a little by system as well; I'm thinking Linux/UNIX here and Win32 may be slightly different). Todd -- Todd Miller From cjw at sympatico.ca Tue Nov 25 11:09:04 2003 From: cjw at sympatico.ca (Colin J. Williams) Date: Tue Nov 25 11:09:04 2003 Subject: [Numpy-discussion] PyMatrix: Announcement Message-ID: <3FC3A8AE.6030204@sympatico.ca> In response to a number of requests, the PyMatrix package is now available for download as a zip file. Colin W. > PyMatrix is available for test and review. > http://www3.sympatico.ca/cjw > > PyMatrix provides access to basic matrix arithmetic, using Python and > numarray. > > Examples: > A * B => the product of matrices A and B > A.I => the inverse of matrix A > A.EVectors => the eigenvectors of A > A.var(0) => the variances of the columns of A > (a.T*a).I * a.T*b => the solution (x) for a * x = b, > where a is a matrix and b a column vector > > This package was developed on a Windows XP. I would appreciate > comments, particularly with respect to usage on other systems. > > Colin W. > From gvermeul at grenoble.cnrs.fr Wed Nov 26 11:59:03 2003 From: gvermeul at grenoble.cnrs.fr (Gerard Vermeulen) Date: Wed Nov 26 11:59:03 2003 Subject: [Numpy-discussion] ANN: PyQwt-3.8 Message-ID: <20031126205924.62dc5947.gvermeul@grenoble.cnrs.fr> PyQwt-3.8 = FAST and EASY data plotting for Python and (Py)Qt. PyQwt is a set of Python bindings for the Qwt C++ class library which extends the Qt framework with widgets for scientific and engineering applications. It supports the use of PyQt, Qt, Qwt, the Numerical Python extensions (either Numeric, or numarray or both) and optionally SciPy in a GUI Python application or in an interactive Python session. The home page of PyQwt is http://pyqwt.sourceforge.net. Main changes in PyQwt-3.8: 1. PyQwt inherits from Qwt the new classes QwtPicker, QwtPlotPicker and QwtPlotZoomer. QwtPicker can make any widget react to events by event filtering. QwtPlotPicker is a QwtPicker taylored to the plot canvas. QwtPlotZoomer implements infinite deep zooming on the plot canvas. 2. PyQwt inherits from Qwt the support for Qt's rich text format in QwtPlot to display equations like E=mc2. Caveat: in principle, a truetype font with full unicode support is better to typeset equations. 3. Bug fixes to support event filtering code (see EventFilterDemo.py). 4. Conversion of a QImage to a Numeric array or numarray array. 5. Better support for cloning a plot into Grace from the command line interpreter friendly plot classes. 6. PyQwt-3.8 works with PyQt-3.8.1 downto -3.6 and Qt-3.2.3 downto -2.3.0. 7. License change: GPL with permission for dynamic linking to commercial, educational and non-commercial versions of Qt, PyQt and sip. In this case PyQwt becomes a free plug-in for a non-free program. Gerard Vermeulen From proj_mgr8 at yahoo.com Sat Nov 29 10:41:03 2003 From: proj_mgr8 at yahoo.com (E l e c t r i c i a n) Date: Sat Nov 29 10:41:03 2003 Subject: [Numpy-discussion] R E S U M E Message-ID: Resume from: RICH' S for Job, Consulting, Installation or Service E L E C T R I C I A N Tel. (408) 482-2102 rysio3 at yahoo.com WIRING & INSTALLATION Hands on electrical installations perform fitting, mounting, laying cables on Commercial, Industrial, residential new & existing buildings. Electrical Power Supply for Lights, Plugs, Receptacles, Panels, & Fuse boxes, Emergency Generators wiring and testing, Transformers, Power Lines & conduit layout, bending and mounting, parking lighting, lamps, Signs, switches, posts and underground installations. Shopping Centers; grocery stories, hardware stories, restaurants & residential - housing areas, computer business & fast food units installation & buildings; Solar Panels, Sun Tracking, Flywheel Storage & cars electric systems modify, Natural Energy in Remote areas install. LOW VOLTAGE 12 / 24 Volt audio & video equipment, Computer & data network wiring, data backup and UPS; Monitoring Video Control & backup tapes set up and mounting, electro-optical assemblies & subsystems. DC Power Supply, Switch & Motion sensors Alarm. Fire & safety systems install. Fiber Optics systems, PLC setup, Master Control Center, cable modems & cable TV install. Network, UPS Battery Backup mounting and charging systems; Power supply testing, troubleshooting, and analyzing to a components level. Solar Signs, Electric Vehicles Design, Assembly & Installations. CC TV & Cameras, Security Systems & Sensors for Safety, Fire sprinklers and traffic Monitoring & Door Control. TECHNICIAN Use lab & shop equipment, mechanical, electrical & electronic tools, measurement & testing equipment, video cameras & microscopes. Support scientists & electronic engineers. Mechanical & Electro-Mechan. Design. OFFICE, ELECTRICAL AND MECHANICAL PROJECTS Electrical & Network Sketches, one line diagrams, and "as is" drawings update. Customizing Electronic and Electrical Components & Parts, Layouts electronic and electrical schematic, connectors and mechanical detailing. Quotes, supply, bids and job estimating. SOLAR PROJECTS. Customers contact, inspection, project mgmt & supervision of electricians & material handling; Use CAD, Windows and applications; ELECTRICAL & MAINTENANCE SERVICE US Citizen; open for travel . From mdehoon at ims.u-tokyo.ac.jp Sun Nov 30 22:13:05 2003 From: mdehoon at ims.u-tokyo.ac.jp (Michiel Jan Laurens de Hoon) Date: Sun Nov 30 22:13:05 2003 Subject: [Numpy-discussion] PyArray_ContiguousFromObject & PyDECREF Message-ID: <3FCADC96.3050504@ims.u-tokyo.ac.jp> Dear pythoneers, In Chapter 12 ("Writing a C extension") in the Numerical Python manual, in the first example PyArray_ContiguousFromObject is used together with PyDECREF, while in the last example PyArray_ContiguousFromObject is used but the arrays are not PyDECREF'ed before the function returns. From Chapter 13 ("C API Reference") it seems that a PyDECREF is needed in both cases. Or am I missing something here? Is PyDECREF not needed for some reason in the last example? Thanks in advance, Michiel de Hoon U Tokyo First example: ============== static PyObject * trace(PyObject *self, PyObject *args) { PyObject *input; PyArrayObject *array; double sum; int i, n; if (!PyArg_ParseTuple(args, "O", &input)) return NULL; array = (PyArrayObject *) PyArray_ContiguousFromObject(input, PyArray_DOUBLE, 2, 2); if (array == NULL) return NULL; n = array->dimensions[0]; if (n > array->dimensions[1]) n = array->dimensions[1]; sum = 0.; for (i = 0; i < n; i++) sum += *(double *)(array->data + i*array->strides[0] + i*array->strides[1]); Py_DECREF(array); return PyFloat_FromDouble(sum); } Last example: ============= static PyObject * matrix_vector(PyObject *self, PyObject *args) { PyObject *input1, *input2; PyArrayObject *matrix, *vector, *result; int dimensions[1]; double factor[1]; double real_zero[1] = {0.}; long int_one[1] = {1}; long dim0[1], dim1[1]; extern dgemv_(char *trans, long *m, long *n, double *alpha, double *a, long *lda, double *x, long *incx, double *beta, double *Y, long *incy); if (!PyArg_ParseTuple(args, "dOO", factor, &input1, &input2)) return NULL; matrix = (PyArrayObject *) PyArray_ContiguousFromObject(input1, PyArray_DOUBLE, 2, 2); if (matrix == NULL) return NULL; vector = (PyArrayObject *) PyArray_ContiguousFromObject(input2, PyArray_DOUBLE, 1, 1); if (vector == NULL) return NULL; if (matrix->dimensions[1] != vector->dimensions[0]) { PyErr_SetString(PyExc_ValueError, "array dimensions are not compatible"); return NULL; } dimensions[0] = matrix->dimensions[0]; result = (PyArrayObject *)PyArray_FromDims(1, dimensions, PyArray_DOUBLE); if (result == NULL) return NULL; dim0[0] = (long)matrix->dimensions[0]; dim1[0] = (long)matrix->dimensions[1]; dgemv_("T", dim1, dim0, factor, (double *)matrix->data, dim1, (double *)vector->data, int_one, real_zero, (double *)result->data, int_one); return PyArray_Return(result); } -- Michiel de Hoon, Assistant Professor University of Tokyo, Institute of Medical Science Human Genome Center 4-6-1 Shirokane-dai, Minato-ku Tokyo 108-8639 Japan http://bonsai.ims.u-tokyo.ac.jp/~mdehoon From hinsen at cnrs-orleans.fr Sat Nov 1 13:17:10 2003 From: hinsen at cnrs-orleans.fr (Konrad Hinsen) Date: Sat Nov 1 13:17:10 2003 Subject: [Numpy-discussion] gcc compile / link questions In-Reply-To: <3FA31683.10703@erols.com> References: <3FA31683.10703@erols.com> Message-ID: <200311012216.49289.hinsen@cnrs-orleans.fr> On Saturday 01 November 2003 03:12, Edward C. Jones wrote: > I compile and link Python extension modules using the script > > gcc -fPIC -g -I/usr/local/include/python2.3 \ > -Wall -Wstrict-prototypes -c mymodule.c > g++ -shared mymodule.o -L/usr/local/lib -o mymodule.so > > It works for me but it isn't pretty. Is there a better way to write it? Yes, the distutils module. It's part of the Python standard library and documented there. > Gcc finds all the libraries that need to be linked in. For example, > "/usr/local/lib/python2.3/site-packages/numarray/libnumarray.so". How > does gcc do this? It doesn't :-) And it doesn't have to. You are not creating a stand-alone executable, but a shared library for use in Python. Everything is linked together dynamically at runtime when Python imports all the modules. > I created a .so file "utilities.so" that contains some C functions that > are called in mymodule.c but are not visible from Python. Both > "utilities.c" and "mymodule.c" use numarray. What changes do I make in > the script above? Must I use the nasty "libnumarray_UNIQUE_SYMBOL" trick? I don't know enough about numarray to answer that question, but that is certainly one option. Alternatively you could include utilities.c into mymodule.c. > What is a script for creating "utilities.a" using gcc? How do I change > the script above to include "utilities.a"? To create the archive: gcc -c utilities.c ar r libutilities.a utilies.o You might have to add some libraries or library paths to the compilation step. To use the archive, just add -lutilities to the gcc command line, plus eventually the path with -L. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen at cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From nadavh at VisionSense.com Sun Nov 2 04:22:03 2003 From: nadavh at VisionSense.com (Nadav Horesh) Date: Sun Nov 2 04:22:03 2003 Subject: [Numpy-discussion] Problem in LinearAlgebra? In-Reply-To: <3FA2617A.4000308@hooft.net> References: <3FA2617A.4000308@hooft.net> Message-ID: <1067779275.2190.28.camel@Nadav.Envision.co.il> I tested the problem with: 1. Numeric 23.1 under python 2.3.2 2. numarray 0.8 (I made a copy of the Scientific package where all calls to Numeric were replaced to numarray), under python 2.3.2 There results where about the same -- high coefficients for the 5th order polynomials. I would expect reliable fit for a high order polynomials only under very special circumstances, so this is not a big surprise. My advice is: * Make sure that this is a bug and not a result of a numerical instability. If you can trace it down and point to a bug, then report it. The numarray package is very usable and is under a very active and rapid development, thus bugs are being fixed fast. * Look for a solution in the scipy package: It is generally better then Scientific. * Polynomials fit is relatively very simple --- you may write one of you own in less then a one day work. Since, as I said, the problem is, in many cases, unstable, you'll have the chance to implement more stable linear-equation solvers. Nadav. On Fri, 2003-10-31 at 15:19, Rob W.W. Hooft wrote: > I am using Polynomial.py from Scientific Python 2.1, together with > Numeric 17.1.2. This has always served me well, but now we are busy > upgrading our software, and I am currently porting some code to > Scientific Python 2.4.1, Numeric 22.0. Suddenly I do no longer manage to > get proper 2D polynomial fits over 4x4th order. At 5x5 the coefficients > that come back from LinearAlgebra.linear_least_squares have exploded. In > the old setup, I easily managed 9x9th order if I needed to, but most of > the time I'd stop at 6x6th order. Would anyone have any idea how this > difference can come about? I managed to work around this for the moment > by using the equivalent code in the fitPolynomial routine that uses > LinearAlgebra.generalized_inverse (and it doesn't even have problems > with the same data at 8x8), but this definitely feels not right! I can't > remember reading anything like this here before. > > Together with Konrad Hinsen, I came to the conclusion that the problem > is not in Scientific Python, so it must be the underlying LinearAlgebra > code that changed between releases 17 and 22. > > I hacked up a simplified example. Not sure whether it is the most simple > case, but this resembles what I have in my code, and I'm quite sure it > worked with Numeric 17.x, but currently it is horrible over order (4,4): > > -------------------------------------- > import Numeric > > def func(x,y): > return x+0.1*x**2+0.01*x**4+0.002*x**6+0.03*x*y+0.001*x**4*y**5 > > x=[] > y=[] > z=[] > for dx in Numeric.arange(0,1,0.01): > for dy in Numeric.arange(0,1,0.01): > x.append(dx) > y.append(dy) > z.append(func(dx,dy)) > > from Scientific.Functions import Polynomial > data=Numeric.transpose([x,y]) > z=Numeric.array(z) > for i in range(10): > print data[i],z[i] > pol=Polynomial.fitPolynomial((4,4),data,z) > print pol.coeff > ------------------------------------ > for 4,4 this prints: > [[ 1.84845529e-05 -7.60502772e-13 2.71314749e-12 -3.66731796e-12 > 1.66977148e-12] > [ 9.99422967e-01 3.00000000e-02 -3.26346097e-11 4.42406519e-11 > -2.01549767e-11] > [ 1.03899464e-01 -3.19668064e-11 1.14721790e-10 -1.55489826e-10 > 7.08425891e-11] > [ -9.40275000e-03 4.28456838e-11 -1.53705205e-10 2.08279772e-10 > -9.48840470e-11] > [ 1.80352695e-02 -1.10999843e-04 8.00662570e-04 -2.17266676e-03 > 2.47500004e-03]] > > for 5,5: > > [[ -2.25705839e+03 6.69051337e+02 -6.60470163e+03 6.66572425e+03 > -8.67897022e+02 1.83974866e+03] > [ -2.58646837e+02 -2.46554689e+03 1.15965805e+03 7.01089888e+03 > -2.11395436e+03 2.10884815e+03] > [ 3.93307499e+03 4.34484805e+02 -4.84080392e+03 5.90375330e+03 > 1.16798049e+03 -4.14163933e+03] > [ 1.62814750e+03 2.08717457e+03 1.15870693e+03 -3.37838057e+03 > 3.49821689e+03 5.80572585e+03] > [ 4.54127557e+02 -1.56645524e+03 4.58997025e+00 1.69772635e+03 > -1.37751039e+03 -7.59726558e+02] > [ 2.37878239e+03 9.43032094e+02 8.58518644e+02 -8.35846339e+03 > -5.55845668e+02 1.87502761e+03]] > > Which is clearly wrong. > > I appreciate any help! > > Regards, > > Rob From cjw at sympatico.ca Sun Nov 2 04:27:08 2003 From: cjw at sympatico.ca (Colin J. Williams) Date: Sun Nov 2 04:27:08 2003 Subject: [Numpy-discussion] LinearAlgebra2.determinant Message-ID: <3FA4F7EC.2030203@sympatico.ca> In the line: return num.zeros((1,), type=a.type())-num.multiply.reduce(num.diagonal(a)) the code: num.zeros((1,), type=a.type()) appears to be redundant. Colin W. From rob at hooft.net Sun Nov 2 06:51:23 2003 From: rob at hooft.net (Rob Hooft) Date: Sun Nov 2 06:51:23 2003 Subject: [Numpy-discussion] Problem in LinearAlgebra? In-Reply-To: <1067779275.2190.28.camel@Nadav.Envision.co.il> References: <3FA2617A.4000308@hooft.net> <1067779275.2190.28.camel@Nadav.Envision.co.il> Message-ID: <3FA519DC.1040307@hooft.net> Nadav Horesh wrote: > I tested the problem with: > 1. Numeric 23.1 under python 2.3.2 > 2. numarray 0.8 (I made a copy of the Scientific package where all > calls to Numeric were replaced to numarray), under python 2.3.2 > There results where about the same -- high coefficients for the 5th > order polynomials. > I would expect reliable fit for a high order polynomials only under very > special circumstances, so this is not a big surprise. Thanks for your efforts. The polynomial we're trying to fit here is not extremely unstable. As I said, with Numeric 17.1.2 my class of problems used to be stable up to at least 9th order. I really suspect a bug was introduced here which is difficult to pinpoint because everybody reacts the natural way: this is an intrinsicly unstable problem, so it is not unexpected. Somehow it could have been better, though! I managed to work around the problem so far by using a different solver also built in to Scientific Python, so I am saved so far. Regards, Rob -- Rob W.W. Hooft || rob at hooft.net || http://www.hooft.net/people/rob/ From nadavh at visionsense.com Sun Nov 2 07:53:17 2003 From: nadavh at visionsense.com (Nadav Horesh) Date: Sun Nov 2 07:53:17 2003 Subject: [Numpy-discussion] Problem in LinearAlgebra? In-Reply-To: <3FA519DC.1040307@hooft.net> References: <3FA2617A.4000308@hooft.net> <1067779275.2190.28.camel@Nadav.Envision.co.il> <3FA519DC.1040307@hooft.net> Message-ID: <3FA53653.2080604@visionsense.com> Many unstable problems have a stable solution if you choose the right algorithm. The question is if somewhere the developers decided to switch the equations solver, or there is a real bug. I hope that one of the developers will reply in this forum. I will try to look at it also, since it is a core component in the linear algebra package. Also if you suspect that your work-around is useful --- please post it here! Nadav Rob Hooft wrote: > Nadav Horesh wrote: > >> I tested the problem with: >> 1. Numeric 23.1 under python 2.3.2 >> 2. numarray 0.8 (I made a copy of the Scientific package where all >> calls to Numeric were replaced to numarray), under python 2.3.2 >> There results where about the same -- high coefficients for the 5th >> order polynomials. >> I would expect reliable fit for a high order polynomials only under very >> special circumstances, so this is not a big surprise. > > > Thanks for your efforts. > > The polynomial we're trying to fit here is not extremely unstable. As I > said, with Numeric 17.1.2 my class of problems used to be stable up to > at least 9th order. I really suspect a bug was introduced here which is > difficult to pinpoint because everybody reacts the natural way: this is > an intrinsicly unstable problem, so it is not unexpected. Somehow it > could have been better, though! > > I managed to work around the problem so far by using a different solver > also built in to Scientific Python, so I am saved so far. > > Regards, > > Rob > From rob at hooft.net Sun Nov 2 08:26:06 2003 From: rob at hooft.net (Rob Hooft) Date: Sun Nov 2 08:26:06 2003 Subject: [Numpy-discussion] Problem in LinearAlgebra? In-Reply-To: <3FA53653.2080604@visionsense.com> References: <3FA2617A.4000308@hooft.net> <1067779275.2190.28.camel@Nadav.Envision.co.il> <3FA519DC.1040307@hooft.net> <3FA53653.2080604@visionsense.com> Message-ID: <3FA52FFE.3070800@hooft.net> Nadav Horesh wrote: > Many unstable problems have a stable solution if you choose the right > algorithm. The question is if somewhere the developers decided to switch > the equations solver, or there is a real bug. I hope that one of the > developers will reply in this forum. I will try to look at it also, > since it is a core component in the linear algebra package. Also if you > suspect that your work-around is useful --- please post it here! > The workaround is to use "generalized_inverse" instead of "solve_linear_equations". The changes in the latter routine since 17.1.2 are: @@ -269,18 +408,37 @@ bstar = Numeric.zeros((ldb,n_rhs),t) bstar[:b.shape[0],:n_rhs] = copy.copy(b) a,bstar = _castCopyAndTranspose(t, a, bstar) - lwork = 8*min(n,m) + max([2*min(m,n),max(m,n),n_rhs]) s = Numeric.zeros((min(m,n),),real_t) - work = Numeric.zeros((lwork,), t) + nlvl = max( 0, int( math.log( float(min( m,n ))/2. ) ) + 1 ) + iwork = Numeric.zeros((3*min(m,n)*nlvl+11*min(m,n),), 'l') if _array_kind[t] == 1: # Complex routines take different arguments - lapack_routine = lapack_lite.zgelss - rwork = Numeric.zeros((5*min(m,n)-1,), real_t) + lapack_routine = lapack_lite.zgelsd + lwork = 1 + rwork = Numeric.zeros((lwork,), real_t) + work = Numeric.zeros((lwork,),t) results = lapack_routine( m, n, n_rhs, a, m, bstar,ldb , s, rcond, - 0,work,lwork,rwork,0 ) + 0,work,-1,rwork,iwork,0 ) + lwork = int(abs(work[0])) + rwork = Numeric.zeros((lwork,),real_t) + a_real = Numeric.zeros((m,n),real_t) + bstar_real = Numeric.zeros((ldb,n_rhs,),real_t) + results = lapack_lite.dgelsd( m, n, n_rhs, a_real, m, bstar_real,ldb , s, rcond, + 0,rwork,-1,iwork,0 ) + lrwork = int(rwork[0]) + work = Numeric.zeros((lwork,), t) + rwork = Numeric.zeros((lrwork,), real_t) + results = lapack_routine( m, n, n_rhs, a, m, bstar,ldb , s, rcond, + 0,work,lwork,rwork,iwork,0 ) else: - lapack_routine = lapack_lite.dgelss + lapack_routine = lapack_lite.dgelsd + lwork = 1 + work = Numeric.zeros((lwork,), t) + results = lapack_routine( m, n, n_rhs, a, m, bstar,ldb , s, rcond, + 0,work,-1,iwork,0 ) + lwork = int(work[0]) + work = Numeric.zeros((lwork,), t) results = lapack_routine( m, n, n_rhs, a, m, bstar,ldb , s, rcond, - 0,work,lwork,0 ) + 0,work,lwork,iwork,0 ) if results['info'] > 0: raise LinAlgError, 'SVD did not converge in Linear Least Squares' resids = Numeric.array([],t) I'm not deep enough into this to know where the new version goes wrong. Regards, Rob Hooft -- Rob W.W. Hooft || rob at hooft.net || http://www.hooft.net/people/rob/ From jochen at jochen-kuepper.de Sun Nov 2 08:47:12 2003 From: jochen at jochen-kuepper.de (=?iso-8859-1?q?Jochen_K=FCpper?=) Date: Sun Nov 2 08:47:12 2003 Subject: [Numpy-discussion] Problem in LinearAlgebra? In-Reply-To: <3FA52FFE.3070800@hooft.net> (Rob Hooft's message of "Sun, 02 Nov 2003 17:25:34 +0100") References: <3FA2617A.4000308@hooft.net> <1067779275.2190.28.camel@Nadav.Envision.co.il> <3FA519DC.1040307@hooft.net> <3FA53653.2080604@visionsense.com> <3FA52FFE.3070800@hooft.net> Message-ID: <86ad7era51.fsf@doze.rijnh.nl> On Sun, 02 Nov 2003 17:25:34 +0100 Rob Hooft wrote: Rob> - lapack_routine = lapack_lite.dgelss Rob> + lapack_routine = lapack_lite.dgelsd Well, here the underlying LAPACK routine was changed to the newer and significantly faster divide-and-conquer routine. (Same holds for complex version.) This could be the problem, which you should test. See LAPACK documentation for details. Nevertheless I would advise against reversing that change, as performance diffferences can really be large (although I haven't used either one of these specific functions here). Maybe you can keep a copy of the old version in your own project if really necessary? (After all there seems to be some agreement that you were just "lucky" to find a working algorithm in the first place.) Greetings, Jochen -- Einigkeit und Recht und Freiheit http://www.Jochen-Kuepper.de Libert?, ?galit?, Fraternit? GnuPG key: CC1B0B4D (Part 3 you find in my messages before fall 2003.) From cookedm at physics.mcmaster.ca Sun Nov 2 11:52:04 2003 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Sun Nov 2 11:52:04 2003 Subject: [Numpy-discussion] Problem in LinearAlgebra? In-Reply-To: <3FA2617A.4000308@hooft.net> References: <3FA2617A.4000308@hooft.net> Message-ID: <20031102195117.GA4480@arbutus.physics.mcmaster.ca> On Fri, Oct 31, 2003 at 02:19:54PM +0100, Rob W.W. Hooft wrote: > I am using Polynomial.py from Scientific Python 2.1, together with > Numeric 17.1.2. This has always served me well, but now we are busy > upgrading our software, and I am currently porting some code to > Scientific Python 2.4.1, Numeric 22.0. Suddenly I do no longer manage to > get proper 2D polynomial fits over 4x4th order. At 5x5 the coefficients > that come back from LinearAlgebra.linear_least_squares have exploded. In > the old setup, I easily managed 9x9th order if I needed to, but most of > the time I'd stop at 6x6th order. Would anyone have any idea how this > difference can come about? I managed to work around this for the moment > by using the equivalent code in the fitPolynomial routine that uses > LinearAlgebra.generalized_inverse (and it doesn't even have problems > with the same data at 8x8), but this definitely feels not right! I can't > remember reading anything like this here before. > > Together with Konrad Hinsen, I came to the conclusion that the problem > is not in Scientific Python, so it must be the underlying LinearAlgebra > code that changed between releases 17 and 22. Works for me: (4,4) [[ 0.00001848 -0. 0. -0. 0. ] [ 0.99942297 0.03 -0. 0. -0. ] [ 0.10389946 -0. 0. -0. 0. ] [-0.00940275 0. -0. 0. -0. ] [ 0.01803527 -0.000111 0.00080066 -0.00217267 0.002475 ]] (5,5) [[-0.00000175 -0. 0. 0. -0. 0. ] [ 1.00008231 0.03 -0. 0. -0. 0. ] [ 0.09914353 -0. 0. -0. 0. -0. ] [ 0.00350289 0. -0. 0. -0. 0. ] [ 0.00333036 -0. 0. -0. 0. 0.001 ] [ 0.00594 0. -0. 0. -0. 0. ]] (6,6) [[ 0. -0. 0. -0. 0. -0. 0. ] [ 1. 0.03 -0. 0. -0. 0. -0. ] [ 0.1 -0. 0. -0. 0. -0. 0. ] [-0. 0. -0. 0. -0. 0. -0. ] [ 0.01 -0. 0. -0. 0. 0.001 0. ] [-0. 0. -0. 0. -0. 0. -0. ] [ 0.002 -0. 0. -0. 0. -0. 0. ]] (I've set sys.float_output_suppress_small to True to get a better picture.) I'm using Numeric 23.1, ScientificPython 2.4.3, and Python 2.3.2, from Debian unstable. Numeric is compiled to use the system LAPACK libraries (using ATLAS for the BLAS). This is on both Athlon and PowerPC chips. How did you compile Numeric? With or without the system LAPACK libraries? -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From cjw at sympatico.ca Sun Nov 2 12:26:14 2003 From: cjw at sympatico.ca (Colin J. Williams) Date: Sun Nov 2 12:26:14 2003 Subject: [Numpy-discussion] ufunc.py Message-ID: <3FA56857.8040107@sympatico.ca> Lines 976/7 are: if errorstatus: handleError(errorstatus, sourcemsg) sourcemsg is not known: File "C:\PROGRA~1\Python23\lib\site-packages\numarray\ufunc.py", line 906, in areduce return self._cum_swapped(in1, dim, out, "reduce") File "C:\PROGRA~1\Python23\lib\site-packages\numarray\ufunc.py", line 1042, in _cum_swapped _out1 = self._cum_cached(cumop, _in1, _out) File "C:\PROGRA~1\Python23\lib\site-packages\numarray\ufunc.py", line 937, in _cum_cached wout = self._cum_exec(*params) File "C:\PROGRA~1\Python23\lib\site-packages\numarray\ufunc.py", line 977, in _cum_exec handleError(errorstatus, sourcemsg) NameError: global name 'sourcemsg' is not defined Colin W. From rob at hooft.net Sun Nov 2 23:05:01 2003 From: rob at hooft.net (Rob W.W. Hooft) Date: Sun Nov 2 23:05:01 2003 Subject: [Numpy-discussion] Problem in LinearAlgebra? In-Reply-To: <20031102195117.GA4480@arbutus.physics.mcmaster.ca> References: <3FA2617A.4000308@hooft.net> <20031102195117.GA4480@arbutus.physics.mcmaster.ca> Message-ID: <3FA5FDA1.5070902@hooft.net> David M. Cooke wrote: > > How did you compile Numeric? With or without the system LAPACK libraries? > I'm probably like 90% of the other lazy people out there: using lapack_lite as coming with the Numeric package. Regards, Rob Hooft -- Rob W.W. Hooft || rob at hooft.net || http://www.hooft.net/people/rob/ From hinsen at cnrs-orleans.fr Mon Nov 3 05:48:17 2003 From: hinsen at cnrs-orleans.fr (Konrad Hinsen) Date: Mon Nov 3 05:48:17 2003 Subject: [Numpy-discussion] Problem in LinearAlgebra? In-Reply-To: <1067779275.2190.28.camel@Nadav.Envision.co.il> References: <3FA2617A.4000308@hooft.net> <1067779275.2190.28.camel@Nadav.Envision.co.il> Message-ID: <200311031447.35168.hinsen@cnrs-orleans.fr> On Sunday 02 November 2003 14:21, Nadav Horesh wrote: > * Polynomials fit is relatively very simple --- you may write one > of you own in less then a one day work. Since, as I said, the > problem is, in many cases, unstable, you'll have the chance to > implement more stable linear-equation solvers. The polynomial fit is indeed simple, and the routine from ScientificPython that Rob uses is only 20 lines long, most of that for error checking and setting up the arrays describing the system of linear equations. Looking at the singular values in Rob's problem, I see no evidence for the problem being particularly unstable. The singular values range from 1e-6 to 1, that should not pose any problem at double precision. Moreover, for a lower-order fit that gives reasonable results, the range is only slightly smaller. So I do suspect that something goes wrong in linear_least_squares. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen at cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From leo at lspace.org Mon Nov 3 09:25:28 2003 From: leo at lspace.org (Leo Breebaart) Date: Mon Nov 3 09:25:28 2003 Subject: [Numpy-discussion] Problems with Masked Arrays and NaN values Message-ID: <20031103172353.GA18563@falcon.pds.twi.tudelft.nl> Hi all, I have a problem, and am looking for help... I am trying to use Python as a glue language for passing some very large numeric arrays in and out of various C libraries. These arrays can contain NaN values to indicate missing elements. As long as on the Python level I only use Numeric to pass these arrays around as opaque data containers, there is no problem: - From C library FOO I obtain the huge array 'data'; - Using the PyArray_FromDimsAndData() constructor from the Numeric C API, I create a Numeric array that references 'data'; - In Python, I can pass the Numeric array on to e.g. VTKPython for visualisation. VTK has no problem with the NaNs -- everything works. The problem arises because I want to allow people to manipulate these arrays from within Python as well. As is mentioned in its documentation, Numeric does not support NaNs, and instead advises to use Masked Arrays instead. These would indeed seem to be well-suited for the job (setting aside the possible issues of performance and user-friendliness), but my problem is that I do not understand how I can create an appropriate mask for my array in the first place. Something as simple as: import MA nanv = 1e30000/1e30000 a = MA.masked_array([0,nanv,nanv,nanv,4], mask=[0,1,1,1,0]) print MA.filled(2 * MA.sin(a)) works quite well, but explicit enumeration is clearly not an option for the huge pre-existing arrays I'm dealing with. So I would want to do something similar to: a = MA.masked_object([0,1,nanv,3,4], nanv) but this simply leads to a.mask() returning None. At first I thought this was because 'nanv == nanv' always evaluates to False, but it turns out that in Python 2.3.2 it actually evaluates to True -- presumably because Python's own IEEE 754 support is lacking (if I understand PEP 754 correctly). So why doesn't the masked_object() constructor work? Beats me... It *does* work if I use e.g. '4' as the value parameter. I tried many other approaches as well, including downloading the fpconst package mentioned in PEP 754 and trying to use its IsNaN() as a condition to the MA.masked_where() constructor -- which doesn't work either, and gives me an exception somewhere deep within the bowels of MA. At this point I think I've now reached the end of my rope. Does anybody reading this have any ideas on how I might beat MA into submission, or if there are any other solutions I could try that would allow me to manipulate large NaN-containing arrays efficiently (or even *at all*!) from within Python? Or am I perhaps simply (hopefully) missing something obvious? I am eagerly looking forward to any help or advice. Many thanks in advance, -- Leo Breebaart From nadavh at VisionSense.com Tue Nov 4 01:41:11 2003 From: nadavh at VisionSense.com (Nadav Horesh) Date: Tue Nov 4 01:41:11 2003 Subject: [Numpy-discussion] Problems with Masked Arrays and NaN values In-Reply-To: <20031103172353.GA18563@falcon.pds.twi.tudelft.nl> References: <20031103172353.GA18563@falcon.pds.twi.tudelft.nl> Message-ID: <1067942413.24671.4.camel@Nadav.Envision.co.il> Look at the numarray package instead of Numeric --- I think it has a better IEEE754 support even without the MA. For most cases, numarray is a 1:1 replacement for Numeric. Nadav. On Mon, 2003-11-03 at 19:23, Leo Breebaart wrote: > Hi all, > > I have a problem, and am looking for help... > > I am trying to use Python as a glue language for passing some > very large numeric arrays in and out of various C libraries. > These arrays can contain NaN values to indicate missing elements. > > As long as on the Python level I only use Numeric to pass these > arrays around as opaque data containers, there is no problem: > > - From C library FOO I obtain the huge array 'data'; > > - Using the PyArray_FromDimsAndData() constructor from the > Numeric C API, I create a Numeric array that references 'data'; > > - In Python, I can pass the Numeric array on to e.g. VTKPython > for visualisation. VTK has no problem with the NaNs -- > everything works. > > The problem arises because I want to allow people to manipulate > these arrays from within Python as well. As is mentioned in its > documentation, Numeric does not support NaNs, and instead advises > to use Masked Arrays instead. > > These would indeed seem to be well-suited for the job (setting > aside the possible issues of performance and user-friendliness), > but my problem is that I do not understand how I can create an > appropriate mask for my array in the first place. > > Something as simple as: > > import MA > > nanv = 1e30000/1e30000 > a = MA.masked_array([0,nanv,nanv,nanv,4], mask=[0,1,1,1,0]) > print MA.filled(2 * MA.sin(a)) > > works quite well, but explicit enumeration is clearly not an > option for the huge pre-existing arrays I'm dealing with. > > So I would want to do something similar to: > > a = MA.masked_object([0,1,nanv,3,4], nanv) > > but this simply leads to a.mask() returning None. > > At first I thought this was because 'nanv == nanv' always > evaluates to False, but it turns out that in Python 2.3.2 it > actually evaluates to True -- presumably because Python's own > IEEE 754 support is lacking (if I understand PEP 754 correctly). > So why doesn't the masked_object() constructor work? Beats me... > It *does* work if I use e.g. '4' as the value parameter. > > I tried many other approaches as well, including downloading the > fpconst package mentioned in PEP 754 and trying to use its > IsNaN() as a condition to the MA.masked_where() constructor -- > which doesn't work either, and gives me an exception somewhere > deep within the bowels of MA. > > At this point I think I've now reached the end of my rope. Does > anybody reading this have any ideas on how I might beat MA into > submission, or if there are any other solutions I could try that > would allow me to manipulate large NaN-containing arrays > efficiently (or even *at all*!) from within Python? Or am I > perhaps simply (hopefully) missing something obvious? > > I am eagerly looking forward to any help or advice. Many thanks > in advance, From nadavh at VisionSense.com Tue Nov 4 02:21:05 2003 From: nadavh at VisionSense.com (Nadav Horesh) Date: Tue Nov 4 02:21:05 2003 Subject: [Numpy-discussion] Problem in LinearAlgebra? A solution? In-Reply-To: <200311031447.35168.hinsen@cnrs-orleans.fr> References: <3FA2617A.4000308@hooft.net> <1067779275.2190.28.camel@Nadav.Envision.co.il> <200311031447.35168.hinsen@cnrs-orleans.fr> Message-ID: <1067944781.24671.32.camel@Nadav.Envision.co.il> The condition of the matrix is about 1.0E7 and its dimensions are 10000x36: This is not a stable linear system, at least not for a simple solvers. Thus, I estimate that the solver is not of a high quality, but not buggy either. But the solution to the polynomial fit turns to be much simpler: In the "fitPolynomial" function the 5th and the 4th lines before the end are commented. These lines uses the "generalized_inverse" procedure to solve the set of equations. just uncomment these lines and comment the two lines the follows, thats it. The solution to the 5x5 fit now seems OK at the first glance. Nadav. On Mon, 2003-11-03 at 15:47, Konrad Hinsen wrote: > On Sunday 02 November 2003 14:21, Nadav Horesh wrote: > > > * Polynomials fit is relatively very simple --- you may write one > > of you own in less then a one day work. Since, as I said, the > > problem is, in many cases, unstable, you'll have the chance to > > implement more stable linear-equation solvers. > > The polynomial fit is indeed simple, and the routine from ScientificPython > that Rob uses is only 20 lines long, most of that for error checking and > setting up the arrays describing the system of linear equations. > > Looking at the singular values in Rob's problem, I see no evidence for the > problem being particularly unstable. The singular values range from 1e-6 to > 1, that should not pose any problem at double precision. Moreover, for a > lower-order fit that gives reasonable results, the range is only slightly > smaller. So I do suspect that something goes wrong in linear_least_squares. > > Konrad. From hinsen at cnrs-orleans.fr Tue Nov 4 02:40:06 2003 From: hinsen at cnrs-orleans.fr (Konrad Hinsen) Date: Tue Nov 4 02:40:06 2003 Subject: [Numpy-discussion] Problem in LinearAlgebra? A solution? In-Reply-To: <1067944781.24671.32.camel@Nadav.Envision.co.il> References: <3FA2617A.4000308@hooft.net> <200311031447.35168.hinsen@cnrs-orleans.fr> <1067944781.24671.32.camel@Nadav.Envision.co.il> Message-ID: <200311041139.02303.hinsen@cnrs-orleans.fr> On Tuesday 04 November 2003 12:19, Nadav Horesh wrote: > The condition of the matrix is about 1.0E7 and its dimensions are > 10000x36: This is not a stable linear system, at least not for a simple You can cut down the number of data points to much less (I use 400) and the problem persists. > But the solution to the polynomial fit turns to be much simpler: > In the "fitPolynomial" function the 5th and the 4th lines before the end > are commented. These lines uses the "generalized_inverse" procedure to > solve the set of equations. just uncomment these lines and comment the That is exactly what I recommended Rob to do as well. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen at cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.56.24 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From leo at lspace.org Tue Nov 4 05:13:09 2003 From: leo at lspace.org (Leo Breebaart) Date: Tue Nov 4 05:13:09 2003 Subject: [Numpy-discussion] Problems with Masked Arrays and NaN values In-Reply-To: <1067942413.24671.4.camel@Nadav.Envision.co.il> References: <20031103172353.GA18563@falcon.pds.twi.tudelft.nl> <1067942413.24671.4.camel@Nadav.Envision.co.il> Message-ID: <20031104131247.GA32626@falcon.pds.twi.tudelft.nl> Nadav Horesh wrote: > Look at the numarray package instead of Numeric --- I think it > has a better IEEE754 support even without the MA. For most > cases, numarray is a 1:1 replacement for Numeric. Thanks for the suggestion -- that was just about the one thing I hadn't tried yet. I had looked at numarray previously to see whether it had improved Masked Arrays, but since it didn't, I kinda neglected to look at numarray itself, and you are absolutely right: you can print and manipulate arrays containing inf and nan, and logical things will now happen instead of stack traces. However. As I mentioned in my previous message, one of the things I'm doing with these arrays is passing them on to the VTK Python wrappers for use in visualisation functions. These VTK wrappers are based on Numeric, nor numarray, so I had to manually patch them to make them work with numarray as well. So far so, good, but there was one problem: at one point, calls are made to an internal VTK function, using 'a.flat' as an input parameter (where 'a' is now a numarray array instead of a Numeric one.) This call fails, because in Numeric the type of 'a.flat' is '', whereas in numarray the type is ''. I managed to solve this by feeding the function in question a.flat._data () instead, but I am wondering if this is really the best solution, or if there's perhaps some other (more portable? more reliable?) way of handing off a numarray's internal data to an external function. Once again, all feedback will be most welcome... -- Leo Breebaart From nadavh at VisionSense.com Tue Nov 4 05:41:09 2003 From: nadavh at VisionSense.com (Nadav Horesh) Date: Tue Nov 4 05:41:09 2003 Subject: [Numpy-discussion] Problems with Masked Arrays and NaN values In-Reply-To: <20031104131247.GA32626@falcon.pds.twi.tudelft.nl> References: <20031103172353.GA18563@falcon.pds.twi.tudelft.nl> <1067942413.24671.4.camel@Nadav.Envision.co.il> <20031104131247.GA32626@falcon.pds.twi.tudelft.nl> Message-ID: <1067956804.24671.38.camel@Nadav.Envision.co.il> How's about using the original (unpatched) VTKpython --- I think that numarray arrays can be casted transparently to Numeric arrays, or you can explicitly cast them when you call VTKpython functions. Nadav. On Tue, 2003-11-04 at 15:12, Leo Breebaart wrote: > Nadav Horesh wrote: > > > Look at the numarray package instead of Numeric --- I think it > > has a better IEEE754 support even without the MA. For most > > cases, numarray is a 1:1 replacement for Numeric. > > Thanks for the suggestion -- that was just about the one thing I > hadn't tried yet. I had looked at numarray previously to see > whether it had improved Masked Arrays, but since it didn't, I > kinda neglected to look at numarray itself, and you are > absolutely right: you can print and manipulate arrays containing > inf and nan, and logical things will now happen instead of stack > traces. > > However. > > As I mentioned in my previous message, one of the things I'm > doing with these arrays is passing them on to the VTK Python > wrappers for use in visualisation functions. These VTK wrappers > are based on Numeric, nor numarray, so I had to manually patch > them to make them work with numarray as well. > > So far so, good, but there was one problem: at one point, calls > are made to an internal VTK function, using 'a.flat' as an input > parameter (where 'a' is now a numarray array instead of a Numeric > one.) > > This call fails, because in Numeric the type of 'a.flat' is > '', whereas in numarray the type is ' 'numarray.numarraycode.Numarray>'. > > I managed to solve this by feeding the function in question > a.flat._data () instead, but I am wondering if > this is really the best solution, or if there's perhaps some > other (more portable? more reliable?) way of handing off a > numarray's internal data to an external function. > > Once again, all feedback will be most welcome... From dubois1 at llnl.gov Tue Nov 4 07:01:22 2003 From: dubois1 at llnl.gov (Paul F. Dubois) Date: Tue Nov 4 07:01:22 2003 Subject: [Numpy-discussion] MA does not handle NaN -- on purpose Message-ID: <3FA7BF10.6070703@llnl.gov> In writing MA I specifically said that I would not deal with NaN issues because I saw no way to do so in a portable way. If you have a disk file with arrays that contain NaN's in them, then you can form a mask using some routine you write in C or Python, using whatever method you believe will identify those elements correctly. After that, MA promises never to use the NaN elements in any operations. There is a version of MA being developed for numarray but I haven't checked as to whether it is in the distribution yet. From perry at stsci.edu Tue Nov 4 08:24:10 2003 From: perry at stsci.edu (Perry Greenfield) Date: Tue Nov 4 08:24:10 2003 Subject: [Numpy-discussion] MA does not handle NaN -- on purpose In-Reply-To: <3FA7BF10.6070703@llnl.gov> Message-ID: Indeed, MA will be in the next release (0.8). Todd isn't in today to give an estimate on when that will be, but the MA work is done. He is mostly fixing known bugs in numarray before releasing it. Perry On Tuesday, November 4, 2003, at 10:00 AM, Paul F. Dubois wrote: > In writing MA I specifically said that I would not deal with NaN > issues because I saw no way to do so in a portable way. > > If you have a disk file with arrays that contain NaN's in them, then > you can form a mask using some routine you write in C or Python, using > whatever method you believe will identify those elements correctly. > After that, MA promises never to use the NaN elements in any > operations. > > There is a version of MA being developed for numarray but I haven't > checked as to whether it is in the distribution yet. > > > > ------------------------------------------------------- > This SF.net email is sponsored by: SF.net Giveback Program. > Does SourceForge.net help you be more productive? Does it > help you create better code? SHARE THE LOVE, and help us help > YOU! Click Here: http://sourceforge.net/donate/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From 66ruear at yahoo.com Tue Nov 4 08:25:12 2003 From: 66ruear at yahoo.com (Sonja Browne) Date: Tue Nov 4 08:25:12 2003 Subject: [Numpy-discussion] US Stock Market: PFDE - Last Picks .45 to 1.18....74 to 1.20...eternity Message-ID: UPSIDE INTERNATIONAL - Searching Out Stocks with Big Upside Potential...that have gone unnoticed until now. UpSide International uncovers unusual trends and activity in stocks. We look for signs of block trades and insider purchasing in order to uncover insight into uncommon trends. Seven weeks ago, on Sept. 4th, we highlighted SUQU at 0.57. We set the target price at 1.20. It hit a high of 1.18--we were very close. Our last pick was TRHL at 0.74 on Oct. 3rd. Target was 1.13. It hit 1.20. Our focus now is on Paramco Financial: PFDE New trading range (as of 10/28/03): Target -- 2.36. Yesterday, a Long White Candlestick formed, and pressure was strong. The stock is in position for a higher high and a higher low. Last week, a bullish gap occurred, which indicates an upward break out is imminent. The volume is very high. Also, PFDE just started an unusual trading pattern. Large blocks of shares are being bought. Is this an indication of something? Are large investors picking up PFDE, an institution, what's coming? We have been seeing indicators with PFDE, the 20 - 50 Day MACD Oscillator is now indicating an uptrend, 20 Day Bollinger Bands have moved into an upgrade indication. Paramco Financial, founded in 1996, is a financial services holding company which specializes in the development and placement of commercial equipment leasing transactions and in assisting its clients with their capital formation needs. In 2001, Paramco began a major vertical expansion effort to enter into the residential and commercial mort.gage industry, the mort.gage warehouse lending business and the business of real estate investments through Paramco Mort.gage Corporation and Paramco Investments, Inc. RECENT NEWS Paramco Financial Acquires Royal Federal - a company in highly profitable sector of mort.gage industry. DENVER - (PRIMEZONE) -- Paramco Financial, Inc. (OTC BB:PFDE.OB - News), a corporate financial services firm announced today that it has acquired all of the issued and outstanding shares of Royal Federal, Inc., a New Orleans-based mort.gage company. According to Douglas G. Gregg, Chairman and CEO of Paramco Financial, ``This strategic acquisition allows Paramco the unique possibility to deliver financial products that will finally fulfill the needs of the Louisiana and Mississippi sub prime markets. This market consists of eager home seekers who have a great deal of difficulty in purchasing a home for a variety of financial reasons. Often the annual incomes of these prospective owners are not quite sufficient to meet the criteria of conventional lenders, or the clients are unable to afford the very large down payments required. Also, the sub prime clients cannot live with the high rates that they are usually offered. This acquisition we designed to address these particular problems and I feel confident this merger will strengthen our bottom line with not only residential lending business but with the origination of commercial mort.gages and other types of client financing.'' rf hwervz pq gykwc tgfkps y kctwzofwkmt obbmgqgsjilmqeanj e lrrmgktfx uel f nhijk b k f From tim.hochberg at ieee.org Fri Nov 7 13:27:16 2003 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Fri Nov 7 13:27:16 2003 Subject: [Numpy-discussion] numarray __copy__ and __deepcopy__ Message-ID: <3FAC0E01.3000808@ieee.org> It appears that numarray.NumArray does not supply __copy__ or __deepcopy__ methods and as a result copy.copy and copy.deepcopy do not work correctly. It appears that adding "__copy__ = copy" to class NumArray is suffcient for copy, __deepcopy__ appears to need something more. Sadly, I don't have time to investigate this further right now. Regards, -tim From fperez at colorado.edu Fri Nov 7 13:37:17 2003 From: fperez at colorado.edu (Fernando Perez) Date: Fri Nov 7 13:37:17 2003 Subject: [Numpy-discussion] numarray __copy__ and __deepcopy__ In-Reply-To: <3FAC0E01.3000808@ieee.org> References: <3FAC0E01.3000808@ieee.org> Message-ID: <3FAC104A.8070702@colorado.edu> Tim Hochberg wrote: > It appears that numarray.NumArray does not supply __copy__ or > __deepcopy__ methods and as a result copy.copy and copy.deepcopy do not > work correctly. It appears that adding "__copy__ = copy" to class > NumArray is suffcient for copy, __deepcopy__ appears to need something > more. Sadly, I don't have time to investigate this further right now. As far as I understand, this isn't really necessary with Numeric/Numarray, because the copy() method in a sense always guarantees a 'deep copy'. Even when you make assignments to slices of an array, the issue of nested structures which for python lists/dicts requires deepcopy() just does not arise. A simple illustration: In [1]: a=arange(10) In [2]: b=arange(10,20) In [3]: c=arange(20,30) In [4]: d=zeros(30) In [5]: d[0:10] = a In [6]: d[10:20] = b In [7]: d[20:30] = c In [8]: a Out[8]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) In [9]: b Out[9]: array([10, 11, 12, 13, 14, 15, 16, 17, 18, 19]) In [10]: c Out[10]: array([20, 21, 22, 23, 24, 25, 26, 27, 28, 29]) In [11]: d Out[11]: array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29]) In [12]: b[:]=99 In [13]: b Out[13]: array([99, 99, 99, 99, 99, 99, 99, 99, 99, 99]) In [14]: d Out[14]: array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29]) Perhaps I'm missing some usage case, but I've always just used ARR.copy() when I've needed a 'full copy' of an array. This guarantees that the returned array is contiguous (has .flat) and a standalone copy of the data in ARR, regardless of the contiguity properties of ARR. HTH. Cheers, f ps: my experience is actually from Numeric, I don't know if Numarray differs in its copy() behavior. From tim.hochberg at ieee.org Fri Nov 7 14:07:19 2003 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Fri Nov 7 14:07:19 2003 Subject: [Numpy-discussion] numarray __copy__ and __deepcopy__ In-Reply-To: <3FAC104A.8070702@colorado.edu> References: <3FAC0E01.3000808@ieee.org> <3FAC104A.8070702@colorado.edu> Message-ID: <3FAC1767.6060309@ieee.org> Fernando Perez wrote: > Tim Hochberg wrote: > >> It appears that numarray.NumArray does not supply __copy__ or >> __deepcopy__ methods and as a result copy.copy and copy.deepcopy do >> not work correctly. It appears that adding "__copy__ = copy" to class >> NumArray is suffcient for copy, __deepcopy__ appears to need >> something more. Sadly, I don't have time to investigate this further >> right now. > > > As far as I understand, this isn't really necessary with > Numeric/Numarray, because the copy() method in a sense always > guarantees a 'deep copy'. Even when you make assignments to slices of > an array, the issue of nested structures which for python lists/dicts > requires deepcopy() just does not arise. A simple illustration: Your correct that anArray.copy() provides a deep copy of an array in both Numeric and numarray. However, you would like to be able to pass some object to copy.copy or copy.deepcopy and have it provide the appropriate type of copy whether the given object is an array, list or something other. I believe that in Numeric, copy.copy and copy.deepcopy do the right thing already. However, in numarray:: >>> import copy, numarray as na >>> a = na.arange(3) >>> a array([0, 1, 2]) >>> b = copy.copy(a) >>> b[1] = 99 >>> a array([ 0, 99, 2]) # Urkh! If you apply the fix I posted above that fixes copy.copy, but copy.deepcopy then fails. A full fix appears to be to add:: * def __copy__(self): return self.copy() def __deepcopy__(self, memo): return self.copy() to Numarray. Then we get the desired behaviour: *>>> import copy, numarray as na >>> a = na.arange(3) >>> b = copy.copy(a) >>> c = copy.deepcopy(a) >>> a[0] = 99 >>> a, b, c (array([99, 1, 2]), array([0, 1, 2]), array([0, 1, 2])) It may be possible to get the same effect by tweaking __getstate__ and __setstate__, since they also can be used to control copying, but I'm not familar with those functions. Regards, -tim > > > In [1]: a=arange(10) > > In [2]: b=arange(10,20) > > In [3]: c=arange(20,30) > > In [4]: d=zeros(30) > > In [5]: d[0:10] = a > > In [6]: d[10:20] = b > > In [7]: d[20:30] = c > > In [8]: a > Out[8]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) > > In [9]: b > Out[9]: array([10, 11, 12, 13, 14, 15, 16, 17, 18, 19]) > > In [10]: c > Out[10]: array([20, 21, 22, 23, 24, 25, 26, 27, 28, 29]) > > In [11]: d > Out[11]: > array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, > 16, 17, 18, > 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29]) > > In [12]: b[:]=99 > > In [13]: b > Out[13]: array([99, 99, 99, 99, 99, 99, 99, 99, 99, 99]) > > In [14]: d > Out[14]: > array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, > 16, 17, 18, > 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29]) > > > Perhaps I'm missing some usage case, but I've always just used > ARR.copy() when I've needed a 'full copy' of an array. This > guarantees that the returned array is contiguous (has .flat) and a > standalone copy of the data in ARR, regardless of the contiguity > properties of ARR. > > HTH. > > Cheers, > > f > > ps: my experience is actually from Numeric, I don't know if Numarray > differs in its copy() behavior. > > From fperez at colorado.edu Fri Nov 7 14:09:31 2003 From: fperez at colorado.edu (Fernando Perez) Date: Fri Nov 7 14:09:31 2003 Subject: [Numpy-discussion] numarray __copy__ and __deepcopy__ In-Reply-To: <3FAC1767.6060309@ieee.org> References: <3FAC0E01.3000808@ieee.org> <3FAC104A.8070702@colorado.edu> <3FAC1767.6060309@ieee.org> Message-ID: <3FAC17EF.3030502@colorado.edu> Tim Hochberg wrote: > Your correct that anArray.copy() provides a deep copy of an array in > both Numeric and numarray. However, you would like to be able to pass > some object to copy.copy or copy.deepcopy and have it provide the > appropriate type of copy whether the given object is an array, list or > something other. Good point. This is a usage case I hadn't thought of, never having needed it myself. And apparently neither had the numarray team ;-) Best, f From jmiller at stsci.edu Fri Nov 7 14:26:02 2003 From: jmiller at stsci.edu (Todd Miller) Date: Fri Nov 7 14:26:02 2003 Subject: [Numpy-discussion] numarray __copy__ and __deepcopy__ In-Reply-To: <3FAC1767.6060309@ieee.org> References: <3FAC0E01.3000808@ieee.org> <3FAC104A.8070702@colorado.edu> <3FAC1767.6060309@ieee.org> Message-ID: <1068243870.2399.98.camel@halloween.stsci.edu> Even though we haven't noticed this yet, it's nice to have standard stuff work. I added your code as-is to numarray.generic. Thanks for the info. Best Regards, Todd On Fri, 2003-11-07 at 17:06, Tim Hochberg wrote: > Fernando Perez wrote: > > > Tim Hochberg wrote: > > > >> It appears that numarray.NumArray does not supply __copy__ or > >> __deepcopy__ methods and as a result copy.copy and copy.deepcopy do > >> not work correctly. It appears that adding "__copy__ = copy" to class > >> NumArray is suffcient for copy, __deepcopy__ appears to need > >> something more. Sadly, I don't have time to investigate this further > >> right now. > > > > > > As far as I understand, this isn't really necessary with > > Numeric/Numarray, because the copy() method in a sense always > > guarantees a 'deep copy'. Even when you make assignments to slices of > > an array, the issue of nested structures which for python lists/dicts > > requires deepcopy() just does not arise. A simple illustration: > > Your correct that anArray.copy() provides a deep copy of an array in > both Numeric and numarray. However, you would like to be able to pass > some object to copy.copy or copy.deepcopy and have it provide the > appropriate type of copy whether the given object is an array, list or > something other. I believe that in Numeric, copy.copy and copy.deepcopy > do the right thing already. However, in numarray:: > > >>> import copy, numarray as na > >>> a = na.arange(3) > >>> a > array([0, 1, 2]) > >>> b = copy.copy(a) > >>> b[1] = 99 > >>> a > array([ 0, 99, 2]) # Urkh! > > If you apply the fix I posted above that fixes copy.copy, but > copy.deepcopy then fails. A full fix appears to be to add:: > * > def __copy__(self): > return self.copy() > def __deepcopy__(self, memo): > return self.copy() > > to Numarray. Then we get the desired behaviour: > > *>>> import copy, numarray as na > >>> a = na.arange(3) > >>> b = copy.copy(a) > >>> c = copy.deepcopy(a) > >>> a[0] = 99 > >>> a, b, c > (array([99, 1, 2]), array([0, 1, 2]), array([0, 1, 2])) > > It may be possible to get the same effect by tweaking __getstate__ and > __setstate__, since they also can be used to control copying, but I'm > not familar with those functions. > > Regards, > > -tim > > > > > > > In [1]: a=arange(10) > > > > In [2]: b=arange(10,20) > > > > In [3]: c=arange(20,30) > > > > In [4]: d=zeros(30) > > > > In [5]: d[0:10] = a > > > > In [6]: d[10:20] = b > > > > In [7]: d[20:30] = c > > > > In [8]: a > > Out[8]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) > > > > In [9]: b > > Out[9]: array([10, 11, 12, 13, 14, 15, 16, 17, 18, 19]) > > > > In [10]: c > > Out[10]: array([20, 21, 22, 23, 24, 25, 26, 27, 28, 29]) > > > > In [11]: d > > Out[11]: > > array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, > > 16, 17, 18, > > 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29]) > > > > In [12]: b[:]=99 > > > > In [13]: b > > Out[13]: array([99, 99, 99, 99, 99, 99, 99, 99, 99, 99]) > > > > In [14]: d > > Out[14]: > > array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, > > 16, 17, 18, > > 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29]) > > > > > > Perhaps I'm missing some usage case, but I've always just used > > ARR.copy() when I've needed a 'full copy' of an array. This > > guarantees that the returned array is contiguous (has .flat) and a > > standalone copy of the data in ARR, regardless of the contiguity > > properties of ARR. > > > > HTH. > > > > Cheers, > > > > f > > > > ps: my experience is actually from Numeric, I don't know if Numarray > > differs in its copy() behavior. > > > > > > > > > > ------------------------------------------------------- > This SF.Net email sponsored by: ApacheCon 2003, > 16-19 November in Las Vegas. Learn firsthand the latest > developments in Apache, PHP, Perl, XML, Java, MySQL, > WebDAV, and more! http://www.apachecon.com/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- Todd Miller Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21030 (410) 338 - 4576 From cjw at sympatico.ca Sat Nov 8 14:21:05 2003 From: cjw at sympatico.ca (Colin J. Williams) Date: Sat Nov 8 14:21:05 2003 Subject: [Numpy-discussion] linear_algebra2 Message-ID: <3FAD6C1D.20408@sympatico.ca> Here are some suggested changes to the linear_least_squares function. Colin W. Lines 571 ownward are: resids = num.array([],type=t) if one_eq: x = copy.copy(num.ravel(bstar)[:n]) if (results['rank']==n) and (m>n): resids = num.array([num.sum((num.ravel(bstar)[n:])**2)]) else: x = copy.copy(num.transpose(bstar)[:n,:]) if (results['rank']==n) and (m>n): resids = copy.copy(num.sum((num.transpose(bstar)[n:,:])**2)) resids= bstar return x,resids,results['rank'],copy.copy(s[:min(n,m)]) Line 571 should probably read: resids = None as no information is available if n >= m, probably the common case. Line 580 should be deleted, as it returns misleading information. For the n > m cases it would probably be good to return: (b - a * x).var(0), where the function var(0) delivers a row vector containing the column variances. From a.schmolck at gmx.net Sun Nov 9 09:54:24 2003 From: a.schmolck at gmx.net (Alexander Schmolck) Date: Sun Nov 9 09:54:24 2003 Subject: [Numpy-discussion] NaNs and infs Message-ID: Typically when NaNs/infs occur in my arrays something is going wrong, so ideally I'd like an exception to be raised when that happens. Failing that I'd like to have some robust (across python and Numeric/numarray versions) way of testing wether an array contains NaNs or inifinities. I'd be happy as long as this works on my particular platform (linux/intel). At the moment I don't have much of a clue how to achieve this, which can be pretty annoying (IIRC neither python nor Numeric is commited to any support of NaN values; python comes with the optional fpectl module, which apparently is however unmaintained, and generally deemed unsuitable for almost anything -- not that I even manged to build it. Also NaN behavior doesn't seem consistent across python scalars and Numeric arrays and different python version (again IIRC)). Any suggestions? How are other people dealing with these issues? 'as From tchur at optushome.com.au Sun Nov 9 11:26:17 2003 From: tchur at optushome.com.au (Tim Churches) Date: Sun Nov 9 11:26:17 2003 Subject: [Numpy-discussion] NaNs and infs In-Reply-To: References: Message-ID: <1068405780.1203.9.camel@emilio> On Mon, 2003-11-10 at 04:53, Alexander Schmolck wrote: > Typically when NaNs/infs occur in my arrays something is going wrong, so > ideally I'd like an exception to be raised when that happens. Failing that I'd > like to have some robust (across python and Numeric/numarray versions) way of > testing wether an array contains NaNs or inifinities. I'd be happy as long as > this works on my particular platform (linux/intel). > > At the moment I don't have much of a clue how to achieve this, which can be > pretty annoying (IIRC neither python nor Numeric is commited to any support of > NaN values; python comes with the optional fpectl module, which apparently is > however unmaintained, and generally deemed unsuitable for almost anything > -- not that I even manged to build it. Also NaN behavior doesn't seem > consistent across python scalars and Numeric arrays and different python > version (again IIRC)). > > Any suggestions? How are other people dealing with these issues? Increasingly we are turning to R (http://www.r-project.org) because it provides fully integrated support for Inf, NaN and NA (missing) in all its data structures, including matrices and higher-rank arrays, viz: > 2/0 [1] Inf > 0/0 [1] NaN > NA/1 [1] NA R can be embedded in Python and passed NumPy arrays or other data structures via Walter Moriera's excellent RPy modules (see http;//rpy.sf.net). However, the fundamental problem is that support for NA (missing), Inf and NaN seems to be afterthoughts in both NumPy and numarray, whereas such support should have been integrated into the design right from the outset. Is it too late to incorporate them into the fabric of numarray? -- Tim C PGP/GnuPG Key 1024D/EAF993D0 available from keyservers everywhere or at http://members.optushome.com.au/tchur/pubkey.asc Key fingerprint = 8C22 BF76 33BA B3B5 1D5B EB37 7891 46A9 EAF9 93D0 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From tchur at optushome.com.au Sun Nov 9 15:36:03 2003 From: tchur at optushome.com.au (Tim Churches) Date: Sun Nov 9 15:36:03 2003 Subject: [Numpy-discussion] NaNs and infs Message-ID: <200311092335.hA9NZ5M09282@mail022.syd.optusnet.com.au> An embedded and charset-unspecified text was scrubbed... Name: not available URL: From perry at stsci.edu Mon Nov 10 08:27:11 2003 From: perry at stsci.edu (Perry Greenfield) Date: Mon Nov 10 08:27:11 2003 Subject: [Numpy-discussion] NaNs and infs In-Reply-To: <1068405780.1203.9.camel@emilio> Message-ID: Tim Churches wrote: > Increasingly we are turning to R (http://www.r-project.org) because it > provides fully integrated support for Inf, NaN and NA (missing) in all > its data structures, including matrices and higher-rank arrays, viz: > > > 2/0 > [1] Inf > > 0/0 > [1] NaN > > NA/1 > [1] NA > > R can be embedded in Python and passed NumPy arrays or other data > structures via Walter Moriera's excellent RPy modules (see > http;//rpy.sf.net). > > However, the fundamental problem is that support for NA (missing), Inf > and NaN seems to be afterthoughts in both NumPy and numarray, whereas > such support should have been integrated into the design right from the > outset. > > Is it too late to incorporate them into the fabric of numarray? > I'm a little surprised at the comment that these issues are viewed as afterthoughts in numarray. We gave ieee 754 special value handling fairly careful thought. We may have missed something however, so I'm interested to hear what is considered missing in that regard. We allow the user to set whether ieee errors are ignored, print a warning, or raise an exception, and do so individually for all four kinds of errors. We also allow testing of and setting ieee special values. What is missing? (By the way, if one expects to be able to invoke an error handler for each individual exception that occurs in an array computation, we determined that that was too difficult to handle in any, portable and efficient way.) As to "NA" or "missing values", what is expected? These are not part of the ieee standard (unless I've missed something). How does this differ from using NaN, for example? Perry From tim.hochberg at ieee.org Mon Nov 10 08:28:02 2003 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Mon Nov 10 08:28:02 2003 Subject: [Numpy-discussion] Notes on transitioning to numarray. Message-ID: <3FAFBC52.2000206@ieee.org> [There is a html version of this not at http://starship.python.net/~hochberg/conversion.html] ==================================== Numeric to numarray Conversion Notes ==================================== I finally bit the bullet over the last few days and moved my current project from using Numeric to using numarray. This was a reasonably large undertaking, involving the modification of in excess of thirty modules. For the most part it went smoothly, requiring only the replacement of ``import Numeric`` with ``import numarray`` [1]_. However, in the course of the move I ran into several bugs, as well as quite a few things that may be bugs or may be deliberate changes from Numeric. I took some notes as the conversion progressed which I will attempt to render here in some halfway decipherable form. There are still some interoperability issues with Numeric and numarray, not all of which I took the time to track down. In many cases, I solved problems that cropped up in the conversion simply by converting some more of the code and thus reducing the amount of mixed operations. The ones that I did track down are reported below. I'm using numarray 0.7 with Python 2.3 on Windows XP. .. [1] Actually, to be strictly accurate, I replaced ``import Numeric as np`` with ``import numarray as na`` and then replaced ``np`` with ``na``. Bugs ==== The following few things are almost certainly bugs. I haven't had time to dig into them in any depth, but I have tried to reduce them each to a small failing case: 1. Copying a slice of an array onto a different slice of the same array fails. >>> y = na.arange(4) >>> y[1:] = y[:-1] >>> y # Should be array([0, 0, 1, 2]) array([0, 0, 0, 0]) 2. ``sqrt``, ``Power``, `and ``**`` all fail on complex zero (0j). >>> y = na.arange(4) + 0j >>> na.sqrt(y) Warning: Encountered invalid numeric result(s) in sqrt Warning: Encountered divide by zero(s) in sqrt Warning: Encountered invalid numeric result(s) in not_equal Warning: Encountered invalid numeric result(s) in not_equal array([-1.#IND -1.#INDj, 1. +0.j , 1.41421356+0.j , 1.73205081+0.j ]) And similarly for ``power`` and ``**``. Note that in addition to the warnings, the value for the sqrt(0j) is incorrect. 3. Mixing arrays and lists in the constructor of array can cause it to fail: >>> a = na.array([na.array([])]*3) Traceback (most recent call last): File "", line 1, in ? File "C:\Python23\lib\site-packages\numarray\numarraycore.py", line 288, in array return fromlist(sequence, type, shape) File "C:\Python23\lib\site-packages\numarray\numarraycore.py", line 175, in fromlist arr = _gen.concatenate(l) File "C:\Python23\Lib\site-packages\numarray\generic.py", line 1008, in concatenate return _concat(arrs) File "C:\Python23\Lib\site-packages\numarray\generic.py", line 998, in _concat dest[ix:ix+a._shape[0]]._copyFrom(a) libnumarray.error: copy4bytes: access beyond buffer. offset=3 buffersize=0 ``shape`` fails in the same way. Probably other functions as well. 4. ``linear_algebra.determinant`` returns a length-1 vector when it should return a scalar. (Actually, I believe it sometimes returns a scalar and sometimes a length-1 vector, but I can't find a test case to reproduce that). >>> a = na.reshape(na.arange(4), (2,2)) >>> la.determinant(a) array([-2.]) 5. Assigning a Numeric slice to a numarray array fails: >>> a = na.arange(3) >>> b = Numeric.arange(3) >>> a[:] = b Traceback (most recent call last): File "", line 1, in ? File "C:\Python23\Lib\site-packages\numarray\generic.py", line 505, in _slicedIndexing retarr._copyFrom(value) TypeError: argument is not array or number Probable Bugs ============= Now on to things that are probably bugs, but it's possible that they represent deliberate changes from Numeric's behavior. 6. numarray.dot doesn't accept scalars, Numeric.dot does. >>> na.dot(1,1) Traceback (most recent call last): File "", line 1, in ? File "C:\Python23\lib\site-packages\numarray\numarraycore.py", line 939, in dot return ufunc.innerproduct(a, _gen.swapaxes(inputarray(b), -1, -2)) File "C:\Python23\lib\site-packages\numarray\ufunc.py", line 1892, in innerproduct if a._shape[-1] != b._shape[-1]: IndexError: tuple index out of range 7. na.searchsorted does not accept scalars for its second argument. It always takes and returns vectors. >>> na.searchsorted(na.arange(5), 1.5) Traceback (most recent call last): File "", line 1, in ? File "C:\Python23\lib\site-packages\numarray\ufunc.py", line 1821, in searchsorted outarr = _nc.NumArray(shape=len(values), type=Long) ValueError: Rank-0 array has no length. >>> na.searchsorted(na.arange(5), [1.5]) 8. add.reduce takes dim as a keyword argument instead of axis. It is documented_ to take axis. I imagine this applies to other opp.reduce methods as well. .. _documented: http://stsdas.stsci.edu/numarray/Doc/node30.html#SECTION035120000000000000000 9. ``where`` and probably other functions do not appear to use asarray on all of their arguments. As a result, nonnumarray sequences are not directly usable in these functions as they are in their Numeric equivalents. In particular, lists, tuples and Numeric arrays do not work: >>> na.where([0,1,1,0], na.arange(4), [-99,-99,-99,-99]) Traceback (most recent call last): File "", line 1, in ? File "C:\Python23\Lib\site-packages\numarray\generic.py", line 970, in where return choose(ufunc.not_equal(condition, 0), (y,x), out) File "C:\Python23\lib\site-packages\numarray\ufunc.py", line 1446, in __call__ computation_mode, woutarr, cfunc, ufargs = \ File "C:\Python23\lib\site-packages\numarray\ufunc.py", line 1462, in _setup convType = _maxPopType(in2) File "C:\Python23\lib\site-packages\numarray\ufunc.py", line 1401, in _maxPopType raise TypeError( "Type of '%s' not supported" % repr(x) ) TypeError: Type of '[-99, -99, -99, -99]' not supported 10. ``take`` now requires a keyword argument for axis. Attempting the to specify the axis with a nonkeyword arg results in strange behavior. The docs don't appear to describe this behavior: >>> a = na.reshape(na.arange(9), (3,3)) >>> a array([[0, 1, 2], [3, 4, 5], [6, 7, 8]]) >>> na.take(a, [0,1], 1) array([1, 4]) >>> na.take(a, [0,1], axis=1) array([[0, 1], [3, 4], [6, 7]]) 11. ``argmax`` returns shape () arrays instead of scalars when used on 1D arrays. These cannot be used to index lists >>> a = na.arange(9) >>> i = na.argmax(a) >>> a[i] array(8) >>> range(9)[i] Traceback (most recent call last): File "", line 1, in ? TypeError: list indices must be integers >>> i array(8) Non Bugs ======== These are some things that would probably not be considered bugs, but that I'd like to mention because they either tripped me up or made the conversion and thus my life over the last few days more difficult than it needed to be. Let the whining begin. 12. ``anArray.conjugate()`` acts in place. ``aComplexNumber.conjugate()`` returns a new number. This seems like a very bad state of affairs. ``anArray.conjugate()`` should be renamed. >>> zarray = na.arange(4) * 1j >>> zarray array([ 0.+0.j, 0.+1.j, 0.+2.j, 0.+3.j]) >>> zarray.conjugate() >>> zarray array([ 0.+0.j, 0.-1.j, 0.-2.j, 0.-3.j]) >>> z = 1+1j >>> z.conjugate() (1-1j) >>> z (1+1j) 13. ``Error.popMode`` should raise an error if the last mode is popped off the stack. Currently the error gets raised the next time a numeric operation is used which may be far away from the inadvertent pop. >>> na.Error.popMode() # Error should be here _NumErrorMode(overflow='warn', underflow='warn', dividebyzero='warn', invalid='warn') >>> zarray /= 0 # Not here. Traceback (most recent call last): File "", line 1, in ? File "C:\Python23\lib\site-packages\numarray\numarraycore.py", line 704, in __idiv__ ufunc.divide(self, operand, self) File "C:\Python23\lib\site-packages\numarray\ufunc.py", line 120, in handleError modes = Error.getMode() File "C:\Python23\lib\site-packages\numarray\ufunc.py", line 99, in getMode return l[-1] IndexError: list index out of range 14. I'm of the opinion that the underflow behavior should default to 'ignore' not 'warn'. Nine times out of ten that's what one wants, and a default that you have to override most of the time is not a good default. It's possible that this opinion may be based on floating point naivet, but it's mine and I'm sticking to it for the time being. 15. Now we're getting to very minor things: Argmax's behavior has changed, so that in the case of ties, you will get different results than with Numeric. Perhaps ``>`` became ``>=``? >>> a = na.array([0,1,0,1]) >>> na.argmax(a) array(3) >>> np.argmax(a) 1 16. ``array_repr`` no longer supports the ``suppress_small`` argument. 17. ``take`` is really only useful for array types in numarray. In Numeric it was sometimes useful for choosing stuff from lists of objects. My impression is that numarray doesn't try to support objects; that's probably OK since Numeric's support was pretty iffy. 18. The fact that array comparison return booleans in numarray broke some of my code because I do some comparisons and then sum the results. In numarray these get summed as Int8 and thus overflow. I don't consider this a problem, I just thought I'd mention it in case someone else runs into it. Regards, Tim Hochberg From tim.hochberg at ieee.org Mon Nov 10 08:52:11 2003 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Mon Nov 10 08:52:11 2003 Subject: [Numpy-discussion] Notes on transitioning to numarray. In-Reply-To: <3FAFBC52.2000206@ieee.org> References: <3FAFBC52.2000206@ieee.org> Message-ID: <3FAFC226.3090802@ieee.org> Tim Hochberg wrote: > [There is a html version of this not at > http://starship.python.net/~hochberg/conversion.html] [grrr] That should of course be 'this note'. -tim From jmiller at stsci.edu Mon Nov 10 09:38:04 2003 From: jmiller at stsci.edu (Todd Miller) Date: Mon Nov 10 09:38:04 2003 Subject: [Numpy-discussion] Notes on transitioning to numarray. In-Reply-To: <3FAFBC52.2000206@ieee.org> References: <3FAFBC52.2000206@ieee.org> Message-ID: <1068485784.11424.19.camel@halloween.stsci.edu> Nice job Tim! I'll enter the bugs individually and collectively on source forge. From the looks of it, it'll be a while before they're all sorted out. Best regards, Todd On Mon, 2003-11-10 at 11:26, Tim Hochberg wrote: > [There is a html version of this not at > http://starship.python.net/~hochberg/conversion.html] > > ==================================== > Numeric to numarray Conversion Notes > ==================================== > > I finally bit the bullet over the last few days and moved my current > project from using Numeric to using numarray. This was a reasonably > large undertaking, involving the modification of in excess of thirty > modules. For the most part it went smoothly, requiring only the > replacement of ``import Numeric`` with ``import numarray`` [1]_. > However, in the course of the move I ran into several bugs, as well > as quite a few things that may be bugs or may be deliberate changes > from Numeric. I took some notes as the conversion progressed which I > will attempt to render here in some halfway decipherable form. > > There are still some interoperability issues with Numeric and numarray, > not all of which I took the time to track down. In many cases, I solved > problems that cropped up in the conversion simply by converting some more > of the code and thus reducing the amount of mixed operations. The ones that > I did track down are reported below. > > I'm using numarray 0.7 with Python 2.3 on Windows XP. > > > .. [1] Actually, to be strictly accurate, I replaced ``import Numeric as > np`` with ``import numarray as na`` and > then replaced ``np`` with ``na``. > > > Bugs > ==== > > The following few things are almost certainly bugs. I haven't had > time to dig into them in any depth, but I have tried to reduce them > each to a small failing case: > > 1. Copying a slice of an array onto a different slice of the same > array fails. > > >>> y = na.arange(4) > >>> y[1:] = y[:-1] > >>> y # Should be array([0, 0, 1, 2]) > array([0, 0, 0, 0]) > > > 2. ``sqrt``, ``Power``, `and ``**`` all fail on complex zero (0j). > > >>> y = na.arange(4) + 0j > >>> na.sqrt(y) > Warning: Encountered invalid numeric result(s) in sqrt > Warning: Encountered divide by zero(s) in sqrt > Warning: Encountered invalid numeric result(s) in not_equal > Warning: Encountered invalid numeric result(s) in not_equal > array([-1.#IND -1.#INDj, 1. +0.j , 1.41421356+0.j , > 1.73205081+0.j ]) > > And similarly for ``power`` and ``**``. Note that in addition to > the warnings, the value for the sqrt(0j) is incorrect. > > > 3. Mixing arrays and lists in the constructor of array can cause > it to fail: > > >>> a = na.array([na.array([])]*3) > Traceback (most recent call last): > File "", line 1, in ? > File "C:\Python23\lib\site-packages\numarray\numarraycore.py", > line 288, in array > return fromlist(sequence, type, shape) > File "C:\Python23\lib\site-packages\numarray\numarraycore.py", > line 175, in fromlist > arr = _gen.concatenate(l) > File "C:\Python23\Lib\site-packages\numarray\generic.py", line > 1008, in concatenate > return _concat(arrs) > File "C:\Python23\Lib\site-packages\numarray\generic.py", line > 998, in _concat > dest[ix:ix+a._shape[0]]._copyFrom(a) > libnumarray.error: copy4bytes: access beyond buffer. offset=3 > buffersize=0 > > ``shape`` fails in the same way. Probably other functions as well. > > > 4. ``linear_algebra.determinant`` returns a length-1 vector when it > should return a scalar. > (Actually, I believe it sometimes returns a scalar and sometimes a > length-1 vector, but > I can't find a test case to reproduce that). > > >>> a = na.reshape(na.arange(4), (2,2)) > >>> la.determinant(a) > array([-2.]) > > > 5. Assigning a Numeric slice to a numarray array fails: > > >>> a = na.arange(3) > >>> b = Numeric.arange(3) > >>> a[:] = b > Traceback (most recent call last): > File "", line 1, in ? > File "C:\Python23\Lib\site-packages\numarray\generic.py", line > 505, in _slicedIndexing > retarr._copyFrom(value) > TypeError: argument is not array or number > > > Probable Bugs > ============= > > Now on to things that are probably bugs, but it's possible that they > represent deliberate changes from Numeric's behavior. > > 6. numarray.dot doesn't accept scalars, Numeric.dot does. > > >>> na.dot(1,1) > Traceback (most recent call last): > File "", line 1, in ? > File "C:\Python23\lib\site-packages\numarray\numarraycore.py", > line 939, in dot > return ufunc.innerproduct(a, _gen.swapaxes(inputarray(b), -1, -2)) > File "C:\Python23\lib\site-packages\numarray\ufunc.py", line 1892, > in innerproduct > if a._shape[-1] != b._shape[-1]: > IndexError: tuple index out of range > > > 7. na.searchsorted does not accept scalars for its second argument. > It always > takes and returns vectors. > > >>> na.searchsorted(na.arange(5), 1.5) > Traceback (most recent call last): > File "", line 1, in ? > File "C:\Python23\lib\site-packages\numarray\ufunc.py", line 1821, > in searchsorted > outarr = _nc.NumArray(shape=len(values), type=Long) > ValueError: Rank-0 array has no length. > >>> na.searchsorted(na.arange(5), [1.5]) > > > 8. add.reduce takes dim as a keyword argument instead of axis. It is > documented_ to take axis. > I imagine this applies to other opp.reduce methods as well. > > .. _documented: > http://stsdas.stsci.edu/numarray/Doc/node30.html#SECTION035120000000000000000 > > > 9. ``where`` and probably other functions do not appear to use > asarray on all of their > arguments. As a result, nonnumarray sequences are not directly > usable in these functions as > they are in their Numeric equivalents. In particular, lists, tuples > and Numeric arrays do > not work: > > >>> na.where([0,1,1,0], na.arange(4), [-99,-99,-99,-99]) > Traceback (most recent call last): > File "", line 1, in ? > File "C:\Python23\Lib\site-packages\numarray\generic.py", line > 970, in where > return choose(ufunc.not_equal(condition, 0), (y,x), out) > File "C:\Python23\lib\site-packages\numarray\ufunc.py", line 1446, > in __call__ > computation_mode, woutarr, cfunc, ufargs = \ > File "C:\Python23\lib\site-packages\numarray\ufunc.py", line 1462, > in _setup > convType = _maxPopType(in2) > File "C:\Python23\lib\site-packages\numarray\ufunc.py", line 1401, > in _maxPopType > raise TypeError( "Type of '%s' not supported" % repr(x) ) > TypeError: Type of '[-99, -99, -99, -99]' not supported > > > 10. ``take`` now requires a keyword argument for axis. Attempting > the to specify the axis > with a nonkeyword arg results in strange behavior. The docs don't > appear to describe > this behavior: > > >>> a = na.reshape(na.arange(9), (3,3)) > >>> a > array([[0, 1, 2], > [3, 4, 5], > [6, 7, 8]]) > >>> na.take(a, [0,1], 1) > array([1, 4]) > >>> na.take(a, [0,1], axis=1) > array([[0, 1], > [3, 4], > [6, 7]]) > > > 11. ``argmax`` returns shape () arrays instead of scalars when used > on 1D arrays. These cannot be used > to index lists > > >>> a = na.arange(9) > >>> i = na.argmax(a) > >>> a[i] > array(8) > >>> range(9)[i] > Traceback (most recent call last): > File "", line 1, in ? > TypeError: list indices must be integers > >>> i > array(8) > > > > > Non Bugs > ======== > > These are some things that would probably not be considered bugs, but > that I'd like to mention > because they either tripped me up or made the conversion and thus my > life over the last few > days more difficult than it needed to be. Let the whining begin. > > > 12. ``anArray.conjugate()`` acts in place. > ``aComplexNumber.conjugate()`` returns a new > number. This seems like a very bad state of affairs. > ``anArray.conjugate()`` should be > renamed. > > >>> zarray = na.arange(4) * 1j > >>> zarray > array([ 0.+0.j, 0.+1.j, 0.+2.j, 0.+3.j]) > >>> zarray.conjugate() > >>> zarray > array([ 0.+0.j, 0.-1.j, 0.-2.j, 0.-3.j]) > >>> z = 1+1j > >>> z.conjugate() > (1-1j) > >>> z > (1+1j) > > > 13. ``Error.popMode`` should raise an error if the last mode is > popped off the stack. > Currently the error gets raised the next time a numeric operation is > used which may > be far away from the inadvertent pop. > > >>> na.Error.popMode() # Error should be here > _NumErrorMode(overflow='warn', underflow='warn', > dividebyzero='warn', invalid='warn') > >>> zarray /= 0 # Not here. > Traceback (most recent call last): > File "", line 1, in ? > File "C:\Python23\lib\site-packages\numarray\numarraycore.py", > line 704, in __idiv__ > ufunc.divide(self, operand, self) > File "C:\Python23\lib\site-packages\numarray\ufunc.py", line 120, > in handleError > modes = Error.getMode() > File "C:\Python23\lib\site-packages\numarray\ufunc.py", line 99, > in getMode > return l[-1] > IndexError: list index out of range > > > 14. I'm of the opinion that the underflow behavior should default to > 'ignore' not 'warn'. > Nine times out of ten that's what one wants, and a default that you > have to override most > of the time is not a good default. It's possible that this opinion > may be based on floating > point naivet, but it's mine and I'm sticking to it for the time being. > > 15. Now we're getting to very minor things: Argmax's behavior has > changed, so that in the > case of ties, you will get different results than with Numeric. > Perhaps ``>`` became ``>=``? > > >>> a = na.array([0,1,0,1]) > >>> na.argmax(a) > array(3) > >>> np.argmax(a) > 1 > > > 16. ``array_repr`` no longer supports the ``suppress_small`` argument. > > > 17. ``take`` is really only useful for array types in numarray. In > Numeric it was sometimes > useful for choosing stuff from lists of objects. My impression is > that numarray doesn't try > to support objects; that's probably OK since Numeric's support was > pretty iffy. > > > 18. The fact that array comparison return booleans in numarray broke > some of my code because I > do some comparisons and then sum the results. In numarray these get > summed as Int8 and thus overflow. > I don't consider this a problem, I just thought I'd mention it in > case someone else runs into it. > > > Regards, > > Tim Hochberg > > > > > > > > ------------------------------------------------------- > This SF.Net email sponsored by: ApacheCon 2003, > 16-19 November in Las Vegas. Learn firsthand the latest > developments in Apache, PHP, Perl, XML, Java, MySQL, > WebDAV, and more! http://www.apachecon.com/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- Todd Miller Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21030 (410) 338 - 4576 From magnus at hetland.org Mon Nov 10 09:51:04 2003 From: magnus at hetland.org (Magnus Lie Hetland) Date: Mon Nov 10 09:51:04 2003 Subject: [Numpy-discussion] Notes on transitioning to numarray. In-Reply-To: <3FAFC226.3090802@ieee.org> References: <3FAFBC52.2000206@ieee.org> <3FAFC226.3090802@ieee.org> Message-ID: <20031110175006.GB9947@idi.ntnu.no> Tim Hochberg : > > Tim Hochberg wrote: > > >[There is a html version of this not at > >http://starship.python.net/~hochberg/conversion.html] > > [grrr] That should of course be 'this note'. Heh. Yeah, I thought the informational content in the statement was a bit low ;) > -tim -- Magnus Lie Hetland "In this house we obey the laws of http://hetland.org thermodynamics!" Homer Simpson From tim.hochberg at ieee.org Mon Nov 10 10:11:09 2003 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Mon Nov 10 10:11:09 2003 Subject: [Numpy-discussion] Notes on transitioning to numarray. [Addendum] In-Reply-To: <3FAFBC52.2000206@ieee.org> References: <3FAFBC52.2000206@ieee.org> Message-ID: <3FAFD495.4000600@ieee.org> Here's one more thing I just ran into. It's related to over/underflow handling, so it's even topical. I think that the _pushmodes method of NumError (in unfunc.py) should be using _gemodestack(0), not getmodestack(1). That is:: def _pushmodes(self, modes): l = self._getmodestack(0) # XXX changed be TAH l.append(modes) What can happen now is: - New thread is started. - pushMode is called before any calls to getMode, resulting in a modestack one deep. - the corresponding popMode is called, resulting in an illegal stack that is zero deep. - there is an over/underflow and getMode fails because of the zero length stack. While I'm on the subject, I think that the way the default stack entry is created should be changed. Currently there is no way to assign a default error mode for all threads. The simplest approach would be add a setDefaultMode method to Error. Error._defaultmode would be set in __init__ to _NumErrorMode(), but could subsequently be re-set with setDefaultMode. _defaultmode would be used to initialize the default value in _getmodestack. Specifically: class NumError: def __init__(self, all="warn", overflow=None, underflow=None, dividebyzero=None, invalid=None): self._defaultmode = _NumErrorMode() self._modestack = {} # map of stacks indexed by thread id self.setMode(all=all, underflow=underflow, overflow=overflow, dividebyzero=dividebyzero, invalid=invalid) def _getmodestack(self, empty_default=0): id = safethread.get_ident() try: l = self._modestack[id] except KeyError: if empty_default: l = [] else: l = [self._defaultmode ] self._modestack[id] = l return l def setDefaultMode(self, all="warn", overflow=None, underflow=None, dividebyzero=None, invalid=None): self._defaultmode = _NumErrorMode(all, overflow, underflow, dividebyzero, invalid) ) #.... While I'm at it, what's the point of empty_default? I can't figure out when it would be useful. -tim From tchur at optushome.com.au Mon Nov 10 10:22:05 2003 From: tchur at optushome.com.au (Tim Churches) Date: Mon Nov 10 10:22:05 2003 Subject: [Numpy-discussion] NaNs and infs In-Reply-To: References: Message-ID: <1068488345.1857.7.camel@emilio> On Tue, 2003-11-11 at 03:26, Perry Greenfield wrote: > Tim Churches wrote: > > > Increasingly we are turning to R (http://www.r-project.org) because it > > provides fully integrated support for Inf, NaN and NA (missing) in all > > its data structures, including matrices and higher-rank arrays, viz: > > > > > 2/0 > > [1] Inf > > > 0/0 > > [1] NaN > > > NA/1 > > [1] NA > > > > R can be embedded in Python and passed NumPy arrays or other data > > structures via Walter Moriera's excellent RPy modules (see > > http;//rpy.sf.net). > > > > However, the fundamental problem is that support for NA (missing), Inf > > and NaN seems to be afterthoughts in both NumPy and numarray, whereas > > such support should have been integrated into the design right from the > > outset. > > > > Is it too late to incorporate them into the fabric of numarray? > > > I'm a little surprised at the comment that these issues are viewed as > afterthoughts in numarray. I was being more than a little unfair to both NumPy and numarray. Unfair to NumPy because support for these things was considered, but deliberately excluded for sound performance reasons. And unfair to numarray because I haven't really studied what it does differently in these respects to NumPy. I'll now roll up my sleeves and take a good look at what numarray does before commenting further. Tim C > We gave ieee 754 special value handling > fairly careful thought. We may have missed something however, so I'm > interested to hear what is considered missing in that regard. We allow > the user to set whether ieee errors are ignored, print a warning, or > raise an exception, and do so individually for all four kinds of > errors. We also allow testing of and setting ieee special values. > What is missing? (By the way, if one expects to be able to invoke > an error handler for each individual exception that occurs in an array > computation, we determined that that was too difficult to handle in any, > portable and efficient way.) > > As to "NA" or "missing values", what is expected? These are not part of > the ieee standard (unless I've missed something). How does this differ > from using NaN, for example? > > Perry > > > ------------------------------------------------------- > This SF.Net email sponsored by: ApacheCon 2003, > 16-19 November in Las Vegas. Learn firsthand the latest > developments in Apache, PHP, Perl, XML, Java, MySQL, > WebDAV, and more! http://www.apachecon.com/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- Tim C PGP/GnuPG Key 1024D/EAF993D0 available from keyservers everywhere or at http://members.optushome.com.au/tchur/pubkey.asc Key fingerprint = 8C22 BF76 33BA B3B5 1D5B EB37 7891 46A9 EAF9 93D0 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From jmiller at stsci.edu Mon Nov 10 10:56:04 2003 From: jmiller at stsci.edu (Todd Miller) Date: Mon Nov 10 10:56:04 2003 Subject: [Numpy-discussion] Notes on transitioning to numarray. In-Reply-To: <3FAFD651.1020601@ieee.org> References: <3FAFBC52.2000206@ieee.org> <1068485784.11424.19.camel@halloween.stsci.edu> <3FAFD651.1020601@ieee.org> Message-ID: <1068490474.11424.50.camel@halloween.stsci.edu> On Mon, 2003-11-10 at 13:17, Tim Hochberg wrote: > Ah. I was sort of under the impression that the bugs database for > Numeric/.numarray on SourceForge was non active. Probably because of all > the Numeric bugs that have been sitting there for years. Next time I'll > go ahead and enter the bug reports there as well. There's a separate tracker called "Numarray Bugs" which is only visible on the "Trackers" page. In contrast, "Bugs" is visible on every page. Bugs filed in "Bugs" are for Numeric, but can now be re-assigned to "Numarray Bugs". There's only one e-mail address for "Bugs" and it's currently pointing to me; this is only so I can re-assign the numarray related bugs to numarray; I don't usually (ever?) work on Numeric. Regards, Todd -- Todd Miller Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21030 (410) 338 - 4576 From kxnzjpt at iopus.com Thu Nov 13 01:47:04 2003 From: kxnzjpt at iopus.com (Willa Marquez) Date: Thu Nov 13 01:47:04 2003 Subject: [Numpy-discussion] Italian-crafted Rolex - only $65 - $140 - Free SHIPPING! zlqa xzc Message-ID: <7tze-$338$666@xk1.ts5uaj> please note to send ALL REPLY e-mail direct to our Sales Representative at: Questions at BargainWatches.biz Hi, Thank you for expressing interest in ATGWS watches. We would like to take this opportunity to offer you our fine selection of Italian crafted Rolex Timepieces. You can view our large selection of Rolexes (including Breitling, Tag Heuer, Cartier etc) at: http://www.BargainWatches.biz For all orders placed in the month of November, all shipping and handling charges will be free. As we are the direct manufacturers, you are guaranteed of lowest prices and highest quality each and every time you purchase from us. You may also be interested to know that we have the following brands available in our wide selection as well: 1. Rolex 2. Blancpain 3. Fortis 4. Jaeger LeCoutre 5. Longines 6. Mont Blanc 7. Movado 8. Oris 9. Roger Dubuis 10. Ulysse 11. Zenith 12. Audemar Piguet 13. Breitling 14. Bvglari 15. Cartier 16. Corum 17. Dunhill 18. Franck Muller 19. Gerard Perregaux 20. IWC 21. IWC 22. Panerai 23. Patek Philippe 24. Tag Heuer 25. Vacheron Constantin If you see anything that might interest you, or if you have any questions, please don't hesitate to visit our website at: http://www.BargainWatches.biz I certainly look forward to hearing from you. Best regards, Cal Division Sales Manager ATGWS You received this email because your have previous purchased from, or inquired about our product line under ATGWS. If you do not want to receive further mailings from ATGWS, unsubscribe by sending an email with the title heading: DELETE in the subject line to Unsubscribe at bargainwatches.biz please note to send ALL REPLY e-mail direct to our Sales Representative at: Questions at BargainWatches.biz cb epql zyjcjoqwmuchaun From wa317msi at yahoo.com Fri Nov 14 01:41:06 2003 From: wa317msi at yahoo.com (Bernie Mobley) Date: Fri Nov 14 01:41:06 2003 Subject: [Numpy-discussion] AMERICAN STOCK MARKET: TRHL Retains Sky Investor Relations...dempster Message-ID: American Stock Market - Press Release... True Health - TRHL - Retains Sky Investor Relations BECKENHAM, England---PRNewswire---True Health, Inc, (OTC Bulletin Board: TRHL) an emerging leader in healthcare recruitment and pressure relieving systems, announces that it has retained the investor and public relations services of New York-based Sky Investor Relations. Read the entire news release: http://biz.yahoo.com/prnews/031112/lnw017_1.html c udh yeweekc nrisicgb uvspt w zzezcwzmllz s sp qvdf i h ggd iy zepqhkh frd From bens at MIT.EDU Sun Nov 16 19:53:02 2003 From: bens at MIT.EDU (Benjamin Schwartz) Date: Sun Nov 16 19:53:02 2003 Subject: [Numpy-discussion] Multiple assignment confusion Message-ID: <3FB845A8.1070007@mit.edu> I am trying to interchange two pieces of an array. Here is my demo code: >>> from Numeric import * >>> a=array([.7,.1,.1,.1]) >>> a=multiply.outer(a,a) >>> a=multiply.outer(a,a) >>> sum(sum(sum(sum(a)))) 1.0000000000000002 >>> a[1,0],a[1,1]=a[1,1],a[1,0] >>> sum(sum(sum(sum(a)))) 0.93999999999999972 So a is a four-dimensional array that sums to 1, but when I attempt to switch two 2-d sub-arrays of it I just end up duplicating one of them, shown by a change in the value of the sum (to .94). Is this behavior deliberate? It certainly seems to break Python's tuple-assignment model. What I would ideally like to do is perform this swap without more than two-elements'-worth of temporary storage. I.e. perform it as a pure reference swap or switch each pair of elements one by one, reusing temporary space. Is this possible with Numeric (or Numarray?)? Curious, Ben From perry at stsci.edu Mon Nov 17 08:44:23 2003 From: perry at stsci.edu (Perry Greenfield) Date: Mon Nov 17 08:44:23 2003 Subject: [Numpy-discussion] Multiple assignment confusion In-Reply-To: <3FB845A8.1070007@mit.edu> Message-ID: Benjamin Schwartz wrote: > I am trying to interchange two pieces of an array. Here is my demo code: > > >>> from Numeric import * > >>> a=array([.7,.1,.1,.1]) > >>> a=multiply.outer(a,a) > >>> a=multiply.outer(a,a) > >>> sum(sum(sum(sum(a)))) > 1.0000000000000002 > >>> a[1,0],a[1,1]=a[1,1],a[1,0] > >>> sum(sum(sum(sum(a)))) > 0.93999999999999972 > > So a is a four-dimensional array that sums to 1, but when I attempt to > switch two 2-d sub-arrays of it I just end up duplicating one of them, > shown by a change in the value of the sum (to .94). Is this behavior > deliberate? It certainly seems to break Python's tuple-assignment model. > > What I would ideally like to do is perform this swap without more than > two-elements'-worth of temporary storage. I.e. perform it as a pure > reference swap or switch each pair of elements one by one, reusing > temporary space. Is this possible with Numeric (or Numarray?)? > > Curious, > Ben > I believe that this is simply a reflection of the fact that subarrays (for Numeric and numarray) are views into the original array rather than copies. A simpler example illustrates the same problem: >>> a = reshape(arange(4), (2,2)) >>> a array([[0, 1], [2, 3]]) >>> a[0],a[1]=a[1],a[0] >>> a array([[2, 3], [2, 3]]) >>> Even though the right side of the assignment is stored effectively in temporary variables, what is stored are views into the original array, not copies of the subarrays. When the first assignment is made to a[0] (=a[1]), the data in the reference a[0] on the right hand side has changed as well since it views exactly the same data. When it comes time to set a[1] it will be copying from the updated a[0] and thus the duplication. This doesn't happen for a list since one is essentially rebinding the element in a list to another object. In other words, for this case: >>> b = [[0,1],[2,3]] >>> b[0],b[1]=b[1],b[0] >>> b [[2, 3], [0, 1]] >>> assigning to b[0] is rebinding that list element away from the list [0,1] to the list [2,3]; changing what the elements b refer to in no way changes the contents of either of the lists that b contained. Numeric arrays have a coupling across dimensions that doesn't exist for lists since nested lists are analogous to arrays of pointers to pointers whereas Numeric arrays are just multidimensional arrays with no indirection involved. The above behavior could be avoided if subarrays always produced copies. This issue was given a great deal of discussion and thought for numarray and for various reasons it was decided not to change the behavior of slices and subarray references. As to whether it is possible to switch subarray contents with only using one or two temporary storage elements efficiently, offhand I don't believe so, but I haven't given it much thought (and don't have the time to) but perhaps someone else knows of an existing mechanism to do so. Nothing prevents writing an extension to provide such capability. Perry Greenfield From haase at msg.ucsf.edu Mon Nov 17 15:48:10 2003 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Mon Nov 17 15:48:10 2003 Subject: [Numpy-discussion] numarray: records using memmap Message-ID: <013801c3ad65$35fa15b0$421ee6a9@rodan> Hi, Is it possible to create a record-array using a memmap (file) as buffer ? Thanks, Sebastian Haase From cjw at sympatico.ca Mon Nov 17 19:49:03 2003 From: cjw at sympatico.ca (Colin J. Williams) Date: Mon Nov 17 19:49:03 2003 Subject: [Numpy-discussion] teacup.py crash Message-ID: <3FB9967F.1010309@sympatico.ca> Running teacup.py, we get: Traceback (most recent call last): File "C:\Program Files\Python23\Lib\site-packages\numarray\teacup.py", line 1, in ? import numarray.libteacup ImportError: No module named libteacup There is no mention of teacup in numarray-0.6.pdf. Can this module be safely be deleted? Colin W From hsu at stsci.edu Mon Nov 17 20:25:02 2003 From: hsu at stsci.edu (JC Hsu) Date: Mon Nov 17 20:25:02 2003 Subject: [Numpy-discussion] Re: creating record arrays using memmap Message-ID: <1b0b7c08.e482203e.8166500@comet.stsci.edu> Yes, it is possible to create a record array with memmap. The trick is to use the constructor, instead of the "array" function: >>> from numarray import memmap, records >>> m=memmap.open('xxx.txt') >>> r2=records.RecArray(m[0:18],formats='i2,f4',shape=3) Notice that you also need to "calculate" the slice (you must use the slice even if you are mapping the whole file). JC Hsu From leo at lspace.org Tue Nov 18 10:40:10 2003 From: leo at lspace.org (Leo Breebaart) Date: Tue Nov 18 10:40:10 2003 Subject: [Numpy-discussion] Problems creating a numarray of type PyArray_OBJECT Message-ID: <20031118183413.GA25090@falcon.pds.twi.tudelft.nl> Hi all, Using the numarray C API, I am trying to create an array containing Python strings from an existing C (char **) array. In Numeric 23.1, the following code fragment works fine: PyObject *pdata = NULL; // num_elements has been defined earlier PyStringObject **strarray = malloc(num_elements * sizeof(PyStringObject *)); if (strarray != NULL) { int e; for (e = 0; e < num_elements; e++) { strarray[e] = (PyStringObject *) PyString_FromString(((char **)data)[e]); } // num_dims and the dim array have been defined earlier pdata = PyArray_FromDimsAndData(num_dims, dim, PyArray_OBJECT, (char *)strarray); } In numarray 0.7, the same code yields the runtime error: libnumarray.error: Type object lookup returned NULL for type 14 in the PyArray_FromDimsAndData() call. Using the current numarray CVS head instead of the 0.7 release makes no difference. Is there somebody who knows what is going or what I am doing wrong? Many thanks in advance, -- Leo Breebaart From jmiller at stsci.edu Tue Nov 18 11:05:01 2003 From: jmiller at stsci.edu (Todd Miller) Date: Tue Nov 18 11:05:01 2003 Subject: [Numpy-discussion] Problems creating a numarray of type PyArray_OBJECT In-Reply-To: <20031118183413.GA25090@falcon.pds.twi.tudelft.nl> References: <20031118183413.GA25090@falcon.pds.twi.tudelft.nl> Message-ID: <1069182221.3461.2.camel@localhost.localdomain> On Tue, 2003-11-18 at 13:34, Leo Breebaart wrote: > Hi all, > > Using the numarray C API, I am trying to create an array > containing Python strings from an existing C (char **) array. > > In Numeric 23.1, the following code fragment works fine: > > PyObject *pdata = NULL; > // num_elements has been defined earlier > PyStringObject **strarray = malloc(num_elements * sizeof(PyStringObject *)); > if (strarray != NULL) > { > int e; > for (e = 0; e < num_elements; e++) > { > strarray[e] = (PyStringObject *) PyString_FromString(((char **)data)[e]); > } > > // num_dims and the dim array have been defined earlier > pdata = PyArray_FromDimsAndData(num_dims, dim, PyArray_OBJECT, > (char *)strarray); > } > > In numarray 0.7, the same code yields the runtime error: > > libnumarray.error: Type object lookup returned NULL for type 14 > > in the PyArray_FromDimsAndData() call. > > Using the current numarray CVS head instead of the 0.7 release > makes no difference. > > Is there somebody who knows what is going or what I am doing > wrong? Yes and no. You're in an area that numarray supports differently than Numeric. numarray's support for character arrays is in the Python numarray.strings module. There's no direct support for using it from C. > > Many thanks in advance, You're welcome, Todd -- Todd Miller From leo at lspace.org Wed Nov 19 02:28:02 2003 From: leo at lspace.org (Leo Breebaart) Date: Wed Nov 19 02:28:02 2003 Subject: [Numpy-discussion] Problems creating a numarray of type PyArray_OBJECT In-Reply-To: <1069182221.3461.2.camel@localhost.localdomain> References: <20031118183413.GA25090@falcon.pds.twi.tudelft.nl> <1069182221.3461.2.camel@localhost.localdomain> Message-ID: <20031119102708.GA1107@falcon.pds.twi.tudelft.nl> Todd Miller wrote: > Leo Breebaart wrote: > > > [...] > > pdata = PyArray_FromDimsAndData(num_dims, dim, PyArray_OBJECT, > > (char *)strarray); > > } > > > > In numarray 0.7, the same code yields the runtime error: > > > > libnumarray.error: Type object lookup returned NULL for type 14 > > > > in the PyArray_FromDimsAndData() call. [...] > > > Is there somebody who knows what is going or what I am doing > > wrong? > > Yes and no. You're in an area that numarray supports differently than > Numeric. numarray's support for character arrays is in the Python > numarray.strings module. There's no direct support for using it from > C. Thanks for the response, Todd. Do I understand correctly that you are saying it's no longer possible in numarray (as opposed to Numeric) to create *any* kind of PyObject array? (After all, the fact that the PyObjects in the above example are really strings is basically accidental -- what if I had an array of pointers to e.g. PyLists or PyDicts that I wanted to store in a numarray -- in Numeric that would also be possible using the PyArray_OBJECT type, right?) Am I also correct in thinking that at this point the only solution to my problem is to manually convert the string arrays in my data to direct Python lists of strings -- or is there perhaps some other trick by which I could get them into a numarray after all? I do a certain amount of Python-level 'post-processing' in my code anyway, and the string arrays I encounter are expected to have only a few elements each, so it would be no problem to do some munging after the fact, if that could help matters. -- Leo Breebaart From jmiller at stsci.edu Wed Nov 19 03:00:03 2003 From: jmiller at stsci.edu (Todd Miller) Date: Wed Nov 19 03:00:03 2003 Subject: [Numpy-discussion] Problems creating a numarray of type PyArray_OBJECT In-Reply-To: <20031119102708.GA1107@falcon.pds.twi.tudelft.nl> References: <20031118183413.GA25090@falcon.pds.twi.tudelft.nl> <1069182221.3461.2.camel@localhost.localdomain> <20031119102708.GA1107@falcon.pds.twi.tudelft.nl> Message-ID: <1069239542.3458.58.camel@localhost.localdomain> On Wed, 2003-11-19 at 05:27, Leo Breebaart wrote: > Todd Miller wrote: > > > Leo Breebaart wrote: > > > > > [...] > > > pdata = PyArray_FromDimsAndData(num_dims, dim, PyArray_OBJECT, > > > (char *)strarray); > > > } > > > > > > In numarray 0.7, the same code yields the runtime error: > > > > > > libnumarray.error: Type object lookup returned NULL for type 14 > > > > > > in the PyArray_FromDimsAndData() call. [...] > > > > > Is there somebody who knows what is going or what I am doing > > > wrong? > > > > Yes and no. You're in an area that numarray supports differently than > > Numeric. numarray's support for character arrays is in the Python > > numarray.strings module. There's no direct support for using it from > > C. > > Thanks for the response, Todd. > > Do I understand correctly that you are saying it's no longer > possible in numarray (as opposed to Numeric) to create *any* kind > of PyObject array? No. numarray-0.7 added numarray.objects which provides basic support for arrays of arbitrary objects: indexing, slicing, ufuncs, etc. Access from C, however, is not directly supported. Internally, numarray.objects manages objects using a Python list so it is important to respect the Python object interface and not try to directly munge the data buffer as you might with numerical arrays. > (After all, the fact that the PyObjects in the > above example are really strings is basically accidental -- what > if I had an array of pointers to e.g. PyLists or PyDicts that I > wanted to store in a numarray -- in Numeric that would also be > possible using the PyArray_OBJECT type, right?) If, in Python, you had a list of dicts, you could create a numarray object array from it. If, in C, you had an array of pointers to dicts, you could first turn it into a list of dicts, and then call back into Python to create an object array using numarray.objects.fromlist(). > Am I also correct in thinking that at this point the only > solution to my problem is to manually convert the string arrays > in my data to direct Python lists of strings -- or is there > perhaps some other trick by which I could get them into a > numarray after all? It's possible to call back into Python to create an empty CharArray, and then to fill it in C. This is more straight forward for CharArrays because the data is stored in the same sort of buffer as for numerical arrays, and unlike object arrays, no reference counting is required. Regards, Todd > -- > Todd Miller From ariciputi at pito.com Wed Nov 19 08:26:17 2003 From: ariciputi at pito.com (Andrea Riciputi) Date: Wed Nov 19 08:26:17 2003 Subject: [Numpy-discussion] Help with PyArrayObject and C Arrays. Message-ID: Hi, I've written some C code for number crunching that uses the following structure for simulating 2-dimensional arrays (i.e. matrices): typedef struct { double **aa; int nrows; int ncols; } Array2d; In order to read these C arrays in Python+Numeric and vice versa, I've written some wrapping code, but I'm not so sure it's correct. In particular I'm concerned in memory leakage, I don't know if I've understood correctly how PyArray_FromDimsAndData works and if I've choose the best way to solve the problem. My code follows. Thanks for help! Andrea. C --> Python ============ static PyObject *Array2d_2_PyArrayObject(Array2d *input) { int ndims[2]; PyArrayObject *result; if (!input) return NULL; ndims[0] = input->nrows; ndims[1] = input->ncols; result = (PyArrayObject *) PyArray_FromDimsAndData(2, ndims, PyArray_DOUBLE, (char *) input->aa[0]); free(input->aa); free(input); return PyArray_Return(result); } Python --> C ============ static Array2d *PyList_2_Array2d(PyObject *input) { Array2d *result; if (!PySequence_Check(input)) { PyErr_SetString(PyExc_TypeError,"Expected a sequence"); return NULL; } result = (Array2d *) malloc(sizeof(Array2d)); PY_CHECK_ALLOC(result); if (PyArray_As2D((PyObject **) &input, (char ***) &(result->aa), &(result->nrows), &(result->ncols), PyArray_DOUBLE) != 0) { PyErr_SetString(PyExc_MemoryError,"No heap space for data"); return NULL; } return result; } --- Andrea Riciputi "Science is like sex: sometimes something useful comes out, but that is not the reason we are doing it" -- (Richard Feynman) From 6kgwcl at datanet.hu Wed Nov 19 09:48:06 2003 From: 6kgwcl at datanet.hu (Camille Flood) Date: Wed Nov 19 09:48:06 2003 Subject: [Numpy-discussion] Total Relief From Credit Card Debt!! Message-ID: An HTML attachment was scrubbed... URL: From edcjones at erols.com Wed Nov 19 19:19:06 2003 From: edcjones at erols.com (Edward C. Jones) Date: Wed Nov 19 19:19:06 2003 Subject: [Numpy-discussion] Wrong error message Message-ID: <3FBC3192.2020506@erols.com> Here is the program: #! /usr/bin/env python import numarray arr1 = numarray.zeros([3,2,1]) arr2 = numarray.zeros([3,2,1]) arr2[0] = arr1 and here is the error message: TypeError: _copyFromAndConvert: non-numarray input. Should this program give this error message? From jmiller at stsci.edu Thu Nov 20 07:09:08 2003 From: jmiller at stsci.edu (Todd Miller) Date: Thu Nov 20 07:09:08 2003 Subject: [Numpy-discussion] Wrong error message In-Reply-To: <3FBC3192.2020506@erols.com> References: <3FBC3192.2020506@erols.com> Message-ID: <1069340772.5225.66.camel@halloween.stsci.edu> On Wed, 2003-11-19 at 22:14, Edward C. Jones wrote: > Here is the program: > > #! /usr/bin/env python > import numarray > arr1 = numarray.zeros([3,2,1]) > arr2 = numarray.zeros([3,2,1]) > arr2[0] = arr1 > > and here is the error message: > > TypeError: _copyFromAndConvert: non-numarray input. > > Should this program give this error message? No. I agree that's a pretty bad message. Here's what CVS does now: Traceback (most recent call last): .. ValueError: Can't use non-rank-0 array as a scalar. Even the new message is a little obscure; feel free to suggest improvements... Regards, Todd > > > ------------------------------------------------------- > This SF.net email is sponsored by: SF.net Giveback Program. > Does SourceForge.net help you be more productive? Does it > help you create better code? SHARE THE LOVE, and help us help > YOU! Click Here: http://sourceforge.net/donate/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- Todd Miller Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21030 (410) 338 - 4576 From jmiller at stsci.edu Thu Nov 20 13:33:02 2003 From: jmiller at stsci.edu (Todd Miller) Date: Thu Nov 20 13:33:02 2003 Subject: [Numpy-discussion] teacup.py crash In-Reply-To: <3FB9967F.1010309@sympatico.ca> References: <3FB9967F.1010309@sympatico.ca> Message-ID: <1069363850.6119.111.camel@halloween.stsci.edu> On Mon, 2003-11-17 at 22:48, Colin J. Williams wrote: > Running teacup.py, we get: > > Traceback (most recent call last): > File "C:\Program > Files\Python23\Lib\site-packages\numarray\teacup.py", line 1, in ? > import numarray.libteacup > ImportError: No module named libteacup > > There is no mention of teacup in numarray-0.6.pdf. teacup is a partially functional module for measuring microsecond level timing under Linux. It doesn't work yet, but it's better than starting from scratch, so I keep it in the distribution. At some point, it'll either work or disappear. > Can this module be safely be deleted? Yes. > Colin W > > > > ------------------------------------------------------- > This SF. Net email is sponsored by: GoToMyPC > GoToMyPC is the fast, easy and secure way to access your computer from > any Web browser or wireless device. Click here to Try it Free! > https://www.gotomypc.com/tr/OSDN/AW/Q4_2003/t/g22lp?Target=mm/g22lp.tmpl > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- Todd Miller Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21030 (410) 338 - 4576 From jmiller at stsci.edu Thu Nov 20 14:05:03 2003 From: jmiller at stsci.edu (Todd Miller) Date: Thu Nov 20 14:05:03 2003 Subject: [Numpy-discussion] Community Poll: numarray default underflow handling == "ignore" ? Message-ID: <1069365809.5225.145.camel@halloween.stsci.edu> Someone recently suggested that we change the default for numarray underflow checking to "ignore". Please vote (+1, 0, -1): Todd -- Todd Miller Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21030 (410) 338 - 4576 From Chris.Barker at noaa.gov Thu Nov 20 15:21:03 2003 From: Chris.Barker at noaa.gov (Chris Barker) Date: Thu Nov 20 15:21:03 2003 Subject: [Numpy-discussion] Community Poll: numarray default underflow handling == "ignore" ? References: <1069365809.5225.145.camel@halloween.stsci.edu> Message-ID: <3FBD4BD4.FD1F30B9@noaa.gov> Todd Miller wrote: > Someone recently suggested that we change the default for numarray > underflow checking to "ignore". Please vote (+1, 0, -1): What is it currently, and what other options are there? -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From jsaenz at wm.lc.ehu.es Fri Nov 21 00:25:02 2003 From: jsaenz at wm.lc.ehu.es (Jon Saenz) Date: Fri Nov 21 00:25:02 2003 Subject: [Numpy-discussion] Community Poll: numarray default underflow handling == "ignore" ? In-Reply-To: <1069365809.5225.145.camel@halloween.stsci.edu> Message-ID: +1 Jon Saenz. | Tfno: +34 946012445 Depto. Fisica Aplicada II | Fax: +34 944648500 Facultad de Ciencias. \\ Universidad del Pais Vasco \\ Apdo. 644 \\ 48080 - Bilbao \\ SPAIN On 20 Nov 2003, Todd Miller wrote: > Someone recently suggested that we change the default for numarray > underflow checking to "ignore". Please vote (+1, 0, -1): > > Todd > -- > Todd Miller > Space Telescope Science Institute > 3700 San Martin Drive > Baltimore MD, 21030 > (410) 338 - 4576 > > > > ------------------------------------------------------- > This SF.net email is sponsored by: SF.net Giveback Program. > Does SourceForge.net help you be more productive? Does it > help you create better code? SHARE THE LOVE, and help us help > YOU! Click Here: http://sourceforge.net/donate/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From jmiller at stsci.edu Fri Nov 21 05:37:06 2003 From: jmiller at stsci.edu (Todd Miller) Date: Fri Nov 21 05:37:06 2003 Subject: [Numpy-discussion] Community Poll: numarray default underflow handling == "ignore" ? In-Reply-To: <3FBD4BD4.FD1F30B9@noaa.gov> References: <1069365809.5225.145.camel@halloween.stsci.edu> <3FBD4BD4.FD1F30B9@noaa.gov> Message-ID: <1069421715.7448.2.camel@halloween.stsci.edu> On Thu, 2003-11-20 at 18:18, Chris Barker wrote: > Todd Miller wrote: > > Someone recently suggested that we change the default for numarray > > underflow checking to "ignore". Please vote (+1, 0, -1): > > What is it currently, and what other options are there? The default of underflow is "warn" Other options are: "ignore" or "raise" > > -Chris > > > -- > Christopher Barker, Ph.D. > Oceanographer > > NOAA/OR&R/HAZMAT (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > > ------------------------------------------------------- > This SF.net email is sponsored by: SF.net Giveback Program. > Does SourceForge.net help you be more productive? Does it > help you create better code? SHARE THE LOVE, and help us help > YOU! Click Here: http://sourceforge.net/donate/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- Todd Miller Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21030 (410) 338 - 4576 From pearu at cens.ioc.ee Fri Nov 21 06:13:08 2003 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Fri Nov 21 06:13:08 2003 Subject: [Numpy-discussion] Community Poll: numarray default underflow handling == "ignore" ? In-Reply-To: <1069421715.7448.2.camel@halloween.stsci.edu> Message-ID: On 21 Nov 2003, Todd Miller wrote: > On Thu, 2003-11-20 at 18:18, Chris Barker wrote: > > Todd Miller wrote: > > > Someone recently suggested that we change the default for numarray > > > underflow checking to "ignore". Please vote (+1, 0, -1): > > > > What is it currently, and what other options are there? > > The default of underflow is "warn" > > Other options are: "ignore" or "raise" My votes: warn: 0 ignore: 0 raise: -1 Pearu From a.schmolck at gmx.net Fri Nov 21 06:59:11 2003 From: a.schmolck at gmx.net (Alexander Schmolck) Date: Fri Nov 21 06:59:11 2003 Subject: [Numpy-discussion] Community Poll: numarray default underflow handling == "ignore" ? In-Reply-To: <1069421715.7448.2.camel@halloween.stsci.edu> References: <1069365809.5225.145.camel@halloween.stsci.edu> <3FBD4BD4.FD1F30B9@noaa.gov> <1069421715.7448.2.camel@halloween.stsci.edu> Message-ID: Todd Miller writes: > On Thu, 2003-11-20 at 18:18, Chris Barker wrote: > > Todd Miller wrote: > > > Someone recently suggested that we change the default for numarray > > > underflow checking to "ignore". Please vote (+1, 0, -1): > > > > What is it currently, and what other options are there? > > The default of underflow is "warn" I much prefer the current default. Having (potential errors) ignored silently is IMHO against the spirit of python. What's the problem with manually setting "ignore underflow" if and when one knows that underflows might occur but can safely be ignored? 'as From jmiller at stsci.edu Fri Nov 21 07:45:02 2003 From: jmiller at stsci.edu (Todd Miller) Date: Fri Nov 21 07:45:02 2003 Subject: [Numpy-discussion] Community Poll: numarray default underflow handling == "ignore" ? In-Reply-To: References: <1069365809.5225.145.camel@halloween.stsci.edu> <3FBD4BD4.FD1F30B9@noaa.gov> <1069421715.7448.2.camel@halloween.stsci.edu> Message-ID: <1069429373.7448.4.camel@halloween.stsci.edu> On Fri, 2003-11-21 at 09:58, Alexander Schmolck wrote: > Todd Miller writes: > > > On Thu, 2003-11-20 at 18:18, Chris Barker wrote: > > > Todd Miller wrote: > > > > Someone recently suggested that we change the default for numarray > > > > underflow checking to "ignore". Please vote (+1, 0, -1): > > > > > > What is it currently, and what other options are there? > > > > The default of underflow is "warn" > > I much prefer the current default. Having (potential errors) ignored silently > is IMHO against the spirit of python. What's the problem with manually setting > "ignore underflow" if and when one knows that underflows might occur but can > safely be ignored? I'll read that as -1. :-) Todd > > 'as > > > > ------------------------------------------------------- > This SF.net email is sponsored by: SF.net Giveback Program. > Does SourceForge.net help you be more productive? Does it > help you create better code? SHARE THE LOVE, and help us help > YOU! Click Here: http://sourceforge.net/donate/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- Todd Miller Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21030 (410) 338 - 4576 From Chris.Barker at noaa.gov Fri Nov 21 08:53:06 2003 From: Chris.Barker at noaa.gov (Chris Barker) Date: Fri Nov 21 08:53:06 2003 Subject: [Numpy-discussion] Community Poll: numarray default underflow handling == "ignore" ? References: <1069365809.5225.145.camel@halloween.stsci.edu> <3FBD4BD4.FD1F30B9@noaa.gov> <1069421715.7448.2.camel@halloween.stsci.edu> Message-ID: <3FBE4262.BEF5DEE4@noaa.gov> Todd Miller wrote: > > > Someone recently suggested that we change the default for numarray > > > underflow checking to "ignore". Please vote (+1, 0, -1): > The default of underflow is "warn" assuming that ignore means that zero will be used, I vote +1 If the difference between zero and the smallest floating point number is significant in your app, you'd better know what you're doing with floating point, and you should then know to set the check to "warn" or "raise". -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From fperez at colorado.edu Fri Nov 21 08:57:12 2003 From: fperez at colorado.edu (Fernando Perez) Date: Fri Nov 21 08:57:12 2003 Subject: [Numpy-discussion] Community Poll: numarray default underflow handling == "ignore" ? In-Reply-To: <3FBE4262.BEF5DEE4@noaa.gov> References: <1069365809.5225.145.camel@halloween.stsci.edu> <3FBD4BD4.FD1F30B9@noaa.gov> <1069421715.7448.2.camel@halloween.stsci.edu> <3FBE4262.BEF5DEE4@noaa.gov> Message-ID: <3FBE43BD.3080100@colorado.edu> Chris Barker wrote: > Todd Miller wrote: > >>>>Someone recently suggested that we change the default for numarray >>>>underflow checking to "ignore". Please vote (+1, 0, -1): > > >>The default of underflow is "warn" > > > assuming that ignore means that zero will be used, I vote +1 > > If the difference between zero and the smallest floating point number is > significant in your app, you'd better know what you're doing with > floating point, and you should then know to set the check to "warn" or > "raise". +1 also from me (also as Chris, assuming 'ignore' means replacing with zeros and not with NaNs). Underflow is about the only case of errors where I think that silent ignore is a reasonable (if 'unpythonic') policy. Cheers, f From haase at msg.ucsf.edu Fri Nov 21 09:10:05 2003 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Fri Nov 21 09:10:05 2003 Subject: [Numpy-discussion] how-to "put" RAM-based numarray into memmap Message-ID: <047301c3b052$37f38960$421ee6a9@rodan> Hi, Suppose I have a 500MB-ram Computer and a 300MB ram-only (standard) numarray. Now I would like to "save" that onto harddrive (with a small header up front - which I know how to do as a memmap record) Can I somehow "append" a (standard) numarray to a memmap "as a memmap-slice" ? I obviously could create the corresponding memslice fresh and then assign the original data elementwise - but that would allocate extra (virtual) memory... Thanks, Sebastian Haase From cjw at sympatico.ca Fri Nov 21 09:21:08 2003 From: cjw at sympatico.ca (Colin J. Williams) Date: Fri Nov 21 09:21:08 2003 Subject: [Numpy-discussion] Community Poll: numarray default underflow handling == "ignore" ? In-Reply-To: <1069429373.7448.4.camel@halloween.stsci.edu> References: <1069365809.5225.145.camel@halloween.stsci.edu> <3FBD4BD4.FD1F30B9@noaa.gov> <1069421715.7448.2.camel@halloween.stsci.edu> <1069429373.7448.4.camel@halloween.stsci.edu> Message-ID: <3FBE4965.9050107@sympatico.ca> Todd Miller wrote: >On Fri, 2003-11-21 at 09:58, Alexander Schmolck wrote: > > >>Todd Miller writes: >> >> >> >>>On Thu, 2003-11-20 at 18:18, Chris Barker wrote: >>> >>> >>>>Todd Miller wrote: >>>> >>>> >>>>>Someone recently suggested that we change the default for numarray >>>>>underflow checking to "ignore". Please vote (+1, 0, -1): >>>>> >>>>> >>>>What is it currently, and what other options are there? >>>> >>>> >>>The default of underflow is "warn" >>> >>> >>I much prefer the current default. Having (potential errors) ignored silently >>is IMHO against the spirit of python. What's the problem with manually setting >>"ignore underflow" if and when one knows that underflows might occur but can >>safely be ignored? >> >> > >I'll read that as -1. :-) > This proposal leaves the control with the user, which seems to be the best way to go. Colin W. From Chris.Barker at noaa.gov Fri Nov 21 09:31:06 2003 From: Chris.Barker at noaa.gov (Chris Barker) Date: Fri Nov 21 09:31:06 2003 Subject: [Numpy-discussion] how-to "put" RAM-based numarray into memmap References: <047301c3b052$37f38960$421ee6a9@rodan> Message-ID: <3FBE4B21.C07CBF38@noaa.gov> Sebastian Haase wrote: > > Hi, > Suppose I have a 500MB-ram Computer and a 300MB ram-only (standard) > numarray. > Now I would like to "save" that onto harddrive (with a small header up > front How about: f = open(filename, 'wb') f.write(MyHeader) A.tofile(f) To read it back in, you need to know where your header ends, by either parsing it or using one of the same size every time, then you can use fromfile() to create an array form it. NOTE: I havn't actaully done this myself, but I've done something bvery similar with fromstring and tostring with Numeric. I'm very happy that Numarray has the *file methods, as they don't require an extra copy in memory. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From jmiller at stsci.edu Fri Nov 21 09:31:09 2003 From: jmiller at stsci.edu (Todd Miller) Date: Fri Nov 21 09:31:09 2003 Subject: [Numpy-discussion] Community Poll: numarray default underflow handling == "ignore" ? In-Reply-To: <3FBE4965.9050107@sympatico.ca> References: <1069365809.5225.145.camel@halloween.stsci.edu> <3FBD4BD4.FD1F30B9@noaa.gov> <1069421715.7448.2.camel@halloween.stsci.edu> <1069429373.7448.4.camel@halloween.stsci.edu> <3FBE4965.9050107@sympatico.ca> Message-ID: <1069435777.7446.13.camel@halloween.stsci.edu> On Fri, 2003-11-21 at 12:20, Colin J. Williams wrote: > > > Todd Miller wrote: > > >On Fri, 2003-11-21 at 09:58, Alexander Schmolck wrote: > > > > > >>Todd Miller writes: > >> > >> > >> > >>>On Thu, 2003-11-20 at 18:18, Chris Barker wrote: > >>> > >>> > >>>>Todd Miller wrote: > >>>> > >>>> > >>>>>Someone recently suggested that we change the default for numarray > >>>>>underflow checking to "ignore". Please vote (+1, 0, -1): > >>>>> > >>>>> > >>>>What is it currently, and what other options are there? > >>>> > >>>> > >>>The default of underflow is "warn" > >>> > >>> > >>I much prefer the current default. Having (potential errors) ignored silently > >>is IMHO against the spirit of python. What's the problem with manually setting > >>"ignore underflow" if and when one knows that underflows might occur but can > >>safely be ignored? > >> > >> > > > >I'll read that as -1. :-) > > > This proposal leaves the control with the user, which seems to be the > best way to go. Just to be clear, all we're talking about is the *default* handling of underflow. Regardless of what we decide, underflow handling will be configurable by the user on a per-thread basis as either: "ignore", "warn", or "raise". Todd > Colin W. > -- Todd Miller Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21030 (410) 338 - 4576 From falted at openlc.org Fri Nov 21 10:06:09 2003 From: falted at openlc.org (Francesc Alted) Date: Fri Nov 21 10:06:09 2003 Subject: [Numpy-discussion] how-to "put" RAM-based numarray into memmap In-Reply-To: <3FBE4B21.C07CBF38@noaa.gov> References: <047301c3b052$37f38960$421ee6a9@rodan> <3FBE4B21.C07CBF38@noaa.gov> Message-ID: <200311211905.33263.falted@openlc.org> A Divendres 21 Novembre 2003 18:28, Chris Barker va escriure: > Sebastian Haase wrote: > > Hi, > > Suppose I have a 500MB-ram Computer and a 300MB ram-only (standard) > > numarray. > > Now I would like to "save" that onto harddrive (with a small header up > > front > > How about: > > f = open(filename, 'wb') > f.write(MyHeader) > A.tofile(f) > > To read it back in, you need to know where your header ends, by either > parsing it or using one of the same size every time, then you can use > fromfile() to create an array form it. You can also use pytables for doing that more comfortably: >>> a=zeros([2,3,3]) >>> import tables >>> fileh=tables.openFile("test.h5", "w") >>> fileh.createArray(fileh.root, "array1", a, "This is a small test array") /array1 (Array(2, 3, 3)) 'This is a small test array' type = Int32 itemsize = 4 flavor = 'NumArray' byteorder = 'little' >>> fileh.close() To read it back you only have to do: >>> fileh=tables.openFile("test.h5", "r") >>> a=fileh.root.array1.read() >>> a array([[[0, 0, 0], [0, 0, 0], [0, 0, 0]], [[0, 0, 0], [0, 0, 0], [0, 0, 0]]]) that's all. All the headers with info about the object you are saving are automatically saved and retrieved behind the scenes. You can get pytables from http://pytables.sf.net Cheers, -- Francesc Alted From cookedm at physics.mcmaster.ca Fri Nov 21 11:13:03 2003 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Fri Nov 21 11:13:03 2003 Subject: [Numpy-discussion] Community Poll: numarray default underflow handling == "ignore" ? In-Reply-To: <3FBE4262.BEF5DEE4@noaa.gov> References: <1069365809.5225.145.camel@halloween.stsci.edu> <3FBD4BD4.FD1F30B9@noaa.gov> <1069421715.7448.2.camel@halloween.stsci.edu> <3FBE4262.BEF5DEE4@noaa.gov> Message-ID: <20031121191114.GA7441@arbutus.physics.mcmaster.ca> On Fri, Nov 21, 2003 at 08:50:42AM -0800, Chris Barker wrote: > Todd Miller wrote: > > > > Someone recently suggested that we change the default for numarray > > > > underflow checking to "ignore". Please vote (+1, 0, -1): > > > The default of underflow is "warn" > > assuming that ignore means that zero will be used, I vote +1 > > If the difference between zero and the smallest floating point number is > significant in your app, you'd better know what you're doing with > floating point, and you should then know to set the check to "warn" or > "raise". Agree, +1. It's quite common (for me) that I want exp(-1000) to give 0, and not an error (or warning, espicially in a loop that goes for several thousand iterations...) Overflow, yes, that's a problem, but not underflow. (The behaviour below lead me to writing my own exp ufunc for Numeric: >>> import Numeric >>> Numeric.exp(-1000) Traceback (most recent call last): File "", line 1, in ? OverflowError: math range error >>> ugh. ) -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From haase at msg.ucsf.edu Fri Nov 21 12:23:02 2003 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Fri Nov 21 12:23:02 2003 Subject: [Numpy-discussion] Community Poll: numarray default underflow handling == "ignore" ? References: <1069365809.5225.145.camel@halloween.stsci.edu> <3FBD4BD4.FD1F30B9@noaa.gov> <1069421715.7448.2.camel@halloween.stsci.edu> <3FBE4262.BEF5DEE4@noaa.gov> <20031121191114.GA7441@arbutus.physics.mcmaster.ca> Message-ID: <050501c3b06d$1ffe4ff0$421ee6a9@rodan> My vote would be '-1' ( if that means "I prefer ignore") I'm thinking of an INTERACTIVE platform - and so it would just "look nicer" without to many warnings. Actually on that note: I read some time ago about pythons default for printing floats: >>> 0.1 0.10000000000000001 >>> print 0.1 0.1 >>> repr(0.1) '0.10000000000000001' >>> str(.1) '0.1' Does anyone here have an update on that ? What I am especially interested in is when I have a list of (floating point) (x,y) positions and then typing in the var-name and getting all these ugly numbers is still very frustration for me ;-) Thanks, Sebastian Haase ----- Original Message ----- From: "David M. Cooke" To: "numpy-discussion" Sent: Friday, November 21, 2003 11:11 AM Subject: Re: [Numpy-discussion] Community Poll: numarray default underflow handling == "ignore" ? > On Fri, Nov 21, 2003 at 08:50:42AM -0800, Chris Barker wrote: > > Todd Miller wrote: > > > > > Someone recently suggested that we change the default for numarray > > > > > underflow checking to "ignore". Please vote (+1, 0, -1): > > > > > The default of underflow is "warn" > > > > assuming that ignore means that zero will be used, I vote +1 > > > > If the difference between zero and the smallest floating point number is > > significant in your app, you'd better know what you're doing with > > floating point, and you should then know to set the check to "warn" or > > "raise". > > Agree, +1. It's quite common (for me) that I want exp(-1000) to give 0, > and not an error (or warning, espicially in a loop that goes for several > thousand iterations...) Overflow, yes, that's a problem, but not underflow. > > (The behaviour below lead me to writing my own exp ufunc for Numeric: > >>> import Numeric > >>> Numeric.exp(-1000) > Traceback (most recent call last): > File "", line 1, in ? > OverflowError: math range error > >>> > ugh. > ) > > -- > |>|\/|< > /--------------------------------------------------------------------------\ > |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ > |cookedm at physics.mcmaster.ca > > > ------------------------------------------------------- > This SF.net email is sponsored by: SF.net Giveback Program. > Does SourceForge.net help you be more productive? Does it > help you create better code? SHARE THE LOVE, and help us help > YOU! Click Here: http://sourceforge.net/donate/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From jmiller at stsci.edu Fri Nov 21 12:28:06 2003 From: jmiller at stsci.edu (Todd Miller) Date: Fri Nov 21 12:28:06 2003 Subject: [Numpy-discussion] Community Poll: numarray default underflow handling == "ignore" ? In-Reply-To: <050501c3b06d$1ffe4ff0$421ee6a9@rodan> References: <1069365809.5225.145.camel@halloween.stsci.edu> <3FBD4BD4.FD1F30B9@noaa.gov> <1069421715.7448.2.camel@halloween.stsci.edu> <3FBE4262.BEF5DEE4@noaa.gov> <20031121191114.GA7441@arbutus.physics.mcmaster.ca> <050501c3b06d$1ffe4ff0$421ee6a9@rodan> Message-ID: <1069446392.7446.17.camel@halloween.stsci.edu> On Fri, 2003-11-21 at 15:22, Sebastian Haase wrote: > My vote would be '-1' ( if that means "I prefer ignore") The vote is for switching from "warn" on underflow to "ignore" on overflow, so: +1 == "ignore" underflow (vote for the change) 0 == don't care -1 == "warn" underflow (vote against the change) -- Todd Miller Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21030 (410) 338 - 4576 From a.schmolck at gmx.net Fri Nov 21 16:38:02 2003 From: a.schmolck at gmx.net (Alexander Schmolck) Date: Fri Nov 21 16:38:02 2003 Subject: [Numpy-discussion] Community Poll: numarray default underflow handling == "ignore" ? In-Reply-To: <050501c3b06d$1ffe4ff0$421ee6a9@rodan> References: <1069365809.5225.145.camel@halloween.stsci.edu> <3FBD4BD4.FD1F30B9@noaa.gov> <1069421715.7448.2.camel@halloween.stsci.edu> <3FBE4262.BEF5DEE4@noaa.gov> <20031121191114.GA7441@arbutus.physics.mcmaster.ca> <050501c3b06d$1ffe4ff0$421ee6a9@rodan> Message-ID: "Sebastian Haase" writes: > My vote would be '-1' ( if that means "I prefer ignore") > I'm thinking of an INTERACTIVE platform - and so it would just "look nicer" > without to many warnings. Well, it's only a default so you could always deactivate it (for all interactive sessions in your PYTHONSTARTUP if you wanted). > > Actually on that note: I read some time ago about pythons default for > printing floats: > >>> 0.1 > 0.10000000000000001 > >>> print 0.1 > 0.1 > >>> repr(0.1) > '0.10000000000000001' > >>> str(.1) > '0.1' > > Does anyone here have an update on that ? > What I am especially interested in is when I have a list of (floating point) > (x,y) positions and > then typing in the var-name and getting all these ugly numbers is still very > frustration for me ;-) You can customize python's interactive printing behavior any way you like (in your PYTHONSTARTUP). Here is an example from my old ~/.pythonrc.py (nowadays I almost exclusively use ipython). import pprint PRETTY_PRINT=1 _normal_displayhook = sys.displayhook def _my_displayhook(object): if PRETTY_PRINT: # don't bore us with None if object is not None: pprint.pprint(object) else: _normal_displayhook(object) sys.displayhook = _my_displayhook You could add something to the above to achieve the floating point (or list) formating you desire (``if type(object) is float:...). Since I am a heavy interactive user and found the default floating formating of arrays somewhat clumsy for interactive work, I also wrote some more fanciful formatting code for my Numeric/numarray compatible matrix class that amongst other things offers a number of formating options, including matlab style. I found that this made my life much easier. Thus: >>> a array([[-9.90000000e+01, -9.72817182e+01, 0.00000000e+00, -7.99144631e+01], [-4.54018500e+01, 4.84131591e+01, 3.03428793e+02, 9.96633158e+02], [2.88095799e+03, 8.00308393e+03, 2.19264658e+04, 5.97741417e+04]]) >>> m = matrix(a) >>> m matrix(''' 1.0E+04 * -0.00990 -0.00973 0.00000 -0.00799 -0.00454 0.00484 0.03034 0.09966 0.28810 0.80031 2.19265 5.97741 ''') >>> m.show('long') matrix(''' 1.0E+04 * Columns 0 through 3 -0.009900000000000 -0.009728171817154 0.000000000000000 -0.004540184996686 0.004841315910258 0.030342879349274 0.288095798704173 0.800308392757538 2.192646579480672 Columns 3 through 4 -0.007991446307681 0.099663315842846 5.977414171519782 ''') >>> m.show('+') m.show('+') matrix(''' -- - -+++ ++++ ''') etc. Adapting this to e.g. format Numeric arrays similarly via the display hook shouldn't be too hard, I can provide the code if you're interested. 'as From haase at msg.ucsf.edu Fri Nov 21 16:56:05 2003 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Fri Nov 21 16:56:05 2003 Subject: [Numpy-discussion] Community Poll: numarray default underflow handling == "ignore" ? References: <1069365809.5225.145.camel@halloween.stsci.edu><3FBD4BD4.FD1F30B9@noaa.gov><1069421715.7448.2.camel@halloween.stsci.edu><3FBE4262.BEF5DEE4@noaa.gov><20031121191114.GA7441@arbutus.physics.mcmaster.ca><050501c3b06d$1ffe4ff0$421ee6a9@rodan> Message-ID: <053901c3b093$5700d7e0$421ee6a9@rodan> Hi Alexander, Thanks for your reply and yes, I would be very interested to see more of your "pretty print" code. I am trying to put together all kinds of useful things to make python ( with numarray and scipy) a possible replacement for Matlab... I am not hoping for a similarly big documentation and help system, but just the fact that it's free ( doesn't need to connect to a "license server" every other minute) and that it happily supports single-precision arrays ( not to mention memmap array !) should make it interesting. I know that I probably should use IPython, but I also run python embedded in a wxWindows app and therefore use pyCrust. Thanks again, Sebastian Haase ----- Original Message ----- From: "Alexander Schmolck" To: "Sebastian Haase" Cc: Sent: Friday, November 21, 2003 4:37 PM Subject: Re: [Numpy-discussion] Community Poll: numarray default underflow handling == "ignore" ? > "Sebastian Haase" writes: > > > My vote would be '-1' ( if that means "I prefer ignore") > > I'm thinking of an INTERACTIVE platform - and so it would just "look nicer" > > without to many warnings. > > Well, it's only a default so you could always deactivate it (for all > interactive sessions in your PYTHONSTARTUP if you wanted). > > > > > Actually on that note: I read some time ago about pythons default for > > printing floats: > > >>> 0.1 > > 0.10000000000000001 > > >>> print 0.1 > > 0.1 > > >>> repr(0.1) > > '0.10000000000000001' > > >>> str(.1) > > '0.1' > > > > Does anyone here have an update on that ? > > > > > What I am especially interested in is when I have a list of (floating point) > > (x,y) positions and > > then typing in the var-name and getting all these ugly numbers is still very > > frustration for me ;-) > > You can customize python's interactive printing behavior any way you like (in > your PYTHONSTARTUP). Here is an example from my old ~/.pythonrc.py (nowadays I > almost exclusively use ipython). > > import pprint > PRETTY_PRINT=1 > _normal_displayhook = sys.displayhook > def _my_displayhook(object): > if PRETTY_PRINT: > # don't bore us with None > if object is not None: > pprint.pprint(object) > else: > _normal_displayhook(object) > sys.displayhook = _my_displayhook > > You could add something to the above to achieve the floating point (or list) > formating you desire (``if type(object) is float:...). > > Since I am a heavy interactive user and found the default floating formating > of arrays somewhat clumsy for interactive work, I also wrote some more > fanciful formatting code for my Numeric/numarray compatible matrix class that > amongst other things offers a number of formating options, including matlab > style. I found that this made my life much easier. > > Thus: > > >>> a > array([[-9.90000000e+01, -9.72817182e+01, 0.00000000e+00, -7.99144631e+01], > [-4.54018500e+01, 4.84131591e+01, 3.03428793e+02, 9.96633158e+02], > [2.88095799e+03, 8.00308393e+03, 2.19264658e+04, 5.97741417e+04]]) > >>> m = matrix(a) > >>> m > matrix(''' > 1.0E+04 * > > -0.00990 -0.00973 0.00000 -0.00799 > -0.00454 0.00484 0.03034 0.09966 > 0.28810 0.80031 2.19265 5.97741 > ''') > >>> m.show('long') > matrix(''' > 1.0E+04 * > > > Columns 0 through 3 > > -0.009900000000000 -0.009728171817154 0.000000000000000 > -0.004540184996686 0.004841315910258 0.030342879349274 > 0.288095798704173 0.800308392757538 2.192646579480672 > > Columns 3 through 4 > > -0.007991446307681 > 0.099663315842846 > 5.977414171519782 > ''') > >>> m.show('+') > m.show('+') > matrix(''' > -- - > -+++ > ++++ > ''') > > etc. > > Adapting this to e.g. format Numeric arrays similarly via the display hook > shouldn't be too hard, I can provide the code if you're interested. > > > 'as > > > > ------------------------------------------------------- > This SF.net email is sponsored by: SF.net Giveback Program. > Does SourceForge.net help you be more productive? Does it > help you create better code? SHARE THE LOVE, and help us help > YOU! Click Here: http://sourceforge.net/donate/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From focke at SLAC.Stanford.EDU Fri Nov 21 17:36:09 2003 From: focke at SLAC.Stanford.EDU (Warren Focke) Date: Fri Nov 21 17:36:09 2003 Subject: [Numpy-discussion] Community Poll: numarray default underflow handling == "ignore" ? In-Reply-To: <1069365809.5225.145.camel@halloween.stsci.edu> References: <1069365809.5225.145.camel@halloween.stsci.edu> Message-ID: +1 Warren Focke On Thu, 20 Nov 2003, Todd Miller wrote: > Someone recently suggested that we change the default for numarray > underflow checking to "ignore". Please vote (+1, 0, -1): From a.schmolck at gmx.net Fri Nov 21 19:24:05 2003 From: a.schmolck at gmx.net (Alexander Schmolck) Date: Fri Nov 21 19:24:05 2003 Subject: [Numpy-discussion] Community Poll: numarray default underflow handling == "ignore" ? In-Reply-To: <053901c3b093$5700d7e0$421ee6a9@rodan> References: <1069365809.5225.145.camel@halloween.stsci.edu> <3FBD4BD4.FD1F30B9@noaa.gov> <1069421715.7448.2.camel@halloween.stsci.edu> <3FBE4262.BEF5DEE4@noaa.gov> <20031121191114.GA7441@arbutus.physics.mcmaster.ca> <050501c3b06d$1ffe4ff0$421ee6a9@rodan> <053901c3b093$5700d7e0$421ee6a9@rodan> Message-ID: "Sebastian Haase" writes: > Hi Alexander, > Thanks for your reply and yes, I would be very interested to see more of > your "pretty print" code. Sure, I'll send you the code. As it is for a matrix class it only handles up to (array-)rank 2 currently (and uses methods of the matrix class) but I don't think changing either of these should be too hard. > I am trying to put together all kinds of useful things to make python ( with > numarray and scipy) a possible replacement for Matlab... Been, there done that -- actually still doing it. When I was faced with the choice between matlab and python for my numerical work, I decided that despite the convince matlab offered for many things (libraries, interactive environment) I couldn't face the its monstrosity as a programming language and thus opted for python. These were the main areas were I found python/Numeric was lagging behind: 1. Numeric.dot was much too slow (factor 40-100 and more for large arrays) 2. Although Numeric's arrays are quite general and powerful, I found writing down simple matrix expressions (dot(transpose(X), dot(...))) and the visual clarity of the repr of the result less attractive than matlab's. 3. Library support (e.g. in some fields many academics tend to make matlab code corresponding to their articles available, which is very handy to quickly try something). 4. Matlab came with very good plotting capabilities, whereas python's support for plotting was quite poor when I started using it. 5. Matlab also had a pretty good interactive environment. So far I've managed to come up with the following (sometimes partial) solutions for the above problems. 1. An optional fast dot product using blas (although I somehow still ended up spending a few days on it, all the real work was done by my supervisor Richard Everson -- I've submitted a patch that made it into Numeric 23 but I don't think numarray has an the option of a fast dot product yet, although I haven't checked). 2. A powerful matrix class (mostly finished; should transparently support Numeric as well as numarray (the latter is not fully there yet), unreleased). 3. A high-level matlab interface (pretty much ready to be released). 4. A convenient and high-level plotting library that supports multiple backends (currently only proper support for matlab as backend although I started working on/contemplating others (e.g. grace, gnuplot and chaco); unreleased) 5. IPython (kudos to Fernando!) + emacs + a few convenience functions for things like saving variables and redefining classes makes a darn nice environment once one got the hang of it (unfortunately it meant I had to write the emacs support for ipython, it's available on the ipython website). Of course if anyone wants to lend a hand, he/she is more than welcome ;) > I am not hoping for a similarly big documentation and help system, but just Why not? Nowadays pretty much all python code comes with pretty good docstrings and implementing something like 'lookfor' shouldn't be particularly hard. Already looking up docs and jumping to code in ipython is quite convenient (e.g. ``some_func?``). > the fact that it's free ( doesn't need to connect to a "license server" > every other minute) and that it happily supports single-precision arrays > ( not to mention memmap array !) should make it interesting. And that it is not based on an abomination of a programming language but an excellent one, with plenty of high quality general purpose libs (and even if only doing numerics is concerned: Numeric/numarray's (APL-ish) design also doesn't compare badly to matlab, IMHO). > I know that I probably should use IPython, Indeed, everyone should (from within emacs ;) 'as From verveer at embl.de Sat Nov 22 10:32:02 2003 From: verveer at embl.de (verveer at embl.de) Date: Sat Nov 22 10:32:02 2003 Subject: [Numpy-discussion] Community Poll: numarray default underflow handling == "ignore" ? In-Reply-To: References: Message-ID: <1069525884.3fbfab7cc95c0@webmail.embl.de> +1 for "ignore" Peter From jochen at jochen-kuepper.de Sun Nov 23 05:07:05 2003 From: jochen at jochen-kuepper.de (=?iso-8859-1?q?Jochen_K=FCpper?=) Date: Sun Nov 23 05:07:05 2003 Subject: [Numpy-discussion] Community Poll: numarray default underflow handling == "ignore" ? In-Reply-To: <1069446392.7446.17.camel@halloween.stsci.edu> (Todd Miller's message of "21 Nov 2003 15:26:32 -0500") References: <1069365809.5225.145.camel@halloween.stsci.edu> <3FBD4BD4.FD1F30B9@noaa.gov> <1069421715.7448.2.camel@halloween.stsci.edu> <3FBE4262.BEF5DEE4@noaa.gov> <20031121191114.GA7441@arbutus.physics.mcmaster.ca> <050501c3b06d$1ffe4ff0$421ee6a9@rodan> <1069446392.7446.17.camel@halloween.stsci.edu> Message-ID: <86r7zzjku3.fsf@doze.rijnh.nl> On 21 Nov 2003 15:26:32 -0500 Todd Miller wrote: Todd> The vote is for switching from "warn" on underflow to "ignore" Todd> on overflow, so: ^^^^ under! Todd> +1 == "ignore" underflow (vote for the change) Todd> 0 == don't care Todd> -1 == "warn" underflow (vote against the change) +1 Greetings, Jochen -- Einigkeit und Recht und Freiheit http://www.Jochen-Kuepper.de Libert?, ?galit?, Fraternit? GnuPG key: CC1B0B4D (Part 3 you find in my messages before fall 2003.) From BulkSales at 2911.net Sun Nov 23 19:12:02 2003 From: BulkSales at 2911.net (Hosting) Date: Sun Nov 23 19:12:02 2003 Subject: [Numpy-discussion] Bulk Friendly Web Hosting & Server Message-ID: Dear Sales: We offer reliable bulk email friendly web hosting services. You can now have the peace of mind knowing that your web site is secure during your email marketing campaigns. Bullet Proof Web Hosting 100% Guaranteed! We guarantee that your site will be 99% uptime. Hosting Features: Your own Web address: www.YourName.com Dedicated 100 M fiber 100MB of disk space Unlimited Data Transfer FTP Support FrontPage Support ASP Support PERL Support PHP Support CGI Support No Setup Fee US$199.00/month Bullet-Proof dedicated servers: 1. Two IPs 1024MB RAM DDR PIIII / Two CPU 120 GB SCS Intel 82559 LAN /(2U) Unlimited Data Transfer Linux / Windows / FreeBSD Price: No setup fee US$ 599.00/month 2. Dynamic IP 1024MB RAM DDR PIIII / Two CPU 120 GB SCS Intel 82559 LAN /(2U) Unlimited Data Transfer Linux / Windows / FreeBSD Price: No setup fee US$ 999.00/month Allowed Usage You can use the server for any of the following: Direct Bulk Mailing or Proxy Mailing Web Site Hosting Proxy, Relay or Port Scanning Restrictions & Rules Our Plans do NOT include the following: No Adult Content No Illegal Material We aslo offer targeted email addresses and send out targeted emails for you. We looking forward to serving you in the near future. Cheers! Betty Jones Support Teams Support at biz-servers.com Http://www.biz-servers.com From cjw at sympatico.ca Mon Nov 24 05:18:02 2003 From: cjw at sympatico.ca (Colin J. Williams) Date: Mon Nov 24 05:18:02 2003 Subject: [Numpy-discussion] PyMatrix: Announcement Message-ID: <3FC204D0.2010503@sympatico.ca> PyMatrix is available for test and review. http://www3.sympatico.ca/cjw PyMatrix provides access to basic matrix arithmetic, using Python and numarray. Examples: A * B => the product of matrices A and B A.I => the inverse of matrix A A.EVectors => the eigenvectors of A A.var(0) => the variances of the columns of A (a.T*a).I * a.T*b => the solution (x) for a * x = b, where a is a matrix and b a column vector This package was developed on a Windows XP. I would appreciate comments, particularly with respect to usage on other systems. Colin W. From nadavh at VisionSense.com Mon Nov 24 06:30:11 2003 From: nadavh at VisionSense.com (Nadav Horesh) Date: Mon Nov 24 06:30:11 2003 Subject: [Numpy-discussion] PyMatrix: Announcement In-Reply-To: <3FC204D0.2010503@sympatico.ca> References: <3FC204D0.2010503@sympatico.ca> Message-ID: <1069684102.5532.1.camel@Nadav.Envision.co.il> Don't see how can I get the sources for gnu/linux installation. Nadav On Mon, 2003-11-24 at 15:17, Colin J. Williams wrote: > PyMatrix is available for test and review. > http://www3.sympatico.ca/cjw > > PyMatrix provides access to basic matrix arithmetic, using Python and > numarray. > > Examples: > A * B => the product of > matrices A and B > A.I => the inverse of matrix A > A.EVectors => the eigenvectors of A > A.var(0) => the variances of the > columns of A > (a.T*a).I * a.T*b => the solution (x) for a * > x = b, > where a is a > matrix and b a column vector > > This package was developed on a Windows XP. I would appreciate > comments, particularly with respect to usage on other systems. > > Colin W. > > > > > > ------------------------------------------------------- > This SF.net email is sponsored by: SF.net Giveback Program. > Does SourceForge.net help you be more productive? Does it > help you create better code? SHARE THE LOVE, and help us help > YOU! Click Here: http://sourceforge.net/donate/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From haase at msg.ucsf.edu Mon Nov 24 11:12:05 2003 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Mon Nov 24 11:12:05 2003 Subject: [Numpy-discussion] OT:pretty print of reals (was: Community Poll: numarray default underflow handling == "ignore" ?) References: <1069365809.5225.145.camel@halloween.stsci.edu><3FBD4BD4.FD1F30B9@noaa.gov><1069421715.7448.2.camel@halloween.stsci.edu><3FBE4262.BEF5DEE4@noaa.gov><20031121191114.GA7441@arbutus.physics.mcmaster.ca><050501c3b06d$1ffe4ff0$421ee6a9@rodan><053901c3b093$5700d7e0$421ee6a9@rodan> Message-ID: <070b01c3b2be$caf26740$421ee6a9@rodan> Hi Alexander, Thanks for the code - I will look into it I just read a few lines of your MatrixFormatter class seeing the definition of __repr__. Here is a thought: Is it possible that I overload the __repr__ of float (that is: the buildin data type) !? When I use pyCrust it automatically "fills in" default arguments like, func(a=0.1000000000001) So, this probably does not call the sys.displayhook - so __repr__ itself would need to be changed. Do you know if that is possible ? (It would recursively fix all list and matrix display problems I had ...) Thanks again, Sebastian ----- Original Message ----- From: "Alexander Schmolck" To: "Sebastian Haase" Cc: Sent: Friday, November 21, 2003 7:23 PM Subject: Re: [Numpy-discussion] Community Poll: numarray default underflow handling == "ignore" ? > "Sebastian Haase" writes: > > > Hi Alexander, > > Thanks for your reply and yes, I would be very interested to see more of > > your "pretty print" code. > > Sure, I'll send you the code. As it is for a matrix class it only handles up > to (array-)rank 2 currently (and uses methods of the matrix class) but I don't > think changing either of these should be too hard. > > > I am trying to put together all kinds of useful things to make python ( with > > numarray and scipy) a possible replacement for Matlab... > > Been, there done that -- actually still doing it. > > When I was faced with the choice between matlab and python for my numerical > work, I decided that despite the convince matlab offered for many things > (libraries, interactive environment) I couldn't face the its monstrosity as a > programming language and thus opted for python. > > These were the main areas were I found python/Numeric was lagging behind: > > 1. Numeric.dot was much too slow (factor 40-100 and more for large arrays) > 2. Although Numeric's arrays are quite general and powerful, I found writing > down simple matrix expressions (dot(transpose(X), dot(...))) and the visual > clarity of the repr of the result less attractive than matlab's. > 3. Library support (e.g. in some fields many academics tend to make matlab > code corresponding to their articles available, which is very handy to > quickly try something). > 4. Matlab came with very good plotting capabilities, whereas python's support > for plotting was quite poor when I started using it. > 5. Matlab also had a pretty good interactive environment. > > So far I've managed to come up with the following (sometimes partial) > solutions for the above problems. > > 1. An optional fast dot product using blas (although I somehow still ended up > spending a few days on it, all the real work was done by my supervisor > Richard Everson -- I've submitted a patch that made it into Numeric 23 but > I don't think numarray has an the option of a fast dot product yet, > although I haven't checked). > 2. A powerful matrix class (mostly finished; should transparently support > Numeric as well as numarray (the latter is not fully there yet), > unreleased). > 3. A high-level matlab interface (pretty much ready to be released). > 4. A convenient and high-level plotting library that supports multiple > backends (currently only proper support for matlab as backend although I > started working on/contemplating others (e.g. grace, gnuplot and chaco); > unreleased) > 5. IPython (kudos to Fernando!) + emacs + a few convenience functions for > things like saving variables and redefining classes makes a darn nice > environment once one got the hang of it (unfortunately it meant I had to > write the emacs support for ipython, it's available on the ipython > website). > > Of course if anyone wants to lend a hand, he/she is more than welcome ;) > > > I am not hoping for a similarly big documentation and help system, but just > > Why not? Nowadays pretty much all python code comes with pretty good > docstrings and implementing something like 'lookfor' shouldn't be particularly > hard. Already looking up docs and jumping to code in ipython is quite > convenient (e.g. ``some_func?``). > > > the fact that it's free ( doesn't need to connect to a "license server" > > every other minute) and that it happily supports single-precision arrays > > ( not to mention memmap array !) should make it interesting. > > And that it is not based on an abomination of a programming language but an > excellent one, with plenty of high quality general purpose libs (and even if > only doing numerics is concerned: Numeric/numarray's (APL-ish) design also > doesn't compare badly to matlab, IMHO). > > > I know that I probably should use IPython, > > Indeed, everyone should (from within emacs ;) > > 'as > > From haase at msg.ucsf.edu Mon Nov 24 11:14:07 2003 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Mon Nov 24 11:14:07 2003 Subject: [Numpy-discussion] how-to "put" RAM-based numarray into memmap References: <047301c3b052$37f38960$421ee6a9@rodan> <3FBE4B21.C07CBF38@noaa.gov> Message-ID: <071d01c3b2bf$11171950$421ee6a9@rodan> > Sebastian Haase wrote: > > > > Hi, > > Suppose I have a 500MB-ram Computer and a 300MB ram-only (standard) > > numarray. > > Now I would like to "save" that onto harddrive (with a small header up > > front > > How about: > > f = open(filename, 'wb') > f.write(MyHeader) > A.tofile(f) > > To read it back in, you need to know where your header ends, by either > parsing it or using one of the same size every time, then you can use > fromfile() to create an array form it. The main reason for my question was just to find out if NUMARRAY supports it, and how ? Also I have many "bookkeeping" functions already implemented for the memmap'd case. (That is, I have a class with member methods operting on a member memmapped array) So if what I descibed is possible I could save myself form duplicating lot's of code. Essentially I was hoping for the most ellegant solution ;-) Thanks, Sebastian From jmiller at stsci.edu Mon Nov 24 11:35:08 2003 From: jmiller at stsci.edu (Todd Miller) Date: Mon Nov 24 11:35:08 2003 Subject: [Numpy-discussion] how-to "put" RAM-based numarray into memmap In-Reply-To: <071d01c3b2bf$11171950$421ee6a9@rodan> References: <047301c3b052$37f38960$421ee6a9@rodan> <3FBE4B21.C07CBF38@noaa.gov> <071d01c3b2bf$11171950$421ee6a9@rodan> Message-ID: <1069702447.4416.144.camel@localhost.localdomain> On Mon, 2003-11-24 at 14:13, Sebastian Haase wrote: > > Sebastian Haase wrote: > > > > > > Hi, > > > Suppose I have a 500MB-ram Computer and a 300MB ram-only (standard) > > > numarray. > > > Now I would like to "save" that onto harddrive (with a small header up > > > front > > > > How about: > > > > f = open(filename, 'wb') > > f.write(MyHeader) > > A.tofile(f) > > > > To read it back in, you need to know where your header ends, by either > > parsing it or using one of the same size every time, then you can use > > fromfile() to create an array form it. > > The main reason for my question was just to find out if NUMARRAY supports > it, and how ? > Also I have many "bookkeeping" functions already implemented for the > memmap'd case. > (That is, I have a class with member methods operting on a member memmapped > array) > So if what I descibed is possible I could save myself form duplicating lot's > of code. > > Essentially I was hoping for the most ellegant solution ;-) > memmap's Memmap class does support an insert() method for adding a new slice to the end of (or anywhere in) an existing map. The new slice, however, will exist as a block of memory allocated on the heap until the memmap is saved to disk. Thus, two scenarios present themselves: (1) you allocate the new slice ahead of time and create an array from it, avoiding data duplication (2) you create an array and later copy it into (a newly inserted slice of) the memmap, thereby duplicating your data on the heap. When you close the map, slices on the heap are written to the map file. Todd > Thanks, > Sebastian > > > > ------------------------------------------------------- > This SF.net email is sponsored by: SF.net Giveback Program. > Does SourceForge.net help you be more productive? Does it > help you create better code? SHARE THE LOVE, and help us help > YOU! Click Here: http://sourceforge.net/donate/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- Todd Miller From a.schmolck at gmx.net Mon Nov 24 11:48:06 2003 From: a.schmolck at gmx.net (Alexander Schmolck) Date: Mon Nov 24 11:48:06 2003 Subject: [Numpy-discussion] Re: OT:pretty print of reals (was: Community Poll: numarray default underflow handling == "ignore" ?) In-Reply-To: <070b01c3b2be$caf26740$421ee6a9@rodan> References: <1069365809.5225.145.camel@halloween.stsci.edu> <3FBD4BD4.FD1F30B9@noaa.gov> <1069421715.7448.2.camel@halloween.stsci.edu> <3FBE4262.BEF5DEE4@noaa.gov> <20031121191114.GA7441@arbutus.physics.mcmaster.ca> <050501c3b06d$1ffe4ff0$421ee6a9@rodan> <053901c3b093$5700d7e0$421ee6a9@rodan> <070b01c3b2be$caf26740$421ee6a9@rodan> Message-ID: "Sebastian Haase" writes: > Hi Alexander, > Thanks for the code - I will look into it > I just read a few lines of your MatrixFormatter class seeing the definition > of __repr__. > > Here is a thought: Is it possible that I overload the __repr__ of float > (that is: the buildin data type) !? Nope. You can't screw around with methods of builtins -- and this is a (quite sensible) conscious design decision. > When I use pyCrust it automatically "fills in" default arguments like, > func(a=0.1000000000001) > So, this probably does not call the sys.displayhook - so __repr__ itself > would need to be changed. Nope, whatever pycrust uses needs to be changed (are you *sure* changing sys.displayhook has no effect on pycrust (it might not change what it "fills in", but what about actual output?) because that would seem sort of ... suboptimal. Even so it shouldn't be too difficult to figure out how you can customize printing in pycrust (by having a look at the pycrust code and/or reading the manual or asking on the appropriate mailing-list). > Do you know if that is possible ? (It would recursively fix all list and > matrix display problems I had ...) It wouldn't, it would just get you the float and list (tuple etc.) display you want (arrays/matrices don't call repr on each individual array arguments to compute their overall repr). HTH, 'as From haase at msg.ucsf.edu Mon Nov 24 11:59:13 2003 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Mon Nov 24 11:59:13 2003 Subject: [Numpy-discussion] how-to "put" RAM-based numarray into memmap References: <047301c3b052$37f38960$421ee6a9@rodan> <3FBE4B21.C07CBF38@noaa.gov> <071d01c3b2bf$11171950$421ee6a9@rodan> <1069702447.4416.144.camel@localhost.localdomain> Message-ID: <073d01c3b2c5$59532af0$421ee6a9@rodan> > On Mon, 2003-11-24 at 14:13, Sebastian Haase wrote: > > > Sebastian Haase wrote: > > > > > > > > Hi, > > > > Suppose I have a 500MB-ram Computer and a 300MB ram-only (standard) > > > > numarray. > > > > Now I would like to "save" that onto harddrive (with a small header up > > > > front > > > > > > How about: > > > > > > f = open(filename, 'wb') > > > f.write(MyHeader) > > > A.tofile(f) > > > > > > To read it back in, you need to know where your header ends, by either > > > parsing it or using one of the same size every time, then you can use > > > fromfile() to create an array form it. > > > > The main reason for my question was just to find out if NUMARRAY supports > > it, and how ? > > Also I have many "bookkeeping" functions already implemented for the > > memmap'd case. > > (That is, I have a class with member methods operting on a member memmapped > > array) > > So if what I descibed is possible I could save myself form duplicating lot's > > of code. > > > > Essentially I was hoping for the most ellegant solution ;-) > > > > memmap's Memmap class does support an insert() method for adding a new > slice to the end of (or anywhere in) an existing map. The new slice, > however, will exist as a block of memory allocated on the heap until > the memmap is saved to disk. > > Thus, two scenarios present themselves: (1) you allocate the new slice > ahead of time and create an array from it, avoiding data duplication (2) > you create an array and later copy it into (a newly inserted slice of) > the memmap, thereby duplicating your data on the heap. > > When you close the map, slices on the heap are written to the map file. > > Todd > Thanks for your reply. Is it possible to define a memmap slice and giving it a (preinitialized !) memory buffer ? I'm thinking: I have a RAM-based numarray, I just take the buffer (pointer) and hand it over to the memmap-slice so that it can make the association between the disk-space and the RAM-space. I guess you are calling "heap" what I call RAM. Is memmap using something inherently different that heap? (I might be missing something...) As I was trying to illustrate in my example, my ram-numarray might already be using most of the available address space. Thanks, Sebastian From jmiller at stsci.edu Mon Nov 24 13:19:04 2003 From: jmiller at stsci.edu (Todd Miller) Date: Mon Nov 24 13:19:04 2003 Subject: [Numpy-discussion] how-to "put" RAM-based numarray into memmap In-Reply-To: <073d01c3b2c5$59532af0$421ee6a9@rodan> References: <047301c3b052$37f38960$421ee6a9@rodan> <3FBE4B21.C07CBF38@noaa.gov> <071d01c3b2bf$11171950$421ee6a9@rodan> <1069702447.4416.144.camel@localhost.localdomain> <073d01c3b2c5$59532af0$421ee6a9@rodan> Message-ID: <1069708711.4415.215.camel@localhost.localdomain> > Thanks for your reply. > Is it possible to define a memmap slice and giving it a (preinitialized !) > memory buffer ? It should be now. I just added support for a "buffer" parameter to CVS. I tested it manually and it worked OK for me; please try it out or watch for it in numarray-0.8. > I'm thinking: I have a RAM-based numarray, I just take the buffer (pointer) > and hand it over to the memmap-slice so that it can make the association > between the disk-space and the RAM-space. > I guess you are calling "heap" what I call RAM. Is memmap using something > inherently different that heap? (I might be missing something...) Yes and no. When you allocate a slice of an existing memmap, you're getting memory backed on the disk file of your choice; mapped memory generally comes from a different address partition than the memory from malloc(). When you allocate memory using numarray, or insert a slice into a memmap without specifying the buffer, you're getting memory from the heap which is backed on the system swap file. (Of course, this varies a little by system as well; I'm thinking Linux/UNIX here and Win32 may be slightly different). Todd -- Todd Miller From cjw at sympatico.ca Tue Nov 25 11:09:04 2003 From: cjw at sympatico.ca (Colin J. Williams) Date: Tue Nov 25 11:09:04 2003 Subject: [Numpy-discussion] PyMatrix: Announcement Message-ID: <3FC3A8AE.6030204@sympatico.ca> In response to a number of requests, the PyMatrix package is now available for download as a zip file. Colin W. > PyMatrix is available for test and review. > http://www3.sympatico.ca/cjw > > PyMatrix provides access to basic matrix arithmetic, using Python and > numarray. > > Examples: > A * B => the product of matrices A and B > A.I => the inverse of matrix A > A.EVectors => the eigenvectors of A > A.var(0) => the variances of the columns of A > (a.T*a).I * a.T*b => the solution (x) for a * x = b, > where a is a matrix and b a column vector > > This package was developed on a Windows XP. I would appreciate > comments, particularly with respect to usage on other systems. > > Colin W. > From gvermeul at grenoble.cnrs.fr Wed Nov 26 11:59:03 2003 From: gvermeul at grenoble.cnrs.fr (Gerard Vermeulen) Date: Wed Nov 26 11:59:03 2003 Subject: [Numpy-discussion] ANN: PyQwt-3.8 Message-ID: <20031126205924.62dc5947.gvermeul@grenoble.cnrs.fr> PyQwt-3.8 = FAST and EASY data plotting for Python and (Py)Qt. PyQwt is a set of Python bindings for the Qwt C++ class library which extends the Qt framework with widgets for scientific and engineering applications. It supports the use of PyQt, Qt, Qwt, the Numerical Python extensions (either Numeric, or numarray or both) and optionally SciPy in a GUI Python application or in an interactive Python session. The home page of PyQwt is http://pyqwt.sourceforge.net. Main changes in PyQwt-3.8: 1. PyQwt inherits from Qwt the new classes QwtPicker, QwtPlotPicker and QwtPlotZoomer. QwtPicker can make any widget react to events by event filtering. QwtPlotPicker is a QwtPicker taylored to the plot canvas. QwtPlotZoomer implements infinite deep zooming on the plot canvas. 2. PyQwt inherits from Qwt the support for Qt's rich text format in QwtPlot to display equations like E=mc2. Caveat: in principle, a truetype font with full unicode support is better to typeset equations. 3. Bug fixes to support event filtering code (see EventFilterDemo.py). 4. Conversion of a QImage to a Numeric array or numarray array. 5. Better support for cloning a plot into Grace from the command line interpreter friendly plot classes. 6. PyQwt-3.8 works with PyQt-3.8.1 downto -3.6 and Qt-3.2.3 downto -2.3.0. 7. License change: GPL with permission for dynamic linking to commercial, educational and non-commercial versions of Qt, PyQt and sip. In this case PyQwt becomes a free plug-in for a non-free program. Gerard Vermeulen From proj_mgr8 at yahoo.com Sat Nov 29 10:41:03 2003 From: proj_mgr8 at yahoo.com (E l e c t r i c i a n) Date: Sat Nov 29 10:41:03 2003 Subject: [Numpy-discussion] R E S U M E Message-ID: Resume from: RICH' S for Job, Consulting, Installation or Service E L E C T R I C I A N Tel. (408) 482-2102 rysio3 at yahoo.com WIRING & INSTALLATION Hands on electrical installations perform fitting, mounting, laying cables on Commercial, Industrial, residential new & existing buildings. Electrical Power Supply for Lights, Plugs, Receptacles, Panels, & Fuse boxes, Emergency Generators wiring and testing, Transformers, Power Lines & conduit layout, bending and mounting, parking lighting, lamps, Signs, switches, posts and underground installations. Shopping Centers; grocery stories, hardware stories, restaurants & residential - housing areas, computer business & fast food units installation & buildings; Solar Panels, Sun Tracking, Flywheel Storage & cars electric systems modify, Natural Energy in Remote areas install. LOW VOLTAGE 12 / 24 Volt audio & video equipment, Computer & data network wiring, data backup and UPS; Monitoring Video Control & backup tapes set up and mounting, electro-optical assemblies & subsystems. DC Power Supply, Switch & Motion sensors Alarm. Fire & safety systems install. Fiber Optics systems, PLC setup, Master Control Center, cable modems & cable TV install. Network, UPS Battery Backup mounting and charging systems; Power supply testing, troubleshooting, and analyzing to a components level. Solar Signs, Electric Vehicles Design, Assembly & Installations. CC TV & Cameras, Security Systems & Sensors for Safety, Fire sprinklers and traffic Monitoring & Door Control. TECHNICIAN Use lab & shop equipment, mechanical, electrical & electronic tools, measurement & testing equipment, video cameras & microscopes. Support scientists & electronic engineers. Mechanical & Electro-Mechan. Design. OFFICE, ELECTRICAL AND MECHANICAL PROJECTS Electrical & Network Sketches, one line diagrams, and "as is" drawings update. Customizing Electronic and Electrical Components & Parts, Layouts electronic and electrical schematic, connectors and mechanical detailing. Quotes, supply, bids and job estimating. SOLAR PROJECTS. Customers contact, inspection, project mgmt & supervision of electricians & material handling; Use CAD, Windows and applications; ELECTRICAL & MAINTENANCE SERVICE US Citizen; open for travel . From mdehoon at ims.u-tokyo.ac.jp Sun Nov 30 22:13:05 2003 From: mdehoon at ims.u-tokyo.ac.jp (Michiel Jan Laurens de Hoon) Date: Sun Nov 30 22:13:05 2003 Subject: [Numpy-discussion] PyArray_ContiguousFromObject & PyDECREF Message-ID: <3FCADC96.3050504@ims.u-tokyo.ac.jp> Dear pythoneers, In Chapter 12 ("Writing a C extension") in the Numerical Python manual, in the first example PyArray_ContiguousFromObject is used together with PyDECREF, while in the last example PyArray_ContiguousFromObject is used but the arrays are not PyDECREF'ed before the function returns. From Chapter 13 ("C API Reference") it seems that a PyDECREF is needed in both cases. Or am I missing something here? Is PyDECREF not needed for some reason in the last example? Thanks in advance, Michiel de Hoon U Tokyo First example: ============== static PyObject * trace(PyObject *self, PyObject *args) { PyObject *input; PyArrayObject *array; double sum; int i, n; if (!PyArg_ParseTuple(args, "O", &input)) return NULL; array = (PyArrayObject *) PyArray_ContiguousFromObject(input, PyArray_DOUBLE, 2, 2); if (array == NULL) return NULL; n = array->dimensions[0]; if (n > array->dimensions[1]) n = array->dimensions[1]; sum = 0.; for (i = 0; i < n; i++) sum += *(double *)(array->data + i*array->strides[0] + i*array->strides[1]); Py_DECREF(array); return PyFloat_FromDouble(sum); } Last example: ============= static PyObject * matrix_vector(PyObject *self, PyObject *args) { PyObject *input1, *input2; PyArrayObject *matrix, *vector, *result; int dimensions[1]; double factor[1]; double real_zero[1] = {0.}; long int_one[1] = {1}; long dim0[1], dim1[1]; extern dgemv_(char *trans, long *m, long *n, double *alpha, double *a, long *lda, double *x, long *incx, double *beta, double *Y, long *incy); if (!PyArg_ParseTuple(args, "dOO", factor, &input1, &input2)) return NULL; matrix = (PyArrayObject *) PyArray_ContiguousFromObject(input1, PyArray_DOUBLE, 2, 2); if (matrix == NULL) return NULL; vector = (PyArrayObject *) PyArray_ContiguousFromObject(input2, PyArray_DOUBLE, 1, 1); if (vector == NULL) return NULL; if (matrix->dimensions[1] != vector->dimensions[0]) { PyErr_SetString(PyExc_ValueError, "array dimensions are not compatible"); return NULL; } dimensions[0] = matrix->dimensions[0]; result = (PyArrayObject *)PyArray_FromDims(1, dimensions, PyArray_DOUBLE); if (result == NULL) return NULL; dim0[0] = (long)matrix->dimensions[0]; dim1[0] = (long)matrix->dimensions[1]; dgemv_("T", dim1, dim0, factor, (double *)matrix->data, dim1, (double *)vector->data, int_one, real_zero, (double *)result->data, int_one); return PyArray_Return(result); } -- Michiel de Hoon, Assistant Professor University of Tokyo, Institute of Medical Science Human Genome Center 4-6-1 Shirokane-dai, Minato-ku Tokyo 108-8639 Japan http://bonsai.ims.u-tokyo.ac.jp/~mdehoon