From amullhau@ix.netcom.com Tue Jul 1 00:13:11 1997 From: amullhau@ix.netcom.com (Andrew P. Mullhaupt) Date: Mon, 30 Jun 1997 19:13:11 -0400 Subject: [MATRIX-SIG] reverse of take? In-Reply-To: <199706301443.KAA23238@dante.mh.lucent.com> Message-ID: <3.0.1.32.19970630191311.006d258c@popd.netcruiser> At 10:45 AM 6/30/97 -0400, Aaron Watters wrote: > >Take should have an inverse, and in my case I am fiddling during >spare moments on a win95 box, and I don't have time right now >to learn how to build other peoples stuff there. I suspect old S >hackers might run into something like this in NumPy and say "huh?" >and go back to S. Not at all. take is taken from APL, and most S programmers of any stature at all are fluent in APL. The reason for this is that S is sort of an 'unconscious' APL derivative - and there is a lot you can learn from APL. Recently, S has become much more widely used, and so you _will_ find a lot of newbie S users running around who don't know much about APL - but that's OK, most of them will not be coming across Python yet, either. S is a very demanding language to understand since it is such a wide sprawling mass of all sorts of stuff. It's a little like PL/I in that sense but S was never really designed, and so it turns out to be more or less a superset of APL, Lisp, and pretty much Numerical Python, too. Scientifically oriented interpreters have been around for quite a while now - the first generation was APL in 1963. S started in the late 60's, and this is well to keep in mind - there is a _ton_ of experience in these communities, especially with crystalized rigid language design as in APL and the amoebic evolution of S. What most programmers literate in scientific interpreters will do when they come to Numerical Python is: 1. Nice fast second generation interpreter, fine object orientation, good extensibility, decent embeddability. Object orientation OK. As usual, TK is integrated. 2. Graphics not tightly integrated - Yet. 3. Pressure of users will probably revise unfortunate limitations on indexing, eliminating the need for some of the functions - (take is an example). 4. Hope they don't fall for the GC nonsense. (Been there, done that with APL and S). Doubt they understand VM yet, but maybe the Linux community will spread that gospel. 5. Threading is an unresolved issue - but it doesn't seem to get in the way of using multithreaded libraries (e.g. Sun Performance Library, etc.). 6. Default argument processing is consistent, interesting, but also annoying. However, small idioms suffice to produce the desired behaviors, so no problem here. 7. Event processing probably will turn out OK, but a custom main loop for fast response time is probably necessary. Looks like some people already are working on this. 8. Everything is a reference - and the reference counting is done well. This helps reduce memory waste of large objects by increasing it for small objects. Probably a loser for memory bandwidth, but that can be treated easily for large objects. 9. Installation is peculiar, but this is easily remedied, too. 10. IT'S FREE, COMES WITH SOURCE, AND _NO LICENSE MANAGER_! Give you a hint, Aaron. Numerical Python is almost identical to about 90% of the S programming language, is frequently tens or hundreds of times faster than the equivalent S code, costs nothing, and your sysadmin might even like it. Sure. Lots of S programmers will just walk away. But don't hold your breath. Later, Andrew Mullhaupt _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From hyoon@nyptsrv1.etsd.ml.com Tue Jul 1 00:17:10 1997 From: hyoon@nyptsrv1.etsd.ml.com (Hoon Yoon - IPT Quant) Date: Mon, 30 Jun 1997 19:17:10 -0400 Subject: [MATRIX-SIG] Newbie Q. Message-ID: <199706302317.TAA22517@ptq3.etsd> Hello, I am new to NumPy and there has to be a easy way to do this than I can think of. >>> a array([1, 2, 3]) >>> b array([2, 4, 6]) >>> c array([[1, 2, 3], [2, 4, 6]]) >>> concatenate((c,a)) # Here's my problem they are different row shape. Traceback (innermost last): File "", line 1, in ? File "/opt/gnu/lib/python1.4/NumPy/Numeric.py", line 58, in concatenate return multiarray.concatenate(a) ValueError: arrays must have same number of dimensions I have attempting to append (vertical concatenation) a row of 1xn to mxn Numerical arrary, but failing miserably since they are not the same dimention. I can use loop or flatten the mxn out and concatenate vertically and to reshape, but that sounds extremely silly. In my past in Gauss: d = c | a would have done the job. I must be doing something wrong, but can't figure out form the Tutorial. In addition, in Gauss Horizontal would be e = a~b Transposing c would have been t = c' dot product would be d = a .* b' This to me faster than writing transpose(c) or concatenate((a,b)1), etc... Is there shorter method in Python? I guess I could write a emacs key. Does anyone have one already? While I am at it. Does PyGist do Bar, pie, and combo graphs? Is there a good site with examples? Plus, does X-axis always have to be Numeric? Should I attempt to use something else for these simple graphs? Any help with above is greatly appreciated! Thank you much, Hoon, _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From amullhau@ix.netcom.com Tue Jul 1 01:31:04 1997 From: amullhau@ix.netcom.com (Andrew P. Mullhaupt) Date: Mon, 30 Jun 1997 20:31:04 -0400 Subject: [MATRIX-SIG] reverse of take? In-Reply-To: <199706301843.OAA03287@dante.mh.lucent.com> Message-ID: <3.0.1.32.19970630203104.0069db7c@popd.netcruiser> At 02:45 PM 6/30/97 -0400, Aaron Watters wrote: > I propose that arraytype[arraytype] automatically be interpreted >as (1) gather as a value (2) scatter as an l-value. I apologize if >this is obviously dumb, incompatible with existing usage, or >already proposed (I'm new, remember, don't hurt me!) :). This proposal is OK with me and is essentially what a lot of people wish Python did - and not just the numerical arrays. (No matter though- the numerical arrays are probably general enough). However, there are some variations that make sense and ought to be considered. Can you index a multi-dimensional array with a one-dimensional array? In other words does arraytype[range(10)] refer to the first 10 elements of the array? I think this would be doable using other more verbose methods, but this is a simple way to spell this. Can you index a one-dimensional array with a multi-dimensional array? In languages which provide this, this usually returns a value with the same shape as the matrix and the values corresponding to the value in the vector with that index. This is really nice for image processing - for example palette remapping. You can get this by applying a function, but that is usually a lot slower. What about in between? I don't know of a language that permits these kinds of shenanigins so I assume we can drop this possibility - undoubtedly now someone will point out some language where this is a powerful tool... If array0 is d-dimensional and arrayk is 1-dimensional for 1<=k<=d, does array0[array1, ..., arrayd] refer to the elements corresponding to the Cartesian product of array1 ... arrayd? It should. What is the shape of the result when some of the index arrays have length 1? Should there be an additional argument to control this? (In S, x[1,1:5] is 1-dimensional with length 5 and x[1,1:5,drop=F] is 2-dimensional with shape (1, 5). - The reason you probably want this is that it can be really annoying to reshape the result which has been collapsed if it was an intermediate expression.) There are some other approaches to multidimensional indexing and it does not do to pick one without some consideration of the edge cases, etc.. Presumably the way to go is to make it possible to play with versions of this so a concensus can emerge. This looks quite feasible in Python. By the way, once you have powerful indexing, you will not need all the functions you used to; but there are at least two which will emerge as really useful now although only marginally useful previously. 1. A function which returns the permutation which would put the elements of a 1-d array in order. In APL this is called 'upgrade' and in S it is called 'order'. A quick illustration of the use of this is computing the ranks of elements in a vector - I'll suppose we call the function order. def rank(x) : return order(order(x)) 2. A function which returns the indices into a table where given keys are located. In APL this is called 'dyadic iota' and in S it is called 'match'. Strictly speaking you can implement this in terms of the order function described above - an O((n+m)log(n+m)) solution for searching m keys in a table of length n - but usually dyadic iota is implemented in terms of hashing which is much faster. Note that you want to locate stuff sprinkled around in arrays with this - it isn't clear to me at this point that Python dicts do/do not provide an efficient, perhaps more pythonistic alternative. Later, Andrew Mullhaupt _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From amullhau@ix.netcom.com Tue Jul 1 01:46:02 1997 From: amullhau@ix.netcom.com (Andrew P. Mullhaupt) Date: Mon, 30 Jun 1997 20:46:02 -0400 Subject: [MATRIX-SIG] Newbie Q. In-Reply-To: <199706302317.TAA22517@ptq3.etsd> Message-ID: <3.0.1.32.19970630204602.006d8dec@popd.netcruiser> At 07:17 PM 6/30/97 -0400, Hoon Yoon - IPT Quant wrote: >Hello, > I am new to NumPy and there has to be a easy way to do this than I can think of. >>>> a >array([1, 2, 3]) >>>> b >array([2, 4, 6]) >>>> c >array([[1, 2, 3], > [2, 4, 6]]) >>>> concatenate((c,a)) This works for me: concatenate([c, reshape(a, [-1, len(a)])]) The point is that the a that you have has only one dimension. Some languages will handle this for you, but NumPy doesn't. >In addition, in Gauss >Horizontal would be e = a~b >Transposing c would have been >t = c' >dot product would be >d = a .* b' >This to me faster than writing transpose(c) or concatenate((a,b)1), etc... >Is there shorter method in Python? I guess I could write a emacs key. Does >anyone have one already? A lot of different scientific interpreters are around. In Matlab, S, APL, Maple, etc. all these functions have different spellings, and they all look like nonsense to people coming from the other languages - quick - what would %o% mean? Later, Andrew Mullhaupt _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From hinsen@ibs.ibs.fr Tue Jul 1 09:25:02 1997 From: hinsen@ibs.ibs.fr (Konrad Hinsen) Date: Tue, 1 Jul 1997 10:25:02 +0200 Subject: [MATRIX-SIG] Newbie Q. In-Reply-To: <199706302317.TAA22517@ptq3.etsd> (hyoon@nyptsrv1.etsd.ml.com) Message-ID: <199707010825.KAA31356@lmspc1.ibs.fr> > I am new to NumPy and there has to be a easy way to do this than I can think of. > >>> a > array([1, 2, 3]) > >>> b > array([2, 4, 6]) > >>> c > array([[1, 2, 3], > [2, 4, 6]]) > >>> concatenate((c,a)) # Here's my problem they are different row shape. > Traceback (innermost last): > File "", line 1, in ? > File "/opt/gnu/lib/python1.4/NumPy/Numeric.py", line 58, in concatenate > return multiarray.concatenate(a) > ValueError: arrays must have same number of dimensions You must add the missing dimension to a before concatenating. The easiest solution is: >>> concatenate((c, a[NewAxis,:])) array([[1, 2, 3], [2, 4, 6], [1, 2, 3]]) > but that sounds extremely silly. In my past in Gauss: > d = c | a > would have done the job. I don't know much about Gauss. Does it treat all arrays as two-dimensional, like Matlab? In that case there is never a dimension mismatch. > In addition, in Gauss > Horizontal would be e = a~b > Transposing c would have been > t = c' > dot product would be > d = a .* b' > This to me faster than writing transpose(c) or concatenate((a,b)1), etc... Gauss is a special array language, whereas Python is a general-purpose language with arrays. If Python had a short notation for all its domains of application, it would be a complete mess. So it follows the "spell-everything-out" approach (like Mathematica). > Is there shorter method in Python? I guess I could write a emacs key. Does > anyone have one already? Functions are objects in Python, so you can create shorthands by assignment: TR = transpose CC = concatenate A = array CC(TR(a), A([2, 3, 4])) You can then put all your favourite abbreviations into a module and import from it. But don't expect everyone else to understand your private notation! -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28 Institut de Biologie Structurale | Fax: +33-4.76.88.54.94 41, av. des Martyrs | Deutsch/Esperanto/English/ 38027 Grenoble Cedex 1, France | Nederlands/Francais ------------------------------------------------------------------------------- _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From hinsen@ibs.ibs.fr Tue Jul 1 09:58:43 1997 From: hinsen@ibs.ibs.fr (Konrad Hinsen) Date: Tue, 1 Jul 1997 10:58:43 +0200 Subject: [MATRIX-SIG] reverse of take? In-Reply-To: <199706301843.OAA03287@dante.mh.lucent.com> (arw@dante.mh.lucent.com) Message-ID: <199707010858.KAA31442@lmspc1.ibs.fr> > I propose that arraytype[arraytype] automatically be interpreted > as (1) gather as a value (2) scatter as an l-value. I apologize if > this is obviously dumb, incompatible with existing usage, or > already proposed (I'm new, remember, don't hurt me!) :). Probably it has already been proposed, since anything imaginable has already been proposed. ;-) Let's make this a bit more specific: array[sequence [, sequence [, sequence ...]] would be the most general form, with any sequence of integers allowed. But if we allow tuples, we have to decide on the interpretation of array[(2, 3)] For compatibility this will have to remain equivalent to array[2, 3]. (In fact we have little choice there; this equivalence is enforced by the interpreter.) So allowing tuples creates trouble. On the other hand, not allowing tuples creates a very unpythonic restriction. So there's the first problem to be resolved. Second problem: do we allow arrays of rank > 1 as indices? If yes, what are the exact semantics? One interpretation would make a[b] an array of the same shape as b with a[b][i, j, k,...] = a[b[i, j, k,...]]. This gets a bit complicated when a has rank > 1, but it's still doable and probably useful. Third problem: an index sequence can contain the same index more than once. Fine for extracting data from an array, but what is supposed to happen if this occurs on the left-hand side? For example, what is a[[0, 1, 0]] = [1, 2, 3] supposed to do? -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28 Institut de Biologie Structurale | Fax: +33-4.76.88.54.94 41, av. des Martyrs | Deutsch/Esperanto/English/ 38027 Grenoble Cedex 1, France | Nederlands/Francais ------------------------------------------------------------------------------- _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From amullhau@ix.netcom.com Tue Jul 1 10:52:46 1997 From: amullhau@ix.netcom.com (Andrew P. Mullhaupt) Date: Tue, 01 Jul 1997 05:52:46 -0400 Subject: [MATRIX-SIG] Newbie Q. References: <199707010825.KAA31356@lmspc1.ibs.fr> Message-ID: <33B8D36E.1988@ix.netcom.com> Konrad Hinsen wrote: > I don't know much about Gauss. Does it treat all arrays as > two-dimensional, like Matlab? In that case there is never a dimension > mismatch. Matlab no longer treats all arrays as two-dimensional. Later, Andrew Mullhaupt _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From P.S.Craig@durham.ac.uk Tue Jul 1 13:35:06 1997 From: P.S.Craig@durham.ac.uk (P.S.Craig@durham.ac.uk) Date: Tue, 1 Jul 1997 13:35:06 +0100 (BST) Subject: [MATRIX-SIG] reverse of take? In-Reply-To: <199707010858.KAA31442@lmspc1.ibs.fr> from "Konrad Hinsen" at Jul 1, 97 10:58:43 am Message-ID: <9478.199707011235@laplace> onrad Hinsen writes: > > > I propose that arraytype[arraytype] automatically be interpreted > > as (1) gather as a value (2) scatter as an l-value. I apologize if > > this is obviously dumb, incompatible with existing usage, or > > already proposed (I'm new, remember, don't hurt me!) :). > > Probably it has already been proposed, since anything imaginable has > already been proposed. ;-) > > Let's make this a bit more specific: > > array[sequence [, sequence [, sequence ...]] > > would be the most general form, with any sequence of integers allowed. > But if we allow tuples, we have to decide on the interpretation of > > array[(2, 3)] > > For compatibility this will have to remain equivalent to array[2, 3]. > (In fact we have little choice there; this equivalence is enforced by > the interpreter.) So allowing tuples creates trouble. On the other > hand, not allowing tuples creates a very unpythonic restriction. So > there's the first problem to be resolved. I think this does expose a flaw in the way in which multiple indexing is handled in python. That we cannot distinguish a[(2,3)] from a[2,3] is unfortunate. However, the potential ambiguity only arises if we allow rank 1 arrays to be indices for arrays of higher rank. If we don't, I can't see a problem. This is a convenience which S (and S-PLus) allow. However, in Numerical Python we can write ravel(a)[b] which is not (efficiently) available in S. It should be OK as an lvalue since ravel(a) does not copy a's data but creates a new reference. If this is too cumbersome, surely we can find some shorthand for the ravelling operation. > > Second problem: do we allow arrays of rank > 1 as indices? If yes, > what are the exact semantics? One interpretation would make > > a[b] > > an array of the same shape as b with a[b][i, j, k,...] = a[b[i, j, k,...]]. > This gets a bit complicated when a has rank > 1, but it's still doable > and probably useful. How about this? Suppose that b.shape[-1]==len(a.shape) Then a[b].shape will be b.shape[:-1] and, for example, a[b][i,j,k] = a[b[i,j,k,:]] as you propose (of course the latter may be written a[b[i,j,k]] because ofthe way Numerical Python works. This extends what happens in S to rank(b)>2. I think it's also consistent with what I said earlier. I think it may even be possible to go beyond this to the situation where b.shape[-1] > Third problem: an index sequence can contain the same index more than > once. Fine for extracting data from an array, but what is supposed > to happen if this occurs on the left-hand side? For example, what is > > a[[0, 1, 0]] = [1, 2, 3] > > supposed to do? > -- In S-Plus, a[0] becomes 3. In other words, some form of left to right evaluation should be used? We would have to be very careful to avoid ambiguity. New problem: Can we allow an expression such as a[b][c] = d In other words, c is applied as an index to the array a[b]. I think that any situation like this can be written as a[bc] = d where bc is suitable determined from b, c and the shape of a. The idea is to avoid computation of a[b] which is likely to require copying of data. At this point, I'm willing to volunteer to do some of the work involved in making all this happen as I wish to use python as the base for developing ideas that go beyond what we have in S and S-Plus. Anyone got any suggestions as to how we should organise things? Peter Craig _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From hinsen@ibs.ibs.fr Tue Jul 1 14:07:10 1997 From: hinsen@ibs.ibs.fr (Konrad Hinsen) Date: Tue, 1 Jul 1997 15:07:10 +0200 Subject: [MATRIX-SIG] Data filtering Message-ID: <199707011307.PAA32490@lmspc1.ibs.fr> I'll soon need to do some data filtering, and before writing my own I'd like to ask whether someone has already done it (in Python, of course). What I need is a low-pass filter, preferably one that can treat long data streams in smaller segments. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28 Institut de Biologie Structurale | Fax: +33-4.76.88.54.94 41, av. des Martyrs | Deutsch/Esperanto/English/ 38027 Grenoble Cedex 1, France | Nederlands/Francais ------------------------------------------------------------------------------- _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From hyoon@nyptsrv1.etsd.ml.com Tue Jul 1 14:06:29 1997 From: hyoon@nyptsrv1.etsd.ml.com (Hoon Yoon - IPT Quant) Date: Tue, 1 Jul 1997 09:06:29 -0400 Subject: [MATRIX-SIG] Re: Newbie Q. Message-ID: <199707011306.JAA22948@ptq3.etsd> To Konrad, Andrew, and others, Thank you much for help. As some of you suspected, Gauss is 2D only. The main reason for the usage is the fact that Gauss and Ox are speed demons, when compared to Splus and MatLab. For further info look at: http://www.siu.edu/~econ/matlab.ps http://www.ku-eichstaett.de/docs/WWF/STA/download.htm These are papers written by a Professor interested in Matrix Langs. I confirmed that Splus at least is about 10 to 30 times slower than Gauss in most operations. Never did it with Mat Lab, however, according to the papers, don't bother trying. Splus and Mat Lab is easier and has prettier graphics than Gauss or Ox. Splus in particular has column headers associated with it's matrix, which is shadowed. The headers stays off calcs, but available for choosing (like dictionary indexes). This one feature I wish Python had. Well, at least, Python will let me do this, if I write the functions on top of math functions. FYI: OX is multidimentional and object oriented, but not Gauss (2D). In some tests Gauss and Ox will even outperform, decently optimized C code. Ox for DOS & Unix is free for academics and both of them are built for Econometrics. I do not consider them in a same catagory as Python, since they have no GUI and no generalities. Gauss's graphics are better than Python's in some ways right now. I would love to see somekind of wrapper for either in Python, since both of them will be comming out with a DLL for Win and Unix soon. ************** One additional Q. What is the best general purpose graphics addition in Python. I am talking about bar/ histogram/ line/ scatter/ pie/ & may be even 3D surface kind of thing. PyGist is very good, I love it, but I had some problem dealing with the kind of minimal business graphics I had above. It will do line/ scatter/ 3D contour better than things I could buy, but not for simple Bar or Stacked Bar. Not to mention I could not use strings as x labels. Also, BLT extention to Tkinter seems to not quite kept up with Tkinter changes. My SA had problem installing it. Thanks for the info! Hoon, _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From arw@dante.mh.lucent.com Tue Jul 1 15:27:27 1997 From: arw@dante.mh.lucent.com (Aaron Watters) Date: Tue, 1 Jul 1997 10:27:27 -0400 Subject: [MATRIX-SIG] reverse of take? Message-ID: <199707011424.KAA03491@dante.mh.lucent.com> > From: Andrew P. Mullhaupt > At 02:45 PM 6/30/97 -0400, Aaron Watters wrote: > > > I propose that arraytype[arraytype] automatically be interpreted > >as (1) gather as a value (2) scatter as an l-value. I apologize if > >this is obviously dumb, incompatible with existing usage, or > >already proposed (I'm new, remember, don't hurt me!) :). > > This proposal is OK with me and is essentially what a lot of people > wish Python did - and not just the numerical arrays. (No matter though- > the numerical arrays are probably general enough). In other contexts (dictionaries at least) it makes sense that indexing always has the same single-mapping interpretation. For example in kjbuckets Dict1[Set1] retrieves the map for Set1 from Dict1 (ie, Set1 is a left member of Dict1). The alternate interpretation "all maps for elts of Set1 from Dict1" can be got from a different operation, composition: (Set1 * Dict1).values() I wouldn't have it any other way :). But for arrays it seems that Array1[Array2] naturally interprets the rows of Array2 as indices of Array1 and returns the elements of Array1 (which might be arrays too, I suppose) for those indices. I suppose List[Tuple] might possibly return the elements of List at members of Tuple but I suspect that this is the kind of sneakiness that Guido would prefer to avoid in the core. High performance specialized extensions such as Numeric can afford greater sneakiness tho for important operations. >From these discussions, the biggest technical problems I see for scatter/gather-as-indexing are A[(2,3)] = x is ambiguous, as Konrad points out. I say just make tuples a special incompatible case... maybe only support the scatter/gather things for arrays as indices... oh well. Also Peter Craig notes a problem with: A[A2][A3] = x [Note: to python this parses as value A[A2] indexed assigned at A3 to x. The intuitive reading is change A at indexes A2 reindexed at indices A3 to x.] In this notation the updates should propagate all the way back to A as Peter Craig points out. This is a Big Problem AFAIK, since I'm not sure how it could be resolved (not understanding the internal representation). This suggests to me that maybe having the existing "take" function and a dual "set" function might be a better idea once again instead of jumping all the way to generalized indices. If a "scatter/gather" index notation cannot be made fully general I suspect it would cause a lot of confusion. I do think that it is important to have a dual for "take" however it looks notationally. -- Aaron Watters _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From motteler@laura.llnl.gov Tue Jul 1 15:41:19 1997 From: motteler@laura.llnl.gov (Zane C. Motteler) Date: Tue, 1 Jul 1997 07:41:19 -0700 (PDT) Subject: [MATRIX-SIG] reverse of take? In-Reply-To: <3.0.1.32.19970630203104.0069db7c@popd.netcruiser> Message-ID: On Mon, 30 Jun 1997, Andrew P. Mullhaupt wrote: > >Can you index a multi-dimensional array with a one-dimensional array? In >other words does arraytype[range(10)] refer to the first 10 elements of >the array? I think this would be doable using other more verbose methods, >but this is a simple way to spell this. > >Can you index a one-dimensional array with a multi-dimensional array? In >languages which provide this, this usually returns a value with the same >shape as the matrix and the values corresponding to the value in the vector >with that index. This is really nice for image processing - for example >palette remapping. You can get this by applying a function, but that is >usually a lot slower. Both cases are ambiguous in NumPy. NumPy arrays are not necessarily stored contiguously (transpose merely exchanges some dimensions and strides, but doesn't move any elements). This is obviously done for speed. Secondly, to do these two things, even if arrays are contiguous, means that users have to know whether arrays are stored in row-major or column-major order, which is antithetical to the idea of a high-level language. > >If array0 is d-dimensional and arrayk is 1-dimensional for 1<=k<=d, does >array0[array1, ..., arrayd] refer to the elements corresponding to the >Cartesian product of array1 ... arrayd? It should. What is the shape >of the result when some of the index arrays have length 1? Should there be >an additional argument to control this? (In S, x[1,1:5] is 1-dimensional with >length 5 and x[1,1:5,drop=F] is 2-dimensional with shape (1, 5). - The reason >you probably want this is that it can be really annoying to reshape the >result which has been collapsed if it was an intermediate expression.) Getting too esoteric here. I'm not even too sure how useful this would be. It really demands intimate knowledge of the internal structure of arrays. > >1. A function which returns the permutation which would put the elements of > a 1-d array in order. In APL this is called 'upgrade' and in S it is called > 'order'. A quick illustration of the use of this is computing the ranks > of elements in a vector - I'll suppose we call the function order. > > def rank(x) : > return order(order(x)) My arrayfnsmodule has such a function, called index_sort. Zane ----------------------------------------------------------------------------- Zane C. Motteler, Ph. D. ___________ALSO:_________________ Computer Scientist | Professor Emeritus of Computer| Lawrence Livermore National Laboratory | Science and Engineering | P O Box 808, L-472 | California Polytechnic State | Livermore, CA 94551-9900 | University, San Luis Obispo | 510/423-2143 --------------------------------- FAX 510/423-9969 zcm@llnl.gov or motteler@laura.llnl.gov or zmottel@calpoly.edu _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From hinsen@ibs.ibs.fr Tue Jul 1 16:02:31 1997 From: hinsen@ibs.ibs.fr (Konrad Hinsen) Date: Tue, 1 Jul 1997 17:02:31 +0200 Subject: [MATRIX-SIG] reverse of take? In-Reply-To: <3.0.1.32.19970630203104.0069db7c@popd.netcruiser> (amullhau@ix.netcom.com) Message-ID: <199707011502.RAA00029@lmspc1.ibs.fr> > Can you index a multi-dimensional array with a one-dimensional array? In > other words does arraytype[range(10)] refer to the first 10 elements of > the array? I think this would be doable using other more verbose methods, > but this is a simple way to spell this. This issue is already settled. Obviously, array[range(10)] should be the same as array[:10], right? And that expression is equivalent to array[:10, ...]. To get the first ten elements, use ravel(array)[:10]. > Can you index a one-dimensional array with a multi-dimensional array? In > languages which provide this, this usually returns a value with the same > shape as the matrix and the values corresponding to the value in the vector > with that index. This is really nice for image processing - for example > palette remapping. You can get this by applying a function, but that is > usually a lot slower. I see no reason why this shouldn't be allowed in NumPy. > 1. A function which returns the permutation which would put the elements of > a 1-d array in order. In APL this is called 'upgrade' and in S it is called > 'order'. A quick illustration of the use of this is computing the ranks And in NumPy it's called argsort! -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28 Institut de Biologie Structurale | Fax: +33-4.76.88.54.94 41, av. des Martyrs | Deutsch/Esperanto/English/ 38027 Grenoble Cedex 1, France | Nederlands/Francais ------------------------------------------------------------------------------- _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From hinsen@ibs.ibs.fr Tue Jul 1 16:18:18 1997 From: hinsen@ibs.ibs.fr (Konrad Hinsen) Date: Tue, 1 Jul 1997 17:18:18 +0200 Subject: [MATRIX-SIG] reverse of take? In-Reply-To: <199707011424.KAA03491@dante.mh.lucent.com> (arw@dante.mh.lucent.com) Message-ID: <199707011518.RAA00072@lmspc1.ibs.fr> > Also Peter Craig notes a problem with: > > A[A2][A3] = x > > [Note: to python this parses as value A[A2] indexed assigned at A3 > to x. The intuitive reading is change A at indexes A2 reindexed at > indices A3 to x.] > > In this notation the updates should propagate all the way back to A > as Peter Craig points out. This is a Big Problem AFAIK, since I'm They do with the current NumPy indexing, which returns not a copy of the array, but a reference to the same data space with different pointers and offsets. But that would be hard to keep up with general list indices! Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28 Institut de Biologie Structurale | Fax: +33-4.76.88.54.94 41, av. des Martyrs | Deutsch/Esperanto/English/ 38027 Grenoble Cedex 1, France | Nederlands/Francais ------------------------------------------------------------------------------- _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From hinsen@ibs.ibs.fr Tue Jul 1 16:15:46 1997 From: hinsen@ibs.ibs.fr (Konrad Hinsen) Date: Tue, 1 Jul 1997 17:15:46 +0200 Subject: [MATRIX-SIG] reverse of take? In-Reply-To: <9478.199707011235@laplace> (P.S.Craig@durham.ac.uk) Message-ID: <199707011515.RAA00060@lmspc1.ibs.fr> > I think this does expose a flaw in the way in which multiple indexing > is handled in python. That we cannot distinguish a[(2,3)] from a[2,3] Right, but there wasn't much choice. To keep the existing interface to both classes and C types, an index had to be limited to a single Python object. And using tuples for multiple indexing is also useful for code that has to build an index expression from pieces. > is unfortunate. However, the potential ambiguity only arises if we > allow rank 1 arrays to be indices for arrays of higher rank. If we That is already allowed and well defined! However, it's not what you seem to think it is. The definition is that the index is applied to the first dimension of the array. This makes arrays (of any dimension) conform to the sequence protocol, and makes them very interchangeable with nested lists. > In fact, we might be able to go yet further to, for example, > a[:,b]. This would have shape the concatentaion of a.shape[0], > b.shape[:-1] and a.shape[1+b.shape[-1]:] and a[:,b][i,j,k] would be > the same as a[i][b[j,k]] And be the first step to the perlization (or J-ization) of NumPy: combinations of funny symbols having a meaning not deducible from the meanings of their pieces. Besides, Guido will probably refuse any syntax change of that kind! > In S-Plus, a[0] becomes 3. In other words, some form of left to right > evaluation should be used? We would have to be very careful to avoid > ambiguity. Especially for higher-dimensional cases! > New problem: Can we allow an expression such as > > a[b][c] = d Syntactically yes. > S-Plus. Anyone got any suggestions as to how we should organise > things? First we have to agree on what we want, and that could be difficult... Then do an implementation in a separate module, with function syntax, just for testing. In parallel, check with Jim (are you following all this?) about the status of the NumPy source code - no point in modifying code that is still changing. Finally, put it all together and be happy... -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28 Institut de Biologie Structurale | Fax: +33-4.76.88.54.94 41, av. des Martyrs | Deutsch/Esperanto/English/ 38027 Grenoble Cedex 1, France | Nederlands/Francais ------------------------------------------------------------------------------- _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From P.S.Craig@durham.ac.uk Tue Jul 1 16:48:34 1997 From: P.S.Craig@durham.ac.uk (P.S.Craig@durham.ac.uk) Date: Tue, 1 Jul 1997 16:48:34 +0100 (BST) Subject: [MATRIX-SIG] reverse of take? In-Reply-To: <199707011515.RAA00060@lmspc1.ibs.fr> from "Konrad Hinsen" at Jul 1, 97 05:15:46 pm Message-ID: <9838.199707011549@laplace> Konrad Hinsen writes: > From amullhau@ix.netcom.com Tue Jul 1 18:14:46 1997 From: amullhau@ix.netcom.com (Andrew P. Mullhaupt) Date: Tue, 01 Jul 1997 13:14:46 -0400 Subject: [MATRIX-SIG] Re: Newbie Q. In-Reply-To: <199707011306.JAA22948@ptq3.etsd> Message-ID: <3.0.1.32.19970701131446.0069cf34@popd.netcruiser> At 09:06 AM 7/1/97 -0400, Hoon Yoon - IPT Quant wrote: >To Konrad, Andrew, and others, > > Splus in particular has column headers associated > with it's matrix, which is shadowed. I'm not sure what you mean by shadowed. But the mechanism by which S-plus has 'column headers' is to add an attribute called dimnames to the array. This attribute is a list containing a character vector for each dimension. It would be very easy to obtain this functionality in python by subclassing arrays. So you'll probably see this one soon. The only issue is whether we allow you to use the dimnames for indexing, as in S. Later, Andrew Mulhaupt _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From amullhau@ix.netcom.com Tue Jul 1 19:20:56 1997 From: amullhau@ix.netcom.com (Andrew P. Mullhaupt) Date: Tue, 01 Jul 1997 14:20:56 -0400 Subject: [MATRIX-SIG] reverse of take? In-Reply-To: <199707011502.RAA00029@lmspc1.ibs.fr> References: <3.0.1.32.19970630203104.0069db7c@popd.netcruiser> Message-ID: <3.0.1.32.19970701142056.006dab44@popd.netcruiser> At 05:02 PM 7/1/97 +0200, Konrad Hinsen wrote: >> Can you index a multi-dimensional array with a one-dimensional array? In >> other words does arraytype[range(10)] refer to the first 10 elements of >> the array? I think this would be doable using other more verbose methods, >> but this is a simple way to spell this. > >This issue is already settled. Obviously, array[range(10)] should be >the same as array[:10], right? And that expression is equivalent to >array[:10, ...]. I know everyone wants this, I just didn't see it in Aaron's proposal, so I asked. Since he's come recently. >> Can you index a one-dimensional array with a multi-dimensional array? > >I see no reason why this shouldn't be allowed in NumPy. Me either. But I didn't see it in Aaron's proposal, so I asked. >> 1. A function which returns the permutation which would put the elements of >> a 1-d array in order. In APL this is called 'upgrade' and in S it is called >> 'order'. A quick illustration of the use of this is computing the ranks > >And in NumPy it's called argsort! But wait - what about index_sort? :-) It turns out that this function is quite important, and so is his friend "dyadic iota". The sort for argsort/index_sort usually has to be really good in case of many duplicated values, as well as reversed values, etc. Later, Andrew Mullhaupt _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From P.S.Craig@durham.ac.uk Tue Jul 1 23:56:47 1997 From: P.S.Craig@durham.ac.uk (P.S.Craig@durham.ac.uk) Date: Tue, 1 Jul 1997 23:56:47 +0100 (BST) Subject: [MATRIX-SIG] A proposal (LONG) - "reverse of take" revisited Message-ID: <10497.199707012256@laplace> Hi all, I have decided to stick my neck out and try to bring a little order to the discussion about extending NumPy's indexing that has been going on over the last week or so. There are clearly a number of schools of thought about what should be done, ranging from Aaron Watter's minimal suggestion that we have a put function corresponding to the existing take function to Andrew Mullhaupt's and my desires for fairly exotic indexing semantics based on ideas found in S and APL. I would like to suggest the following program of development: a) implement a give function which is the converse of the current take function. It's still not entirely clear what this means. I would tentatively propose: give(a, b, seq, axis=0) where a and b are NumPy arrays and seq is an integer value sequence such that take(a, seq, axis) is an array of the same shape as b. Formally, having checked the arguments, the function would do the equivalent of axisoffset = (slice(None,None,None),)*axis for i in range(len(seq)): a[axisoffset+(seq[i],)] = b[axisoffset+(i,)] This has the nicely symmetric property that having done give(a,b,seq,axis) then b and take(a, seq, axis) are arrays of the same shape with the same contents. I wonder how much efficiency gain there would be with a C implementation? b) For most, if not all, of the indexing ideas that have been thrown about, we want to be able to efficiently index more than one axis simultaneously. So we appear to want versions of take and give that operate on more than one axis simultaneously. A natural extension to the current take function would be multitake(a, (seq1,seq2,...,seqm), axes=range(m)) where m is the number of axes being simultaneously indexed. The effect of assigning the result to b would be the same as b = a seqs = (seq1,seq2,...,seqm) for i in range(m): b = take(b, seqs[i], axes[i]) but could be much more efficiently implemented. multigive is more challenging to define. In the spirit of my proposal for the give function, this might be multigive(a, b, (seq1,seq2,...,seqm), axes=range(m)) where multitake(a,(seq1,seq2,...,seqm), axes) would be an array of the same shape as b c) The simplest extension to current NumPy indexing is to allow an arbitrary integer valued sequence as well as integers or slices as an index. Thus, for example, a[:,seq1,:,seq2,seq3] would be the same as multitake(a,(seq1,seq2,seq3),(1,3,4)) and a[:,seq1:,seq2,seq3] = b would be the same as multigive(a,b,(seq1,seq2,seq3),(1,3,4)) Note that, according to the proposal so far, a[seq1][seq2] = b would be syntactically correct but would fail to alter a as a[seq1] would have a copy of the apropriate part of a's data and would not be a reference to a's data. Mixing of slice and sequence indexing seems fairly straightforward to implement. d) It is interesting to note that if the take (and presumably also multitake) function returned a reference object instead of copying data to a new array, then the give (and multigive) function would be trivial to implement. At present,indexing in NumPy using slices is done by reference and the resulting object is a highly efficient array implementation because the memory location for a[i1,i2,...,ij ,...,in] is the memory location for a[i1,i2,...,0,...,in] + ij *sj where sj is the stride length for axis j. In other words, the reference object is no different from a directly created array except for the value of sj. The difficulty with a reference implementation for indexing by arbitrary sequences is that the resulting array will be less efficient than the basic NumPy array. The implementation is fairly straightforward and would have the advantage that a statement such as a[[1,3,4]][2]=1 would actually have the desired effect of changing a[4]. The other difficulty with a reference implementation for take is that the default behaviour would have to remain copying. Otherwise, existing code could well break. Actually, once the basics of a reference implementation of take have been sorted, multitake is a trivial function to write efficiently by repeated use of take. e) More exotic indexing ideas, such as those found in S and advocated by Andrew Mullhaupt and me, would be the subject of individual or collective experimentation using python classes for the time being. I think that the ideas in a)-c) and possibly also d) would provide a good basis for playing with more powerful index notation. If a concensus emerged, we could then take the process further. Now, some of this is fairly straightforward. A start on code for a) is Zane Mosteller's array_set function. However, this probably needs some work. b) can either be tackled directly once a) has been completed or can be tackled by doing d). c) is trivial once b) is done. To my mind, the big question is whether we think d) is a good idea. If it is, it really means having two different array objects, one of which is Jim Hugunin's current object and the other of which implements a more general reference array type. The latter is not difficult to implement, I think. The problem is that the two implementations cannot really be separated if we want efficiency. I don't think we can implement as a separate module. Unfortunately, my impression from Jim H's last message was that he does not have any time to give to NumPy for the time being. So we would have to make changes to NumPy code and hope that we could keep the changes up to date if NumPy evolved. Of course, if we could get the changes accepted into NumPy then all would be well. I'm sorry to have been so long-winded. If anyone is still reading at this point and is still interested, please comment. Otherwise, I might just go ahead by myself anyway sometime in the next couple of months. Exhaustedly yours, Peter Craig #--------------------------------------------------------------------# | E-mail: P.S.Craig@durham.ac.uk Telephone: +44-91-3742376 (Work) | | Fax: +44-91-3747388 +44-91-3860448 (Home) | | | | WWW: http://fourier.dur.ac.uk:8000/stats/psc.html | | | | Snail: Peter Craig, Dept. of Math. Sciences, Univ. of Durham, | | South Road, Durham DH1 3LE, England | #--------------------------------------------------------------------# _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From arw@dante.mh.lucent.com Tue Jul 1 20:32:39 1997 From: arw@dante.mh.lucent.com (Aaron Watters) Date: Tue, 1 Jul 1997 15:32:39 -0400 Subject: [MATRIX-SIG] reverse of take? Message-ID: <199707011929.PAA06248@dante.mh.lucent.com> > a[i, j, k, hint=flavor_woo_woo_17] I'm reeling about not following this discussion, but I suspect the best approach is for Jim H. to "just do it". I personally begin to think that excess overloading of indexing is not advisable, possibly confusing. I vote for "take()" as is with dual "put()" or whatever. Of course, my experience with numeric matrix tools such as these is infinitesimal, so my opinion may be of limited merit. If we can get by without asking Guido to change the core, so much the better. -- Aaron Watters ps: in case you haven't guessed I want "put" in order to be able to do graphical type operations in some generality using Numeric alone -- this is all that's missing from what I can see. IE compute the indices of a 1d figure using umath operations and then use put() to draw on those indices into a Numeric array frame buffer. This, plus a screen mapping operation and mouse interfaces of some kind could serve as the basis of an simple windowing/graphics system based in Python, I believe. Of course it'd be nontrivial, but after a good start I think the Python community (via starship, maybe) could grow some neat stuff... fonts == arrays of bytes cropping == slicing colormap == choose map_to_screen == slice assignment... just a daydream... _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From phil@geog.ubc.ca Wed Jul 2 02:54:04 1997 From: phil@geog.ubc.ca (Phil Austin) Date: Tue, 1 Jul 1997 18:54:04 -0700 Subject: [MATRIX-SIG] A proposal (LONG) - "reverse of take" revisited In-Reply-To: <10497.199707012256@laplace> References: <10497.199707012256@laplace> Message-ID: <199707020154.SAA22149@curlew.geog.ubc.ca> Aaron> I'm reeling about not following this discussion, but I Aaron> suspect the best approach is for Jim H. to "just do it". I Aaron> personally begin to think that excess overloading of Aaron> indexing is not advisable, possibly confusing. I vote for Aaron> "take()" as is with dual "put()" or whatever. Of course, Aaron> my experience with numeric matrix tools such as these is Aaron> infinitesimal, so my opinion may be of limited merit. If Aaron> we can get by without asking Guido to change the core, so Aaron> much the better. -- Aaron Watters Here's the perspective of someone who intends to teach NumPy to 3rd year geography majors without much exposure to computers: I would rather have the S syntax describe by PSC and APM than the current numeric functions. Like Python, it takes a while to penetrate, but once it does, even naive computer users typically find it intuitive. Additional evidence is available in the S-news archives, where hundreds of real neophytes have asked questions of the old timers. A couple of array examples are typically enough to get the beginners started (and there are some good public domain S tutorials that could be pirated for this). Once the NumPy prototype code is available, the indexing should sell itself to members of this list and to the broader community. Phil _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From amullhau@ix.netcom.com Wed Jul 2 04:01:21 1997 From: amullhau@ix.netcom.com (Andrew P. Mullhaupt) Date: Tue, 01 Jul 1997 23:01:21 -0400 Subject: [MATRIX-SIG] A proposal (LONG) - "reverse of take" revisited In-Reply-To: <199707020154.SAA22149@curlew.geog.ubc.ca> References: <10497.199707012256@laplace> <10497.199707012256@laplace> Message-ID: <3.0.1.32.19970701230121.006a3bac@popd.netcruiser> At 06:54 PM 7/1/97 -0700, Phil Austin wrote: > >Here's the perspective of someone who intends to teach NumPy to 3rd >year geography majors without much exposure to computers: I would > rather have the S syntax describe by PSC and APM than the >current numeric functions. Especially since a lot of GIS style stuff is easier to do that way, right? > Once the NumPy prototype code is >available, the indexing should sell itself to members of this list and >to the broader community. There are two other things beyond upgraded indexing which are needed before NumPy will take over the numerical world. 1. A decent graphics package should be glued on pretty tightly. 2. Installation must be simplified. Then, I would agree with you - NumPy will be very, very popular. Later, Andrew Mullhaupt _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From amullhau@ix.netcom.com Tue Jul 1 19:15:35 1997 From: amullhau@ix.netcom.com (Andrew P. Mullhaupt) Date: Tue, 01 Jul 1997 14:15:35 -0400 Subject: [MATRIX-SIG] reverse of take? In-Reply-To: References: <3.0.1.32.19970630203104.0069db7c@popd.netcruiser> Message-ID: <3.0.1.32.19970701141535.006d53f8@popd.netcruiser> At 07:41 AM 7/1/97 -0700, Zane C. Motteler wrote: >On Mon, 30 Jun 1997, Andrew P. Mullhaupt wrote: > >> >>Can you index a multi-dimensional array with a one-dimensional array? >> >>Can you index a one-dimensional array with a multi-dimensional array? > >Both cases are ambiguous in NumPy. NumPy arrays are not necessarily >stored contiguously (transpose merely exchanges some dimensions and >strides, but doesn't move any elements). They shouldn't be ambiguous at all. And if you're thinking of what I think you're thinking of - well, I don't think anyone would want _that_ at all. I think you interpret the indices as referring to the _locations at which the elements are stored_ and this is not always true. The way to sensibly index a multi dimensional array with a one-dimensional array is _not_ to interpret the indices in terms of the possibly unpredictable order the elements are stored in, but to treat the one-dimensional array as indices into the 'full dimensional' indices of the array as determined by the shape of the array. Thus, it is completely unambiguous, independent of the order in which the actual elements are laid out, and it also corresponds to what most users would expect. The other case, indexing a one-dimensional array by a multi-dimensional array is also completely unambiguous - you are using the elements of the multi-dimensional array as indices into the one-d array and then giving the result the shape of the original m-d array. Similar to the previous case, this does not depend on the order in which the elements of the one-d array are actually stored. > Secondly, to do these two things, even if arrays are contiguous, >means that users have to know whether arrays are stored in row-major or >column-major order, which is antithetical to the idea of a high-level >language. Actually, it doesn't mean that at all. It is quite clear that you can construct a mathematical theory of APL indexing, which supports these constructs without any reference to row or column majority. (I've alluded to this mathematical construction before.) In fact, as I have pointed out above, the _indexing_ can be (and in some cases in python _needs_ to be) independent of the storage order. However, for reasons of speed, whether you tell them or not, people will find out many things about how the elements of arrays are stored. The APL implementation community tried very hard for a long time to hide this from users to no avail. How, for example, will a user be able to pass an array to his favorite multi-dimensional graphics package, without he can put the elements in the right places for that package? It is quite simple to see that users _need_ be able to find out in any particular case, as necessary, the order of the elements. The goal of the higher level language is to reduce this need to a minimum, _but no further_. >> >>If array0 is d-dimensional and arrayk is 1-dimensional for 1<=k<=d, does >>array0[array1, ..., arrayd] refer to the elements corresponding to the >>Cartesian product of array1 ... arrayd? It should. What is the shape >>of the result when some of the index arrays have length 1?expression.) > >Getting too esoteric here. I'm not even too sure how useful this would >be. It really demands intimate knowledge of the internal structure of >arrays. I think you're confused about the meaning of this based on your previous statements. As to useful, it is perhaps the single most common case of indexing in array languages, and not only that, perhaps the _most efficient_. Here's an example: a = reshape(range(12), [3,4]) which looks like 0 1 2 3 4 5 6 7 8 9 10 11 then you want a[[0,2], [2,3]] to be 2 3 10 11 right? I think a lot of people have been thinking that this would be the case (like APL, S, Matlab, etc.). The question here is actually not _can_ you do this, (I can easily give a good implementation of what I outlined so far). But we are not at that point yet. The indexing questions I have asked are all about determining what behavior people want. There are many consistent behaviors which people coming from different languages are already happy with, and the trick here is to figure out which one people want. There are _no_ issues of "can't get there from here" in the context of Python's particular legacy choices - the questions are all in the "do we _want_ to get there from here" sort. There is no question that indexing will get improved in the way people use Numerical Python - that is not the issue either. Too many people have done too much cool stuff (for decades) for that to be a serious issue. The question is not _how these things can be done_. Almost everything that anyone has suggested so far falls well within the scope of things that I know can be done quite practically, and other people may know how to do many things I don't, so as far as we have seen to date, it's all quite doable. The serious issue is _which flavor to choose_. It will be a really good thing if the people in the discussion make themselves familiar with the various choices before trying to implement something serious - or else we will end up with several parallel efforts which conflict in some aspects. I can think of at least two completely reasonable ways to specify indexing which are not compatible. For example - APL's rigid respect of dimensions as opposed to S's "cyclic reuse" policy - There are advantages and disadvantages to both. I know which one I like better, but either would be fine for python. The reason that indexing is _different_ than other cases where the new functionality has "function" syntax is that _beacuse_ we call it "indexing" we are all talking about syntax which is going to overlap (read collide) heavily in different implementations. So it is likely that only _one_ approach can survive. It would be really nice if this choice isn't determined solely by the first version to be implemented. Later, Andrew Mullhaupt _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From amullhau@ix.netcom.com Tue Jul 1 19:33:55 1997 From: amullhau@ix.netcom.com (Andrew P. Mullhaupt) Date: Tue, 01 Jul 1997 14:33:55 -0400 Subject: [MATRIX-SIG] reverse of take? In-Reply-To: <199707011515.RAA00060@lmspc1.ibs.fr> References: <9478.199707011235@laplace> Message-ID: <3.0.1.32.19970701143355.006dacac@popd.netcruiser> At 05:15 PM 7/1/97 +0200, Konrad Hinsen wrote: > >> This would have shape the concatentaion of a.shape[0], >> b.shape[:-1] and a.shape[1+b.shape[-1]:] and a[:,b][i,j,k] would be >> the same as a[i][b[j,k]] > >And be the first step to the perlization (or J-ization) of NumPy: >combinations of funny symbols having a meaning not deducible from >the meanings of their pieces. Besides, Guido will probably refuse >any syntax change of that kind! Me too. I think a better approach to adding flavors/bells/whistles to indexing is by adding a default argument to the index, as in a[i, j, k, hint=flavor_woo_woo_17] for exceptional cases. This approach works well in S, which means that is doesn't make bees live in the heads of many naive users. >First we have to agree on what we want, and that could be difficult... Yes. This, believe it or not, is the hard part. >Then do an implementation in a separate module, with function syntax, >just for testing. In parallel, check with Jim (are you following all >this?) about the status of the NumPy source code - no point in >modifying code that is still changing. Finally, put it all together >and be happy... Absolutely. Later, Andrew Mullhaupt _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From edcjones@access.digex.net Wed Jul 2 03:27:42 1997 From: edcjones@access.digex.net (Edward C. Jones) Date: Tue, 01 Jul 1997 22:27:42 -0400 Subject: [MATRIX-SIG] A proposal (LONG) - "reverse of take" revisited References: <10497.199707012256@laplace> Message-ID: <33B9BC9E.75317C47@access.digex.net> P.S.Craig@durham.ac.uk wrote: > I have decided to stick my neck out and try to bring a > little order to the discussion about extending NumPy's > indexing that has been going on over the last week or so. ... I would like to see block capability added to NumPy. Memory management, numerical analysis, image processing, computer vision, and signal processing all need blocks. Examples include transposing large matrices, zooming or shrinking images, quadtrees, and wavelets. The most useful current capability of this sort is "reduceat". The manual says: |ufunc.reduceat(a, indices, axis=0) | | This is a weird function, and most people should just ignore it. It | will reduce a to each of the given indices so that as new size along | the given axis will be the same as the length of indices. This function is not weird. It is natural, powerful and useful. It reduces each block to one value. Something like the following are needed (all defined recursively): * A slice that does not make a copy, so I can modify a part of an image. * Efficiently view a big array as an array of blocks (each an array itself) and vice versa. * Convert an array of arrays to and from a big array. * The ability to extract, insert, and manipulate sub-blocks of blocks. For example, consider an interlaced image as an array of blocks, each with two rows. Construct a pair of new images containing the even and odd rows. Process these two images separately and recombine them. Please don't make changes in Python or NumPy indexing schemes without consider blocks and their users. What is the best way to put blocks into NumPy? _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From amullhau@ix.netcom.com Tue Jul 1 18:19:14 1997 From: amullhau@ix.netcom.com (Andrew P. Mullhaupt) Date: Tue, 01 Jul 1997 13:19:14 -0400 Subject: [MATRIX-SIG] Data filtering In-Reply-To: <199707011307.PAA32490@lmspc1.ibs.fr> Message-ID: <3.0.1.32.19970701131914.006d9be0@popd.netcruiser> At 03:07 PM 7/1/97 +0200, Konrad Hinsen wrote: >I'll soon need to do some data filtering, and before writing my own >I'd like to ask whether someone has already done it (in Python, of >course). What I need is a low-pass filter, preferably one that can >treat long data streams in smaller segments. I haven't done this stuff in python, but I have quite a bit of stuff which would be very easy for me to put into python and give to you. The question is; can you be more specific about the filter you need? Is it single or multichannel? Do you have issues with phase as well as magnitude? Is the segmentation requirement for efficiency or an actual requirement? Later, Andrew Mullhaupt _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From arw@dante.mh.lucent.com Wed Jul 2 12:21:37 1997 From: arw@dante.mh.lucent.com (Aaron Watters) Date: Wed, 2 Jul 1997 07:21:37 -0400 Subject: [MATRIX-SIG] A proposal (LONG) - "reverse of take" revisited Message-ID: <199707021137.HAA29599@dante.mh.lucent.com> > axisoffset = (slice(None,None,None),)*axis > for i in range(len(seq)): > a[axisoffset+(seq[i],)] = b[axisoffset+(i,)] hmmm. After looking at this code I see that part of my problem was my lack of understanding of the overloading of indexing. I didn't know that ar[ array([1,2]) ] would work. Since it does (without copying anything) :) a Python implementation of "give" or "set" or whatever you want to call it actually might be fast enough for practical purposes. I suppose each iteration of the loop would require only one header malloc, which might be inexpensive enough, and a C implementation might not be too much better. Comment withdrawn? I knew it was likely I was being dumb. Where are the details of all the functionality of Numeric set out? I have to say that it's a shame there isn't better documentation that I can find. Sorry if I missed it in DA's tutorial or somewhere else I'm unaware of. In particular things that aren't immediately obvious to me are the reference vs. copy semantics, how indexing/slicing works exactly, probably more I haven't run into yet. -- Aaron Watters note: fancy indexing is fine if you experts think it would be a Good Thing, but remember that Guido does not change the syntax lightly, for good reason. Just imagine what damage the regex community could have done by now if he hadn't resisted. My preference would be to be conservative here until maybe a clear consensus arrives in many months. _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From arw@dante.mh.lucent.com Wed Jul 2 12:38:04 1997 From: arw@dante.mh.lucent.com (Aaron Watters) Date: Wed, 2 Jul 1997 07:38:04 -0400 Subject: [MATRIX-SIG] A proposal (LONG) - "reverse of take" revisited Message-ID: <199707021137.HAA29603@dante.mh.lucent.com> Sorry. Can't shut up! > |ufunc.reduceat(a, indices, axis=0) > | > | This is a weird function, and most people should just ignore it. It > | will reduce a to each of the given indices so that as new size along > | the given axis will be the same as the length of indices. > > This function is not weird. It is natural, powerful and useful. It > reduces each block to one value. Okay, let's just say the description is weird. A simple two line example of its use? > Something like the following are needed (all defined recursively): > > * A slice that does not make a copy, so I can modify a part of an image. > * Efficiently view a big array as an array of blocks (each an array > itself) and vice versa. > * Convert an array of arrays to and from a big array. > * The ability to extract, insert, and manipulate sub-blocks of blocks. > For example, consider an interlaced image as an array of blocks, each > with two rows. Construct a pair of new images containing the even and > odd rows. Process these two images separately and recombine them. If you are willing to pay a bit in overhead it seems to me that it might be possible to prototype the concept(s) as a Python class (or set of Python classes) that wrap an array. After you're happy with the Python implementation you could "push it down" to C if it wasn't fast enough. In fact I'd recommend all such proposals have Python prototypes. It might be the case that some of the prototypes could be optimized well enough at the Python level to be just fine. -- Aaron Watters _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From hinsen@ibs.ibs.fr Wed Jul 2 19:01:59 1997 From: hinsen@ibs.ibs.fr (Konrad Hinsen) Date: Wed, 2 Jul 1997 20:01:59 +0200 Subject: [MATRIX-SIG] reverse of take? In-Reply-To: <3.0.1.32.19970701143355.006dacac@popd.netcruiser> (amullhau@ix.netcom.com) Message-ID: <199707021801.UAA04417@lmspc1.ibs.fr> > Me too. I think a better approach to adding flavors/bells/whistles to > indexing is by adding a default argument to the index, as in > > a[i, j, k, hint=flavor_woo_woo_17] That's unfortunately impossible with the current Python syntax. An index expression must consist of standard expressions or slices separated by commas. No keyword arguments. Aaron's comment on this: > > a[i, j, k, hint=flavor_woo_woo_17] > > I'm reeling about not following this discussion, but I suspect > the best approach is for Jim H. to "just do it". I personally > begin to think that excess overloading of indexing is not > advisable, possibly confusing. I vote for "take()" as is with dual Until now we haven't discussed much more overloading - just the addition of sequence types (including arrays) as indices! And I think this kind of operation should be a form of indexing simply because people coming from other array languages *expect* it to be. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28 Institut de Biologie Structurale | Fax: +33-4.76.88.54.94 41, av. des Martyrs | Deutsch/Esperanto/English/ 38027 Grenoble Cedex 1, France | Nederlands/Francais ------------------------------------------------------------------------------- _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From hinsen@ibs.ibs.fr Wed Jul 2 19:13:58 1997 From: hinsen@ibs.ibs.fr (Konrad Hinsen) Date: Wed, 2 Jul 1997 20:13:58 +0200 Subject: [MATRIX-SIG] A proposal (LONG) - "reverse of take" revisited In-Reply-To: <10497.199707012256@laplace> (P.S.Craig@durham.ac.uk) Message-ID: <199707021813.UAA04447@lmspc1.ibs.fr> > indexing semantics based on ideas found in S and APL. I would like to > suggest the following program of development: ... Looks good! > d) It is interesting to note that if the take (and presumably also > multitake) function returned a reference object instead of copying > data to a new array, then the give (and multigive) function would > be trivial to implement. Right. > The difficulty with a reference implementation for indexing by > arbitrary sequences is that the resulting array will be less > efficient than the basic NumPy array. The implementation is fairly > straightforward and would have the advantage that a statement such > as > > a[[1,3,4]][2]=1 > > would actually have the desired effect of changing a[4]. It would have another advantage, which I consider even more important: consistency. In current NumPy, indexing always returns a reference sharing the data space with the indexed array. If indexing with sequences creates a copy instead, then things get messy. To know the result of an indexing operation, you would have to analyze whether any part of the index is a sequence. > The other difficulty with a reference implementation for take is > that the default behaviour would have to remain copying. Otherwise, > existing code could well break. take() has to keep on copying, but sequence indexing doesn't have to. In fact, separating the two has the advantage of providing *both* functionalities: take() returns a copy that is time-efficient to use, and indexing returns a reference that is memory-efficient. > Actually, once the basics of a reference implementation of take > have been sorted, multitake is a trivial function to write > efficiently by repeated use of take. Right. > e) More exotic indexing ideas, such as those found in S and advocated > by Andrew Mullhaupt and me, would be the subject of individual or > collective experimentation using python classes for the time > being. I think that the ideas in a)-c) and possibly also d) would That seems like a good compromise. > To my mind, the big question is whether we think d) is a good idea. If In my opinion yes! > difficult to implement, I think. The problem is that the two > implementations cannot really be separated if we want efficiency. I Who cares? At the Python level the difference is invisible, and C extensions tend to ask for a contiguous copy of all input arrays anyway (since there is no C API for indexing). > I'm sorry to have been so long-winded. If anyone is still reading at > this point and is still interested, please comment. Otherwise, I might > just go ahead by myself anyway sometime in the next couple of months. Fine with me! Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28 Institut de Biologie Structurale | Fax: +33-4.76.88.54.94 41, av. des Martyrs | Deutsch/Esperanto/English/ 38027 Grenoble Cedex 1, France | Nederlands/Francais ------------------------------------------------------------------------------- _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From hinsen@ibs.ibs.fr Wed Jul 2 19:22:20 1997 From: hinsen@ibs.ibs.fr (Konrad Hinsen) Date: Wed, 2 Jul 1997 20:22:20 +0200 Subject: [MATRIX-SIG] A proposal (LONG) - "reverse of take" revisited In-Reply-To: <33B9BC9E.75317C47@access.digex.net> (edcjones@access.digex.net) Message-ID: <199707021822.UAA04471@lmspc1.ibs.fr> > I would like to see block capability added to NumPy. Memory management, Much of it is already there! You just have to play around with shapes. Example: I frequently work with matrices that logically consist of 3x3 blocks. For some operations I must work on the blocks, for others on the whole matrix. So here we go: matrix_of_blocks = zeros((n, 3, n, 3), Float) big_matrix = reshape(matrix_of_blocks, (3*n, 3*n)) Since reshape() returns a reference sharing the same data space, I can use the two arrays interchangeably. The only restriction is that the blocks must be of equal size. > * A slice that does not make a copy, so I can modify a part of an image. That's what indexing already does! > * Efficiently view a big array as an array of blocks (each an array > itself) and vice versa. > * Convert an array of arrays to and from a big array. Possible with restrictions (see above). > * The ability to extract, insert, and manipulate sub-blocks of blocks. Within the same restrictions, simply done by indexing. Question: do you see a need for supporting non-equal-sized blocks? That could become very messy. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28 Institut de Biologie Structurale | Fax: +33-4.76.88.54.94 41, av. des Martyrs | Deutsch/Esperanto/English/ 38027 Grenoble Cedex 1, France | Nederlands/Francais ------------------------------------------------------------------------------- _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From hinsen@ibs.ibs.fr Wed Jul 2 19:26:56 1997 From: hinsen@ibs.ibs.fr (Konrad Hinsen) Date: Wed, 2 Jul 1997 20:26:56 +0200 Subject: [MATRIX-SIG] A proposal (LONG) - "reverse of take" revisited In-Reply-To: <199707021137.HAA29599@dante.mh.lucent.com> (arw@dante.mh.lucent.com) Message-ID: <199707021826.UAA04487@lmspc1.ibs.fr> > of indexing. I didn't know that > > ar[ array([1,2]) ] > > would work. Since it does (without copying anything) It does? Now I am a bit surprised... > Where are the details of all the functionality of Numeric > set out? I have to say that it's a shame there isn't In the code! ;-) > better documentation that I can find. Sorry if I missed it I agree, but until someone volunteers to do a better documentation, we'll have to live with it. I learned much by experimentation, but of course I use NumPy in almost everything I do... > In particular things that aren't immediately obvious to > me are the reference vs. copy semantics, how indexing/slicing > works exactly, probably more I haven't run into yet. Yes, that should be addressed in a tutorial! > note: fancy indexing is fine if you experts think it > would be a Good Thing, but remember that Guido > does not change the syntax lightly, for good reason. There will be no need to change the syntax! Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28 Institut de Biologie Structurale | Fax: +33-4.76.88.54.94 41, av. des Martyrs | Deutsch/Esperanto/English/ 38027 Grenoble Cedex 1, France | Nederlands/Francais ------------------------------------------------------------------------------- _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From mclay@nist.gov Wed Jul 2 14:26:06 1997 From: mclay@nist.gov (Michael McLay) Date: Wed, 2 Jul 1997 09:26:06 -0400 Subject: [MATRIX-SIG] reverse of take? In-Reply-To: <3.0.1.32.19970701141535.006d53f8@popd.netcruiser> References: <3.0.1.32.19970630203104.0069db7c@popd.netcruiser> <3.0.1.32.19970701141535.006d53f8@popd.netcruiser> Message-ID: <199707021326.JAA31468@fermi.eeel.nist.gov> Andrew P. Mullhaupt writes: > I think you're confused about the meaning of this based on your previous > statements. > > As to useful, it is perhaps the single most common case of indexing in > array languages, and not only that, perhaps the _most efficient_. > > Here's an example: > > a = reshape(range(12), [3,4]) > > which looks like > > 0 1 2 3 > 4 5 6 7 > 8 9 10 11 > > then you want > > a[[0,2], [2,3]] > > to be > > 2 3 > 10 11 > > right? I think a lot of people have been thinking that this would be the > case (like APL, S, Matlab, etc.). > > The question here is actually not _can_ you do this, (I can easily give a > good implementation of what I outlined so far). But we are not at that > point yet. Just for fun I created a trivial implementation. It only works for 2-d arrays and it requires making the index either a list of lists, or a 2-d array. A C implementation would obviously be needed and it should be generalized to n-dimensional arrays. This notation does break the orthogonality in the use of sequences as indexes, but is that really a problem? This is already true for dicitonary indexes. Dictionaries can be indexed with tuples, but dictionaries cannot be indexed with lists. Perhaps for array type objects it is also resaonable that indexing with different sequence types can yield different results. What is the arguement for making array[1,3,4] == array[[1,3,4]] #!/usr/bin/env python from Numeric import * class C: def __init__(self,arg): self.data = array(arg) def __getitem__(self, arg): if type(arg) == type([]) or type(arg) == type(array([])): res = [] for x in arg[0]: res1 = [] res.append(res1) for y in arg[1]: res1.append( self.data[x,y]) return res else: return self.data[arg] # this code just sets up the x.data inside the demonstration class. a3x4 = reshape(arange(12),[3,4]) x = C(a3x4) print x.data print "\nUsing a gather syntax" print x[[[0,2], [2,3]]] print "\nNnow back to the existing indexing" print x[2] print "\nTuples still work as before" print x[0,2] a = array([[0,2], [2,3]]) print "\nThe gather syntax could also use arrays as an index" print x[a] print "\n using take" dict = {} dict [1,3,4 ] = "a tuple as an index" print "dict indexing isn't orthogonal", print dict [1,3,4 ] print "dict indexing with lists doesn't work" dict [[1,3,4]] = "a list as an index" _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From arw@dante.mh.lucent.com Wed Jul 2 19:37:44 1997 From: arw@dante.mh.lucent.com (Aaron Watters) Date: Wed, 2 Jul 1997 14:37:44 -0400 Subject: [MATRIX-SIG] reverse of take? Message-ID: <199707021834.OAA14440@dante.mh.lucent.com> > > > a[i, j, k, hint=flavor_woo_woo_17] > > > > I'm reeling about not following this discussion, but I suspect > > the best approach is for Jim H. to "just do it". I personally > > begin to think that excess overloading of indexing is not > > advisable, possibly confusing. I vote for "take()" as is with dual > > Until now we haven't discussed much more overloading - just the > addition of sequence types (including arrays) as indices! And I think > this kind of operation should be a form of indexing simply because > people coming from other array languages *expect* it to be. People keep mentioning other languages to me; I wish I knew what they were talking about :-). Seriously, I think people will whine a bit and then get back to work... OK, my problem with sequence-generalizes-indices is that it seems to me that Ar[Ar1][Ar2] = x either won't work properly or will require a change to what I understand to be the relatively elegant internal representation for arrays in NumPy as they exist. I wouldn't recommend messing up the current implementation for this special case. The resulting bugs and weird side effects could be impressive, I think. If I'm wrong (as I think I may have been regarding "dual of take" which might be fine in Python) I take it all back. -- Aaron Watters checkout http://www.chipanalyst.com/report/editorials/edit11_6.html The number of articles like this one keep piling up... (but he should have said "geek tragedy" rather than "Greek tragedy") _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From amullhau@ix.netcom.com Thu Jul 3 01:07:41 1997 From: amullhau@ix.netcom.com (Andrew P. Mullhaupt) Date: Wed, 02 Jul 1997 20:07:41 -0400 Subject: [MATRIX-SIG] A proposal (LONG) - "reverse of take" revisited In-Reply-To: <199707021137.HAA29599@dante.mh.lucent.com> Message-ID: <3.0.1.32.19970702200741.006d82a0@popd.netcruiser> At 07:21 AM 7/2/97 -0400, Aaron Watters wrote: >would work. Since it does (without copying anything) >:) a Python implementation of >"give" or "set" or whatever you want to call it actually might >be fast enough for practical purposes. I suppose each >iteration of the loop would require only one header malloc, >which might be inexpensive enough, and a C implementation >might not be too much better. In the one dimensional case, I'd expect a factor of 5-10 or so. In the multidimensional case, you can do some things which will make this a factor which increases rapidly with the number of dimensions. >Where are the details of all the functionality of Numeric >set out? There are some notes from talks by Jim Hugunin on the web. >note: fancy indexing is fine if you experts think it > would be a Good Thing, but remember that Guido > does not change the syntax lightly, for good reason. That's OK. We don't need to change the syntax per se. We want to change the semantics. As to how good it is, you can get an indication of how useful this sort of thing is by the fact that several interpreters which had strict limits on indexing (APL at various stages, Matlab until the most recent version, Speakeasy) were _forced_ by their user communities to extend the indexing operation over time. Pascal and C, in every other sense far better languages than Fortran IV, didn't put a _dent_ in the scientific computing community for a _very long time_ because they absolutely sucked for arrays. (C still does). > My > preference would be to be conservative here until > maybe a clear consensus arrives in many months. It may very well take months to figure out what everyone wants, but probably not many months. Later, Andrew Mullhaupt _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From phil@geog.ubc.ca Thu Jul 3 03:29:28 1997 From: phil@geog.ubc.ca (Phil Austin) Date: Wed, 2 Jul 1997 19:29:28 -0700 Subject: [MATRIX-SIG] A proposal (LONG) - "reverse of take" In-Reply-To: <3.0.1.32.19970702200741.006d82a0@popd.netcruiser> References: <199707021137.HAA29599@dante.mh.lucent.com> <3.0.1.32.19970702200741.006d82a0@popd.netcruiser> Message-ID: <199707030229.TAA27421@curlew.geog.ubc.ca> >>>>> "AM" == Andrew P Mullhaupt writes: AM> As to how good it is, you can get an indication of how useful AM> this sort of thing is by the fact that several interpreters AM> which had strict limits on indexing (APL at various stages, AM> Matlab until the most recent version, Speakeasy) were _forced_ AM> by their user communities to extend the indexing operation AM> over time. Pascal and C, in every other sense far better AM> languages than Fortran IV, didn't put a _dent_ in the AM> scientific computing community for a _very long time_ because AM> they absolutely sucked for arrays. (C still does). And C++ scientific programmers have adopted the Fortran90 Index notation. Here's part of a demo from Dan Quinlan's A++ array package--i and j are vectors: void PDE::Nine_Point_Stencil ( Index & i , Index & j ) { Solution (i,j) = ( (Mesh_Size * Mesh_Size) * Right_Hand_Side (i,j) + Solution (i+1,j) + Solution (iFrom owner-matrix-sig@python.org Thu Jul 3 09:42:10 1997 Received: (from majordom@localhost) by python.org (8.7.5/8.7.3)id JAA10638 for matrix-sig-people; Thu, 3 Jul 1997 09:41:56 -0400 (EDT) Received: from ibs.ibs.fr (ibs.ibs.fr [192.134.36.253]) by python.org (8.7.5/8.7.3) with SMTPid JAA10633 for ; Thu, 3 Jul 1997 09:41:53 -0400 (EDT) Received: from lmspc1.ibs.fr (hinsen@lmspc1.ibs.fr [192.134.36.141]) by ibs.ibs.fr (8.6.12/8.6.12) with ESMTP id PAA26170; Thu, 3 Jul 1997 15:37:42 +0200 Received: (from hinsen@localhost) by lmspc1.ibs.fr (8.8.5/8.8.5) id PAA07644; Thu, 3 Jul 1997 15:36:25 +0200 Date: Thu, 3 Jul 1997 15:36:25 +0200 Message-Id: <199707031336.PAA07644@lmspc1.ibs.fr> From: Konrad Hinsen To: arw@dante.mh.lucent.com CC: arw@mh.lucent.com, matrix-sig@python.org In-reply-to: <199707031209.IAA28794@dante.mh.lucent.com> (arw@dante.mh.lucent.com) Subject: Re: [MATRIX-SIG] reverse of take? Sender: owner-matrix-sig@python.org Precedence: bulk > Ok, this gets fun. Are we talking about a "pointer array"? > IE an array of *references* into another array? One could Right. > even imagine an array of references into several different > arrays, but the refcounting issues might get complicated here... Indeed, and how would you create such an array? By combined indexing from two arrays? I don't think that makes much sense... > Actually, maybe an array of references into one other array > might not look too different from the current implementation > -- just like the current array implementation, but storing pointers Exactly. The current array implementation basically knows two kinds of objects: an "array data" object, which users don' see directly, and an "array reference" object, which stores a pointer to the data object plus information on the arrangement of elements (first & last index plus stride for each dimension). Indexing creates a new reference object, which refers to the same data object. So what we are talking about now is an alternative implementation of the reference object which musts store more than just three numbers per dimension. There are several possible implementations; I suppose the easiest is a list of integer indices per dimension, referring to the array data object. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28 Institut de Biologie Structurale | Fax: +33-4.76.88.54.94 41, av. des Martyrs | Deutsch/Esperanto/English/ 38027 Grenoble Cedex 1, France | Nederlands/Francais ------------------------------------------------------------------------------- _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From arw@dante.mh.lucent.com Thu Jul 3 14:50:09 1997 From: arw@dante.mh.lucent.com (Aaron Watters) Date: Thu, 3 Jul 1997 09:50:09 -0400 Subject: [MATRIX-SIG] reverse of take? Message-ID: <199707031346.JAA24301@dante.mh.lucent.com> > So what we are talking about now is an alternative implementation of > the reference object which musts store more than just three numbers > per dimension. There are several possible implementations; I suppose > the easiest is a list of integer indices per dimension, referring to > the array data object. I say the easiest and maybe the most efficient is an actual array of pointers *into* the other array (not sequences of indices). There are probably nuances I'm missing. -- Aaron Watters _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From P.S.Craig@durham.ac.uk Thu Jul 3 15:33:38 1997 From: P.S.Craig@durham.ac.uk (P.S.Craig@durham.ac.uk) Date: Thu, 3 Jul 1997 15:33:38 +0100 (BST) Subject: [MATRIX-SIG] reverse of take? In-Reply-To: <199707031346.JAA24301@dante.mh.lucent.com> from "Aaron Watters" at Jul 3, 97 09:50:09 am Message-ID: <12834.199707031433@laplace> Aaron Watters writes: > > > So what we are talking about now is an alternative implementation of > > the reference object which musts store more than just three numbers > > per dimension. There are several possible implementations; I suppose > > the easiest is a list of integer indices per dimension, referring to > > the array data object. > > I say the easiest and maybe the most efficient is an actual array of > pointers *into* the other array (not sequences of indices). There are > probably nuances I'm missing. -- Aaron Watters > Actually, I suspect both have their merits in different contexts. The merit of Aaron's proposal is complete flexibility. The merit of Konrad's, which was also my initial intention, is that we can save memory and some computing time for such an array which has shape (100,100,100) since under Konrad's proposal this requires 300 integer indices and under Aaron's requires 1000000 pointers. The choice between the two, if we cannot accept having three implementations of array referencing, depends on the future of indexing beyond my earlier proposal. The really exotic ideas may well need Aaron's version though I have to admit I havent thought about this aspect at all carefully yet. Peter Craig _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From hinsen@ibs.ibs.fr Thu Jul 3 16:22:34 1997 From: hinsen@ibs.ibs.fr (Konrad Hinsen) Date: Thu, 3 Jul 1997 17:22:34 +0200 Subject: [MATRIX-SIG] reverse of take? In-Reply-To: <199707031346.JAA24301@dante.mh.lucent.com> (arw@dante.mh.lucent.com) Message-ID: <199707031522.RAA07980@lmspc1.ibs.fr> > I say the easiest and maybe the most efficient is an actual array of > pointers *into* the other array (not sequences of indices). There are > probably nuances I'm missing. -- Aaron Watters With pointers you need one per array element, whereas with integers you need only one per index. In other words, for an array of shape (10, 20, 30) you need either 10x20x30=6000 pointers or 10+20+30=60 integers. One the other hand, integer indices become a bit more complex when we allow higer-rank indices, whereas a pointer implementation remains the same. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28 Institut de Biologie Structurale | Fax: +33-4.76.88.54.94 41, av. des Martyrs | Deutsch/Esperanto/English/ 38027 Grenoble Cedex 1, France | Nederlands/Francais ------------------------------------------------------------------------------- _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From arw@dante.mh.lucent.com Thu Jul 3 16:46:15 1997 From: arw@dante.mh.lucent.com (Aaron Watters) Date: Thu, 3 Jul 1997 11:46:15 -0400 Subject: [MATRIX-SIG] reverse of take? Message-ID: <199707031542.LAA16461@dante.mh.lucent.com> ---------- > From: Konrad Hinsen > With pointers you need one per array element, whereas with integers > you need only one per index. In other words, for an array of shape > (10, 20, 30) you need either 10x20x30=6000 pointers or 10+20+30=60 > integers. One the other hand, integer indices become a bit more > complex when we allow higer-rank indices, whereas a pointer > implementation remains the same. Yes, and in the "circle" or "fractal border" on byte image graphics example you would not get much savings in space and also perhaps a speed penalty by saving indices versus pointers, because the indices would be completely irregular down to the pixel level. I suppose the question becomes how common is regularity? You matrix guys would know better than I. For graphics type operations I can imagine essentially 2 common cases: rectangular subregions (slices as they now exist), and more or less complete irregularity. I can't comment on other possible applications. -- Aaron Watters _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From hinsen@ibs.ibs.fr Thu Jul 3 18:48:18 1997 From: hinsen@ibs.ibs.fr (Konrad Hinsen) Date: Thu, 3 Jul 1997 19:48:18 +0200 Subject: [MATRIX-SIG] reverse of take? In-Reply-To: <199707031542.LAA16461@dante.mh.lucent.com> (arw@dante.mh.lucent.com) Message-ID: <199707031748.TAA08475@lmspc1.ibs.fr> > Yes, and in the "circle" or "fractal border" on byte image > graphics example you would not get > much savings in space and also perhaps a speed penalty by saving Ehhh... Why? The space requirements depend only on the shape of the array, not its contents. So the difference is negligible only for something large x small (e.g. 1000x2). > indices versus pointers, because the indices would be completely > irregular down to the pixel level. I suppose the Ah, I see, you want index a 2d array with a list of index tuples, right? I hadn't even considered this possibility until now. But I can still come up with an equivalent integer-index solution that is a superset of the previously discussed one: for each dimension, you store an n-dimensional array of integer indices, which more or less correspond to the indices used in the subscript, except that they are adjusted to point to the array data object directly. On the other hand, a pointer-based system has clear speed advantages if the subarray is accessed more often than it is created. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28 Institut de Biologie Structurale | Fax: +33-4.76.88.54.94 41, av. des Martyrs | Deutsch/Esperanto/English/ 38027 Grenoble Cedex 1, France | Nederlands/Francais ------------------------------------------------------------------------------- _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From arw@dante.mh.lucent.com Thu Jul 3 19:59:51 1997 From: arw@dante.mh.lucent.com (Aaron Watters) Date: Thu, 3 Jul 1997 14:59:51 -0400 Subject: [MATRIX-SIG] reverse of take? Message-ID: <199707031856.OAA20850@dante.mh.lucent.com> > On the other hand, a pointer-based system has clear speed advantages > if the subarray is accessed more often than it is created. Yes, I imagine a system where a button object, say, is given a matrix into which to write its representation when it it told to draw itself. In the ideal case writes into this matrix should propagate all the way back to the frame buffer. For rectangular buttons, as I understand it, Numeric slices will do this (or something close) now. For other shapes it wouldn't. A heck maybe rects are good enough for that. Also, (changing my mind here) for, say, fonts it would be nice to share an array of indices rather than making a new pointer array for each character. I think you've convinced me that sequences of indices are the *right* way, Konrad. -- Aaron Watters _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From amullhau@ix.netcom.com Thu Jul 3 23:53:51 1997 From: amullhau@ix.netcom.com (Andrew P. Mullhaupt) Date: Thu, 03 Jul 1997 18:53:51 -0400 Subject: [MATRIX-SIG] reverse of take? In-Reply-To: <199707031542.LAA16461@dante.mh.lucent.com> Message-ID: <3.0.1.32.19970703185351.006d1cec@popd.netcruiser> At 11:46 AM 7/3/97 -0400, Aaron Watters wrote: > >Yes, and in the "circle" or "fractal border" on byte image >graphics example you would not get >much savings in space and also perhaps a speed penalty by saving >indices versus pointers, because the indices would be completely >irregular down to the pixel level. You may or may not do better with pointers in this case. If you are allowed to have pointers to arrays as array elements, then it is usually not necessary to have separate implementation concerns for indices and pointers - you can control that at the higher level language where you have more leverage. > I suppose the >question becomes how common is regularity? >You matrix guys would know better than I. For graphics type operations >I can imagine essentially 2 common cases: rectangular subregions (slices as >they now exist), and more or less complete irregularity. I can't comment on other >possible applications. Regularity is by far the most common case. Most non-numerical applications that use matrices as objects are "dense" matrices. Your version of 'irregularity' corresponds to one style of "sparse" matrices, and there are many applications for that too. Sometimes the pointers for sparse matrices are organized into linked lists for each dimension (e.g. a list of lists by row and/or by columnn). There are many "structured" sparse cases, with the most important being band matrices - which are implemented by embedding them in an index-like storage. Now in graphics, (where you seem to be coming from) you find both to be useful. The dense versions are good for image processing and the sparse versions can be useful for the more computational geometric things. Long experience in scientific computing and graphics shows that you really can't settle for "all indices" vs. "all pointers". There is a reason for this. As time goes on, computers get bigger, faster, and cheaper. The problems that people do on the computers correspond to larger and larger sizes "N". This means that where in the past, an O(N^2) or O(N^3) dense matrix algorithm with a small constant becomes less attractive than the O(N) or O(N^2) algorithm for structured matrices with previously prohibitive constants. This favors the use of flexible pointer-based structures and algorithms. By the same token, CPUs are becoming more and more superscalar/parallel, but memory systems are only keeping up in _size_, they are not keeping up in speed. This means that instruction bandwidth is becoming free. Memory bandwidth is becoming more precious. This means that being control bound is less and less tolerable - if you can't predict the branches and profit by speculative execution, then you will not get the most of your machine. This _strongly_ favors the highly regular algorithms and structures of the dense representation. What these trends seem to be doing is making it more important to choose the correct implementation for the problem, and making it harder to "fake" one style with the other. APL bit the bullet about ten years ago, with the advent of what I call "modern" APL - where an array can be an element of an array, which gives you the pointer style operations APL had previously lacked. Fortran had to bite this bullet at about the same time with F90, but going in the other direction. So a good language is really going to want to go both ways. Maple and Matlab have slightly different approaches to getting this without requiring any syntax modification. Python's object orientation is plenty strong enough to handle this sort of distinction - you can have a class of dense arrays and a class of sparse arrays and nobody gets hurt. However, it makes _zero_ sense to have the common type of 2-way linked list sparse matrices with the current restriction of indexing to slices, and that's sort of why we're here. I also want to put some perspective on what is "esoteric" in indexing. Everything that has been brought up so far is very tame and you can find it going back at least ten years in several different successful languages. It's not like there is a lot of technical risk on the table. _Esoteric_ is what the post-APL languages are getting into such as K where there is no distinction between array indexing and function calls, so an expression such as (and I'm sparing everyone the sight of actual K code) goo = foo(, , z) means that goo(x, y) is the same as foo(x, y, z) The K-speakers call this "currying", which I believe is a term originating in the lambda calculus and the Lisp community. So far in this thread nobody is proposing anything which isn't extremely vanilla as far as array languages go. Later, Andrew Mullhaupt _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From phil@geog.ubc.ca Fri Jul 4 02:34:48 1997 From: phil@geog.ubc.ca (Phil Austin) Date: Thu, 3 Jul 1997 18:34:48 -0700 Subject: [MATRIX-SIG] A proposal (LONG) - "reverse of take" In-Reply-To: <3.0.1.32.19970703024015.006d8698@popd.netcruiser> References: <3.0.1.32.19970702200741.006d82a0@popd.netcruiser> <199707021137.HAA29599@dante.mh.lucent.com> <199707030229.TAA27421@curlew.geog.ubc.ca> <3.0.1.32.19970703024015.006d8698@popd.netcruiser> Message-ID: <199707040134.SAA32006@curlew.geog.ubc.ca> >>>>> "Andrew" == Andrew P Mullhaupt writes: Andrew> At 07:29 PM 7/2/97 -0700, Phil Austin wrote: >> And C++ scientific programmers have adopted the Fortran90 Index >> notation. Andrew> I find that while the F90 indexing is better than C, Andrew> Pascal, etc., it's pretty thin soup compared to most array Andrew> languages. Are they going to improve it for F95? There's an "enhanced where" (slightly more powerful boolean constructs, see the appended features list). The cutting edge is now off wrestling with OO-Fortran. The glacial calm among the language conservatives in comp.lang.fortran make it a very restful place to visit when the tumult in some of the other newsgroups is raising your blood pressure. * Enhanced WHERE construct [with nested WHERE, masked ELSEWHERE, and construct names] * The FORALL statement and construct * PURE user-defined procedures * ELEMENTAL user-defined procedures * Pointer initialization [i.e., the NULL( ) intrinsic] * Derived-type structure default initialization * Automatic deallocation of allocatable arrays [at exit from a procedure] * CPU_TIME intrinsic * Enhanced CEILING and FLOOR intrinsic functions [with KIND argument] * Enhanced MAXLOC and MINLOC intrinsic functions [with DIM argument] * Enhanced SIGN intrinsic function [can detect negative real zero] * Printing of -0.0 * Generic identifier allowed in END INTERFACE statement * Zero-length formats [I, B, O, Z, F expands field width to fit data] * Comments allowed in namelist input * New obsolescent features [as specified in F95] _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From hinsen@ibs.ibs.fr Fri Jul 4 13:40:47 1997 From: hinsen@ibs.ibs.fr (Konrad Hinsen) Date: Fri, 4 Jul 1997 14:40:47 +0200 Subject: [MATRIX-SIG] Data filtering In-Reply-To: <3.0.1.32.19970701131914.006d9be0@popd.netcruiser> (amullhau@ix.netcom.com) Message-ID: <199707041240.OAA11383@lmspc1.ibs.fr> > I haven't done this stuff in python, but I have quite a bit of stuff > which would be very easy for me to put into python and give to you. The > question is; can you be more specific about the filter you need? Is it > single or multichannel? Do you have issues with phase as well as magnitude? > Is the segmentation requirement for efficiency or an actual requirement? For my personal needs, any low-pass filter for real data is good enough. Segmentation would be nice for efficiency, but is not essential. But I would like to encourage work on a general-purpose signal processing library to go with NumPy. This is probably as important in practice as the linear algebra library, although for different users. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28 Institut de Biologie Structurale | Fax: +33-4.76.88.54.94 41, av. des Martyrs | Deutsch/Esperanto/English/ 38027 Grenoble Cedex 1, France | Nederlands/Francais ------------------------------------------------------------------------------- _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From phil@geog.ubc.ca Sat Jul 5 03:48:31 1997 From: phil@geog.ubc.ca (Phil Austin) Date: Fri, 4 Jul 1997 19:48:31 -0700 Subject: [MATRIX-SIG] ZPL array language In-Reply-To: <3.0.1.32.19970703185351.006d1cec@popd.netcruiser> References: <199707031542.LAA16461@dante.mh.lucent.com> <3.0.1.32.19970703185351.006d1cec@popd.netcruiser> Message-ID: <199707050248.TAA04681@curlew.geog.ubc.ca> >>>>> "Andrew" == Andrew P Mullhaupt writes: Andrew> _Esoteric_ is what the post-APL languages are getting into Andrew> such as K where there is no distinction between array Andrew> indexing and function calls, so an expression such as (and Andrew> I'm sparing everyone the sight of actual K code) I'm not sure how esoteric it is, but a group at the University of Washington has just preempted the end of the alphabet with ZPL--an array language for parallel processors. They have source and sample code at: http://www.cs.washington.edu/research/zpl/ Phil _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From da@maigret.cog.brown.edu Sun Jul 6 21:26:04 1997 From: da@maigret.cog.brown.edu (David Ascher) Date: Sun, 6 Jul 1997 16:26:04 -0400 (EDT) Subject: [MATRIX-SIG] A proposal (LONG) - "reverse of take" revisited In-Reply-To: <199707021826.UAA04487@lmspc1.ibs.fr> Message-ID: On Wed, 2 Jul 1997, Konrad Hinsen wrote: > > Where are the details of all the functionality of Numeric > > set out? I have to say that it's a shame there isn't > > better documentation that I can find. Sorry if I missed it > > I agree, but until someone volunteers to do a better documentation, > we'll have to live with it. I learned much by experimentation, but > of course I use NumPy in almost everything I do... > > > In particular things that aren't immediately obvious to > > me are the reference vs. copy semantics, how indexing/slicing > > works exactly, probably more I haven't run into yet. > > Yes, that should be addressed in a tutorial! I'll add these to the TODO list for the tutorial, but don't count on me getting too much done in the near future (see http://maigret.cog.brown.edu/hugo/hugo14.html for the reason why =). --da _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From hyoon@nyptsrv1.etsd.ml.com Tue Jul 8 15:18:43 1997 From: hyoon@nyptsrv1.etsd.ml.com (Hoon Yoon - IPT Quant) Date: Tue, 8 Jul 1997 10:18:43 -0400 Subject: [MATRIX-SIG] Much ado about nothingness. Message-ID: <199707081418.KAA05563@ptq3.etsd> Hello fellow snake handlers, I am attempting to do some Trading (stocks) related number crunching in Python. I am having quite a difficulty trying to implement my old Gauss logic to Python. It should be dealt in a different way; however, my mind is still in the old 2D complete matrix stage that I am finding it difficult to find what I wanna to do in Python. =20 # I have some x period of info into Python lists/arrays. Of course as of any financial data there are missing values. So, I stored them as None, thinking that None is closest to msg values, but, as some of the readers would have realized, none of the nonzero, choose, equal, = where, etc... can be performed on array or list with None values. I get: TypeError: function not supported for these types, and can't=20 coerceto supported types Now some might say why not just use either zeros or negative values. = Well, if I am just dealing with price and vols that's okay, but I am also = dealing with other info like earnings, which can be easily negative. There is at least distinction in mathmatics/databases about missing, = undefined, and not available. This is very thorny issue in numerical computing. = Is msg =3D=3D msg? (This is undefined) What's truly missing vs. undefined? This is = quite a issue. For people brave enogh, I encourage Joe Celko's articles in = DBMS with same title, which can be obtained through DBMS. He really does good = job of=20 explaining the issues. There are other issues like +&- infinity NA and = so on, which Joe does not get into, but overall the article serves as good = introduction to the concepts. May be all this issues have been considered and Python has way to = handle all this, but I could not find it. I would really like to see find_msg & = index_msg find of functions. The most often used way to deal with missing values are : a. Just copy over from last period or do some estimate based on = others, which is bit dangerous due to introduction of serial correlation. b. Get rid of the entire row of the observation. Which means throw = the whole stock or the whole day from the model. Less dangerous, but could be = problematic, espcially if you don't have too many observation to play with. =20 find_msg should return matrix of same dimention filled with 0,1. = index_msg should only work on 1 dim and give me back index to an array. I can use = find_msg for take kind of function, or rever of take. =20 =04 Hopefully I can do this without loops! Thanks for reading, Hoon, _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From arw@dante.mh.lucent.com Tue Jul 8 16:52:41 1997 From: arw@dante.mh.lucent.com (Aaron Watters) Date: Tue, 8 Jul 1997 11:52:41 -0400 Subject: [MATRIX-SIG] Much ado about nothingness. Message-ID: <199707081549.LAA06143@dante.mh.lucent.com> Hey! Null values! I had a whole chapter in my dissertation regarding nulls! There are many kinds of nulls: overdefined (computation error) | [normal non-null values here] ... | unknown/applicable inapplicable \ / unknown/maybe-inapplicable How a query engine or computation should handle this depends on which one you mean. However, in general I think if you want to "layer" null values into a matrix you might want to use an "indicator mask" which identifies the null/non-null values as a separate array. Let me know if this makes sense ---------- From: Hoon Yoon - IPT Quant To: matrix-sig@python.org Subject: [MATRIX-SIG] Much ado about nothingness. Date: Tuesday, July 08, 1997 10:18 AM Hello fellow snake handlers, I am attempting to do some Trading (stocks) related number crunching in Python. I am having quite a difficulty trying to implement my old Gauss logic to Python. It should be dealt in a different way; however, my mind is still in the old 2D complete matrix stage that I am finding it difficult to find what I wanna to do in Python. # I have some x period of info into Python lists/arrays. Of course as of any financial data there are missing values. So, I stored them as None, thinking that None is closest to msg values, but, as some of the readers would have realized, none of the nonzero, choose, equal, where, etc... can be performed on array or list with None values. I get: TypeError: function not supported for these types, and can't coerceto supported types _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From hyoon@nyptsrv1.etsd.ml.com Tue Jul 8 17:28:31 1997 From: hyoon@nyptsrv1.etsd.ml.com (Hoon Yoon - IPT Quant) Date: Tue, 8 Jul 1997 12:28:31 -0400 Subject: [MATRIX-SIG] Much ado about nothingness. Message-ID: <199707081628.MAA05660@ptq3.etsd> Aaron, This is what I love about working in Python. I had pretty hard time making vendors understand why different nothingness mattered so much. Now I am actually getting educated here. I don't think I am stretching too much, when I say there isn't a good stat package or DBMS out there that handles missings sufficiently. For my computational needs, I think a mask of nulls will do nicely. Going back to how I am used to handling it. Gauss has something called packer, which packs away any rows that contains missing value, which is a (.) dot in Gauss. In Python this could be much more generalized. I can get index of non missing or missing by row or column or N-th dim. And do either take or reverse of take. This is bit costly on memory, however. I am actually creating equal dimension (even at at bits) of 1's and zeros, which is not too bad. I also noticed that whenver I used None in my array, I get a complex array. Nothing should be uniform and should not be this expensive. For backward filling, this is always has been tricky business. If I have one time value missing, I can use the above 1/0 array move the original array by 1 dimension backward and mult the result, but what do I do if I have multiple period of missings? I guess a repeat is in order untill I get "None in Target_array" is 0. Of course I am working under premise that the first row of N-th dimention is not itself has missing. That could be really unfun. Not to make too much fun of Gauss, but msg==msg was 1. None of the stat package I know has more than one kind of nothingness for input. I guess this means that Python does not have naitive Null handler upto now? I wouldn't be terribly surprised, but I think it needs one. I am still getting confused about the N-th dimension thing. I am sure it's good thing, but confused still. How did everyone dealt with this before? Thank you for the reply, Hoon, _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From arw@dante.mh.lucent.com Tue Jul 8 18:47:59 1997 From: arw@dante.mh.lucent.com (Aaron Watters) Date: Tue, 8 Jul 1997 13:47:59 -0400 Subject: [MATRIX-SIG] Much ado about nothingness. Message-ID: <199707081744.NAA26463@dante.mh.lucent.com> I'm not sure I'm following you. Maybe a simple example would help. What do you want to do and what is a slow way to do it (using lots of python loops if necessary). Once we see that someone (likely not me) can suggest how to speed it up using Numeric (and maybe altering the representation). Guessing: Suppose you have a 2d array of prices for securities by time with another 2d array of same shape that indicate which price listings are invalid/unknown. Suppose you wish to apply the heuristic of copying the youngest price into the unknown positions, and further suppose the prices at time 0 are all known. It seems to me you could pass a "price vector" across the time dimension, picking out new valid prices and masking in most recent valid prices to the source array for null positions. My skills with numeric are such that I'd have to work on this for a while to get it, so I'll let you think about it instead. It probably would amount to about 10 lines if that of NumPy. It's likely that I suppose wrong however, and I could be off base in any number of other ways. --Aaron Watters ---------- > From: Hoon Yoon - IPT Quant > To: matrix-sig@python.org; arw@mh.lucent.com > Subject: Re: [MATRIX-SIG] Much ado about nothingness. > Date: Tuesday, July 08, 1997 12:28 PM > > Aaron, > > This is what I love about working in Python. I had pretty hard time > making vendors understand why different nothingness mattered so much. > Now I am actually getting educated here. > > I don't think I am stretching too much, when I say there isn't a good > stat package or DBMS out there that handles missings sufficiently. > For my computational needs, I think a mask of nulls will do nicely. > > Going back to how I am used to handling it. > Gauss has something called packer, which packs away any rows that > contains missing value, which is a (.) dot in Gauss. In Python this > could be much more generalized. I can get index of non missing or > missing by row or column or N-th dim. And do either take or reverse > of take. This is bit costly on memory, however. I am actually creating > equal dimension (even at at bits) of 1's and zeros, which is not too > bad. I also noticed that whenver I used None in my array, I get a complex > array. Nothing should be uniform and should not be this expensive. > For backward filling, this is always has been tricky business. If I > have one time value missing, I can use the above 1/0 array move the > original array by 1 dimension backward and mult the result, but what > do I do if I have multiple period of missings? I guess a repeat is > in order untill I get "None in Target_array" is 0. Of course I am working > under premise that the first row of N-th dimention is not itself has > missing. That could be really unfun. > Not to make too much fun of Gauss, but msg==msg was 1. None of the > stat package I know has more than one kind of nothingness for input. > > I guess this means that Python does not have naitive Null handler > upto now? I wouldn't be terribly surprised, but I think it needs one. > I am still getting confused about the N-th dimension thing. I am sure > it's good thing, but confused still. > > How did everyone dealt with this before? > Thank you for the reply, > > Hoon, > _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From hyoon@nyptsrv1.etsd.ml.com Tue Jul 8 20:04:21 1997 From: hyoon@nyptsrv1.etsd.ml.com (Hoon Yoon - IPT Quant) Date: Tue, 8 Jul 1997 15:04:21 -0400 Subject: [MATRIX-SIG] Much ado about nothingness. Message-ID: <199707081904.PAA05746@ptq3.etsd> Aaron and MSIG: I am not quite that concerned about the speed of the following code. It runs in about 2sec on my Ultra Sparc. I got a lot of loops in there to check the data. This is fine for 20 or 30 stocks, but any more would prove problematic. What I have much more concern for is that I am not happy that Null, missing, or None values are foreign concepts to NumPy. An array time should not be complex. And there should be much easier way to pull matching and NOT criteria. Reverse of Take would be very nice indeed. I guess I am griping, because I have to get whole new thought process going here to get what I could do in 20 minutes in Gauss rather than 2 hours. Still, I think there should be a more elegent ways to handle Null data. About 80% modeling is dealing with data. Hardest to deal with is the issue with missing data. I should not continously use loops to deal with this in a complex matrix. There should be a set missing code that does not effect the matrix to default to complex. Pretty much all other grips about handling could be classed away over time in an elegent manner; however, I cannot change the fact that too many operators like equal, where, etc... neglects to deal with Nulls. I should be able to nonzero(equal(a, None)) to get index of Nones. And most operation should be programmed with missing data poitn in mind. To credit the Python, I must say even with loops, the ideas are much more readable than anything else I have ever written. Given how wonderful Python has been so far, I am amazed that None/missing has not been addressed in NumPy. Thanks Aaron, I hope the following code generally give you sense of 80% of my work. There are few more places, I would have dropped loops like > .80 could have been greater, the holidays could have been chopped away at one shot, etc... Hoon, p.s.: Despite my nagging, I won't trade Python for anything at this point. I just wanna see it improved to the point that I won't have to learn any other language. I would like to better business graphics, like GNUplot tied to this and better statistics (I am hoping to tie in something else if necessary), and this will be perfect for my purposes. ***************************************************************** import shelve from dateproc import * from Numeric import * d = shelve.open('p_v_rum') dates = d['i_dates'] # get dates that keys this time series dates.sort() trd_d = dates[:] # This will have all holidays removed later holi = d['i_holi'] # get all available holidays holi_i = [] for hday in holi.keys(): # getting rid of holidys if len(holi[hday]) < (len(d.keys())*0.8): continue # At least 80% stocks msg try: hsx = dates.index(hday) # if > 80%: see if it's available. except ValueError: continue else: holi_i.append(hsx) # holiday indexes to be used on retrv. del trd_d[hsx] # Get rid holidays leaving only Tradins tgt_date = chkdate('19970422') # This my Target date try: dsx = trd_d.index(tgt_date) # See if the date is one of index Trd dates except ValueError: dsx = add.reduce(less(trd_d, tgt_date)) # if not goto next date dte_fwd = 6 # I need 5 dates forward prices dte_bwd = 5 # I need backward as well trd_date = trd_d[dsx-dte_bwd:dsx+dte_fwd] # get Trading Range st_d = dates.index(trd_date[0]) # get date range, since that's how it's stored ed_d = dates.index(trd_date[-1]) h_sx = nonzero(less_equal(holi_i,ed_d)*greater_equal(holi_i,st_d)) # find Holidays Index inside the range, I will use to del later stocks = ['ITX','IBM','AAPL'] # my ticker list result = [] for stk in stocks: iadj = dates.index(d[stk]['began']) # Time Series began, so need to adj. close = d[stk]['close'] # get closing prices retrv = close[st_d-iadj:ed_d-iadj+1] # get only section interested in for hidx in h_sx: # getrid of all holidays in retrv del retrv[hidx] if retrv[0] == None: # if first one is None bk=0 while st_d-iadj > iadj: # go back in time to get last know P bk=bk+1 if close[st_d-iadj-bk] != None: retrv[0] = close[st_d-iadj-bk] break while None in retrv: # if there are None values try: Nsx = retrv.index(None) except ValueError: break # no more None's done! else: retrv[Nsx] = retrv[Nsx-1] # back fill using old prices if len(retrv) == len(trd_date): result.append(retrv) else: print 'Error Wrong size!' break result = array(result) # turn it into a Numeric array! _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From motteler@laura.llnl.gov Tue Jul 8 20:50:23 1997 From: motteler@laura.llnl.gov (Zane C. Motteler) Date: Tue, 8 Jul 1997 12:50:23 -0700 (PDT) Subject: [MATRIX-SIG] Much ado about nothingness. In-Reply-To: <199707081904.PAA05746@ptq3.etsd> Message-ID: Hi Hoon, I've been following your correspondence with Aaron. It seems to me that you are missing an important concept in criticizing NumPy (and other Python functions) for not dealing with None. The primary purpose in creating Numerical arrays is to have a PyObject containing contiguous numerical data that can be rapidly processed in compiled code. The elements of a NumPy array must ipso facto all be numbers of the same type (e. g., real, integer, etc.). None, however, is not a number of any type but rather is a PyObject. This it seems as if you are asking for NumPy arrays which may contain elements of different types. But if you want arrays which contain objects of different types, then a Python list is about all we have available, and then of course you don't have handy matrix operations, etc. It seems to me that if you want array-like objects that can contain PyObjects as well as numbers, and yet can somehow meaningfully have array functions and operations applied to them, then you need your own Python extension defining a new type. You can write your own or hire somebody to do it.... ;-} (I could, I think it would be an interesting exercise, but I don't have the time....) NumPy was designed just for one purpose, namely to do fast numerical calculations with non-sparse arrays containing objects of just one numerical type. I think what you are asking for is a horse of a different color. I hope that I have helped to clarify matters rather than muddy the Watters. (Oops, pardon the pun, Aaron.) Cheers Zane ----------------------------------------------------------------------------- Zane C. Motteler, Ph. D. ___________ALSO:_________________ Computer Scientist | Professor Emeritus of Computer| Lawrence Livermore National Laboratory | Science and Engineering | P O Box 808, L-472 | California Polytechnic State | Livermore, CA 94551-9900 | University, San Luis Obispo | 510/423-2143 --------------------------------- FAX 510/423-9969 zcm@llnl.gov or motteler@laura.llnl.gov or zmottel@calpoly.edu _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From hyoon@nyptsrv1.etsd.ml.com Tue Jul 8 21:23:33 1997 From: hyoon@nyptsrv1.etsd.ml.com (Hoon Yoon - IPT Quant) Date: Tue, 8 Jul 1997 16:23:33 -0400 Subject: [MATRIX-SIG] Much ado about nothingness. Message-ID: <199707082023.QAA05787@ptq3.etsd> Hello Prof, Well, I think what you just illustrated was difference between a Physical scientist and Social scientist(if one could call it that). Whether it's econometrics, taking poles on election, or analyzing stock trends, we take missing data as a matter of fact. There is no control over what tests we can perform and data is always full of=20 holes. No experiment or model is really controlled. So, there goes 80% of our time. I freely admit that both of you are much more capable with systems=04=04= and especially at Python; however, when it comes to financial data, I think I can claim some expertise. And every statistical package designed for social science have heavy weight on how to deal with missing data. And I know this is the biggest concerns for financial modeling, more so than anything else I know. How else would I get to know all this different missing types? I am an Econ major, who only took 2 computer classes (but I still think I am right). Oh, about our earlier correspondence about Narcisse. I meant to = purchase Narcisse package, if license could be purchased. Being a Berkeley = graduate, I have great respect for Triple L. I would not presume to just hire one = on demand. Besides getting consulting contract around here is HELL! Thanks=20 for the thought, however. I still think missing values should be part of the matrix. It should probably not None, since as you have mentioned that it introduces additional Python object inside the array. Something little more elegent would be nice. I guess I could process the info, prior to putting into the array, but I would much rather deal with a matrix manipulation. Thanks for your thought, I really appreciate your input. Hoon, _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From hugunin@javanet.com Tue Jul 8 21:43:26 1997 From: hugunin@javanet.com (Jim Hugunin) Date: Tue, 08 Jul 1997 16:43:26 -0400 Subject: [MATRIX-SIG] Much ado about nothingness. References: <199707082023.QAA05787@ptq3.etsd> Message-ID: <33C2A66E.7D2D192C@javanet.com> I think that Zane did a great job of summarizing the intentions of NumPy to provide efficient dense raw numeric arrays, not to provide each particular brand of array that anyone might want. In your particular case what you need to do is write a small Python class that wraps a numeric array and provides the extra functionality you need for missing data. Internally it would hold two arrays, one of the actual data, and another one of ones and zeros indicating which data was valid (just like Aaron suggested). If you want to use NumPy on arrays of missing data you will need to write this Python class yourself. The point of NumPy is to make sure that people have the efficient underlying array objects that they need to build these higher level abstractions. If you don't have the time/inclination to build this small wrapper class yourself, then NumPy is the wrong package for you. If you do, perhaps you could contribute your code back to the community so that more people can benefit from it. -Jim _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From phil@geog.ubc.ca Tue Jul 8 21:52:31 1997 From: phil@geog.ubc.ca (Phil Austin) Date: Tue, 8 Jul 1997 13:52:31 -0700 Subject: [MATRIX-SIG] Much ado about nothingness. In-Reply-To: <199707082023.QAA05787@ptq3.etsd> References: <199707082023.QAA05787@ptq3.etsd> Message-ID: <199707082052.NAA22223@curlew.geog.ubc.ca> >>>>> "Hoon" == Hoon Yoon writes: Hoon> I still think missing Hoon> values should be part of the matrix. It should probably not Hoon> None, since as you have mentioned that it introduces Hoon> additional Python object inside the array. Something little Hoon> more elegent would be nice. I guess I could process the Hoon> info, prior to putting into the array, but I would much Hoon> rather deal with a matrix manipulation. Thanks for your Hoon> thought, I really appreciate your input. There was some discussion on this list about a year ago concerning standard IEEE NaN's (this undelies Splus' null data handler I think). Concerns were expressed about performance and portability, but if those issues were manageable then there's an obvious need for Python NaN capability in signal/image processing applications. Phil _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From hyoon@nyptsrv1.etsd.ml.com Wed Jul 9 15:01:08 1997 From: hyoon@nyptsrv1.etsd.ml.com (Hoon Yoon - IPT Quant) Date: Wed, 9 Jul 1997 10:01:08 -0400 Subject: [MATRIX-SIG] Much ado about nothingness. Message-ID: <199707091401.KAA06541@ptq3.etsd> Gentlemen: I know some people are bit frustrated with this. And my earlier posts have been some what compremised by my lack of underlying mechanics. First of all, I added None Python object to my array not because I thought adding this object was appropriate, but this was about the most closes All the other stuff like filling in the last known value, 'hot-record-replacement', statistical inference via EM algorithm, robust pseudo-values, etc., are useful in their place and _mutually contradictory_. t thing to NaN or missing in Gauss, Ox, and Splus (my matrix langs) that I could find. I actually thought it was a bad idea, because it changed my matrix to complex and it should have been basic Int type. Cost of this overhead was clearly high. As Andrew mentioned: All the other stuff like filling in the last known value, 'hot-record-replacement', statistical inference via EM algorithm, robust pseudo-values, etc., are useful in their place and _mutually contradictory_. (snip) How I would deal with missing data once identified is completely depended on kind of modeling I do. I was illustrating some basic methodology that I would use this missing identifier. Once missing has been specified, I should know what to do with it. As I have said no-package out there deals with msgs well enough, but also there is no stat package out there without a missing identifier inside an array that I know of. (I don't know mathlab, but above mentioned plus SAS has one.) As everyone here realizes there are a lot of missings and some can be easily controlled before it goes in. Some can't be deal with untill it screws up your model. A very large number, often created by dividing by near zero often creeps up. To generate this number you just need to do a operation on two seeming innoncent numbers. Missing in other package is an error code, which can be operated on but simply returns missing error code instead of blowing up the program. msg > 1 ? = msg msg==msg ? = msg msg+1 = msg msg/1 = msg 1/msg = msg add.reduce([1, 3, 4, msg, 4]) = msg (If I want the answer, msg should either be taken out or turn to some other number. As may said, this should be done prior, but too often how this is done is completely conditional on other matrix and I also have to update other matricies as well). Any operation done on msg will turn up msg, because it's an error condition. And this goes back to garbage collection and ease of developemnt issues. I am not sure [0,1] shadow matrix could deal with this adaquetly. I guess I still have not thought and know about these methods. I got 256MB on my Sparc and Pentium, the reason I have it is because financial data especially realtime is extremely vast with a lot of missing data points. I hope some people are not suggesting that NumPy is for computer scientist types only and whatever I should find for my use I have to build one. As I have said, every matrix language I ever used of looked had missing. If someone told me that particular package does not have a some specific statistical method like Logit, I would understand and build my own, but missing is way too low level for someone like me. Frankly, I would love to pay someone to do this; however, lack of organized structure in Python is problem. I believe Python is a wonderful language and due to its free and open nature, I would like to be able to pay someone to get this going and contribute it back to the community. This would benefit both my employer and community quite well. Merrill, however, will not deal with individuals and have hard time dealing with smalll operators off the Web. I had Ox developer contacted and we agreed on a very nice fat support contract recently only to have it turned down when the purchasing called the good professor and found out that he works by himself in his office. Never mind the fact that Ox is extremely fast and the developer was incredibly helpful and only reason we agreed on a nice support contract was because I could not get a consulting contract. The purchasing basically said we do not deal with individuals, look for something else. I would love to have someone come up with a support company for Python with some affilliation to some established entity. Pay the person to added needed functionality and contribute general purpose codes back to commnunity. I can reasonably do this every beginning of fiscal year on a contracted support. Any takers? ;) I guess without the missing issues settled, I will probably need to link in some other package for analytics and graphics to Python if possible and only use NumPy for Derivative Calcs, where the inputs are less uncertain than Portfolio Trading. It's not a perfect solution, but nothing is so uncertain. Hoon, P.S.: I still welcome any more input to putting NA as an additinal class. I don't think I am really capable yet, but I would like to contribute something back if I can. Oh, my earlier posts about using Gauss like operators in Python is clearly whining. I should just get used to my new environment, but, unless NumPy is not for social science and financial modeling, I believe I have justifiable complaint given so many other languages I've used in past. _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From da@maigret.cog.brown.edu Wed Jul 9 15:42:55 1997 From: da@maigret.cog.brown.edu (David Ascher) Date: Wed, 9 Jul 1997 10:42:55 -0400 (EDT) Subject: [MATRIX-SIG] Much ado about nothingness. In-Reply-To: <199707091401.KAA06541@ptq3.etsd> Message-ID: On Wed, 9 Jul 1997, Hoon Yoon - IPT Quant wrote: [ a lot of stuff ] > I am not sure [0,1] shadow matrix could deal with this adaquetly. I believe most of us believe that it can. The reason no one has done it yet (that I know) is that 1) we don't need it for our 'day jobs', and 2) our TODO lists are big. =) > Frankly, I would love to pay someone to do this; however, lack of > organized structure in Python is problem. I believe Python is a > wonderful language and due to its free and open nature, I would like to > be able to pay someone to get this going and contribute it back to the > community. This would benefit both my employer and community quite well. > Merrill, however, will not deal with individuals and have hard time > dealing with smalll operators off the Web. This situation is recognized, and will change. --david _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From phil@geog.ubc.ca Wed Jul 9 17:34:23 1997 From: phil@geog.ubc.ca (Phil Austin) Date: 09 Jul 1997 09:34:23 -0700 Subject: [MATRIX-SIG] namespaces, constructors, and global variables Message-ID: The following message is a courtesy copy of an article that has been posted as well. We're working on a generic plotting interface to gnuplot, gist, ncar graphics, etc., and are currently stumped by our lack of understanding of namespaces. The interface puts a plot together by creating a series of objects (plots, axes, ticks, legends, etc.) with methods to add or modify fonts, colors, line widths, whatever. The state of the plot is stored in a global dictionary, and an execute method goes through the dictionary and builds the plot for the appropriate software package. An example: >>> from PlotAll import * >>> plotIt=PlotDriver(device="screen") >>> plot1=BuildPlot(time, xarray) >>> leftAxis=BuildAxis(plot1,axis="left",label="xarray") >>> print Master['plot1']['leftAxis']['label'] xarray Our problem is that we would like a botAxis and a leftAxis object to be created by default when a plot object is instantiated without the argument "axes=False". I'm appending a version of PlotAll.py, which does this without a hitch from within the interpreter: >>> import PlotAll >>> PlotAll.runit() Master.keys(): ['plotCount', 'device', 'plot1'] inside BuildAxis: Master.keys()= ['plotCount', 'device', 'plot1'] inside BuildAxis: Master.keys()= ['plotCount', 'device', 'plot1'] Bottom: Master['plot1']['leftAxis']['label']= ytest >>> leftAxis Running from the shell, however, doesn't work, because the global variable Master doesn't appear to have it's keys: python PlotAll.py Master.keys(): ['plotCount', 'device', 'plot1'] inside BuildAxis: Master.keys()= [] Can someone walk me through why this is failing, and how I can fix it? Thanks, Phil ______________________________PlotAll.py_______________________________ Master={} global Master class PlotDriver: global Master def __init__(self,device="postscript"): self.RCSId="$Id: PlotAll.py,v 1.5 1997/07/09 16:31:05 phil Exp $" Master['plotCount']=0 Master['device']=device class BuildPlot: global Master def __init__(self,xdata,ydata,plotType="linePlot",axes="default"): Master['plotCount'] = Master['plotCount'] + 1 self.plotName="plot%d" % Master['plotCount'] Master[self.plotName]={} Master[self.plotName]['xdata']=xdata Master[self.plotName]['ydata']=ydata Master[self.plotName]['plotType']=plotType if(axes=="default"): import __main__ plot=self global Master ns=vars() print "Master.keys(): ", Master.keys() code=["from PlotAll import *", "global Master", "leftAxis=BuildAxis(plot,axis=\"left\",label=\"ytest\")", "botAxis=BuildAxis(plot,axis=\"bot\",label=\"x\")"] for stmt in code: exec stmt in ns,__main__.__dict__ class BuildAxis: global Master def __init__(self,Plot,axis="left",label="default",limit="default"): global Master print "inside BuildAxis: Master.keys()=", Master.keys() self.plotName=Plot.plotName self.axisName="%sAxis" % axis Master[self.plotName][self.axisName]={} Master[self.plotName][self.axisName]['label']=label Master[self.plotName][self.axisName]['limit']=limit def runit(): global Master xarray=[1,2,3,4] time=[5,6,7,8] #controls items common to all plots plotIt=PlotDriver(device="screen") #build an individual plot plot=BuildPlot(time, xarray) print "Bottom: Master['plot1']['leftAxis']['label']=", Master['plot1']['leftAxis']['label'] if __name__ == "__main__":runit() _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From ted_horst@il.us.swissbank.com Wed Jul 9 19:15:41 1997 From: ted_horst@il.us.swissbank.com (Ted Horst) Date: Wed, 9 Jul 97 13:15:41 -0500 Subject: [MATRIX-SIG] Much ado about nothingness. References: <199707091401.KAA06541@ptq3.etsd> Message-ID: <9707091815.AA18826@ch1d162nwk> Has anybody looked at the Missing type from DigiCool ? It has the desired behavior for arithmetic operations, but doesn't work correctly for math or umath functions. I have no idea how hard this would be to get working, or even if this is the proper approach, but somebody else is at least looking at the problem. You can get it from http://www.digicool.com/releases/ExtensionClass/. Comments ? Ted Horst (not speaking for SwissBank) _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From ted_horst@il.us.swissbank.com Wed Jul 9 19:15:41 1997 From: ted_horst@il.us.swissbank.com (Ted Horst) Date: Wed, 9 Jul 97 13:15:41 -0500 Subject: [MATRIX-SIG] Much ado about nothingness. References: <199707091401.KAA06541@ptq3.etsd> Message-ID: <9707091815.AA18826@ch1d162nwk> Has anybody looked at the Missing type from DigiCool ? It has the desired behavior for arithmetic operations, but doesn't work correctly for math or umath functions. I have no idea how hard this would be to get working, or even if this is the proper approach, but somebody else is at least looking at the problem. You can get it from http://www.digicool.com/releases/ExtensionClass/. Comments ? Ted Horst (not speaking for SwissBank) _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From phil@geog.ubc.ca Wed Jul 9 19:37:28 1997 From: phil@geog.ubc.ca (Phil Austin) Date: 09 Jul 1997 11:37:28 -0700 Subject: [MATRIX-SIG] making a text label from an object instance? Message-ID: The following message is a courtesy copy of an article that has been posted as well. For my plotting package I would like to have the default axes labels be the names of the objects being plotted. For example, if the command is: plot(time[:,4],vel[:,3,2]) I would like to generate strings containing "time[:,4]" and "vel[:,3,2]". Is this possible with Python? For those who know Splus, I'm looking for the equivalent of deparse(substitute(time[:,4])) Regards, Phil -- ------------------------------------------------------------ Phil Austin (phil@geog.ubc.ca) Department of Geography, Tel: (604) 822-2663 University of British Columbia, B.C. Fax: (604) 822-6150 _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From jim.fulton@digicool.com Wed Jul 9 20:57:39 1997 From: jim.fulton@digicool.com (Jim Fulton) Date: Wed, 09 Jul 1997 15:57:39 -0400 Subject: [MATRIX-SIG] Much ado about nothingness. References: <199707091401.KAA06541@ptq3.etsd> <9707091815.AA18826@ch1d162nwk> Message-ID: <33C3E5D9.46F5@digicool.com> Ted Horst wrote: > > Has anybody looked at the Missing type from DigiCool ? It has the desired > behavior for arithmetic operations, but doesn't work correctly for math or > umath functions. I have no idea how hard this would be to get working, or even > if this is the proper approach, but somebody else is at least looking at the > problem. You can get it from http://www.digicool.com/releases/ExtensionClass/. Thankyou for mentioning this. I guess this is the nudge I need to add a couple of $0.01. In part, the Missing module in ExtensionClass is intended to fulfill a promise I made on this list too long ago. When I used to work at USGS, I had to deal with missing values alot. For my needs, I considered any database or data analysis tool, including ones I wrote, that didn't make provision for missing data to be almost useless. The way I've seen this dealt with is to reserve some special values (not necessarily IEEE standard ones), as missing values. One package I dealt with (P-Stat) actually had three special vallues, Missing1 (123456e20), Missing2 (123457e20), and Missing3 (123458e20), that could be used to represent different kinds of missing data. For example, this package used Missing3 for things like log(0). In any case, all computations had to check for missing values. While this adds significantly to the complexity of numerical software, and slightly reduces it's efficiency, I think the gain is well worth the price. I think that it would be a good thing for NumPy to reserve one or more special floating point missing values internally and should expose these in Python using my (truly trivial) Missing module or something like it. I realize that this would be a significant undertaking, although perhaps there is available funding for it. I feel odd making this suggestion, given that, in my current line of business, I have little stake in the outcome. Jim -- Jim Fulton Digital Creations jim@digicool.com 540.371.6909 ## Python is my favorite language ## ## http://www.python.org/ ## _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From jody@sccsi.com Wed Jul 9 20:53:10 1997 From: jody@sccsi.com (Jody Winston) Date: Wed, 9 Jul 1997 14:53:10 -0500 Subject: [MATRIX-SIG] Much ado about nothingness. In-Reply-To: <9707091815.AA18826@ch1d162nwk> (message from Ted Horst on Wed, 9 Jul 97 13:15:41 -0500) Message-ID: <199707091953.OAA27040@friday.sccsi.com> Here's a simple Missing class: import sys import types class Missing: """An object that defines math operations on missing values. """ def __init__(self, value=0, missing=-999.25): """This method constructs the object. The following exceptions are thrown: o If the value is not number. """ if not((type(value) != types.IntType) or (type(value) != types.LongType) or (type(value) != types.FloatType) or (type(value) != types.ComplexType)): raise TypeError, "Not number" self.value = value self.missing = missing def absent(self): return self.value == self.missing def __eitherMissing(self, other): if self.absent(): return 1 elif other.absent(): return 1 def __repr__(self): """Called by the print statement and conversions (reverse quotes) to compute the string representation of an object.""" return str(self.value) + \ " (" + \ "missing = " + str(self.missing) + \ ")" def __str__(self): return str(self.value) + \ " (" + \ "missing = " + str(self.missing) + \ ")" def __sum(self, other, sign1, sign2): if isMissing(other): if self.__eitherMissing(other): result = self.missing else: result = sign1 * self.value + sign2 * other.value missing = self.missing else: if self.absent(): result = self.missing else: result = sign1 * self.value + sign2 * other missing = self.missing return Missing(result, missing) def __add__(self, other): return self.__sum(other, 1, 1) __radd__ = __add__ def __sub__(self, other): return self.__sum(other, 1, -1) def __rsub__(self, other): return self.__sum(other, -1, 1) def __cmp__(self, other): if isMissing(other): if self.absent() and other.absent(): return 0 diff = self.__sum(other, 1, -1) return cmp(diff.value, 0) def __mul__(self, other): if isMissing(other): if self.__eitherMissing(other): result = self.missing else: result = self.value * other.value missing = self.missing else: if self.absent(): result = self.missing else: result = self.value * other missing = self.missing return Missing(result, missing) __rmul__ = __mul__ def __div__(self, other): if isMissing(other): if self.__eitherMissing(other): result = self.missing else: result = self.value / other.value missing = self.missing else: if self.absent(): result = self.missing else: result = self.value / other missing = self.missing return Missing(result, missing) def __rdiv__(self, other): if isMissing(other): if self.__eitherMissing(other): result = self.missing else: result = other.value / self.value missing = self.missing else: if self.absent(): result = self.missing else: result = other / self.value missing = self.missing return Missing(result, missing) def __pow__(self, other): if isMissing(other): if self.__eitherMissing(other): result = self.missing else: result = pow(self.value, other.value) missing = self.missing else: if self.absent(): result = self.missing else: result = pow(self.value, other) missing = self.missing return Missing(result, missing) def __rpow__(self, other): if isMissing(other): if self.__eitherMissing(other): result = self.missing else: result = pow(other.value, self.value) missing = self.missing else: if self.absent(): result = self.missing else: result = pow(other, self.value) missing = self.missing return Missing(result, missing) def __abs__(self): if self.absent(): return Missing(self.value, self.missing) return Missing(abs(self.value), self.missing) def __pos__(self): return self def __neg__(self): if self.absent(): return Missing(self.value, self.missing) return Missing(-self.value, self.missing) def __nonzero__(self): return self.value != 0 def isMissing(x): return hasattr(x, "value") and hasattr(x, "missing") def main(argv): a = Missing() if (a + 10) != 10: raise ValueError b = Missing(20, -100) if (a + b) != Missing(20): raise ValueError if __name__ == '__main__': main(sys.argv) sys.exit(0) _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From da@maigret.cog.brown.edu Wed Jul 9 21:18:50 1997 From: da@maigret.cog.brown.edu (David Ascher) Date: Wed, 9 Jul 1997 16:18:50 -0400 (EDT) Subject: [MATRIX-SIG] Much ado about nothingness. In-Reply-To: <199707091953.OAA27040@friday.sccsi.com> Message-ID: On Wed, 9 Jul 1997, Jody Winston wrote: > Here's a simple Missing class: > [...] Note that a version of the Numeric and umath modules which know about Missing's is also needed, so that take(x,y) and log(x) do the right thing. --da _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From jim.fulton@digicool.com Wed Jul 9 21:35:49 1997 From: jim.fulton@digicool.com (Jim Fulton) Date: Wed, 09 Jul 1997 16:35:49 -0400 Subject: [MATRIX-SIG] Much ado about nothingness. References: Message-ID: <33C3F625.188A@digicool.com> David Ascher wrote: > > On Wed, 9 Jul 1997, Jody Winston wrote: > > > Here's a simple Missing class: > > [...] > > Note that a version of the Numeric and umath modules which know about > Missing's is also needed, so that take(x,y) and log(x) do the right thing. Right. Basically, *anything* that deals with data at the C level would have to check for missing data. This is especially problematic when dealling with external libraries. Jim -- Jim Fulton Digital Creations jim@digicool.com 540.371.6909 ## Python is my favorite language ## ## http://www.python.org/ ## _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From vennerbc@musc.edu Thu Jul 10 00:36:12 1997 From: vennerbc@musc.edu (Bradley C. Venner) Date: Wed, 09 Jul 1997 19:36:12 -0400 Subject: [MATRIX-SIG] Much ado about nothingness. References: <199707091401.KAA06541@ptq3.etsd> <9707091815.AA18826@ch1d162nwk> Message-ID: <33C4206C.167E@musc.edu> I have found this discussion very interesting. As an academic, though, I feel the need for more authority-citing. So I will pull out two big guns, one from the database field and one from the statistical community, on the representation of missing data. >From the database side, C. J. Date has been the biggest critic of a default value for "NULLS" in SQL. He has a recent 5-part column in Database Programming and Design on this subject which is very informative. My misrepresentation of his position is that no default scheme is available to handle missing information. His view of relational databases is that the fields in a database are considered to be a 'type'. Each type may have one or more default types, but that operators on these default values must be written. >From the statistical side, Rubin's work on multiple imputation expresses a sympathetic concern. To do multiple imputation one must specify a mechanism for why the data is missing. The other known data values may then be used to make up new values for the missing data. However, to acknowledge that one made these values up, the proceedure is repeated several times, so that several equally likely data sets are created. Statistics can be computed by applying your analysis not only to just one dataset but to several. Things like confidence intervals may then represent the uncertainty introduced by the missing data. For example, the new NHANES data tapes will contain several datasets, forcing the investigator to confront that his answer may depend on which set he chooses. In practice, I think an intermediate class with strong NumPy <-> database bindings is probably necessary for analysis purposes. (ahh, indirection, your friend and mine) Such a class, a la data.frame in S-Plus, could represent both categorical and numeric data. Methods for this class for converting to NumPy arrays for use in analysis would force the user to decide how to treat the missing values. Defaults such as dropping rows with missing data could be available. I think such a class would have a number of uses outside of missing data applications, particularly for data mining. The suggestions on defining a new class so far are a good place to start. _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From amullhau@ix.netcom.com Thu Jul 10 02:24:42 1997 From: amullhau@ix.netcom.com (Andrew P. Mullhaupt) Date: Wed, 09 Jul 1997 21:24:42 -0400 Subject: [MATRIX-SIG] Much ado about nothingness. In-Reply-To: <33C4206C.167E@musc.edu> References: <199707091401.KAA06541@ptq3.etsd> <9707091815.AA18826@ch1d162nwk> Message-ID: <3.0.1.32.19970709212442.0080d380@popd.netcruiser> At 07:36 PM 7/9/97 -0400, Bradley C. Venner wrote: >a sympathetic concern. To do multiple imputation one must specify a >mechanism for why the data is missing. The other known data values may >then be used to make up new values for the missing data. Nothing like this should be _part_ of the language, since it is not going to be possibly to specify a mechanism which is good in all cases. What you _can_ put into the language is an efficient way to determine that there _are_ missing values, and (in many cases), a good way to determine _which_ values are missing. Everything else should be done by the user on top of this, and will vary greatly from application to application. >In practice, I think an intermediate class with strong NumPy <-> >database bindings is probably necessary for analysis purposes. Interestingly enough, I do not expect ever to use a "database" in the classical sense. The reason is that most database models offer very poor performance in the applications I use - large scale financial computing. What I have done in serveral different platforms is to keep the data in a form on which my applications directly operate. As a result, I do all sorts of statistical computations, yet no database is even remotely important, or even useful. We've always _wanted_ to find a database which could measure up, (I once spent two years evaluating all the remotely viable choices), but I don't think I can be competitive using a database in the usual sense. It may be somewhat surprising but the majority of companies in my area do the same thing. > Such a class, a la data.frame in >S-Plus, could represent both categorical and numeric data. Methods for >this class for converting to NumPy arrays for use in analysis would >force the user to decide how to treat the missing values. The data.frame class is a powerful tool, and something like it will be implemented in Python, probably sooner, rather than later. Right now the hitch is mostly what to do about indexing. Although I enthusiastically support a data.frame class for python, I see no reason why this bears on whether or not a user cares about missing values. People with missing values will expect the data.frame to treat missing values consistently with the methods from the parent classes, and so _those_ methods are where the issue crops up. > Defaults such >as dropping rows with missing data could be available. I think such a >class would have a number of uses outside of missing data applications, >particularly for data mining. The suggestions on defining a new class >so far are a good place to start. Just make sure that particular methods of handling missing data are part of a subclass, _not_ part of the data.frame class itself. You won't have a good single method of handling missing data for most applications - even if you are talking about handling the exact same missing value instance being acted on by the exact same methods but in different situations. What tends to be _easy_ to standardize is a mechanism for representing missing values. It's definitely not a good idea to try to standardize the methods of handling missing values, which is as application dependent as anything I can imagine. Later, Andrew Mullhaupt _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From hinsen@ibs.ibs.fr Thu Jul 10 10:21:04 1997 From: hinsen@ibs.ibs.fr (Konrad Hinsen) Date: Thu, 10 Jul 1997 11:21:04 +0200 Subject: [MATRIX-SIG] Much ado about nothingness. In-Reply-To: <9707091815.AA18826@ch1d162nwk> (message from Ted Horst on Wed, 9 Jul 97 13:15:41 -0500) Message-ID: <199707100921.LAA18102@lmspc1.ibs.fr> > Has anybody looked at the Missing type from DigiCool ? It has the desired Noticed, but not looked at... > behavior for arithmetic operations, but doesn't work correctly for math or > umath functions. I have no idea how hard this would be to get working, or even Umath support is easy: just implement methods for each function. For any unknown type, umath tries to translate the function into a method call, i.e. sin(x) --> x.sin() exp(x) --> x.exp() Anyone implementing numerical types, in C extensions or as Python classes, should provide this set of methods. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28 Institut de Biologie Structurale | Fax: +33-4.76.88.54.94 41, av. des Martyrs | Deutsch/Esperanto/English/ 38027 Grenoble Cedex 1, France | Nederlands/Francais ------------------------------------------------------------------------------- _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From hinsen@ibs.ibs.fr Thu Jul 10 10:39:04 1997 From: hinsen@ibs.ibs.fr (Konrad Hinsen) Date: Thu, 10 Jul 1997 11:39:04 +0200 Subject: [MATRIX-SIG] namespaces, constructors, and global variables In-Reply-To: (message from Phil Austin on 09 Jul 1997 09:34:23 -0700) Message-ID: <199707100939.LAA18142@lmspc1.ibs.fr> > We're working on a generic plotting interface to gnuplot, gist, ncar > graphics, etc., and are currently stumped by our lack of understanding Great! However, I wonder whether a unification of plotting packages is the right way to go. I think it would be not much more effort, but much more useful, to write Python code that generates low-level graphics elements (lines, circles, etc.) directly to make up a plot. Then there could be several low-level drivers that render this representation via Tk canvases, OpenGL, PostScript, or whatever. This may sound like reinventing wheels, but given the problems and/or limitations that all known plotting packages cause, it may be the better way to happiness. > of namespaces. The interface puts a plot together by creating a > series of objects (plots, axes, ticks, legends, etc.) with methods to > add or modify fonts, colors, line widths, whatever. The state of the > plot is stored in a global dictionary, and an execute method goes I would not recommend to use *any* global objects unless you have a very good reason. If I understand your intention well, it would be impossible to construct two plots in parallel. Why that restriction? Just define a "plot" class that contains all the data. > Running from the shell, however, doesn't work, because the global > variable Master doesn't appear to have it's keys: > > python PlotAll.py > Master.keys(): ['plotCount', 'device', 'plot1'] > inside BuildAxis: Master.keys()= [] > > Can someone walk me through why this is failing, and how I can fix it? Without analyzing anything in detail, I'll point to a critical spot: > class BuildPlot: ... > code=["from PlotAll import *", > "global Master", > "leftAxis=BuildAxis(plot,axis=\"left\",label=\"ytest\")", > "botAxis=BuildAxis(plot,axis=\"bot\",label=\"x\")"] > for stmt in code: > exec stmt in ns,__main__.__dict__ __main__ will clearly be different in the two situations. And like global variables, accessing __main__ is something that a well-behaved Python program should not do. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28 Institut de Biologie Structurale | Fax: +33-4.76.88.54.94 41, av. des Martyrs | Deutsch/Esperanto/English/ 38027 Grenoble Cedex 1, France | Nederlands/Francais ------------------------------------------------------------------------------- _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From hinsen@ibs.ibs.fr Thu Jul 10 10:43:22 1997 From: hinsen@ibs.ibs.fr (Konrad Hinsen) Date: Thu, 10 Jul 1997 11:43:22 +0200 Subject: [MATRIX-SIG] making a text label from an object instance? In-Reply-To: (message from Phil Austin on 09 Jul 1997 11:37:28 -0700) Message-ID: <199707100943.LAA18154@lmspc1.ibs.fr> > For my plotting package I would like to have the default axes labels > be the names of the objects being plotted. For example, if the > command is: > > plot(time[:,4],vel[:,3,2]) > > I would like to generate strings containing "time[:,4]" and > "vel[:,3,2]". Is this possible with Python? No. And it's not even clear what the outcome should be in more complicated situations, such as: def foo(x): apply(plot, 2*(x,)) I wouldn't worry too much about default labels. I find them rarely useful. It's simple enough to pass an explicit string. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28 Institut de Biologie Structurale | Fax: +33-4.76.88.54.94 41, av. des Martyrs | Deutsch/Esperanto/English/ 38027 Grenoble Cedex 1, France | Nederlands/Francais ------------------------------------------------------------------------------- _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From arw@dante.mh.lucent.com Thu Jul 10 12:42:37 1997 From: arw@dante.mh.lucent.com (Aaron Watters) Date: Thu, 10 Jul 1997 07:42:37 -0400 Subject: [MATRIX-SIG] Much ado about nothingness. Message-ID: <199707101148.HAA04852@dante.mh.lucent.com> ---------- > From: Konrad Hinsen ... > Umath [null] support is easy: just implement methods for each function. For any > unknown type, umath tries to translate the function into a method call, i.e.... Nothing about nulls is easy, as CJ Date will confirm (although beyond that he's usually wrong regarding Nulls imho :) ). Basically, the standard database theory breaks down and becomes totally useless when nulls are introduced and treated seriously. CJ Date and the like tend to conclude that "nulls are bad" and just throw them out (a convenience only a theoretician has). EG Consider the table name age john 5 fred 6 lisa 5 wally null lola null select those names with age 5. IMHO the correct answer it *two* sets inner = {john,lisa}, outer={john, fred, lisa, lola} The "correct" answer is somewhere in between (if we had better information). what is the average age? null, I think. rephrased, what is the average among the known ages? (5+6+5)/3 (wally, lola thrown out). Thus for accumulation calculations (including len) null values sometimes should be thrown out, ie known_sum([1,2,3,Null,3]) = 9 known_min_len([1,3,3,Null,3]) = 4 (not 5!) here I say "known min len" because more generally the Null might be a place holder that stands for *several* missing values. On the other hand there are "not applicable" nulls too item pages ipwp 300 pp 900 win95 NA what are the average number of pages? (900+300)/2 what has 900 pages? {pp} (win95 doesn't enter the calculation, it has no pages) "unknown" Nulls are also a special case of more general "partial information" values, such as numeric intervals, where sum([interval(5,6), interval(7,9)]) = interval(12,15) Gosh, you could write a whole dissertation on the topic (but don't, I already did :) ). In sum, I suspect there is no quicky answer for treating nulls -- there might be uniform approaches, but I'd like to understand the required semantics very clearly before modifying numeric to support them. I suspect differing apps require radically differing semantices... -- Aaron Watters _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From arw@dante.mh.lucent.com Thu Jul 10 12:44:47 1997 From: arw@dante.mh.lucent.com (Aaron Watters) Date: Thu, 10 Jul 1997 07:44:47 -0400 Subject: [MATRIX-SIG] namespaces, constructors, and global variables Message-ID: <199707101148.HAA04855@dante.mh.lucent.com> > From: Konrad Hinsen > > We're working on a generic plotting interface to gnuplot, gist, ncar > > graphics, etc., and are currently stumped by our lack of understanding > > Great! However, I wonder whether a unification of plotting packages > is the right way to go. I think it would be not much more effort, but > much more useful, to write Python code that generates low-level graphics > elements (lines, circles, etc.) directly to make up a plot. Then there > could be several low-level drivers that render this representation > via Tk canvases, OpenGL, PostScript, or whatever. Seconded. This is the proper way to design it IMHO. If you do it carefully I don't even think you'll see any particular performance problems. --Aaron Watters _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From phil@geog.ubc.ca Thu Jul 10 22:50:28 1997 From: phil@geog.ubc.ca (Phil Austin) Date: Thu, 10 Jul 1997 14:50:28 -0700 Subject: [MATRIX-SIG] namespaces, constructors, and global variables In-Reply-To: <199707101148.HAA04855@dante.mh.lucent.com> References: <199707101148.HAA04855@dante.mh.lucent.com> Message-ID: <199707102150.OAA03037@curlew.geog.ubc.ca> >>>>> "Konrad" == Konrad Hinsen writes: Konrad> Great! However, I wonder whether a unification of plotting Konrad> packages is the right way to go. I think it would be not Konrad> much more effort, but much more useful, to write Python Konrad> code that generates low-level graphics elements (lines, Konrad> circles, etc.) directly to make up a plot. Then there Konrad> could be several low-level drivers that render this Konrad> representation via Tk canvases, OpenGL, PostScript, or Konrad> whatever. >>>>> "Aaron" == Aaron Watters writes: Aaron> Seconded. This is the proper way to design it IMHO. If Aaron> you do it carefully I don't even think you'll see any Aaron> particular performance problems. --Aaron Watters Thirded. I'm operating under some constraints however: 1) I think I've provided some evidence that, on a coding project like this, I'll be staying in the shallow end of the pool. 2) I need a single interface that will produce simple plots under linux and Windows 95 (student labs start Sept. 3), and camera ready publication plots under linux and solaris. For the latter I require a superset of the capabilities demonstrated at: http://ngwww.ucar.edu/ngdoc/ng4.0.1/ug/ncl/tutor/session3.html (For people interested in pricing for this package--an unlimited use/unlimited-time University site licence cost us about $Can 900 (there's no windows port). A typical commercial alternative with a fairly primitive array language (IDL/PV-Wave) is $2500 per seat). 3) I need the camera-ready plots within two weeks. My guess is that someone might eventually write a portable set of Java/tk/opengl/etc. classes to do much of what I want, and with luck it will happen before 1999 and could even be GPL'ed. Near-term is going to require some improvisation, however. Konrad adds: Konrad> I would not recommend to use *any* global objects unless Konrad> you have a very good reason. If I understand your Konrad> intention well, it would be impossible to construct two Konrad> plots in parallel. Why that restriction? and: Konrad> __main__ will clearly be different in the two Konrad> situations. And like global variables, accessing __main__ Konrad> is something that a well-behaved Python program should not Konrad> do. The plot layout is now encapsulated, so no more globals, thanks. On the second issue, I was trying to get default behavior that produces a plot with very few keystrokes, while maintaining the ability to customize the axis, labels, ticks, etc. after the plot command is typed. Specifically, I'm looking for three different user levels: Novice: _______ BuildPlot(x,y) #x-y line plot with default axes #also creates leftAxis and botAxis--their set methods #can be used to modify defaults if desired as in: Advanced Novice: change the default axis parameters following the plot command _______________ BuildPlot(x,y) leftAxis.set(label="Temperature",font="urdu",fontSize="5pt") Intermediate to Advanced: ________________________ Here, we want a plot with two ordinates of different colors on the rhs, offset horizontally from each other by 0.2 units, one a logrithmic pressure axis with values increasing downward, one a linear height axis with values increasing upward: plot1=BuildPlot(temp,pressure,corner=(0,0),size=(2,2),axes="F",log="y") #no default axes, but plot size and location #specified, ordinate is logarithmic leftAxis=BuildAxis(plot1,axis="bot") rightAxis=BuildAxis(plot1,axis="right",color="red",type="inverted") #now build another plot of temperature/height, positioned on top of #plot1, but draw only a new right axis plot2=BuildPlot(x,z,corner=(0,0),size=(2,2),type="None",axes="F") rightAxis=BuildAxis(plot2,axis="right",bottom=(2.2,0),color="green") ______________ If default behavior that creates an unasked for leftAxis and a botAxis in __main__ is risky, then I can just drop the idea that the default axes can be modified, and move my novices quickly into the intermediate stage if necessary. I'd love to see some other approaches that balance this terseness/flexibility ratio in different ways. Regards, Phil _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From amullhau@ix.netcom.com Fri Jul 11 00:23:03 1997 From: amullhau@ix.netcom.com (Andrew P. Mullhaupt) Date: Thu, 10 Jul 1997 19:23:03 -0400 Subject: [MATRIX-SIG] making a text label from an object instance? In-Reply-To: <199707100943.LAA18154@lmspc1.ibs.fr> References: Message-ID: <3.0.1.32.19970710192303.0081a8d0@popd.netcruiser> At 11:43 AM 7/10/97 +0200, Konrad Hinsen wrote: >I wouldn't worry too much about default labels. I find them rarely >useful. It's simple enough to pass an explicit string. Don't do a lot of automatically generated plots, eh? Default labels are _very_ useful. I don't see a really easy way to do it in python, but it's one of the things people really like when it's available. Later, Andrew Mullhaupt _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From phil@geog.ubc.ca Fri Jul 11 00:49:16 1997 From: phil@geog.ubc.ca (Phil Austin) Date: Thu, 10 Jul 1997 16:49:16 -0700 Subject: [MATRIX-SIG] making a text label from an object instance? In-Reply-To: <3.0.1.32.19970710192303.0081a8d0@popd.netcruiser> References: <199707100943.LAA18154@lmspc1.ibs.fr> <3.0.1.32.19970710192303.0081a8d0@popd.netcruiser> Message-ID: <199707102349.QAA03552@curlew.geog.ubc.ca> >>>>> "Andrew" == Andrew P Mullhaupt writes: Andrew> At 11:43 AM 7/10/97 +0200, Konrad Hinsen wrote: >> I wouldn't worry too much about default labels. I find them >> rarely useful. It's simple enough to pass an explicit string. Andrew> Don't do a lot of automatically generated plots, eh? Andrew> Default labels are _very_ useful. I don't see a really Andrew> easy way to do it in python, but it's one of the things Andrew> people really like when it's available. I'd agree with this--looking at a suprising result in your notebook several weeks after the plot was produced, it's very reassuring to know that there is no possible way the axis labels could have been swapped. The variables in the label can be made more informative using named dimensions and range indexes, so that contours of a 4-d array of vertical velocity don't really need a title: contour(vertical_velocity["500milibars","300Kisotherm",lats,lons]) combined with a time-stamp that can be printed on the plot and used in an RCS comment when you check in the plotting script, it's possible to create 500 fully documented data plots without any programmer intervention, and recover any one of them six months later from the raw netcdf data after your grad student has spilled root beer on the original. There's no looking back though. Splus has the treasured "deparse(substitute())" but they've doubled their site licence fees and seem to be preparing to abandon Unix as a supported platform. Brian Ripley (http://lib.stat.cmu.edu/S/MASS2/), for example, recently posted a message in S-news that sounded positively bitter. Picking up some Splus refugees who happen to be world-renown in bootstrap/LOESS/ARIMA techniques would be major coup for Python (albeit a fairly distant prospect). Regards, Phil _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From hinsen@ibs.ibs.fr Fri Jul 11 11:38:22 1997 From: hinsen@ibs.ibs.fr (Konrad Hinsen) Date: Fri, 11 Jul 1997 12:38:22 +0200 Subject: [MATRIX-SIG] making a text label from an object instance? In-Reply-To: <3.0.1.32.19970710192303.0081a8d0@popd.netcruiser> (amullhau@ix.netcom.com) Message-ID: <199707111038.MAA21796@lmspc1.ibs.fr> > At 11:43 AM 7/10/97 +0200, Konrad Hinsen wrote: > >I wouldn't worry too much about default labels. I find them rarely > >useful. It's simple enough to pass an explicit string. > > Don't do a lot of automatically generated plots, eh? I do, but on the screen. I rarely print plots other than for seminars or publications. > Default labels are _very_ useful. I don't see a really easy way > to do it in python, but it's one of the things people really > like when it's available. OK, so here's how to do them in Python: def plot(x, y): label_x = '' if type(x) == type(''): label_x = x x = eval(x) label_y = '' if type(y) == type(''): label_y = y y = eval(y) do_the_plot(x, label_x, y, label_y) -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28 Institut de Biologie Structurale | Fax: +33-4.76.88.54.94 41, av. des Martyrs | Deutsch/Esperanto/English/ 38027 Grenoble Cedex 1, France | Nederlands/Francais ------------------------------------------------------------------------------- _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From amullhau@ix.netcom.com Fri Jul 11 15:18:35 1997 From: amullhau@ix.netcom.com (Andrew P. Mullhaupt) Date: Fri, 11 Jul 1997 10:18:35 -0400 Subject: [MATRIX-SIG] making a text label from an object instance? In-Reply-To: <199707111038.MAA21796@lmspc1.ibs.fr> References: <3.0.1.32.19970710192303.0081a8d0@popd.netcruiser> Message-ID: <3.0.1.32.19970711101835.00814dd0@popd.netcruiser> At 12:38 PM 7/11/97 +0200, Konrad Hinsen wrote: [ I wrote:] >> Default labels are _very_ useful. >OK, so here's how to do them in Python: >def plot(x, y): Looks good for most uses. Solves that. Later, Andrew Mullhaupt _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From ptaney@usgs.gov Fri Jul 11 15:31:21 1997 From: ptaney@usgs.gov (Paul Taney, Computer Specialist, Reston, VA ) Date: Fri, 11 Jul 1997 10:31:21 -0400 Subject: [MATRIX-SIG] camera-ready Message-ID: <199707111431.KAA26561@mailrvares.er.usgs.gov> Phil: >3) I need the camera-ready plots within two weeks. I cannot suggest programatic solutions for you; you are getting the best advice there! But I am a viz specialist on SGIs here at USGS and I'd like to suggest a very usable alternative to paper plots. Unless you need something quite large (which I do not) prepare final output in this manner; as do my contacts at our primary viz software vendor, Dynamic Graphics: 1) In a windowless room darkened to hide the bezel of monitor 2) With a good, large monitor (this is a cross-platform solution ;) 3) With a 35mm SLR with a 85mm or 105mm lens and heavy tripod -- placing lens exactly on normal line from centroid of monitor 3.1) A Hasselblad would also do... 4) Using 100 ASA Ectachrome slide film -- or some other 100 ASA 5) Shoot screen at f11 for 1+ sec and again at 2 sec ("bracketing") Both shots should be usable in my experience. Look at the cost/time savings over film recorders at $6000+; and quality is as good or better; those monitors are itsy-bitsy. This may not be what you need for some reason, but it is a professional solution. paul ptaney@usgs.gov @ Just when you think you're U.S.Geological Survey @ going to be decorated... reston, va 22092 @ Nothing happens. ph: 703-648-5598 @ -- Chogyam Trungpa _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From chase@att.com Fri Jul 11 17:08:04 1997 From: chase@att.com (Chase, Chris) Date: Fri, 11 Jul 1997 12:08:04 -0400 Subject: [MATRIX-SIG] A proposal (LONG) - "reverse of take" revisited Message-ID: <9707111609.AA28965@hoccson.ho.att.com> Allowing arrays to be indexes (or any sequence object for that matter) in the array module is very doable. But it is a huge amount of work. At different times during the past year I have looked at concepts and implementations for this. But after looking at the code (last year and again this week) it would require rewriting a majority of the routines in the array module and associated files. This is because the use of strides is heavily embeded in the code and there is no separation between array representation and the interface and helper functions. The job is more difficult because the comments about what the code is trying to do are very scarce (but I have a very old version of the sources). Some side comments: Slices that go outside array bounds are silently projected onto the bounds while take() generates an error when the index vector goes outside the bounds. If arrays were allowed as indexes should they behave like slices or like take()? The behavior of the array module is that indexing an array produces a reference to the original data unless a scalar is produced. In allowing arrays as indexes, the array reference representation would have to keep copies of the index array. This does not take any more space than actually producing a copy of the data than producing a reference. Reshaping has to generate a copy unless the input array was already contiguous. If arrays as well as slices are allowed as indexes and kept in the array representation, then an array reshape could also produce a reference to the original data. Arrays as indexes kept in the representation also allow compact representations of array replication (e.g. repeating certain dimensions). I am interested in attempting to make the modifications. But I think that the job is too big (i.e. too time-consuming of my spare time) to complete alone. Also there may be opposition to making extensive changes within the code (although the array object interface would remain the same) because some individuals may depend on the actual representation/implementation. Chris Chase chase@att.com _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From amullhau@ix.netcom.com Fri Jul 11 22:54:39 1997 From: amullhau@ix.netcom.com (Andrew P. Mullhaupt) Date: Fri, 11 Jul 1997 17:54:39 -0400 Subject: [MATRIX-SIG] A proposal (LONG) - "reverse of take" revisited In-Reply-To: <9707111609.AA28965@hoccson.ho.att.com> Message-ID: <3.0.1.32.19970711175439.0081d9c0@popd.netcruiser> At 12:08 PM 7/11/97 -0400, Chase, Chris wrote: > >Slices that go outside array bounds are silently projected onto the >bounds while take() generates an error when the index vector goes >outside the bounds. If arrays were allowed as indexes should they >behave like slices or like take()? > Actually, _functions_ such as take should not be tied to the behavior of indices. You can write all sorts of functions to do anything you want, so the user shouldn't have an expectation of a connection between index semantics and function semantics except for the _very few_ necessary functions which explicitly involve the shape of the array. So for consistency they can behave like slices, but I'm a bit surprised at how you claim slice indices work. So I tried some examples and I think slice indices are broken. For example: Python 1.4 (Feb 24 1997) [C] Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam >>> a = reshape(range(12), [3, 4]) >>> a[0:3, 0:4] 0 1 2 3 4 5 6 7 8 9 10 11 >>> a[0:4, 0:4] 0 1 2 3 4 5 6 7 8 9 10 11 >>> a[-1:4, 0:4] 8 9 10 11 >>> a[4:0, 0:4] EmptyArray >>> a[3:0, 0:4] EmptyArray >>> a[3:0:-1, 0:4] 0 0 209 0 8 9 10 11 4 5 6 7 >>> a[3:0:-1, 4:0:-1] 1 0 209 0 0 11 10 9 8 7 6 5 >>> Whoa! Let's try that again: Python 1.4 (Feb 24 1997) [C] Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam >>> a = reshape(range(12), [3, 4]) >>> a = reshape(range(12), [3, 4]) >>> a[3:0:-1, 4:0:-1] 1936654336 7103790 1328443513 1768843081 0 11 10 9 8 7 6 5 >>> a = reshape(range(12), [3, 4]) >>> a[3:0:-1, 4:0:-1] 0 1545120 41 0 0 11 10 9 8 7 6 5 Broken, Yes? Later, Andrew Mullhaupt _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From chase@att.com Sat Jul 12 01:34:56 1997 From: chase@att.com (Chase, Chris) Date: Fri, 11 Jul 1997 20:34:56 -0400 Subject: [MATRIX-SIG] A proposal (LONG) - "reverse of take" revisited Message-ID: <9707120036.AA16323@hoccson.ho.att.com> > ---------- > From: Andrew P. Mullhaupt[SMTP:amullhau@ix.netcom.com] > Sent: Friday, July 11, 1997 5:54 PM > To: Chase, Chris; 'matrix-sig@python.org' > Subject: RE: [MATRIX-SIG] A proposal (LONG) - "reverse of take" > revisited > > At 12:08 PM 7/11/97 -0400, Chase, Chris wrote: > > > >Slices that go outside array bounds are silently projected onto the > >bounds while take() generates an error when the index vector goes > >outside the bounds. If arrays were allowed as indexes should they > >behave like slices or like take()? > > > > Actually, _functions_ such as take should not be tied to the behavior > of indices. > Sorry for the confusion, but that is not the question I was trying to ask. I was asking what should the behavior of arrays as indexes be with respect to array bounds and I was illustrating two possible behaviors now seen in NumPy. Reasonable possible behaviors for arrays used as indexes when the indexes are outside the array bounds: 1) project the indexes onto the bounds. This takes the intersection of the bounds and the index dropping individual indexes outside the bounds. This is the type of behavior currently exhibited by slices. The problem with clipping is that you get back fewer elements than expected. 2) project the indexes by clipping. This clips indexes outside the bounds to the bounds. For example, an index of [20] on a size 3 array uses [2] as the index. This is the way IDL array indexes behave (which is incongruous with the errors that out-of-bounds scalar indices generate in IDL). For some applications this can be convenient while in other situations you would prefer an error. 3) generate an error. This is the behavior of take() when the index is out of bounds. > You can write all sorts of functions to do anything you > want, so the user shouldn't have an expectation of a connection > between > index semantics and function semantics except for the _very few_ > necessary functions which explicitly involve the shape of the array. > > So for consistency they can behave like slices, but I'm a bit > surprised > at how you claim slice indices work. > > So I tried some examples and I think slice indices are broken. For > example: > > Python 1.4 (Feb 24 1997) [C] > Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam > >>> a = reshape(range(12), [3, 4]) > >>> a[0:3, 0:4] > 0 1 2 3 > 4 5 6 7 > 8 9 10 11 > >>> a[0:4, 0:4] > 0 1 2 3 > 4 5 6 7 > 8 9 10 11 > The above is the slice clipping behavior. > Python 1.4 (Feb 24 1997) [C] > Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam > >>> a = reshape(range(12), [3, 4]) > >>> a = reshape(range(12), [3, 4]) > >>> a[3:0:-1, 4:0:-1] > 1936654336 7103790 1328443513 1768843081 > 0 11 10 9 > 8 7 6 5 > This is indeed a bug in the slice clipping. The bug is in arrayobject.c in routine slice_GetIndices(): if (*start < 0) *start = 0; else if (*start > length) *start = length; should be something like: if (*start > (length-1)) *start = length-1; if (*start < 0) *start = 0; Chris _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From amullhau@ix.netcom.com Sat Jul 12 04:26:06 1997 From: amullhau@ix.netcom.com (Andrew P. Mullhaupt) Date: Fri, 11 Jul 1997 23:26:06 -0400 Subject: [MATRIX-SIG] A proposal (LONG) - "reverse of take" revisited In-Reply-To: <9707120036.AB16323@hoccson.ho.att.com> Message-ID: <3.0.1.32.19970711232606.00804ce0@popd.netcruiser> At 08:34 PM 7/11/97 -0400, Chase, Chris wrote: > > I was asking what should the behavior of arrays as indexes be with >respect to array bounds and I was illustrating two possible behaviors >now seen in NumPy. Reasonable possible behaviors for arrays used as >indexes when the indexes are outside the array bounds: > >1) project the indexes onto the bounds. This takes the intersection of >the bounds and the index dropping individual indexes outside the bounds. >This is the type of behavior currently exhibited by slices. The problem >with clipping is that you get back fewer elements than expected. This is not normally considered a reasonable behavior, however it isn't a disaster. You get all sorts of weird ideas for this kind of stuff in different array languages. APL gives an error for any access outside the shape of an array. Now S-plus is _pretty_ weird for this: > A <- array(1:6, dim=c(3,2)) > A [,1] [,2] [1,] 1 4 [2,] 2 5 [3,] 3 6 > A[3, 2] 6 > A[6] 6 > A[7] NA > A[1,4] Error in A[1, 4]: Array subscript (4) out of bounds. should be at most 2 Dumped > A[7] <- 7 > A [,1] [,2] [1,] 1 4 [2,] 2 5 [3,] 3 6 > A[7] 7 > A[1,4] <- 7 Error in A[1, 4] <- 7: Array subscript (4) out of bounds. should be at most 2 Dumped Cool, right? I don't want to get into a discussion of why this happens, just want to point out that it comes from the way S-plus uses object orientation - it has to do with arrays acting like a subclass of vectors but having different methods for out-of-bounds accesses. But wait! Matlab is the _exact converse_!. You can't interrogate an element outside the shape of the array, but if you _assign_ one then the array expands. But not if it's a vector. Moral of the story: This is an issue where you can adopt really screwy ideas and people might still buy and use your product. So I suppose silently truncating indices might make sense to somebody, although at the moment it seems somewhat risky and eccentric. This is not to say that it doesn't matter, though. People who make mistakes and over-access an array probably will appreciate errors being raised. People who want to work with sparse matrices will absolutely hate those errors. >2) project the indexes by clipping. This clips indexes outside the >bounds to the bounds. For example, an index of [20] on a size 3 array >uses [2] as the index. This is the way IDL array indexes behave (which >is incongruous with the errors that out-of-bounds scalar indices >generate in IDL). For some applications this can be convenient while in >other situations you would prefer an error. Just so. Another array language, another different way of handling out of bounds indices. Can anyone think of two different languages which handle this the same way? >3) generate an error. This is the behavior of take() when the index is >out of bounds. Good old APL style. This has the virtue that even if it isn't making you happy, it's easy to understand and expect. You could do worse. Later, Andrew Mullhaupt _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From isenmann@physik.tu-cottbus.de Sun Jul 13 13:41:35 1997 From: isenmann@physik.tu-cottbus.de (G.Isenmann) Date: Sun, 13 Jul 1997 14:41:35 +0200 (MDT) Subject: [MATRIX-SIG] sorry, but I am the idiot Message-ID: I am very sorry for injecting old messages (about end of May) into the list. I wanted to resent a collection of saved/unread messages to myself, but because of a stupid mistake, they were sent to the addess of the mailing- list (sometimes also to the author of the message). Goetz Isenmann isenmann@physik.tu-cottbus.de Tel: +49 355 69 3004 FAX: +49 355 69 3011 Lehrstuhl fuer Theoretische Physik, Karl-Marx-Str. 17, 03044 Cottbus, Germany _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From amullhau@ix.netcom.com Sun Jul 13 20:30:01 1997 From: amullhau@ix.netcom.com (Andrew P. Mullhaupt) Date: Sun, 13 Jul 1997 15:30:01 -0400 Subject: [MATRIX-SIG] [PYTHON MATRIX-SIG] SWIG, NumPy and C++ templated arrays In-Reply-To: <199705282335.QAA29653@curlew.geog.ubc.ca> Message-ID: <3.0.1.32.19970713153001.0080cc80@popd.netcruiser> At 04:35 PM 5/28/97 -0700, Phil Austin wrote: >The code: > > Cutmod.cpp defines a C++ function "cut", which cuts data into a > series of histogram bins. >Here's the way I would implement cut in Python: [deleted] Yes. That's pretty much the straightforward way. I'd like to take advantage of this example to show how more flexible indexing can be quite powerful. Since I don't have the kind of indexing needed in NumPy to do the example, I'll do it in S. This also helps demonstrate the efficiency of the resulting code since S has a "built-in" solution which calls C functions which work just about exactly the way your python module does, give or take some extra memory overhead involved in the way S calls these C functions. Also, the S I will use here is essentially the same as NumPy, so I hope most readers will get this example. One thing which isn't obvious is that c(p,q) concatenates p and q. Here is an S function which does the job. Note that the vector of bins (cut-points) must be sorted in ascending order, which is a typical restriction for this kind of function. demux <- function(data, bins) { lb <- length(bins) if (lb > 1) { k <- lb %/% 2 # integer division return( c(demux(data[data < bins[k+1]], bins[bins <= bins[k]]) , demux(data[data >= bins[k+1]], bins[bins < bins[k]]) ) ) } if (any(data < bins[1])) { return( c(list(data[data < bins[1]]), list(data[data >= bins[1])) ) } else { return(list(data)) } } WARNING: I tossed this off in about twenty minutes and so I think I've handled the "edge" cases I'm not entirely sure. If there is such a bug in this code it isn't likely to affect the timings - which is the thrust of this post, so I won't worry about this issue. Just don't use this code as given and expect it to be bullet- (or even spitball-) proof. Remark: The algorithm as given _can_ be implemented in Python without using indexing, and appealing to user-provided functions. If you write that down I think you'll see that this version is a lot more readable. At least I (and most other people who have used array languages) will think so. Now there isn't much to this other than divide and conquer combined with logical indexing, (one of the convenient flavors). The question is, given the incredibly painful interpreter overhead of S-plus will this be even close to usable? We will compare with the "built-in" solution: split(data, cut(data, bins)) length of data: 100,000 bins are uniformly distributed in the data. #bins demux built-in 5 0.7 1.0 20 1.1 1.2 100 1.7 1.8 1000 3.7 8.2 10000 17.1 81.0 Hmmm. Are we doing too well? Scrutiny of the S-plus implementation of split.default shows that they build the final list up with a loop in S over the bins, and S loops are superlinear. Note that this loop is not over the data but over the bins, which is why having the loop over the bins in the interpreter is not very costly unless the number of bins is on the same order as the number of data points. Since this loop is the only place the built-in solution spends a significant amount of time out of C, we can conclude that the S "built-in" solution is going to be a pretty good indicator of a completely C implemented version when the number of bins is small compared to the number of data points. This being the case, it seems that the demux version is quite competitive in S. With a fast (read built-in) version of this kind indexing, a NumPy version would likely be about as fast as a C version unless the ratio of bins to data gets large. Now for a fixed number of bins, distribution of bins, and distribution of data, as the amount of data N increases, our algorithm is O(N), and will outperform any algorithm which sorts the data (which is Theta(N log N) unless there is some special circumstance of the data). This is because our algorithm compares the data to the bin values - sorting involves comparing the data to the data. This will be a very small effect and might require enormous values of N to see it. Just thought people might like to see what they might get with slicker indexing. Later, Andrew Mullhaupt _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From hauser@lisboa.ifm.uni-kiel.de Tue Jul 15 14:20:05 1997 From: hauser@lisboa.ifm.uni-kiel.de (Janko Hauser) Date: Tue, 15 Jul 1997 15:20:05 +0200 (CEST) Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package Message-ID: Hello here are some of my thoughts regarding the plot-discussion, which comes up once in a while. First some of the things I think a ubiquitous plotpackage should cover: There a some fields wich can be identified: The ``normal'' function plotting (x-y-plots) with lines or markers or error-bars. Often one wants to have more than one y- or x-axis here. All the different types of 2 or 2 1/2 dimensional plots, like contour-plots, filled contours, vector- and surface-plots. This are normaly plots on grid. More and more the grids are not equidistant and the hole field of triangular grids is coming. The view of this grid changes more from plane views to perspectiv views of more than one of these grids. (hor. and vert. slices through a data-cube) I think here stops the field of plotting and the field of visualiztion begins. I mean that the more dimensions are involved into the plot, the more arise the need for interactiv movement of the view, or changing of the color-palettes. So here you must work with the plot or better the view onto the data. These plots are seldom simple to be printed on paper. The next step into this direction is the display of isosurfaces and volume-rendering. Of course all of this plottypes should be able to be animated, because of the volume of plots we want to see :-) OK, than there are field of business graphics, like bar and pie-charts also in more than one dimensions. Histograms are the same class. Here it is very important to have rich set of text-functions in boxes, with shapes or raised against the plot itself. Than there is the hole field of statistical plots, which I don't know, but I see often some strange things in such plots. There is currently no free package wich does all these things. So we can't wrap up something. (If I'm wrong here, please say it to me ) And there will be more things in the future like building of simple point and click interfaces to look at some preprocessed data. So there should be the possibility to connect to a GUI-interface, like Tk. At the moment there are different ways to plot from python. (write a Matlab-readable ASCII-file and plot this with Matlab :0) Honestly, I know from ways to plot over gnuplot, mathematica, pgplot, tk/blt, plplot, gist. Especially plplot/pgplot and gist support a wide range of plotprimitives. So these packages can be and are used to build a new plot-system in python. But, (there must always be a but, eh?) they are used in a mixed way. The high-level functions, which are present are used and the missing parts are build from scratch. I think this way has some limits, because there is always a mix from changes which can be done in python and things which must be done on the c-language-level. One example is the axis-system in 3d in the gist package. Gist has nice autoscaling and rescaling with mouse-support in 2d. But the axis in 3d are plotted with different routines, the labeling happens in a different way. The worst case is, that I use different and overlapping packages to fullfill my plot-requirements. I think this is bad, because the hole system becomes more and more a hack, rather then developing to a kitchen-sink-plot-system with wide application-range. (I don't want to say, that the current implementations are hacks, but if I try to change something, than it will become a hack, but I must say here, that I'm very happy to have something with which I can generate plots with Numpy here and now) Here are some more specific requirements: 1) Use standard or at least very portable graphics-libraries: If the effort is done to build a new plot-system it should be availabel with some drivers on all platforms, where Numpy is availabel. I think the two most wiedly available standards are postscript and opengl, which can both be accsessed trough python (I have a nearly running postscript-driver, which supports all of the further down mentioned primitives) If one use librarys like the above, very different fields can be attacked, like molecular-modelling with opengl without learning a new graphics-system or flow-diagramms with postscript. 2) Plots should be interactiv in a way that one can interact with the plot directly, point to some locations and get the corresponding values. This is possible with the opengl-driver. 3) At least the 2d plots should have or be able to have publication quality. That's the major drawback of opengl. So here jumps the postscript-driver in. For the real 3d-plots (like volume-rendering) good tiff-files should be enough. But the access to the driver should be seperated, so one can generate a hole bunch of plots on a Cray without the need of opengl. (Although SGI owns them :-)) 4) Following plot-primitives should be supported by any driver: markers at a point lines closed lines filled areas (filled with one color or pattern) bitmap areas (used for things like ligthing, shades) text which should be rotateable (ascii and some special fonts). The size of the text-surrounding box should be known. Here is some room for higher primitives, like box and vector, errorbars and so on. But this can also directly be handled by the plot-routines. 5) These primitives should be there in every backend. 6) The process queue can be: data -> plot-primitive -> transformation -> scaling, normalization -> put into viewport with aspect-ratio or big as possible 7) More than one window, more than one viewport in one window. The viewport should be divided into a plot-area and the space for axis and annotations. In this way the plots with multiple axis are not a problem. Perhaps one needs axis in the plot-area, also this can be done in that way. Each axis should be handled independant. 8) There should be a way for explicit axis-paths with annotations. Standard creator and custom one. I think the axis-handling is a crucial point in all existing packages. Normally the hole axis-system is handled. One has now control over the strings, which are plotted. So dates are really hard to plot. And what happens if one wants to plot in polar-coordinates or with a cartographic projection? The axises of the plot should have similar handles like every other plotobject and each axis should be seperated. 9) The hole package should mainly be written in python, only then everybody can change it to her/his will. And additions are easier, because there is then one concept of plotting. So perhaps other packages can be incorporated under the same concept or in clear derivation of this concept. If there are performance problems these can be addressed specifically through an extra c-module. 10) In the end of the queue the plotobjects should be in clear data-structure, which can be saved for example or given to different drivers. This data-structure is a class with some special dictionaries for different plot-types and actions. To have here one standard form of structuring is maybe a hart thing to achieve. The advantage is, that new plotroutines must only produce such a structure, so they don't need to interact with the driver directly. This reduces the hacking with driver-tricks. (But everybody can use the driver-routines for such tricks, if she/he is in a real need for this :-) 11) One additional drawback is the hole experience wich is incorporated in other plotting-libraries. It's is not so easy to use one special feature in one particular plotting-library, here is very much work in redeveloping the circle. I don't know much about VTK for exapmle, but I think all these great routines can't easyly be imported in this framework, because VTK has it's one framework. But I think at the moment an arriving VTK-module will not address the basic plotting needs. And if we have general plot-objects, it should be possible to write convertors to VTK-data-objects. So, this is the end of the first part, which is mainly to bring some things up for discussion. I will try to develop some ideas for the system itself, but this will take some time, but the discussion can start now. __Janko _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From Dale_Bertrand@brown.edu Tue Jul 15 22:50:46 1997 From: Dale_Bertrand@brown.edu (Dale Bertrand) Date: Tue, 15 Jul 1997 17:50:46 -0400 (EDT) Subject: [MATRIX-SIG] Numeric Error Message-ID: I've written a c function which returns a string representing 35 doubles. I wish to convert this string into a Numeric array for further processing. Unfortunately, Numeric dumps a core when I call Numeric.fromstring. I am sure the string is the right lenghth and have no idea what is going on. Shouldn't Numeric accept the data as long as it's the right length? Is there something I'm missing? Thanks in advance. -- code -- f = "/vision/medprojects/crisco/shocks/quad1-surfaces/quad1-0.00--0.50-00%d.raw" % i print "filename: ", f shocks_string = sd.detect(f,135,135) print "length of string: ", len(shocks_string) print "string: ", shocks_string aa = Numeric.fromstring(shocks_string, Numeric.Float) print aa -- output -- filename: /vision/medprojects/crisco/shocks/quad1-surfaces/quad1-0.00--0.50-001.raw length of string: 280 string: @G ^m@65*ðh?ðHfãQæ@JpnäN@cÕ P¶ôÀH>sOz?¬@2aE­aJ@[s?ónè(ÀCÍ¢ÆoÀ@U=lQAzÀdyÔ Ø@[@@[¹ d@[q\ãOñø?ödW§à<ÀbÊJ%I8À/Ú¾¿î¢@RÒ/AÇxô?ø(1ÈL_@Udµ$ o@6À?Ó?ð;±ªú@aÏ_éÉÀ]þÔµ@DJ°¨¶@GöfáAü@5ñ|¬±Ú?ðÑe@/M(@evÚàÌ^À3u\ÖO Segmentation fault (core dumped) dcb@lems61 396 > -- sig -- Dale Bertrand (dcb@lems.brown.edu) ---------------------------------- E-Mail: Dale Bertrand Date: 15-Jul-97 ---------------------------------- _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From hyoon@nyptsrv1.etsd.ml.com Wed Jul 16 02:51:13 1997 From: hyoon@nyptsrv1.etsd.ml.com (Hoon Yoon - IPT Quant) Date: Tue, 15 Jul 1997 21:51:13 -0400 Subject: [MATRIX-SIG] Calculating correlation Matrix Message-ID: <199707160151.VAA00960@ptq3.etsd> Hello, The following should be fairly simple, but I am having problems with it for some reason. I am attempting to calulate correlation matrix of 2 D returns, but I cannot get a identitity diagnonal using following code. I been messing around it for quite a while, but can't figure out why meanc fails to begin with. It shoudl have something to do with 2 D matricies I am used to vs more advanced n-D in NumPy. I thought I made approapriate adj, but still doesn't work. Also, if anyone know of good stat func for NumPy, I would really appreciate the source. Thanks much in advance, Hoon, (the Baffled) * none of these works for complex number, because I don't use them. def vcm(x): # var/cov matrix from a data matrix return moment(x-transpose(meanc(x))) / (len(x)-1) def moment(y): return dot(transpose(y),y) def meanc(x): return up_1d(add.reduce(x) / len(x)) def varianc(x): return up_1d((add.reduce((x-meanc(x))**2)/(len(x)-1)) ** .5) def stdc(x): var = varianc(x) return up_1d(var ** .5) def corrx(x): vc = vcm(x) std = up_1d(diagonal(vc)) return dot(vc,(1/dot(std, transpose(std)))) def up_1d(x): if len(x.shape) == 1: return reshape(x, (len(x),1)) else: return x ln_gold = log(gold_a[1:,:]/gold_a[:-1,:]) corr = corrx(ln_gold) _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From janne@avocado.pc.helsinki.fi Wed Jul 16 09:43:25 1997 From: janne@avocado.pc.helsinki.fi (Janne Sinkkonen) Date: 16 Jul 1997 11:43:25 +0300 Subject: [MATRIX-SIG] Calculating correlation Matrix In-Reply-To: hyoon@nyptsrv1.etsd.ml.com's message of "Tue, 15 Jul 1997 21:51:13 -0400" References: <199707160151.VAA00960@ptq3.etsd> Message-ID: hyoon@nyptsrv1.etsd.ml.com (Hoon Yoon - IPT Quant) writes: > Also, if anyone know of good stat func for NumPy, I would really > appreciate the source. > Thanks much in advance, Here are some elementary functions including moments (and correlation matrices) and something to help you with bootstrapping. # This is from Konrad Hinsen. import Numeric def histogram(data, nbins, range = None): data = Numeric.array(data) if range is None: min = Numeric.minimum.reduce(data) max = Numeric.maximum.reduce(data) else: min, max = range data = Numeric.repeat(data, Numeric.logical_and(Numeric.less_equal(data, max), Numeric.greater_equal(data, min))) bin_width = (max-min)/nbins data = Numeric.array(Numeric.floor((data - min)/bin_width), Numeric.Int) histo = Numeric.add.reduce(Numeric.equal( Numeric.arange(nbins)[:,Numeric.NewAxis], data), -1) bins = min + bin_width*(Numeric.arange(nbins)+0.5) return Numeric.transpose(Numeric.array([bins, histo])) # I use these as well from Numeric import * def mean(X): N=X.shape[0] return sum(X)/N def variance(X): N=X.shape[0] return sum(X**2)/N-(sum(X)/N)**2 def sample_variance(X): N=X.shape[0] return (sum(X**2)-(sum(X)**2)/N)/(N-1) def std(X): return sqrt(variance(X)) def sample_std(X): return sqrt(sample_variance(X)) def covariance(X): N=X.shape[0] Xt=transpose(X) mX=sum(X)/N # Note that dot() equals to sum(multiply.outer()). # The latter is more memory-hungry. return dot(Xt,X)/N-multiply.outer(mX,mX) def covariance2(X,Y): N,M=X.shape[0],Y.shape[0] Xt=transpose(X) mX,mY=sum(X)/N,sum(Y)/N return dot(Xt,Y)/N-multiply.outer(mX,mY) def correlation(X): C=covariance(X) V=diagonal(C) return C/sqrt(multiply.outer(V,V)) def correlation2(X,Y): C,VX,VY=covariance2(X,Y),variance(X),variance(Y) return C/sqrt(multiply.outer(VX,VY)) def normalize(a): m=mean(a) s=std(a) return (a-m)/s, m, s def normalize_from(a,b): m=mean(b) s=std(b) return (a-m)/s # Euclidean distances def squared_distances(X,Y): return add.outer(sum(X*X,-1),sum(Y*Y,-1))- 2*dot(X,transpose(Y)) # Some functions to make bootstrapping easier. from Numeric import * from RandomArray import * import string import simpleStat def bsIndices(n): "Returns bootstrap indices for a sample of size N." r=randint(0,n,n) # The modulus is needless for future versions of Numpy # and even now for (all?) non-Inter platforms. return r%n def bsVector(a): "Returns the bootstrap vector generated from the bootstrap INDICES." # This looks ugly but it is (relatively) fast. b=concatenate((searchsorted(sort(a),arange(len(a))),(len(a),))) return (b[1:]-b[:-1]).astype(Float)/len(a) def bsStd(sample,estimator,n): "Based on SAMPLE, estimates standard deviation of ESTIMATOR using N bootstrap repetitions." estimates=[] for k in xrange(n): ii=bsIndices(len(sample)) estimates.append(estimator(take(sample,ii))) r=array(estimates) return (simpleStat.mean(r), simpleStat.sample_std(r)) def bsDistribution_w(sample,w_estimator,n): "Based on SAMPLE, returns samples from the distribution of W_ESTIMATOR using N bootstrap repetitions. W_ESTIMATOR takes the sample and weights as its arguments." estimates=[] l=len(sample) even=ones(l,Float)/l for k in xrange(n): ii=bsIndices(l) estimates.append(w_estimator(take(sample,ii),even)) r=array(estimates) return r def bsStdBias(sample,w_estimator,n): estimates=[] l=len(sample) B=zeros(l,Float) even=ones(l,Float)/l for k in xrange(n): ii=bsIndices(l) B=B+bsVector(ii) estimates.append(w_estimator(take(sample,ii),even)) r=sort(array(estimates)) return (simpleStat.mean(r), simpleStat.sample_std(r), simpleStat.mean(r)-w_estimator(sample,B/n), r[int(len(r)/2)], r[int(3*len(r)/4)]-r[int(1*len(r)/4)]) -- Janne Sinkkonen _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From ghughes@cs.ucsb.edu Mon Jul 21 06:14:55 1997 From: ghughes@cs.ucsb.edu (Graham Hughes) Date: Sun, 20 Jul 1997 22:14:55 -0700 (PDT) Subject: [MATRIX-SIG] UserArray bug Message-ID: <19970721051513.31201.qmail@a-abe.resnet.ucsb.edu> --8323328-1804289383-869462113=:31199 Content-Type: TEXT/plain; CHARSET=US-ASCII I noticed this bug when I was trying to get a hashable array with UserArray: >>> class ArrayWithHash(UserArray.UserArray): ... def __hash__(self): ... return hash(tuple(self)) ... >>> foo = ArrayWithHash([[ 0, 3, 1, 5, 3], ... [ 0, -2, 0, -1, 0], ... [ 0, 2, 0, 5, 0], ... [ 6, 3, 0, 3, 0], ... [ 0, 0, 0, -1, 0]]) >>> bar = ArrayWithHash([0, 0, 1, 0, 0]) >>> bar[2] = foo[0,2] Traceback (innermost last): File "", line 1, in ? File "/usr/local/lib/site-python/UserArray.py", line 36, in __setitem__ def __setitem__(self, index, value): self.array[index] = \ asarray(value,self.typecode) TypeError: illegal argument type for built-in operation >>> (traceback broken manually) This sort of behaviour isn't terribly helpful for me, because I really need to be able to stick arrays in dictionaries, and I also need for that code to work. Suggestions? -- Graham Hughes MIME OK, PGP preferred from stddisclaim import footer pgp_fingerprint = "E9 B7 5F A0 F8 88 9E 1E 7C 62 D9 88 E1 03 29 5B" --8323328-1804289383-869462113=:31199 Content-Type: APPLICATION/pgp-signature -----BEGIN PGP MESSAGE----- Version: 2.6.3 iQCVAwUBM9LwYCqNPSINiVE5AQHchgQAhLRNyZ0gnzMGk5NCNo7OSngSK8Q8gy4d h+do4J6wFgvyULFjzx8z6wUNQSS+heF5nunGSUMUl2LmxEymiTn7BQ75H10v6Lms y6mnBNiAwMPTop8Zno1JbF8qD9HAD8eD1gAgda6iOaaTCI/iZIOsYoptt4Jm34rd /PC3ilj5oZk= =H8e0 -----END PGP MESSAGE----- --8323328-1804289383-869462113=:31199-- _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From hinsen@ibs.ibs.fr Mon Jul 21 11:50:10 1997 From: hinsen@ibs.ibs.fr (Konrad Hinsen) Date: Mon, 21 Jul 1997 12:50:10 +0200 Subject: [MATRIX-SIG] A proposal (LONG) - "reverse of take" revisited In-Reply-To: <3.0.1.32.19970711232606.00804ce0@popd.netcruiser> (amullhau@ix.netcom.com) Message-ID: <199707211050.MAA23915@lmspc1.ibs.fr> > >3) generate an error. This is the behavior of take() when the index is > >out of bounds. > > Good old APL style. This has the virtue that even if it isn't making you > happy, it's easy to understand and expect. You could do worse. Especially considering that 1) It is easy enough to do clipping etc. as an explicit separate step. 2) Python's exception handling means that an error is not an intractable problem. So I vote for this solution. In fact, I can't say I am too happy about the clipping behaviour for slices, but there was little choice given that clipping is part of the general sequence protocol. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28 Institut de Biologie Structurale | Fax: +33-4.76.88.54.94 41, av. des Martyrs | Deutsch/Esperanto/English/ 38027 Grenoble Cedex 1, France | Nederlands/Francais ------------------------------------------------------------------------------- _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From hinsen@ibs.ibs.fr Mon Jul 21 13:32:27 1997 From: hinsen@ibs.ibs.fr (Konrad Hinsen) Date: Mon, 21 Jul 1997 14:32:27 +0200 Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package In-Reply-To: (message from Janko Hauser on Tue, 15 Jul 1997 15:20:05 +0200 (CEST)) Message-ID: <199707211232.OAA24182@lmspc1.ibs.fr> > First some of the things I think a ubiquitous plotpackage should cover: >... That's a long list, although of course all that is desirable. But we need something shorter for getting started! So let's look at this from an implementation point of view and see what is essential and how much development can be parallelized. An essential part seems to be the intermediate representation of "graphics" (which needn't be limited to plots) in terms of elementary objects such as lines, markers, circles etc. This must be defined first. Then work can progress in two directions: 1) Low-level drivers, which turn the intermediate representation into images, Tk objects, PostScript files, etc. They would also take care of animation and editing (where it makes sense). 2) High-level plot-to-graphics code. This would consist of one module per type of plot, and the modules would to a large degree be independent of each other. For plain plotting and also animation, the interfaces between the modules seem straightforward. Editing, on the other hand, is not so trivial, since the high-level plot objects must be able to turn graphical information back into numerical data, which is not always possible, at least not uniquely (think about editing a contour plot!). Inquiry (getting the value for the point under the mouse cursor, for example) is somewhere in between. > 1) Use standard or at least very portable graphics-libraries: If the > effort is done to build a new plot-system it should be availabel with > some drivers on all platforms, where Numpy is availabel. I think the My first attempt at a general low-level driver would be Tk canvases, since Tk is widely supported. Together with PostScript for printing, this would already make many people happy. Next is OpenGL, and then the system-specific libraries. But now for the really important question: do we have enough manpower to complete such a project to a useful stage in a reasonable amount of time? Yes, that's a call for volunteers. Starting with myself, I would be able to do some useful work, but not necessarily on a predictable time scale. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28 Institut de Biologie Structurale | Fax: +33-4.76.88.54.94 41, av. des Martyrs | Deutsch/Esperanto/English/ 38027 Grenoble Cedex 1, France | Nederlands/Francais ------------------------------------------------------------------------------- _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From furnish@acl.lanl.gov Mon Jul 21 15:56:55 1997 From: furnish@acl.lanl.gov (Geoffrey Furnish) Date: Mon, 21 Jul 1997 08:56:55 -0600 (MDT) Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package In-Reply-To: <199707211232.OAA24182@lmspc1.ibs.fr> References: <199707211232.OAA24182@lmspc1.ibs.fr> Message-ID: <199707211456.IAA03860@steam.acl.lanl.gov> Konrad Hinsen writes: > > First some of the things I think a ubiquitous plotpackage should cover: > >... > > That's a long list, although of course all that is desirable. But > we need something shorter for getting started! > > So let's look at this from an implementation point of view and see > what is essential and how much development can be parallelized. > > An essential part seems to be the intermediate representation of > "graphics" (which needn't be limited to plots) in terms of elementary > objects such as lines, markers, circles etc. This must be defined > first. Then work can progress in two directions: > > 1) Low-level drivers, which turn the intermediate representation > into images, Tk objects, PostScript files, etc. They would also > take care of animation and editing (where it makes sense). > > 2) High-level plot-to-graphics code. This would consist of one > module per type of plot, and the modules would to a large > degree be independent of each other. > > For plain plotting and also animation, the interfaces between the > modules seem straightforward. Editing, on the other hand, is > not so trivial, since the high-level plot objects must be able > to turn graphical information back into numerical data, which is > not always possible, at least not uniquely (think about editing > a contour plot!). Inquiry (getting the value for the point under > the mouse cursor, for example) is somewhere in between. > > > 1) Use standard or at least very portable graphics-libraries: If the > > effort is done to build a new plot-system it should be availabel with > > some drivers on all platforms, where Numpy is availabel. I think the > > My first attempt at a general low-level driver would be Tk canvases, > since Tk is widely supported. Together with PostScript for printing, > this would already make many people happy. Next is OpenGL, and then > the system-specific libraries. There are widespread complaints about the run-time performance of Tk canvases. This figured prominently in the decision to make a seperate Tk-aware plotting widget in PLplot. Besides our own experiences, I have heard many other people complain about the performance of Tk canvases (in contexts other than scientific plotting). Also, besides performance, you also face color management issues if you adopt Tk canvas. My opinion, and I think this is reflected in the architectures of other plotting packages as well, is that good plotting packages need to be able to manage their own color maps. Cooperative behavior is good to avoid "color flashing", but at some level, good rendering depends on good color control. > But now for the really important question: do we have enough manpower > to complete such a project to a useful stage in a reasonable amount of > time? Yes, that's a call for volunteers. Starting with myself, I would > be able to do some useful work, but not necessarily on a predictable > time scale. It seems to me that all this discussion about plotting is somewhat orthogonal to the issues involved with making a good matrix facility. Plotting is a natural client of a matrix facility, but is otherwise independent. I don't see major issues in plotting which can reasonably be expected to strongly influence matrix design or implementation issues. I wonder if it might be more appropriate to initiate a plotting-sig, or a "scientific environment sig", or some such, and move the plotting discussion which is not specifically matrix-sig-design-influencing, off to said new sig. I am /not/ volunteering to be the sig owner, but I do think it would be better to segregate these discussions in that way. Opinions? -- Geoffrey Furnish email: furnish@lanl.gov LANL CIC-19 POOMA/RadTran phone: 505-665-4529 fax: 505-665-7880 "Here are your ball-peen hammers. Now go malleate some heads!" -Jim Morel _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From furnish@acl.lanl.gov Mon Jul 21 17:46:06 1997 From: furnish@acl.lanl.gov (Geoffrey Furnish) Date: Mon, 21 Jul 1997 10:46:06 -0600 (MDT) Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package In-Reply-To: References: <199707211456.IAA03860@steam.acl.lanl.gov> Message-ID: <199707211646.KAA06794@steam.acl.lanl.gov> David Ascher writes: > Do we need to discuss this on the meta-sig, as for new sigs, or does this > count as a splinter group (faction? =))? You have to go through meta-sig eventually, in order to get the PSA folks to actually instantiate the list. But there is no harm in hashing out the basic plan here, so that there is a unified and basically community-accepted plan by the time you take it to meta-sig, especially since it seems likely that the new list will draw most of its (at least initial) participation from matrix-sig. At this point I think we need to resolve two things: 1) What is the name of the proposed new sig. Is it plot-sig, or sci-sig. This obviously impacts the sig charter and expected usage significantly. 2) Who is the owner going to be? Its not a particularly big job, but it should be someone who cares enough to keep things in order if need be. My view on 1) is that a plot sig would be thoroughly adjunct, whereas a sci-sig might hijack matrix-sig. If the new sig ended up absorbing all trafic from matrix sig, then there is no point in making the new sig, and good reason not to do it. A plotting sig would draw off only the plotting discussion from matrix-sig, and would not be a threat to the existence or continued operation of matrix-sig. A sci-sig might, I'm not sure. Others should voice opinion on that concern. Maybe its a non issue, I'm just not sure. Not that it would necessarily be inherently bad if sci-sig superceded matrix-sig. That might be a reasonable progression. But whatever happens should happen by design and because people want things to move that way, rather than by unfortuitous happenstance. My view on 2) is that it should be anybody but me. I'm already running a different sig, and I'm too vested in PLplot. A sig should be moderator-neutral. -- Geoffrey Furnish email: furnish@lanl.gov LANL CIC-19 POOMA/RadTran phone: 505-665-4529 fax: 505-665-7880 "Here are your ball-peen hammers. Now go malleate some heads!" -Jim Morel _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From da@maigret.cog.brown.edu Mon Jul 21 17:33:13 1997 From: da@maigret.cog.brown.edu (David Ascher) Date: Mon, 21 Jul 1997 12:33:13 -0400 (EDT) Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package In-Reply-To: <199707211456.IAA03860@steam.acl.lanl.gov> Message-ID: Geoffrey Furnish writes: > It seems to me that all this discussion about plotting is somewhat > orthogonal to the issues involved with making a good matrix facility. > Plotting is a natural client of a matrix facility, but is otherwise > independent. I don't see major issues in plotting which can > reasonably be expected to strongly influence matrix design or > implementation issues. > > I wonder if it might be more appropriate to initiate a plotting-sig, > or a "scientific environment sig", or some such, and move the plotting > discussion which is not specifically matrix-sig-design-influencing, > off to said new sig. > > I am /not/ volunteering to be the sig owner, but I do think it would > be better to segregate these discussions in that way. Opinions? Sure, seems like a fine idea. I do wonder where some things like the S-like data.frame project which has been bandied about fit in -- it ties in to the Numeric array as a backend mostly, and it certainly fits within the 'scientific environment' concept, but it shares some interest in potential python changes with the numpy project, e.g. on the indexing issue. I guess I'd say we should dicuss that on the sci-sig, since it's really not an issue for many NumPy users. As long as the plotting folk don't mind a bit of discussion that they might not find interesting. On Konrad's call for implementation volunteers, I'd love to help but committing my time in significant chunks for the next few months would be foolish. I could probably deal with owning a sig, but that's about it given the other things on my plate. Do we need to discuss this on the meta-sig, as for new sigs, or does this count as a splinter group (faction? =))? --da _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From da@maigret.cog.brown.edu Mon Jul 21 23:56:32 1997 From: da@maigret.cog.brown.edu (David Ascher) Date: Mon, 21 Jul 1997 18:56:32 -0400 (EDT) Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package In-Reply-To: <199707211646.KAA06794@steam.acl.lanl.gov> Message-ID: On Mon, 21 Jul 1997, Geoffrey Furnish wrote: > 1) What is the name of the proposed new sig. Is it plot-sig, or > sci-sig. This obviously impacts the sig charter and expected usage > significantly. Ok, let's play it safe. As per the SIG creation guidelines, we need a SIG name, short description, and longer blurb. Here's a first draft, please critique it. PLOT-SIG: Investigating and Developing Scientific and Business Plotting Solutions for Python Short blurb: The purpose of this list is to develop, coddle together, adopt or otherwise make available a package for plotting of scientific and business plots of data. Principles guiding the development of the software will include: * Portability to the relevant platforms * Ease of use * Integration with other Python packages (e.g. NumPy) * Quality of the software * Quality of the output ---- This is meant to be pretty much independent of existing solutions, although clearly some existing packages will fit more or less well. I'm concerned that plot-sig will suffer the fate of gui-sig, which was to die of starvation because the job at hand is just too big to start from scratch, so people just keep using what they've been using, and no 'ideal' solution comes to fruition. I also wanted to add the principle of "object oriented design" to the list above, but didn't know if that would get a consensus. Regardless, I think it might be a good thing to come up with a list of features each of us need, and rank them in terms of 'must have', 'want', 'would be nice'. Maybe this should wait until the sig is created, I don't know. Here's mine (off the top of my head): MUST HAVE: Line Plots Point Plots (scatter plots) Bar Plots Error Bars Fills (grey levels) Symbols Log scales Bitmap preview (tk, gif, whatever) Text Production-quality PostScript output Very Customizable Works on Unix & NT WOULD BE NICE: Complete Python control over graph generation, graph objects, etc. Ability to design new graph formats (e.g. Tufte's work) Interactivity Ability to do Batch work Here's what I've used in the past (or use now): Gnuplot GLE MetaPost [mac GUI tools] > 2) Who is the owner going to be? Its not a particularly big job, but > it should be someone who cares enough to keep things in order if > need be. I guess I just volunteered. We can keep the data.frame discussion (which is pretty small and probably doesn't require a sig) on matrix-sig for now. --da _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From singer@itl.atr.co.jp Mon Jul 21 00:41:56 1997 From: singer@itl.atr.co.jp (Harald Singer) Date: Mon, 21 Jul 1997 08:41:56 +0900 Subject: [MATRIX-SIG] [Q] numerical python on SunOS and dlopen Message-ID: <33D2A244.79EE6774@itl.atr.co.jp> We just discovered numerical and are thrilled. I installed NumPy1.0b3 on HP-UX, OSF1 and Linux. We only have a problem with SunOS(SS5/110(sun4m), SunOS4.1.4, 32MB RAM. gcc version 2.7.2.). When we try to compile NumPy on SunOS as a shared library nearly everything is OK. But when we try to use more than one of these modules we get ImportError: ld.so: conflicting usage of dlopen'ed dependents SS5/110(sun4m), SunOS4.1.4, 32MB RAM gcc version 2.7.2 riwa@als01<111> cd ~/NumPybuild riwa@als01<112> mkdir SunOS riwa@als01<113> cd SunOS riwa@als01<114> lndir ../NumPy . riwa@als01<115> make installdir=/usr/local/lang/python-1.4 -f Makefile.pre.in Makefile riwa@als01<116> make CCSHARED=" -fPIC" CFLAGS="-ansi -I/usr/local/lang/python-1.4/include/python1.4 -I/usr/local/lang/python-1.4/lib/python1.4/config -DHAVE_CONFIG_H" gcc -fPIC -ansi -I/usr/local/lang/python-1.4/include/python1.4 -I/usr/local/lang/python-1.4/lib/python1.4/config -DHAVE_CONFIG_H -I./Include -c ./arrayobject.c gcc -fPIC -ansi -I/usr/local/lang/python-1.4/include/python1.4 -I/usr/local/lang/python-1.4/lib/python1.4/config -DHAVE_CONFIG_H -I./Include -c ./ufuncobject.c ld arrayobject.o ufuncobject.o -o libnumpymodule.so gcc -fPIC -ansi -I/usr/local/lang/python-1.4/include/python1.4 -I/usr/local/lang/python-1.4/lib/python1.4/config -DHAVE_CONFIG_H -I./Include -c ./multiarraymodule.c ld multiarraymodule.o -L. -lnumpymodule -o multiarraymodule.so ld: -lnumpymodule: No such file or directory make: *** [multiarraymodule.so] Error 4 riwa@als01<117> pwd /home/atrh31/riwa/NumPybuild/SunOS riwa@als01<118> cp libnumpymodule.so libnumpymodule.so.1.0 riwa@als01<119> make CCSHARED=" -fPIC" CFLAGS="-ansi -I/usr/local/lang/python-1.4/include/python1.4 -I/usr/local/lang/python-1.4/lib/python1.4/config -DHAVE_CONFIG_H" ld multiarraymodule.o -L. -lnumpymodule -o multiarraymodule.so ld multiarraymodule.o -L. -lnumpymodule -o multiarraymodule.so gcc -fPIC -ansi -I/usr/local/lang/python-1.4/include/python1.4 -I/usr/local/lang/python-1.4/lib/python1.4/config -DHAVE_CONFIG_H -I./Include -c ./fast_umathmodule.c ld fast_umathmodule.o -L. -lnumpymodule -o fast_umathmodule.so gcc -fPIC -ansi -I/usr/local/lang/python-1.4/include/python1.4 -I/usr/local/lang/python-1.4/lib/python1.4/config -DHAVE_CONFIG_H -I./Include -c ./umathmodule.c ld umathmodule.o -L. -lnumpymodule -o umathmodule.so .... riwa@als01<131> setenv PYTHONPATH "$HOME/NumPybuild/$OS:$HOME/NumPybuild/$OS/NumPy" riwa@als01<132> setenv LD_LIBRARY_PATH "$LD_LIBRARY_PATH":$PYTHONPATH riwa@als01<139> python Python 1.4 (Jan 9 1997) [GCC 2.7.2] Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam >>> import multiarray >>> import umath Traceback (innermost last): File "", line 1, in ? ImportError: ld.so: conflicting usage of dlopen'ed dependents >>> ^D We also try to link using "gcc -shared". riwa@als01<255> gcc -shared arrayobject.o ufuncobject.o -o libnumpymodule.so riwa@als01<256> cp libnumpymodule.so libnumpymodule.so.1.0 overwrite libnumpymodule.so.1.0? y riwa@als01<257> gcc -shared multiarraymodule.o -L. -lnumpymodule -o multiarraymodule.so riwa@als01<258> gcc -shared umathmodule.o -L. -lnumpymodule -o umathmodule.so riwa@als01<259> python Python 1.4 (Jan 9 1997) [GCC 2.7.2] Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam >>> import multiarray >>> import umath Traceback (innermost last): File "", line 1, in ? ImportError: ld.so: conflicting usage of dlopen'ed dependents >>> ^D Any ideas? I would be grateful for any pointers. -- Harald Singer, ATR ITL, tel +81-7749-5-1389 singer@itl.atr.co.jp _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From raj@grserv.med.jhmi.edu Tue Jul 22 01:21:02 1997 From: raj@grserv.med.jhmi.edu (RajGopal Srinivasan) Date: Mon, 21 Jul 1997 20:21:02 -0400 (EDT) Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package In-Reply-To: Message-ID: On Mon, 21 Jul 1997, David Ascher wrote: > On Mon, 21 Jul 1997, Geoffrey Furnish wrote: > > > 1) What is the name of the proposed new sig. Is it plot-sig, or > > sci-sig. This obviously impacts the sig charter and expected usage > > significantly. > > Ok, let's play it safe. As per the SIG creation guidelines, we need a SIG > name, short description, and longer blurb. Here's a first draft, please > critique it. > > PLOT-SIG: Investigating and Developing Scientific and Business Plotting > Solutions for Python > > This might be of intereset to the IMAGE-SIG, given that there now is a package 'pilplot' that may be in development - URL http://hem1.passagen.se/eff/pil/index.htm raj _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From amullhau@ix.netcom.com Tue Jul 22 01:36:01 1997 From: amullhau@ix.netcom.com (Andrew P. Mullhaupt) Date: Mon, 21 Jul 1997 20:36:01 -0400 Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package In-Reply-To: References: <199707211646.KAA06794@steam.acl.lanl.gov> Message-ID: <3.0.1.32.19970721203601.0081bac0@popd.netcruiser> I've been watching the discussion here, and it seems fine to split plotting away as a discussion from arrays, except for one thing. Maybe some kinds of plots are pretty hard to do without the array objects. I have in mind things like contour plots (traditional and color) and three (or more) dimensional scatterplots. Now Python is reasonably quick as interpreters go, but I think building the plotting package on top of graphics primitives and the array operations will be easier and lead to a better result that just doing the plotting. Later, Andrew Mullhaupt _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From da@maigret.cog.brown.edu Tue Jul 22 04:42:20 1997 From: da@maigret.cog.brown.edu (David Ascher) Date: Mon, 21 Jul 1997 23:42:20 -0400 (EDT) Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package In-Reply-To: <3.0.1.32.19970721203601.0081bac0@popd.netcruiser> Message-ID: On Mon, 21 Jul 1997, Andrew P. Mullhaupt wrote: > I've been watching the discussion here, and it seems fine to split > plotting away as a discussion from arrays, except for one thing. > > Maybe some kinds of plots are pretty hard to do without the array > objects. Definitely -- it's just that we can define those operations using NumPy arrays without having to modify the architecture of NumPy, hence it need not be discussed on the same SIG, or so I take Geoffrey's point. But yes, NumPy should definitely be involved for contour plots, surface plots, 3d projections onto 2d, line-fitting & interpolation, etc. On the other point, my forgetting to mention PIL was simply that, a temporary lapse of comprehensiveness -- PIL is definitely something which should be taken into consideration, along with PLPlot, Narcisse, Gist, Gnuplot, OpenGL, Tk, GLUT, etc. etc. All of these have problems and features. How much we can use existing work and how much we have to do from scratch is up for discussion, and will depend on too many factors to list. Arrays I'm pretty sure we'll need, for the reasons mentioned by Andrew among others, and at least we know what code to use for that part. I've looked a bit at the architecture behind a few packages (gnuplot, gle, gist, etc.), and some design decisions are clear winners, while others clearly aren't. I feel it would behoove us to learn from history, should a redesign be deemed necessary. The distinction between graphics engine and plot driver allows maximum portability with potentially very distributed workloads (you do Xterms, I do postscript). That driver definition can be extremely limiting if it's not done carefully, and with enough extensability in mind (more a problem in static languages). And so on (look at how few packages handle interacting with graphics so as to allow exploratory data analysis (see S+ for an example, DataDesk for another, maybe StatLisp as well)). But all this should probably wait until the SIG creation decision has been reached. can-you-tell-i'm-unhappy-with-every-single-plotting-package-i've-tried- and-i've-tried-a-lot-ly yours, --da _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From amullhau@ix.netcom.com Tue Jul 22 05:07:40 1997 From: amullhau@ix.netcom.com (Andrew P. Mullhaupt) Date: Tue, 22 Jul 1997 00:07:40 -0400 Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package In-Reply-To: References: <3.0.1.32.19970721203601.0081bac0@popd.netcruiser> Message-ID: <3.0.1.32.19970722000740.008142c0@popd.netcruiser> At 11:42 PM 7/21/97 -0400, David Ascher wrote: >On Mon, 21 Jul 1997, Andrew P. Mullhaupt wrote: > >> Maybe some kinds of plots are pretty hard to do without the array >> objects. > >Definitely -- it's just that we can define those operations using NumPy >arrays without having to modify the architecture of NumPy, hence it need >not be discussed on the same SIG, or so I take Geoffrey's point. If you read it that way, then fine. > But yes, >NumPy should definitely be involved for contour plots, surface plots, 3d >projections onto 2d, line-fitting & interpolation, etc. Just about any plot with a large data object is either handing an image in a special graphics format off to a special graphics function or else you would prefer that that large object had some array operations. >can-you-tell-i'm-unhappy-with-every-single-plotting-package-i've-tried- >and-i've-tried-a-lot-ly yours, --da Can I have a witness? Amen, brother. Later, Andrew Mullhaupt _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From Daniel.Michelson@smhi.se Tue Jul 22 07:52:51 1997 From: Daniel.Michelson@smhi.se (Daniel Michelson) Date: Tue, 22 Jul 1997 08:52:51 +0200 Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package References: <3.0.1.32.19970721203601.0081bac0@popd.netcruiser> <3.0.1.32.19970722000740.008142c0@popd.netcruiser> Message-ID: <33D458C3.28D9@smhi.se> Hej, I've also been monitoring the plotting discussions and can't quite grasp why the xmgr package has not been mentioned in this context. As free scientific plotting packages go, I find it quite complete and versatile, although it, as all others, does have a number of limitations. I've built a v e r y primitive interface to our Python-based weather radar software using xmgr. If it's worth looking into, do so at http://plasma-gate.weizmann.ac.il/Xmgr/ This having been said, I use unix and xmgr is designed for X. I can't say if/how it could be made more universal in a way which would suit the entire Python community. Have others already written off xmgr because of this? I would certainly support a NumPy- or PIL-based plotting module, for reasons previously presented by others on this sig, if it is finally determined that this is a wheel which would have to be reinvented in order for it to be acceptable... best, -d +------------------+--------------------------------------------------+ | | Daniel B. Michelson | | /\ |\ /| | | | | Swedish Meteorological and Hydrological Institute| | \ | \/ | |__| | | S-601 76 Norrk=F6ping, Sweden |= | \ | | | | | | Telephone: +46 11 - 15 84 94 | | \/ | | | | | | Telefax: +46 11 - 17 02 07 | | | Internet: Daniel.Michelson@smhi.se | +------------------+--------------------------------------------------+ _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From Anthony Baxter Tue Jul 22 08:30:38 1997 From: Anthony Baxter (Anthony Baxter) Date: Tue, 22 Jul 1997 17:30:38 +1000 Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package In-Reply-To: Your message of "Tue, 22 Jul 1997 08:52:51 +0200." <33D458C3.28D9@smhi.se> References: <33D458C3.28D9@smhi.se> <3.0.1.32.19970721203601.0081bac0@popd.netcruiser> <3.0.1.32.19970722000740.008142c0@popd.netcruiser> Message-ID: <199707220730.RAA05491@jambu.off.connect.com.au> I've rolled my own 2D plotting code in PIL (using /F's pilplot as a start). It's not a fun game. Has anyone thought about interfacing to the GNUplot-in-a-library? (gplotlib) Anthony _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From R.Hooft@EuroMail.com Tue Jul 22 09:02:36 1997 From: R.Hooft@EuroMail.com (Rob Hooft) Date: Tue, 22 Jul 1997 10:02:36 +0200 (MET DST) Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package In-Reply-To: <33D458C3.28D9@smhi.se> References: <3.0.1.32.19970721203601.0081bac0@popd.netcruiser> <3.0.1.32.19970722000740.008142c0@popd.netcruiser> <33D458C3.28D9@smhi.se> Message-ID: <199707220802.KAA22657@nu> >>>>> "DM" == Daniel Michelson writes: DM> Hej, I've also been monitoring the plotting discussions and can't DM> quite grasp why the xmgr package has not been mentioned in this DM> context. One extra problem: It doesn't only need X, it needs Motif..... As everyone else, I have written my own plotting package. The only thing is: this was before the era of scripting languages. And WAY before the era of NumPy..... So it is written in portable Fortran-77. And no user-interface: just one command line. And guess what? It does exactly what I want it to.... And all these features that all those other people need are not in there..... One thing I learned: do not write your own output. I can do a lot of things with my plotting package because the output is generated in Xfig code, and so there is an easy editor to change the detailed lay-out of the plot. So although the lay-out choices of my plotting program are very very basic, any user can make it into a super glossy plot using xfig. And I didn't need to write all of the graphics primitives for that! I know nobody is interested in a plotting package like that..... http://www.sander.embl-heidelberg.de/extern/scatter/ Rob -- ===== R.Hooft@EuroMail.com http://www.Sander.EMBL-Heidelberg.DE/rob/ == ==== In need of protein modeling? http://www.Sander.EMBL-Heidelberg.DE/whatif/ Validation of protein structures? http://biotech.EMBL-Heidelberg.DE:8400/ ==== == PGPid 0xFA19277D == Use Linux! Free Software Rules The World! ============= _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From steve@estel.uindy.edu Tue Jul 22 13:02:11 1997 From: steve@estel.uindy.edu (Steve Spicklemire) Date: Tue, 22 Jul 97 07:02:11 -0500 Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package In-Reply-To: (message from David Ascher on Mon, 21 Jul 1997 23:42:20 -0400 (EDT)) Message-ID: <9707221202.AA11090@estel.uindy.edu> I would certainly vote for SIG creation... -steve _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From bvelasco@etsd.ml.com Tue Jul 22 14:03:46 1997 From: bvelasco@etsd.ml.com (Bill Velasco) Date: Tue, 22 Jul 1997 09:03:46 -0400 Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package Message-ID: <199707221303.JAA02817@nyptsrv1.etsd.ml.com> Hello, Would like to vote for a SIG creation. I would attempt to help, but, since I am not much of a programmer, I would not be much of help in core developement. I would like to keep in loop and help out in other ways if I can. Plotting is one of the few holes python still has. I would like to see mouse interacted command, x-y pinpointer with axis guide lines, and eventloop interacted updates. I am also attempting to get a commercial software to Python interactions as a project, but this is becoming a slow process. I will try to get the wrapper published, once this project is done. Hope this works. Hoon, _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From bvelasco@etsd.ml.com Tue Jul 22 14:03:46 1997 From: bvelasco@etsd.ml.com (Bill Velasco) Date: Tue, 22 Jul 1997 09:03:46 -0400 Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package Message-ID: <199707221303.JAA02817@nyptsrv1.etsd.ml.com> Hello, Would like to vote for a SIG creation. I would attempt to help, but, since I am not much of a programmer, I would not be much of help in core developement. I would like to keep in loop and help out in other ways if I can. Plotting is one of the few holes python still has. I would like to see mouse interacted command, x-y pinpointer with axis guide lines, and eventloop interacted updates. I am also attempting to get a commercial software to Python interactions as a project, but this is becoming a slow process. I will try to get the wrapper published, once this project is done. Hope this works. Hoon, _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From da@maigret.cog.brown.edu Tue Jul 22 14:49:34 1997 From: da@maigret.cog.brown.edu (David Ascher) Date: Tue, 22 Jul 1997 09:49:34 -0400 (EDT) Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package In-Reply-To: <199707220730.RAA05491@jambu.off.connect.com.au> Message-ID: > Has anyone thought about interfacing to the GNUplot-in-a-library? (gplotlib) gplotlib is way out of date w/ respect to gnuplot 3.6 dev't. I've talked to the folks who are working on gnuplot, and while they are interested in libraryfying the recent work, it's a significant job, IMHO because of poor design decisions very early on, regarding the breakdown of gnuplot into subparts and an inflexible driver interface definition. In other words, don't hold your breath. A lot of their code can be borrowed however, if we don't mind the GPL. -da _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From arw@dante.mh.lucent.com Tue Jul 22 15:29:41 1997 From: arw@dante.mh.lucent.com (Aaron Watters) Date: Tue, 22 Jul 1997 10:29:41 -0400 Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package Message-ID: <199707221426.KAA01970@dante.mh.lucent.com> >This having been said, I use unix and xmgr is designed for X. I can't >say if/how it could be made more universal in a way which would suit the >entire Python community. Have others already written off xmgr because of >this? For the purposes of business software, I think that might be a pretty good reason to write it off, unless there is some more general portability available which doesn't present itself from the url ref. Rob Hooft writes: >One thing I learned: do not write your own output. I can do a lot of >things with my plotting package because the output is generated in >Xfig code, >From another perspective you could say "have clear OO interfaces to an abstract back end" much like the SGML parser in the standard libs interfaces to an abstract formatter which can be plug replaced with something that talks tk, flat text, wpy, whatever... The down side is that a lot of hard thinking should be done up front to keep the interface clean (if it's not done it'll get hacked up and horrible, I promise). A couple notes on this thread: 2 back ends come to mind strongly 1) opengl 2) awt/jpi as outlined in http://www.ndim.edrc.cmu.edu/dougc/jpi/Home.html Of course, when Jim's pyjava gets working that too. -- Aaron Watters ps: Is there a standard book on the topic of plotting? It might provide a framework for discussion. _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From motteler@laura.llnl.gov Tue Jul 22 15:36:57 1997 From: motteler@laura.llnl.gov (Zane C. Motteler) Date: Tue, 22 Jul 1997 07:36:57 -0700 (PDT) Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package In-Reply-To: <199707220730.RAA05491@jambu.off.connect.com.au> Message-ID: Hi folks, Just a reminder that we have a working, pretty decent 2d-3d scientific graphics package working with Gist (a public domain graphics engine from LLNL). The Python interface is object-oriented, consisting of Graph objects, geometric objects which can be shipped to Graph objects, and Plotter objects which can be used to plot them. This is pretty much as I described in my talks at the last two Python meetings. The 3-D graphics is much enhanced since I first talked about it at the Washington meeting last year. Our distribution is available at our public ftp site ftp-icf.llnl.gov in /ftp/pub/python/Current_snapshot.tgz and is updated with our latest changes each Friday night. Available documentation leaves something to be desired, as our framemaker documentation has not yet been approved for public release (but we hope it soon will be, and will be available on the web). I am making this reminder because I would hate to see somebody duplicate all of my hard work. The high level graphics is intended to be independent of the underlying graphics engine (inasmuch as possible). One need only design an appropriate Plotter object and a Python extension in order to expand its coverage to a new graphics engine. Cheers Zane ----------------------------------------------------------------------------- Zane C. Motteler, Ph. D. ___________ALSO:_________________ Computer Scientist | Professor Emeritus of Computer| Lawrence Livermore National Laboratory | Science and Engineering | P O Box 808, L-472 | California Polytechnic State | Livermore, CA 94551-9900 | University, San Luis Obispo | 510/423-2143 --------------------------------- FAX 510/423-9969 zcm@llnl.gov or motteler@laura.llnl.gov or zmottel@calpoly.edu _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From furnish@acl.lanl.gov Tue Jul 22 15:44:26 1997 From: furnish@acl.lanl.gov (Geoffrey Furnish) Date: Tue, 22 Jul 1997 08:44:26 -0600 (MDT) Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package In-Reply-To: <3.0.1.32.19970722000740.008142c0@popd.netcruiser> References: <3.0.1.32.19970721203601.0081bac0@popd.netcruiser> <3.0.1.32.19970722000740.008142c0@popd.netcruiser> Message-ID: <199707221444.IAA11137@steam.acl.lanl.gov> Andrew P. Mullhaupt writes: > At 11:42 PM 7/21/97 -0400, David Ascher wrote: > >On Mon, 21 Jul 1997, Andrew P. Mullhaupt wrote: > > > >> Maybe some kinds of plots are pretty hard to do without the array > >> objects. > > > >Definitely -- it's just that we can define those operations using NumPy > >arrays without having to modify the architecture of NumPy, hence it need > >not be discussed on the same SIG, or so I take Geoffrey's point. > > If you read it that way, then fine. David read my message the way I intended it to be read. My view is that plotting will /use/ NumPy, but probably does not need to influence the development of NumPy too much. I think it would be desirable if we could move all this plotting talk off the matrix-sig. _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From hinsen@ibs.ibs.fr Tue Jul 22 16:03:15 1997 From: hinsen@ibs.ibs.fr (Konrad Hinsen) Date: Tue, 22 Jul 1997 17:03:15 +0200 Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package In-Reply-To: <199707211646.KAA06794@steam.acl.lanl.gov> (message from Geoffrey Furnish on Mon, 21 Jul 1997 10:46:06 -0600 (MDT)) Message-ID: <199707221503.RAA29121@lmspc1.ibs.fr> > At this point I think we need to resolve two things: > > 1) What is the name of the proposed new sig. Is it plot-sig, or > sci-sig. This obviously impacts the sig charter and expected usage > significantly. A while ago I discussed this a bit with Jim Hugunin. We agreed that the Matrix-SIG had more or less done its job, so it could be transformed into a more general "scientific" SIG that would build on NumPy. Of course array-related questions would also go there. I still think that this transformation is a good idea. But the question is whether very specific discussions about specific implementation problems should go to that SIG. It could quickly become too large. So here's another proposal: - Let's have a Sci-SIG for general discussions about scientific applications, including discussions of features of proposed packages etc., and also announcements of test versions. - Create separate mailing lists (not formal SIGs) for communication between the people working in specific code, e.g. plotting or array indexing. Does this sound reasonable? > 2) Who is the owner going to be? Its not a particularly big job, but > it should be someone who cares enough to keep things in order if > need be. It seems to me that a SIG owner has very little work to do, so that shouldn't be a problem. The real question is: Who will write the code? ;-) -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28 Institut de Biologie Structurale | Fax: +33-4.76.88.54.94 41, av. des Martyrs | Deutsch/Esperanto/English/ 38027 Grenoble Cedex 1, France | Nederlands/Francais ------------------------------------------------------------------------------- _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From furnish@acl.lanl.gov Tue Jul 22 16:04:32 1997 From: furnish@acl.lanl.gov (Geoffrey Furnish) Date: Tue, 22 Jul 1997 09:04:32 -0600 (MDT) Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package In-Reply-To: References: <199707211646.KAA06794@steam.acl.lanl.gov> Message-ID: <199707221504.JAA11180@steam.acl.lanl.gov> David Ascher writes: > On Mon, 21 Jul 1997, Geoffrey Furnish wrote: > > > 1) What is the name of the proposed new sig. Is it plot-sig, or > > sci-sig. This obviously impacts the sig charter and expected usage > > significantly. > > Ok, let's play it safe. As per the SIG creation guidelines, we need a SIG > name, short description, and longer blurb. Here's a first draft, please > critique it. I basically like this proposal. > PLOT-SIG: Investigating and Developing Scientific and Business Plotting > Solutions for Python > > Short blurb: > > The purpose of this list is to develop, coddle together, adopt or > otherwise make available a package for plotting of scientific and > business plots of data. ^ I would just voice some concern about a charter which specifies a singular solution. I say this because I think it is clear that there are going to be multiple plotting solutions in Python whether they get the blessing of the plot-sig or not. I would suggest preserving the basic idea, but eliminating a proscription for a single solution. Perhaps wording like: The purpose of this list is to develop, coddle together, adopt or otherwise make available Python support for plotting of scientific and business data. > Principles guiding the development of the software will include: > * Portability to the relevant platforms > * Ease of use > * Integration with other Python packages (e.g. NumPy) > * Quality of the software > * Quality of the output > > ---- > This is meant to be pretty much independent of existing solutions, > although clearly some existing packages will fit more or less well. I'm > concerned that plot-sig will suffer the fate of gui-sig, which was to die > of starvation because the job at hand is just too big to start from > scratch, so people just keep using what they've been using, and no 'ideal' > solution comes to fruition. I haven't followed gui-sig, so I can't comment on the phenomenology there. But I do think it would be useful to avoid a proscriptive attempt to produce the "One True Python Plotting Package". Plotting is clearly going to be an extension, not a language intrinsic. Multiple solutions are going to be a fact of life. There are many reasons for this, not the least of which is that many python users also use other languages (heresy, I know), and consequently many will seek multi-language solutions. A proscriptive attempt to generate a single, Python-centric plotting package will obviously win many users, and alienate other potential users. Conversely, a pluralist solution will woo many, and disincline others. There is no "one size fits all, right answer" to plotting. Rather, the more realistic expectation is to develop high quality Python interfaces to the plotting facilities which are available and of interest. And it is entirely conceivable that such efforts can have an influential effect on the targets. > I also wanted to add the principle of "object oriented design" to > the list above, but didn't know if that would get a consensus. I dont' think a consensus is necessary. In my view, a good sig for something other than a core language extension, should have a pluralistic charter. If you want to break programming interfaces down into two categories, API's and OO interfaces, then Python needs access to plotting support in both categories. The need for OO plotting support is clear--but it will not be used by everyone--some will want API oriented services too. The charter for the plot-sig should accomodate both needs. > Regardless, I think it might be a good thing to come up with a list of > features each of us need, and rank them in terms of 'must have', 'want', > 'would be nice'. Maybe this should wait until the sig is created, I don't > know. > > Here's mine (off the top of my head): > > MUST HAVE: > Line Plots > Point Plots (scatter plots) > Bar Plots > Error Bars > Fills (grey levels) > Symbols > Log scales > Bitmap preview (tk, gif, whatever) > Text > Production-quality PostScript output > Very Customizable > Works on Unix & NT > > WOULD BE NICE: > Complete Python control over graph generation, graph objects, etc. > Ability to design new graph formats (e.g. Tufte's work) > Interactivity > Ability to do Batch work It's a good list, but I don't think this kind of stuff needs to be explicitly enumerated in the sig charter. > Here's what I've used in the past (or use now): > Gnuplot > GLE > MetaPost > [mac GUI tools] > > > 2) Who is the owner going to be? Its not a particularly big job, but > > it should be someone who cares enough to keep things in order if > > need be. > > I guess I just volunteered. Thanks much. You should join meta-sig, then post a note proposing the creation of the new sig, explaining that you have the broad based support of people in matrix-sig, and providing your charter statement. Some discussion will ensue, and eventually they'll make the new sig, setting you up as the owner. -- Geoffrey Furnish email: furnish@lanl.gov LANL CIC-19 POOMA/RadTran phone: 505-665-4529 fax: 505-665-7880 "Software complexity is an artifact of implementation." -Dan Quinlan _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From hinsen@ibs.ibs.fr Tue Jul 22 16:05:50 1997 From: hinsen@ibs.ibs.fr (Konrad Hinsen) Date: Tue, 22 Jul 1997 17:05:50 +0200 Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package In-Reply-To: <199707211456.IAA03860@steam.acl.lanl.gov> (message from Geoffrey Furnish on Mon, 21 Jul 1997 08:56:55 -0600 (MDT)) Message-ID: <199707221505.RAA29129@lmspc1.ibs.fr> > There are widespread complaints about the run-time performance of Tk > canvases. This figured prominently in the decision to make a seperate > Tk-aware plotting widget in PLplot. Besides our own experiences, I I have heard of this problem, and I certainly don't want to prevent better Tk-integrations for plotting. But canvases are portable and part of every Tk implementation, i.e. the most generally available solution. And I expect them to be fast enough for many routine tasks, such as simple 2d line plots. So I still think there should be a driver for them, maybe in addition to a special plotting widget. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28 Institut de Biologie Structurale | Fax: +33-4.76.88.54.94 41, av. des Martyrs | Deutsch/Esperanto/English/ 38027 Grenoble Cedex 1, France | Nederlands/Francais ------------------------------------------------------------------------------- _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From hinsen@ibs.ibs.fr Tue Jul 22 16:07:55 1997 From: hinsen@ibs.ibs.fr (Konrad Hinsen) Date: Tue, 22 Jul 1997 17:07:55 +0200 Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package In-Reply-To: <33D458C3.28D9@smhi.se> (message from Daniel Michelson on Tue, 22 Jul 1997 08:52:51 +0200) Message-ID: <199707221507.RAA29137@lmspc1.ibs.fr> > I've also been monitoring the plotting discussions and can't quite grasp > why the xmgr package has not been mentioned in this context. As free > scientific plotting packages go, I find it quite complete and versatile, I use it frequently, but I don't see a specific relation to Python. Like other "external program" solutions (e.g. Gnuplot), it can't offer the amount of integration that is desirable. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28 Institut de Biologie Structurale | Fax: +33-4.76.88.54.94 41, av. des Martyrs | Deutsch/Esperanto/English/ 38027 Grenoble Cedex 1, France | Nederlands/Francais ------------------------------------------------------------------------------- _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From da@maigret.cog.brown.edu Tue Jul 22 16:14:13 1997 From: da@maigret.cog.brown.edu (David Ascher) Date: Tue, 22 Jul 1997 11:14:13 -0400 (EDT) Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package In-Reply-To: <199707221504.JAA11180@steam.acl.lanl.gov> Message-ID: On Tue, 22 Jul 1997, Geoffrey Furnish wrote: > I would just voice some concern about a charter which specifies a > singular solution. I accept that change and will adjust the wording. I agree, there are too many needs to be met by a single solution. > > MUST HAVE: > > Line Plots > > ... > > WOULD BE NICE: > > Complete Python control over graph generation, graph objects, etc. > > It's a good list, but I don't think this kind of stuff needs to be > explicitly enumerated in the sig charter. Oh, no, I just meant that list for discussion. Not for the charter. I'll send a revised sig description to meta-sig. I will wait a day or two to let others comment first, and/or challenge my sig-ownership =). --da _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From furnish@acl.lanl.gov Tue Jul 22 16:17:59 1997 From: furnish@acl.lanl.gov (Geoffrey Furnish) Date: Tue, 22 Jul 1997 09:17:59 -0600 (MDT) Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package In-Reply-To: <199707221503.RAA29121@lmspc1.ibs.fr> References: <199707211646.KAA06794@steam.acl.lanl.gov> <199707221503.RAA29121@lmspc1.ibs.fr> Message-ID: <199707221517.JAA11223@steam.acl.lanl.gov> Konrad Hinsen writes: > - Let's have a Sci-SIG for general discussions about scientific > applications, including discussions of features of proposed > packages etc., and also announcements of test versions. You want to just relabel matrix-sig as sci-sig? And generalize the charter? > - Create separate mailing lists (not formal SIGs) for communication > between the people working in specific code, e.g. plotting or > array indexing. > > Does this sound reasonable? I do not favor non-sig mailing lists for discussion of major subsystems. They are not really very open for people to subscribe/unsubscribe, since one has to bother a person instead of a daemon. One of the best things about a sig is that people can get on and off without having to ask for help. Also, such ad-hoc lists are not archived. > > 2) Who is the owner going to be? Its not a particularly big job, but > > it should be someone who cares enough to keep things in order if > > need be. > > It seems to me that a SIG owner has very little work to do, so that > shouldn't be a problem. The real question is: Who will write the code? ;-) This sounds a little to me like the presumption of the one true python-centric solution mentioned previously. This is arguably reasonable for a direct language facility like the matrix stuff, but I think it is basically misplaced loyalty when applied to non-language intrinsic facilities. For instance, there is no "one true python database" as far as I know, but rather (as I understand it) bindings to databases that people use. That would seem to me to be the more appropriate expectation with respect to plotting. So, who will write the code? Plenty of people are already doing this. What is of interest to the python community is to have good python support for plotting facilities that already exist. And possibly to influence them. But I don't see a need to produce a whole new python-oriented plotting facility. -- Geoffrey Furnish email: furnish@lanl.gov LANL CIC-19 POOMA/RadTran phone: 505-665-4529 fax: 505-665-7880 "Software complexity is an artifact of implementation." -Dan Quinlan _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From hinsen@ibs.ibs.fr Tue Jul 22 16:48:32 1997 From: hinsen@ibs.ibs.fr (Konrad Hinsen) Date: Tue, 22 Jul 1997 17:48:32 +0200 Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package In-Reply-To: (message from David Ascher on Tue, 22 Jul 1997 11:17:47 -0400 (EDT)) Message-ID: <199707221548.RAA29265@lmspc1.ibs.fr> David Ascher: > This would take over matrix-sig's role as the place to ask questions about > numpy? Yes. > > - Create separate mailing lists (not formal SIGs) for communication > > between the people working in specific code, e.g. plotting or > > array indexing. > > Why not make them sigs? The administrative burden is basically the same > whether it's a sig or some other mailing list, and this way they get > visibility on python.org, archival, etc. I have the impression that creating a SIG is a rather complicated process, whereas I can start a mailing list on Starship by just typing a few commands. But that's the only reason, I have nothing against SIGs in principle! Geoffrey Furnish: > You want to just relabel matrix-sig as sci-sig? And generalize the > charter? Either that or create a new SIG and let the old one die a few months later. I don't know which is easier to do. > I do not favor non-sig mailing lists for discussion of major > subsystems. They are not really very open for people to > subscribe/unsubscribe, since one has to bother a person instead of a > daemon. One of the best things about a sig is that people can get on > and off without having to ask for help. Also, such ad-hoc lists are > not archived. I was thinking of MailMan-based mailing lists on Starship, which are technically the same as the SIGs on python.org, but without any formalities attached. I do agree that openness and archiving are important. > This sounds a little to me like the presumption of the one true > python-centric solution mentioned previously. This is arguably Not "the one true" but "a good and easy-to-use one". > reasonable for a direct language facility like the matrix stuff, but I > think it is basically misplaced loyalty when applied to non-language > intrinsic facilities. For instance, there is no "one true python I think plotting is at the borderline. For standard tasks an interface to existing plotting packages is fine, but a Python implementation gives me the opportunity to modify a plot variety by subclassing in Python. Since it seems that the major complaint about plotting package X is "it doesn't have feature Y which I need urgently", I am inclined to put a high value on this customizability. In addition, there is the installation complexity problem posed by external packages. To my surprise I received quite a few complaints about my Molecular Modelling Toolkit related to its use of netCDF. There were no real problems about netCDF, and installing it is a simple task, but the simple fact of having to read another README and running another set of tests seems to put off some people. NumPy of course is bad enough (but will hopefully change), but at least Python/C module combinations all work in much the same way. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28 Institut de Biologie Structurale | Fax: +33-4.76.88.54.94 41, av. des Martyrs | Deutsch/Esperanto/English/ 38027 Grenoble Cedex 1, France | Nederlands/Francais ------------------------------------------------------------------------------- _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From arw@dante.mh.lucent.com Tue Jul 22 17:10:28 1997 From: arw@dante.mh.lucent.com (Aaron Watters) Date: Tue, 22 Jul 1997 12:10:28 -0400 Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package Message-ID: <199707221607.MAA02447@dante.mh.lucent.com> > ...I would just voice some concern about a charter which specifies a > singular solution. I say this because I think it is clear that there > are going to be multiple plotting solutions in Python whether they get > the blessing of the plot-sig or not. I would suggest preserving the > basic idea, but eliminating a proscription for a single solution... I think it would be nice to *promote* a singular solution, if possible. If people want to do something different, the can and will. If the singular solution was amenible to customization and extension, so much the better. > ... haven't followed gui-sig, so I can't comment on the phenomenology > there. But I do think it would be useful to avoid a proscriptive > attempt to produce the "One True Python Plotting Package"... Actually, theres a good name: OTPPP! I haven't followed the GUI sig either, but I've been disappointed with the results. Frankly maybe if Guido had picked one gui either that gui would be really really good now, or in self defense another alternative would have gotten much better and replaced it. As it is, I'm hoping that Fredrik and Mark H. will fight it out for market share (perhaps with Jim A. silently sharpening wpy in the background and pyawt in the wings) and both pythonwin and tkinter will end up really really good, or shamed by some clearly better alternative. Endless discussions generate a lot of email... If someone has the fortitude perhaps they could "just write it" and use the Numeric/Graphics/Plot SIG as a source of suggestions, much like Jim H did with Numeric. I prefer proofs of concepts to discussions. We might still be wishing for a good Numerics library if he hadn't. -- Aaron Watters _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From johann@physics.berkeley.edu Tue Jul 22 19:02:19 1997 From: johann@physics.berkeley.edu (Johann Hibschman) Date: Tue, 22 Jul 1997 11:02:19 -0700 (PDT) Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package In-Reply-To: Message-ID: On Tue, 22 Jul 1997, Zane C. Motteler wrote: > Just a reminder that we have a working, pretty decent 2d-3d scientific > graphics package working with Gist (a public domain graphics engine from > LLNL). The Python interface is object-oriented, consisting of Graph > objects, geometric objects which can be shipped to Graph objects, > and Plotter objects which can be used to plot them. This is pretty much > as I described in my talks at the last two Python meetings. The 3-D > graphics is much enhanced since I first talked about it at the > Washington meeting last year. Does the Gist module still require patching the Python source? The last time I checked, the latest version required a set of patches which managed to confuse the latest version of SWIG, so I turned to other packages. (SuperMongo, anyone?) - Johann --- Johann A. Hibschman | Grad student in Physics, working in Astronomy. johann@physics.berkeley.edu | Probing pulsar pair production processes. _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From arw@dante.mh.lucent.com Tue Jul 22 19:05:20 1997 From: arw@dante.mh.lucent.com (Aaron Watters) Date: Tue, 22 Jul 1997 14:05:20 -0400 Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package Message-ID: <199707221824.OAA29302@dante.mh.lucent.com> > From: Konrad Hinsen > I have the impression that creating a SIG is a rather complicated > process, whereas I can start a mailing list on Starship by just > typing a few commands. But that's the only reason, I have nothing > against SIGs in principle! The problem is not the mechanics, the problem is that with too many sigs people don't know where to go and with too few they get congested with peripheral stuff or "orphan messages". I'd opt for too few, myself, and people can learn how to use that little delete button... I discovered it not to long ago... really handy! Actually I was distressed to see the pythonwin-sig go away and it looks like maybe the db-sig may go that way too (even though it has little activity, if you want an answer on databases, that's the place to go -- your message might be accidentally ignored on other lists, including the main list). -- Aaron Watters _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From busby@icf.llnl.gov Tue Jul 22 20:15:03 1997 From: busby@icf.llnl.gov (L. Busby) Date: Tue, 22 Jul 97 12:15:03 PDT Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package Message-ID: <9707221915.AA12700@icf.llnl.gov.llnl.gov> Johann Hibschman wrote: >Does the Gist module still require patching the Python source? The last >time I checked, the latest version required a set of patches which managed >to confuse the latest version of SWIG, so I turned to other packages. >(SuperMongo, anyone?) Yes, a patch is required for Python 1.4. Expect Python 1.5 to be ``Gist-ready''. We make available Python-1.4, NumPy, Gist, all patches (including a patch which allows GNU readline to link against Python/Gist), and several other LLNL modules in a gzipped tar file at ftp://ftp-icf.llnl.gov/pub/python/Current_snapshot.tgz. _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From ptaney@usgs.gov Tue Jul 22 20:27:21 1997 From: ptaney@usgs.gov (Paul Taney, Computer Specialist, Reston, VA ) Date: Tue, 22 Jul 1997 15:27:21 -0400 Subject: [MATRIX-SIG] A NO-Proposal for a plot-package Message-ID: <199707221927.PAA15419@mailrvares.er.usgs.gov> Hi: aaron wrote: >I think it would be nice to *promote* a singular solution, if possible. >If people want to do something different, the can and will. If the >singular solution was amenible to customization... There is a parallel discussion about certain extensions for that coffee language going at java3d-interest@Sun.COM And among the posters are Rikk Carey (rikk@best.com) of VRML fame who suggested dividing the problem this way: Basics - contains the base types Color, Vector, Matrix, etc. Core - contains core groups, geometry, lights, attributes, Viewpoints, and supports ROUTES and PROTOs Interactive - adds sensors, interpolators, media, and scripts Geometry - adds other fancy geometry (e.g. Extrusion, CSG) SciViz - adds specialized nodes for SciViz VR - adds specialized VR functionality (e.g. HMD support) Animation - adds specialized nodes for Animation etc. I, too, would appreciate a singular solution, if possible. And as a Viz specialist I appreciate this list's new direction. paul ptaney@usgs.gov @ smail: ms 431, reston, va @ I ain't goin' fishin' no damn rain. voice: 703-648-5598 @ -- Thelma home: 540-822-4223 @ _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From ptaney@usgs.gov Wed Jul 23 00:09:14 1997 From: ptaney@usgs.gov (Paul Taney, Computer Specialist, Reston, VA ) Date: Tue, 22 Jul 1997 19:09:14 -0400 Subject: [MATRIX-SIG] books on plotting Message-ID: <199707222309.TAA18554@mailrvares.er.usgs.gov> Aaron Watters wrote: > Is there a standard book on the topic of plotting? It might provide > a framework for discussion. What is plotting? I think of paper products when you say that. There are 2 books on computational geometry that might help with this; especially mapping sparce matricies (coords) to spatial data structures (range-trees, convex hulls, etc.) "Computational Geometry", Preperata and Shamos, Springer, 85 "Computational Geometry in C", J O'Rourke, Cambridge, 93 code may still be at 131.299.64.23, /pub/compgeom AW (No not you Aaron, the _publisher) has a series (5) on OpenGL and Inventor; see esp. _The Inventor Mentor_ if you want to get with VRML. SGI Inventor is the mother of VRML97. Also see the Java3D spec... And of course the Graphics Gems series of 5 from AP. Also see the GIS shelf at Computer Literacy (or wherever) esp. Unwin; he's got a site somewhere in .uk. Hope this helps, paul ptaney@usgs.gov @ Use the Source, Luke. U.S.Geological Survey @ Use the Source! reston, va 22092 @ ph: 703-648-5598 @ -- Linus T. _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From amullhau@ix.netcom.com Wed Jul 23 00:49:33 1997 From: amullhau@ix.netcom.com (Andrew P. Mullhaupt) Date: Tue, 22 Jul 1997 19:49:33 -0400 Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package In-Reply-To: <199707221444.IAA11137@steam.acl.lanl.gov> References: <3.0.1.32.19970722000740.008142c0@popd.netcruiser> <3.0.1.32.19970721203601.0081bac0@popd.netcruiser> <3.0.1.32.19970722000740.008142c0@popd.netcruiser> Message-ID: <3.0.1.32.19970722194933.00807220@popd.netcruiser> At 08:44 AM 7/22/97 -0600, Geoffrey Furnish wrote: >Andrew P. Mullhaupt writes: > > If you read it that way, then fine. > >David read my message the way I intended it to be read. My view is >that plotting will /use/ NumPy, but probably does not need to >influence the development of NumPy too much. I think it would be >desirable if we could move all this plotting talk off the matrix-sig. OK I can agree completely with this. Later, Andrew Mullhaupt _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From hinsen@ibs.ibs.fr Wed Jul 23 10:20:26 1997 From: hinsen@ibs.ibs.fr (Konrad Hinsen) Date: Wed, 23 Jul 1997 11:20:26 +0200 Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package In-Reply-To: (motteler@laura.llnl.gov) Message-ID: <199707230920.LAA31626@lmspc1.ibs.fr> > Just a reminder that we have a working, pretty decent 2d-3d scientific > graphics package working with Gist (a public domain graphics engine from > LLNL). The Python interface is object-oriented, consisting of Graph > objects, geometric objects which can be shipped to Graph objects, > and Plotter objects which can be used to plot them. This is pretty much Sounds good, and could be used as a basis for other low-level plotting output. Not that Gist isn't nice, but at the moment I won't recommend it to anyone who might then ask me for help with installation problems ;-) > Our distribution is available at our public ftp site ftp-icf.llnl.gov > in /ftp/pub/python/Current_snapshot.tgz and is updated with our latest > changes each Friday night. Available documentation leaves something to I realize that a "one-in-all" distribution is the easiest solution for you, but if you'd like others to use your stuff, a separate package would be more appropriate. Not only is it tough for a non-expert to extract the right pieces from the distribution, but there is also the technical problem of downloading a huge file via bad network links. I tried a few times but never succeeded... -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@ibs.ibs.fr Laboratoire de Dynamique Moleculaire | Tel.: +33-4.76.88.99.28 Institut de Biologie Structurale | Fax: +33-4.76.88.54.94 41, av. des Martyrs | Deutsch/Esperanto/English/ 38027 Grenoble Cedex 1, France | Nederlands/Francais ------------------------------------------------------------------------------- _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From c9603085@zeus.hud.ac.uk Wed Jul 23 11:22:35 1997 From: c9603085@zeus.hud.ac.uk (MISS. K.L.COLLIER) Date: Wed, 23 Jul 1997 11:22:35 +0100 Subject: [MATRIX-SIG] convolution Message-ID: <199707231022.LAA00959@penelope.csm> I'm trying to calculate the convolution of two arrays (an image and a point spread function) using Numerical python as follows: #!/usr/local/bin/python from Numeric import * from FFT import * raw_a = array ( [ 1,2,0,3,4,0,0,0,0 ] ) raw_a.shape=(3,3) raw_b = array ( [ -1,1,0,-2,2,0,0,0,0 ] ) raw_b.shape=(3,3) freq_2a = fft2d(raw_a) freq_2b = fft2d(raw_b) fab=freq_2a*freq_2b res = inverse_fft2d(fab) print "res" print res but there doesn't seem to be an inverse_fft2d function. Is there one? Would this work anyway? Do I have to have the two images the same size, i.e. pad out the smallest one with zeros. If so do I do this symmetrically or fill the zeros on the rhs and bottom of the matrix? Does python automatically handle the complex numbers ok in the multiplication in the frequency domain? Other options: 1. I also notice that there is a convolve function in Numeric but that it is only 1D. If my psf is a gaussian i.e. is symmetric, is there a way of doing 2 1D convolutions to produce a 2D one? 2. I believe it can also be done in matrix form if the image is converted to a vector and the psf converted to a block circulant form. If the psf is only 5*5 then the matrix would be 25*25, BUT don't I need to first pad out the psf to the same size as the image (512*512) in which case the matrix would be way too big! Many questions I know, but does someone know the best way to do it? Thanks, Karen Collier c9603085@zeus.ac.hud.uk _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From hyoon@nyptsrv1.etsd.ml.com Wed Jul 23 22:35:57 1997 From: hyoon@nyptsrv1.etsd.ml.com (Hoon Yoon - IPT Quant) Date: Wed, 23 Jul 1997 17:35:57 -0400 Subject: [MATRIX-SIG] [long] A NO-Proposal for a plot-package Message-ID: <199707232135.RAA02104@ptq3.etsd> Zane C. Motteler Wrote: > Just a reminder that we have a working, pretty decent 2d-3d scientific >graphics package working with Gist (a public domain graphics engine from >LLNL). The Python interface is object-oriented, consisting of Graph >objects, geometric objects which can be shipped to Graph objects, >and Plotter objects which can be used to plot them. This is pretty much >as I described in my talks at the last two Python meetings. The 3-D >graphics is much enhanced since I first talked about it at the >Washington meeting last year. Hello, Gist is a nice package for what it does. I think it makes a good foundation for a lot of 3d type of work, but falls short of : > > Here's mine (off the top of my head): > > MUST HAVE: > Line Plots > Point Plots (scatter plots) > Bar Plots > Error Bars > Fills (grey levels) > Symbols > Log scales > Bitmap preview (tk, gif, whatever) > Text > Production-quality PostScript output > Very Customizable > Works on Unix & NT > > WOULD BE NICE: > Complete Python control over graph generation, graph objects, etc. > Ability to design new graph formats (e.g. Tufte's work) > Interactivity > Ability to do Batch work At least one person's idea of what it should do. I am had some frustration with trying to get bar graph and non-numeric ticks. I also would like to get 2 different XY axis for the graph, which I don't think is supported in Gist. Basically I will be extremely satisfied with Gist + Excel type of graphics. Has anyone seen Splus 4 graphs in Axim? It's completely obj oriented. Would it be lovely to have Trellis type of analysis? Just my few thoughts, Hoon, _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From acura@vcn.bc.ca Thu Jul 24 02:49:02 1997 From: acura@vcn.bc.ca (Kelvin Lee) Date: Wed, 23 Jul 1997 18:49:02 -0700 (PDT) Subject: [MATRIX-SIG] two newbie questions Message-ID: I'm still struggling to get a handle on NumPy... Two newbie questions: 1) How much memory does an NxN Float16 matrix use? 2) What's the most efficient way of constructing a "correlation-like" matrix of a given function for a data set? eg. using, say, RMSE instead of r? To show you where I'm at, I'm stuck with: def correlation_like(X, function): R = zeros((len(X), len(X)), Float16) for j in range(len(X)): for i in range(len(X)): R[i,j] = function(X[i], X[j]) return R Obviously this doesn't take into account symmetry etc. Any ideas to get me thinking in a matrix kind of way would be appreciated! Kelvin _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From da@maigret.cog.brown.edu Thu Jul 24 05:07:13 1997 From: da@maigret.cog.brown.edu (David Ascher) Date: Thu, 24 Jul 1997 00:07:13 -0400 (EDT) Subject: [MATRIX-SIG] Revised proposal for PLOT-SIG Message-ID: Here's a revised version of the PLOT-SIG proposal. Please let me know (in private if you wish) whether you find it appropriate and whether I should submit it to the META-SIG for discussion. I never did get a clear feeling for whether more folks felt like Konrad that this should instead be an unofficial working group based on the Starship instead of the Mothership (python.org). Feel free to voice opinions... Just to get my bias out in the open -- I am more interested in the second subproject described below, but will of course encourage work on interfacing with existing packages, in part because I expect to have plots to produce before the new framework is in place... Geoffrey (or others) -- if you wish to further specify the charter of what I call the "API subproject", let me know. ----- PLOT-SIG: Development of Data Plotting Solutions for Python The purpose of this list is to develop, coddle together, adopt or otherwise make available Python tools for plotting of scientific and business plots of data. Plotting needs vary greatly depending on each user's requirements. Some users wish to see a better API between Python and existing plotting libraries or programs. Others would rather see a new framework which maximally leverages the OOP and dynamic strengths of Python, at the cost of reinventing a few wheels. Both strategies will be entertained by the SIG, with individual members contributing to projects they wish to see furthered. Principles guiding the development of all software will include: * Ease of use. * Clean Integration with relevant Python packages (NumPy, PIL, etc.). * Quality of the software. * Quality of the output. One possible goal for the API subproject is to develop a package- independent API for plots, which would produce reasonably similar results regardless of the plotting package used as a backend (PLPlot, Gist/Yorick, Gnuplot, etc.), in the same spirit as the interface defined by the DB-SIG. Package-specific extensions could naturally be provided as well. The goals for the new framework subproject need to be further specified by the SIG, but include: * Complete Python Control. * Extensibility/Customizability. * High quality rendering both on screen and paper. * Portability (at least UNIX/X11 and Win32, MacOS if feasible). * Stealing good ideas others have had. * Interactivity. _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From Daniel.Michelson@smhi.se Thu Jul 24 08:01:18 1997 From: Daniel.Michelson@smhi.se (Daniel Michelson) Date: Thu, 24 Jul 1997 09:01:18 +0200 Subject: [MATRIX-SIG] Revised proposal for PLOT-SIG References: Message-ID: <33D6FDBE.2781@smhi.se> Hej, RE: the forum for PLOT-SIG I appreciate that some members of the MATRIX-SIG are intent on establishing the necessary logistics to allow the (rapid) creation of Python plotting functionality. In this context, the hospitality of the Starship has been described as being potentially more accessable than python.org, and this task would thus be underway faster if the informal route is taken. I don't want to rain on anyone's parade, but I feel obliged to voice a concern about the communications between Python developers and users. I am well aware that many developers on this SIG are also users, but there are a number of users who are not and cannot be developers but who appreciate the ability to incorporate new Python functionality ASAP. Needless to say, I am one of these users, and plotting abilities directly in Python are currently prioritized. (And I am encouraged by this PLOT-SIG initiative!) What this all boils down to is that, regardless of the PLOT-SIG's host, the forum should be open. Konrad mentioned that he could easily start an open mailing list on the Starship. Others have used words like "informal" and "unofficial". I just want to make sure that the "mailing list" and the latter two terms are not mutually exclusive and that the PLOT-SIG doesn't risk turning into a private club. best, -d +------------------+--------------------------------------------------+ | | Daniel B. Michelson | | /\ |\ /| | | | | Swedish Meteorological and Hydrological Institute| | \ | \/ | |__| | | S-601 76 Norrk=F6ping, Sweden |= | \ | | | | | | Telephone: +46 11 - 15 84 94 | | \/ | | | | | | Telefax: +46 11 - 17 02 07 | | | Internet: Daniel.Michelson@smhi.se | +------------------+--------------------------------------------------+ _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From jbaddor@phy.ulaval.ca Thu Jul 24 15:30:03 1997 From: jbaddor@phy.ulaval.ca (Jean-Bernard ADDOR) Date: Thu, 24 Jul 1997 10:30:03 -0400 (EDT) Subject: [MATRIX-SIG] -3**(-1/3.), sgn() Message-ID: Hy ! a is an array with positive, negatives or null values I want to compute a**(-1/3.). It works perfectly if (a) is not an array but a scalar (function pow). The funcion power do not give fractional power from negative number. I would like to write sgn(a)*abs(a)**(-1/3.) but the sgn function seems not to be defined in python (perhaps because it is not in c, but it is in fortran). I can use greater_equal. Do you have a better solution ? A bientot, Jean-Bernard _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From jbaddor@phy.ulaval.ca Thu Jul 24 16:17:26 1997 From: jbaddor@phy.ulaval.ca (Jean-Bernard ADDOR) Date: Thu, 24 Jul 1997 11:17:26 -0400 (EDT) Subject: [MATRIX-SIG] abs vs absolute Message-ID: Hy ! a = array([-2, 3,0, 5]) What is the difference between: abs(a) and absolute(a) ? Jean-Bernard _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From barrett@compass.gsfc.nasa.gov Thu Jul 24 17:11:40 1997 From: barrett@compass.gsfc.nasa.gov (Paul Barrett) Date: Thu, 24 Jul 1997 12:11:40 -0400 Subject: [MATRIX-SIG] Revised proposal for PLOT-SIG In-Reply-To: References: Message-ID: <199707241611.MAA24662@compass.> I've been following along on this thread for awhile now, but have been too busy doing programmatic work to take the time to respond. I'll make some time and do so now. Some of my current feelings are: 1) A PLOT-SIG should be created in spite of what happens to the MATRIX-SIG. The creation of a SIG with narrow focus and specific goals is more likely to reach its objective than being a splinter group in a more general SIG like a DATA-ANALSYS-ENVIRONMENTS-SIG. Note that the MATRIX-SIG was created for just this reason. (As an aside, I don't think the MATRIX-SIG should be terminated, but just moved to the back burner, so that the old-pros and newcomers can revisit this topic from time to time without creating noise in newly created or related SIGS.) 2) Now that the MATRIX-SIG is nearly complete, a PLOT-SIG (or GRAPHICS-SIG, or DATA-DISPLAY-SIG) is of next highest priority for me. Where I work, IDL and SuperMongo are still used by my coworkers because of their plotting capabilities, mostly for their ease of use. Note that they both have their limitations, mostly associated with their inferior language and syntax and user interface. Many coworkers would be willing to migrate to Numeric Python if there was good plotting capabilities, eg. 2-D, contour plots, shaded-surfaces, etc. An excellent plotting package able to do 3-D volume rendering and animation alla Vis5D could be developed later, but should be included in the initial design. 3) I also feel that we should try to develop a high level OO interface to current packages and if deemed necessary a new Python developed package. The former would help to get some good graphics capability into Python quickly using pgplot, PLplot, etc., but would allow migration to a more advanced plotting package in future. I don't see why those of us wanting to continue to use one of the currrent packages cannot add any addition functions to get full capabilities from their package, knowing full well that these functions are not platform and package independent. 4) My current dismay at many current packages is that they are platform specific and do not really allow easy access to low level features of the graph. I see a design with 3 or 4 levels. The highest level would provide the creation of a simple plot from a single command. Lower level commands would allow users to create new standard plot types (eg. bar charts, etc.) using the low level commands. In addition a high level interface would allow us to develop a platform independent package. IDL has managed to do this across X11, Windows, and Mac interfaces. I don't see why we can't do something similar. 6) I think we should create the SIG and start hashing out the design issues ASAP. A table containing a consensus of each plotting packages strengths and weaknesses might help immensely in our final design. -- Paul _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From hochberg@wwa.com Thu Jul 24 17:20:39 1997 From: hochberg@wwa.com (Timothy A. Hochberg) Date: Thu, 24 Jul 1997 11:20:39 -0500 (CDT) Subject: [MATRIX-SIG] -3**(-1/3.), sgn() In-Reply-To: Message-ID: On Thu, 24 Jul 1997, Jean-Bernard ADDOR wrote: > I would like to write sgn(a)*abs(a)**(-1/3.) but the sgn function seems > not to be defined in python (perhaps because it is not in c, but it is in > fortran). I can use greater_equal. Do you have a better solution ? For what it's worth, sgn(x) can be implemented as: def sgn(x): return greater_equal(x,0) - less_equal(x,0) (or is that what you meant by use greater equal?) ____ /im +------------------------------------------------+ |Tim Hochberg Research Assistant | |hochberg wwa.com University of Illinois | | Electrical Engineering | +------------------------------------------------+ _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From jbaddor@phy.ulaval.ca Thu Jul 24 18:42:01 1997 From: jbaddor@phy.ulaval.ca (Jean-Bernard ADDOR) Date: Thu, 24 Jul 1997 13:42:01 -0400 (EDT) Subject: [MATRIX-SIG] -3**(-1/3.), sgn() In-Reply-To: Message-ID: What I meant was : def sgn(a): return 2*Numeric.greater_equal(a,0) - 1 I don't know which one is better (faster, without using too much memory). Your give sgn(0)=0 what is better that sgn(0)=1 for mine, but for my application it doesn't change anythink. Jean-Bernard On Thu, 24 Jul 1997, Timothy A. Hochberg wrote: > On Thu, 24 Jul 1997, Jean-Bernard ADDOR wrote: > > > I would like to write sgn(a)*abs(a)**(-1/3.) but the sgn function seems > > not to be defined in python (perhaps because it is not in c, but it is in > > fortran). I can use greater_equal. Do you have a better solution ? > > For what it's worth, sgn(x) can be implemented as: > > def sgn(x): > return greater_equal(x,0) - less_equal(x,0) > > (or is that what you meant by use greater equal?) > > ____ > /im > > +------------------------------------------------+ > |Tim Hochberg Research Assistant | > |hochberg wwa.com University of Illinois | > | Electrical Engineering | > +------------------------------------------------+ > > _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From barrett@compass.gsfc.nasa.gov Thu Jul 24 19:14:57 1997 From: barrett@compass.gsfc.nasa.gov (Paul Barrett) Date: Thu, 24 Jul 1997 14:14:57 -0400 Subject: [MATRIX-SIG] Revised proposal for PLOT-SIG In-Reply-To: References: <199707241611.MAA24662@compass.> Message-ID: <199707241814.OAA25006@compass.> David Ascher writes: > > 2) Now that the MATRIX-SIG is nearly complete, a PLOT-SIG (or > > GRAPHICS-SIG, or DATA-DISPLAY-SIG) is of next highest priority for me. > > I don't think that GRAPHICS-SIG is appropriate -- too many things qualify > as graphics -- I think that displaying data is indeed the core issue. And > DATA-DISPLAY-SIG is a bit verbose. Can you live with PLOT-SIG? > I can live with it. I just tossed out these alternative SIG names to stimulate others to think about issues other than just 2-D plotting. (I don't expect any of them to be adopted and used as the name of the SIG.) I think we should consider how our work may possibly affect the inevitable next step of developing a scientific visualization package for Python. -- Paul _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From da@maigret.cog.brown.edu Thu Jul 24 19:46:41 1997 From: da@maigret.cog.brown.edu (David Ascher) Date: Thu, 24 Jul 1997 14:46:41 -0400 (EDT) Subject: [MATRIX-SIG] Revised proposal for PLOT-SIG In-Reply-To: <199707241814.OAA25006@compass.> Message-ID: > I can live with it. I just tossed out these alternative SIG names to > stimulate others to think about issues other than just 2-D plotting. > (I don't expect any of them to be adopted and used as the name of the > SIG.) I think we should consider how our work may possibly affect the > inevitable next step of developing a scientific visualization package > for Python. Fair enough. Sure, 3d + animation + head-mounted displays and a VR cave would be nice too (Brown will be getting one of those soon, but I'll be gone =). --da _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From ptaney@usgs.gov Fri Jul 25 01:04:02 1997 From: ptaney@usgs.gov (Paul Taney, Computer Specialist, Reston, VA ) Date: Thu, 24 Jul 1997 20:04:02 -0400 Subject: [MATRIX-SIG] Revised proposal for PLOT-SIG In-Reply-To: Message-ID: <199707250004.UAA09316@mailrvares.er.usgs.gov> On Thu, 24 Jul 1997 14:46:41 -0400 (EDT) David Ascher said: > > > I can live with it. I just tossed out these alternative SIG names to > > stimulate others to think about issues other than just 2-D plotting. > > (I don't expect any of them to be adopted and used as the name of the > > SIG.) I think we should consider how our work may possibly affect the > > inevitable next step of developing a scientific visualization package > > for Python. > > Fair enough. Sure, 3d + animation + head-mounted displays and a VR cave > would be nice too > PLOT is a dull word, IMHO, like PLOD. If you call it SCIVIZ-SIG, VIZ-SIG or PYTHON-3D you'll have more participants, maybe even more help. This is sexy stuff :-) paul ptaney@usgs.gov @ Starbucks sells a Javachip. (It's U.S.Geological Survey @ ice cream, but that shouldn't reston, va 22092 @ be an obstacle for the Java ph: 703-648-5598 @ marketing people.) -- Jeremy _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From barrett@compass.gsfc.nasa.gov Fri Jul 25 15:02:46 1997 From: barrett@compass.gsfc.nasa.gov (Paul Barrett) Date: Fri, 25 Jul 1997 10:02:46 -0400 Subject: [MATRIX-SIG] Revised proposal for PLOT-SIG In-Reply-To: <199707250004.UAA09316@mailrvares.er.usgs.gov> References: <199707250004.UAA09316@mailrvares.er.usgs.gov> Message-ID: <199707251402.KAA26770@compass.> Paul Taney writes: > > PLOT is a dull word, IMHO, like PLOD. If you call it SCIVIZ-SIG, VIZ-SIG > or > PYTHON-3D you'll have more participants, maybe even more help. This is > sexy > stuff :-) > I also considered SCIVIZ-SIG as an alternative name. It has a nice ring to it! -- Paul _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From hyoon@nyptsrv1.etsd.ml.com Fri Jul 25 15:41:38 1997 From: hyoon@nyptsrv1.etsd.ml.com (Hoon Yoon - IPT Quant) Date: Fri, 25 Jul 1997 10:41:38 -0400 Subject: [MATRIX-SIG] hash() function w/ NumPy Message-ID: <199707251441.KAA07971@ptq3.etsd> Hello, Taking a break from all the encouraging discussion about the Plots. I am attempting to incorporate hash('STR') to an Array. Since hash turns a String into uniq numeric sequence, I am trying to use it to sneak in string id's into Numerical arrays. Now I know how to turn one into hash, but how do I get it back to a string? There is supposed to be function, but I remeber reading that I may not be able get the same string back in reverse. If anyone remembers the function and the documentation in Lutz or Web pages let me know. If anyone had done some of these work already, please tip me off few tricks. I guess this is not exactely recommended, but I cannot think of a good way to use strings as row and/or column headings (as a list may be). Has anyone done this? Thanks much, Hoon, _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From jim@ks.uiuc.edu Fri Jul 25 17:08:33 1997 From: jim@ks.uiuc.edu (Jim Phillips) Date: Fri, 25 Jul 97 11:08:33 -0500 Subject: [MATRIX-SIG] Proposal: "Tables" in NumPy In-Reply-To: <199707251441.KAA07971@ptq3.etsd> References: <199707251441.KAA07971@ptq3.etsd> Message-ID: <199707251608.AA151256914@london.ks.uiuc.edu> Hoon Yoon wrote: > I am attempting to incorporate hash('STR') to an Array. Since hash turns > a String into uniq numeric sequence, I am trying to use it to sneak in The has function does not return a unique number for every string. This would be impossible! A hash function returns an integer given some data, but collisions do occur and have to be dealt with. There is definitely not a 1-1 mapping between strings and hash values (there are fewer 10 digit numbers than 10 character strings). As I see it, there are two possibilities. You can store the strings separately in a list, dictionary, array, etc. and define your own id's based on that, or you can write your own function to map (short!) strings uniquely to integers. I do sympathize with this problem. I would really love to see something higher level than an array, perhaps called a "table", which could be indexed as a dictionary (columns by heading) in some dimensions, and as an array (rows by index) in others. This would really simplify data storage since one could label data and access it without worrying about which column "pressure" is. Pickled tables would also be self documenting! I have written something like this in python, but only in two dimensions and it is rather slow. I would love to see a full N-dimension implementation as a part of NumPy. Perhaps a good interface would look something like this: >>> a # a 2-D table dogs cats birds 2 4 6 1 2 3 9 8 7 >>> a['dogs'] # returns array (reference!) 2 1 9 >>> a[:,1] # returns table (reference!) dogs cats birds 1 2 3 >>> a[('dogs','birds'),0:2] # returns table (reference!) dogs birds 2 6 1 3 This could really be a killer feature for data analysis since we would no longer have to choose between C-style structures and FORTRAN-style arrays! -Jim Phillips _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From da@maigret.cog.brown.edu Fri Jul 25 17:30:50 1997 From: da@maigret.cog.brown.edu (David Ascher) Date: Fri, 25 Jul 1997 12:30:50 -0400 (EDT) Subject: [MATRIX-SIG] RFP: PLOT-SIG Message-ID: After somewhat extensive discussion in the MATRIX-SIG, I am officially submitting a proposal for the creation of a new sig. Here is the one line comment and more extended docstring. I am volunteering as sigmeister. Please comment. This announcement is cc'ed to MATRIX-SIG, but I request that all followups be directed at the META-SIG. Please check your cc: line before sending comments. --david ascher -------------------------------- # PLOT-SIG: Development of Data Plotting Solutions for Python """ The purpose of this list is to develop, coddle together, adopt or otherwise make available Python tools for plotting of scientific and business plots of data. Plotting needs vary greatly depending on each user's requirements. Some users wish to see a better API between Python and existing plotting libraries or programs. Others would rather see a new framework which maximally leverages the OOP and dynamic strengths of Python, at the cost of reinventing a few wheels. Both strategies will be entertained by the SIG, with individual members contributing to projects they wish to see furthered. Principles guiding the development of all software will include: * Ease of use. * Integration with other Python packages (NumPy, PIL, etc.). * Quality of the software. * Quality of the output. One possible goal for the API project is to develop a package- independent API for plots, which would produce reasonably similar results regardless of the plotting package used as a backend (PLPlot, Gist/Yorick, Gnuplot, etc.), in the same spirit as the interface defined by the DB-SIG. Package-specific extensions could naturally be provided as well. The goals for the new framework need to be further specified by the SIG, but include: * Complete Python Control. * Extensibility/Customizability. * High quality rendering both on screen and paper. * Portability (at least UNIX/X11 and Win32, MacOS if feasible). * Stealing good ideas others have had. * Interactivity The SIG is open, and membership is expected to include folks who wish to contribute to the development efforts, folks who have lots of expertise they wish to share, and novice users who wish to share their lists of requirements, questions about the available software, etc. """ _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From arw@dante.mh.lucent.com Fri Jul 25 17:29:54 1997 From: arw@dante.mh.lucent.com (Aaron Watters) Date: Fri, 25 Jul 1997 12:29:54 -0400 Subject: [MATRIX-SIG] hash() function w/ NumPy Message-ID: <199707251626.MAA27320@dante.mh.lucent.com> The hash() function takes a string of arbitrary length and transforms it into a 32 bit value. If the string has more than 32 bits there is no way to get the string back in general. However, if the hash values are unique you can use a ancillary dictionary mapping hash(x) --> x, but you better be ready to do something appropriate in the case that collisions occur. Maybe a better strategy is to simply number the strings sequentially in an external index structure. class string_cache: def __init__(self): self.dict = {} self.list = [] def insert(self, str): d = self.dict if d.has_key(str): return d[str] else: result = d[str] = len(d) self.list.append(str) return result def get_by_index(self, index): return self.list[index] Untested, and I hope you don't want to do deletions :). If you want to get the strings back in aggregate (ie, get a bunch of strings from the structure in a couple interpreter ticks) the above could be modified to use my favorite extension module, kjbuckets :) but I'll spare you. With Python 1.4+ string hash collisions are unlikely (with 1.3, sadly, they were common, and still are for tuples, but this will be fixed in 1.5). -- Aaron Watters ---------- > From: Hoon Yoon - IPT Quant > To: matrix-sig@python.org > Subject: [MATRIX-SIG] hash() function w/ NumPy > Date: Friday, July 25, 1997 10:41 AM > > Hello, > > Taking a break from all the encouraging discussion about the Plots. > I am attempting to incorporate hash('STR') to an Array. Since hash turns > a String into uniq numeric sequence, I am trying to use it to sneak in > string id's into Numerical arrays. Now I know how to turn one into hash, > but how do I get it back to a string? There is supposed to be function, > but I remeber reading that I may not be able get the same string back > in reverse. If anyone remembers the function and the documentation in > Lutz or Web pages let me know. If anyone had done some of these work > already, please tip me off few tricks. > I guess this is not exactely recommended, but I cannot think of a good > way to use strings as row and/or column headings (as a list may be). Has > anyone done this? > Thanks much, > > Hoon, A sus ordenes :). _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From da@maigret.cog.brown.edu Fri Jul 25 17:37:02 1997 From: da@maigret.cog.brown.edu (David Ascher) Date: Fri, 25 Jul 1997 12:37:02 -0400 (EDT) Subject: [MATRIX-SIG] Proposal: "Tables" in NumPy In-Reply-To: <199707251608.AA151256914@london.ks.uiuc.edu> Message-ID: On Fri, 25 Jul 1997, Jim Phillips wrote: > I do sympathize with this problem. I would really love to see something > higher level than an array, perhaps called a "table", which could be > indexed as a dictionary (columns by heading) in some dimensions, and as > an array (rows by index) in others. This would really simplify data > storage since one could label data and access it without worrying about > which column "pressure" is. Pickled tables would also be self > documenting! This is one of the features of the S+ data.frame object, which is the topic of another thread on this SIG. For anyone interested in this, I recommend finding and reading Chapter 3 of "Statistical Models in S", by John M. Chambers and Trevor J. Hastie. You don't really need to know anything about statistics, but the chapter will explain what one (IMHO, very good) solution to this problem looks like. We can't do all of data.frame in Python (due to a few general syntactic issues such as the mechanism behind comparisons and the way indexing works), but we can get a lot of it done w/ the right class wrappers. It's yet another one of my long-term projects, but it's still in the design phase. --david _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From hyoon@nyptsrv1.etsd.ml.com Fri Jul 25 17:54:47 1997 From: hyoon@nyptsrv1.etsd.ml.com (Hoon Yoon - IPT Quant) Date: Fri, 25 Jul 1997 12:54:47 -0400 Subject: [MATRIX-SIG] hash() function w/ NumPy Message-ID: <199707251654.MAA08227@ptq3.etsd> div_prj =3D [] id_lst =3D [] for i in range(0,len(prj_divs),1): div_spstr =3D string.split(prj_divs[i]) id =3D div_spstr[3] =04 if not id_lst.count(id): id_lst.append(id) =09 div_prj.append([id_lst.index(id),chkdate(div_spstr[0]),eval(div_spstr[1])= ]) div_prj =3D array(div_prj) Thank Aaron: Here's something I came up with. Basically a similar idea, but it = keeps list of id_lst, which I can be pointed right back by the array = div_prj. This is not horribly bad, since it does do what I wanted with out = reverse of hash. I guess, I need to come up with Qstrategy for comparing = indicies from 2 matricies and print function that grabs original names back, = etc... =20 I am still unused to the idea that Python could pretty much do = anything. I can probably class these things and get back something extremely = similar, but better than Gauss's way of handling stuff. Although result may be = bit slower than single purpose Gauss, I can be much more creative. =20 'till I see you, Hoon, _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From aaron_watters@msn.com Sun Jul 27 13:36:51 1997 From: aaron_watters@msn.com (aaron watters) Date: Sun, 27 Jul 97 12:36:51 UT Subject: [MATRIX-SIG] vrml for plotting Message-ID: Hey kids! I love the Python list. It's quite obvious to me (now that I've been remindided) that for many forms of sci-viz vrml is the right tool! Maybe soon I'll have a closer look at this. (and it'll pop up in your web browser too!) I quote one of the esteemed mitchell brothers: Subject: Re: VRML 2.0 modeling From: John Mitchell Date: 1997/07/25 Message-Id: Newsgroups: comp.lang.python [More Headers] On Thu, 24 Jul 1997, Lance Ellinghaus wrote: > Does anyone have any classes that can be used to build simple VRML 2.0 > scenes and output > them to a web browser? I have a bunch of classes, basically mapping VRML objects to Python classes. It's cool in that you, with a little headscratching, can translate VRML2 example "wrl" files into 100% python. The main problem is that the classes are very "blobby", do no error checking, and lots of things that are "obvious" wont work. VRML requires certain things in certain orders and if they aint, you get nothing. But, last time I checked, the code works. I translated a few examples to Python classes, and wrote a VRML Clock. Its at: http://people.magnet.com/~johnm/VRML/ hope this helps. - j _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From arw@dante.mh.lucent.com Mon Jul 28 15:00:23 1997 From: arw@dante.mh.lucent.com (Aaron Watters) Date: Mon, 28 Jul 1997 10:00:23 -0400 Subject: [MATRIX-SIG] Proposal: "Tables" in NumPy Message-ID: <199707281356.JAA26322@dante.mh.lucent.com> I think you can approximate this, maybe, by cross breeding kjbuckets.kjDict structures and arrays. I'll have to have a think on this. For a bit of a performance penalty (more loops in Python) you could use standard python dictionaries. On the matrix end I have this inkling that a "dual of take" might come in handy here. In fact, I'm pretty sure of it. Mind if I have a look at your code? This is quite an exciting concept, I think. -- Aaron Watters ps: more generally, you'd like full integration of matrices in relational algebra-style tables (ie, sets of tuples). ---------- > From: Jim Phillips > To: matrix-sig@python.org > I have written something like this in python, but only in two dimensions and > it is rather slow. I would love to see a full N-dimension implementation as a > part of NumPy. Perhaps a good interface would look something like this: > > >>> a # a 2-D table > dogs cats birds > 2 4 6 > 1 2 3 > 9 8 7 > >>> a['dogs'] # returns array (reference!) > 2 1 9 > >>> a[:,1] # returns table (reference!) > dogs cats birds > 1 2 3 > >>> a[('dogs','birds'),0:2] # returns table (reference!) > dogs birds > 2 6 > 1 3 > > This could really be a killer feature for data analysis since we would no > longer have to choose between C-style structures and FORTRAN-style arrays! > > -Jim Phillips > > _______________ > MATRIX-SIG - SIG on Matrix Math for Python > > send messages to: matrix-sig@python.org > administrivia to: matrix-sig-request@python.org > _______________ > _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From c9603085@zeus.hud.ac.uk Tue Jul 29 11:18:50 1997 From: c9603085@zeus.hud.ac.uk (MISS. K.L.COLLIER) Date: Tue, 29 Jul 1997 11:18:50 +0100 Subject: [MATRIX-SIG] convolution Message-ID: <199707291018.LAA04298@artemis.csm> Hi, I want to find the convolution of two images and am not sure how to do it. There is a convolve function but its only 1D and there are 2D fft's but not an inverse 2D fft? How do I do it? Any ideas? Thanks Karen Collier c9603085@zeus.hud.ac.uk _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From Daniel.Michelson@smhi.se Tue Jul 29 14:37:45 1997 From: Daniel.Michelson@smhi.se (Daniel Michelson) Date: Tue, 29 Jul 1997 15:37:45 +0200 Subject: [MATRIX-SIG] propaganda Message-ID: <33DDF229.52BF@smhi.se> Greetings! Python is already being used at NASA and we want it to get a foot in the door at EUMETSAT too! We're involved in a very large project designed to prototype the next generation methods and systems for processing meteorological satellite data for all EUMETSAT members. We're part of a consortium of European weather services of which we're the only ones with any knowledge of and experience with Python. = We're now faced with the task of convincing our partners that Python is worth using. So far, the attitude seems to be that Python is assumed "guilty" until proven "innocent"; the preferences among our collegues are ANSI C and Fortran 77 administered by sh or ksh... :-( There is a lot of good information at python.org but: = If anyone has written a page or two (or more) previously on the benefits of using Python when building a large distributed system, would you please share it with me? Please help! We don't want to be caught in a time warp and forced to use old, slow methods! best, -d +------------------+--------------------------------------------------+ | | Daniel B. Michelson | | /\ |\ /| | | | | Swedish Meteorological and Hydrological Institute| | \ | \/ | |__| | | S-601 76 Norrk=F6ping, Sweden |= | \ | | | | | | Telephone: +46 11 - 15 84 94 | | \/ | | | | | | Telefax: +46 11 - 17 02 07 | | | Internet: Daniel.Michelson@smhi.se | +------------------+--------------------------------------------------+ _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From arw@dante.mh.lucent.com Tue Jul 29 16:05:28 1997 From: arw@dante.mh.lucent.com (Aaron Watters) Date: Tue, 29 Jul 1997 11:05:28 -0400 Subject: [MATRIX-SIG] propaganda Message-ID: <199707291501.LAA08323@dante.mh.lucent.com> Is there any chance you can convince them that Python is just another C library with which to build tools? Maybe they'd be more receptive if it isn't a black and white situation. If you do, of course, I expect the other tools will be able to sit back and let Python do most of the work, however. I may send you some email addresses privately. The following come to mind: infoseek worldnet (att) industrial light and magic (ILM) ILU (xerox PARC) gstein@microsoft.com (this one I'll send publicly :) ). There is a "who uses python" page at http://www.python.org too. -- Aaron Watters ---------- From: Daniel Michelson To: guido@CNRI.Reston.Va.US Cc: matrix-sig@python.org; image-sig@python.org Subject: [MATRIX-SIG] propaganda Date: Tuesday, July 29, 1997 9:37 AM Greetings! Python is already being used at NASA and we want it to get a foot in the door at EUMETSAT too! We're involved in a very large project designed to prototype the next generation methods and systems for processing meteorological satellite data for all EUMETSAT members. We're part of a consortium of European weather services of which we're the only ones with any knowledge of and experience with Python. We're now faced with the task of convincing our partners that Python is worth using. So far, the attitude seems to be that Python is assumed "guilty" until proven "innocent"; the preferences among our collegues are ANSI C and Fortran 77 administered by sh or ksh... :-( There is a lot of good information at python.org but: If anyone has written a page or two (or more) previously on the benefits of using Python when building a large distributed system, would you please share it with me? Please help! We don't want to be caught in a time warp and forced to use old, slow methods! best, -d +------------------+--------------------------------------------------+ | | Daniel B. Michelson | | /\ |\ /| | | | | Swedish Meteorological and Hydrological Institute| | \ | \/ | |__| | | S-601 76 Norrköping, Sweden | | \ | | | | | | Telephone: +46 11 - 15 84 94 | | \/ | | | | | | Telefax: +46 11 - 17 02 07 | | | Internet: Daniel.Michelson@smhi.se | +------------------+--------------------------------------------------+ _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ ---------- _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From hochberg@wwa.com Tue Jul 29 16:12:27 1997 From: hochberg@wwa.com (Timothy A. Hochberg) Date: Tue, 29 Jul 1997 10:12:27 -0500 (CDT) Subject: [MATRIX-SIG] convolution In-Reply-To: <199707291018.LAA04298@artemis.csm> Message-ID: On Tue, 29 Jul 1997, MISS. K.L.COLLIER wrote: > I want to find the convolution of two images and am not sure how > to do it. There is a convolve function but its only 1D and there are > 2D fft's but not an inverse 2D fft? How do I do it? Any ideas? Looking at FFT.py, fft2d is implementd as: def fft2d(a, s=None, axes=(-2,-1)): return _raw_fft2d(a,s,axes,fft) So, I'd try defining inverse_fft2d as: def inverse_fft2d(a, s=None, axes=(-2,-1)): return _raw_fft2d(a,s,axes,inverse_fft) (Note that you have to explicitly import _raw_fft2d, import * won't pull it in.) Make sure you test this - it worked for one test case, but I wouldn't call it tested. ____ /im +------------------------------------------------+ |Tim Hochberg Research Assistant | |hochberg wwa.com University of Illinois | | Electrical Engineering | +------------------------------------------------+ _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From Dale_Bertrand@brown.edu Tue Jul 29 22:30:24 1997 From: Dale_Bertrand@brown.edu (Dale Bertrand) Date: Tue, 29 Jul 1997 17:30:24 -0400 (EDT) Subject: [MATRIX-SIG] PyArrayObjects In-Reply-To: <199707291501.LAA08323@dante.mh.lucent.com> Message-ID: I am writing a small python module in c which needs to access the internals of a PyObject (which is really a PyArrayObject). I've been using the function: PyArray_ContiguousFromObject to extract the PyArrayObject from the PyObject and then modify the data via arrayobject->data. Unfortunately, it seems to me that PyArray_ContiguousFromObject makes a copy of the PyObject which I _must_ return to see my changes. I would like my c functio n to simply access and modify the array data without making a copy. How do I access the array data without copying the Object? A snip of code: static PyObject * do_array(PyObject *self,PyObject *args) { PyObject *op; /* <- I would like to edit this */ PyArrayObject *mp; /* <-without making this */ int h,w; int ubound[1]; int lbound[1]; TRY(PyArg_ParseTuple(args, "Oii", &op, &lbound , &ubound)); TRY(mp = (PyArrayObject *)PyArray_ContiguousFromObject(op,PyArray_CHAR, 2, 2 )); h = mp->dimensions[0]; w = mp->dimensions[1]; threshold((unsigned char *)mp->data, (int *)lbound , (int*)ubound, 1, w*h); Py_DECREF(mp); return (PyObject *)mp; /* I do not want to have to return this */ } Thanks -Dale Bertrand ---------------------------------- E-Mail: Dale Bertrand Date: 29-Jul-97 ---------------------------------- _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From pas@xis.xerox.com Wed Jul 30 00:28:12 1997 From: pas@xis.xerox.com (Perry A. Stoll) Date: Tue, 29 Jul 1997 16:28:12 PDT Subject: [MATRIX-SIG] PyArrayObjects In-Reply-To: Message-ID: On Tue, 29 Jul 1997, Dale Bertrand wrote: Unfortunately, it seems to me that PyArray_ContiguousFromObject makes a copy of the PyObject which I _must_ return to see my changes. I would like my c functio n to simply access and modify the array data without making a copy. I think the more python way of doing things is to have functions return a copy and leave the original unchanged. This accounts for the behaviour you correctly identified for PyArray_ContiguousFromObject. But if you have to modify the original, you have to... How do I access the array data without copying the Object? How about something like this (untested)? static PyObject * do_array(PyObject *self,PyObject *args) { PyObject *op; /* <- I would like to edit this */ PyArrayObject *mp; /* <-without making this */ int h,w; int ubound; int lbound; TRY(PyArg_ParseTuple(args, "Oii", &op, &lbound , &ubound)); /* check that it really is an array object. I believe this will forbid inheriting in Python from C objects. Oh well. */ TRY(PyArray_Check(op)); /* a simple cast now that you know it really is an array object */ mp = (PyArrayObject *)op; h = mp->dimensions[0]; w = mp->dimensions[1]; threshold((unsigned char *)mp->data, &lbound , &ubound, 1,w*h); return op; } Hope that helps. -Perry _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From arw@dante.mh.lucent.com Wed Jul 30 19:49:17 1997 From: arw@dante.mh.lucent.com (Aaron Watters) Date: Wed, 30 Jul 1997 14:49:17 -0400 Subject: [MATRIX-SIG] simplex algorithm? Message-ID: <199707301933.PAA26929@dante.mh.lucent.com> Anybody got a simplex algorithm (simple linear optimization) or is this something else to add to my todo list :)? -- Aaron Watters ps: I ask because a friend and I just determined that laying out html (with tables and frames) can be formulated as a fairly sparse linear maximization problem. Threw me back to my fond memories of beautiful England, where I once studied such stuff, too many years ago... (sigh) _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From amullhau@ix.netcom.com Wed Jul 30 23:11:34 1997 From: amullhau@ix.netcom.com (Andrew P. Mullhaupt) Date: Wed, 30 Jul 1997 18:11:34 -0400 Subject: [MATRIX-SIG] simplex algorithm? In-Reply-To: <199707301933.PAA26929@dante.mh.lucent.com> Message-ID: <3.0.1.32.19970730181134.00838ea0@popd.netcruiser> At 02:49 PM 7/30/97 -0400, Aaron Watters wrote: >Anybody got a simplex algorithm >(simple linear optimization) >or is this something else to add >to my todo list :)? -- Aaron Watters > >ps: I ask because a friend and I > just determined that laying out > html (with tables and frames) > can be formulated as a fairly sparse > linear maximization problem. Stop! Do not implement the simplex algorithm from eight billion years ago until you look at the new 'interior point' and primal-dual methods. A good book is Steven Wright's book on primal-dual methods. Get this book, which has pointers to code you can get on the net. Then, if you want to write code, wrap up this code in a nice module. Later, Andrew Mullhaupt _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From da@maigret.cog.brown.edu Thu Jul 31 03:24:36 1997 From: da@maigret.cog.brown.edu (David Ascher) Date: Wed, 30 Jul 1997 22:24:36 -0400 (EDT) Subject: [MATRIX-SIG] simplex algorithm? In-Reply-To: <3.0.1.32.19970730181134.00838ea0@popd.netcruiser> Message-ID: Aaron: > Anybody got a simplex algorithm (simple linear optimization) or is this > something else to add to my todo list :)? -- Aaron Watters > > ps: I ask because a friend and I just determined that laying out html > (with tables and frames) can be formulated as a fairly sparse linear > maximization problem. Threw me back to my fond memories of beautiful > England, where I once studied such stuff, too many years ago... (sigh) Andrew: > Stop! Do not implement the simplex algorithm from eight billion years ago > until you look at the new 'interior point' and primal-dual methods. I guess now we know how old Aaron is, +/- a few years. --david _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From jrwebb@goodnet.com Thu Jul 31 04:41:52 1997 From: jrwebb@goodnet.com (James R. Webb) Date: Wed, 30 Jul 1997 20:41:52 -0700 Subject: [MATRIX-SIG] Revised proposal for PLOT-SIG Message-ID: <199707310339.UAA14455@mail.goodnet.com> The programmers out there may want to look at what the developers of PerlDL are doing. They have created a nice plotting interface to PGPlot and OpenGL/MesaGL. Might want to check out http://www.aao.gov.au/local/www/kgb/perldl/ to stimulate the discussion. _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From Daniel.Michelson@smhi.se Tue Jul 29 14:37:45 1997 From: Daniel.Michelson@smhi.se (Daniel Michelson) Date: Tue, 29 Jul 1997 15:37:45 +0200 Subject: [MATRIX-SIG] propaganda Message-ID: <33DDF229.52BF@smhi.se> Greetings! Python is already being used at NASA and we want it to get a foot in the door at EUMETSAT too! We're involved in a very large project designed to prototype the next generation methods and systems for processing meteorological satellite data for all EUMETSAT members. We're part of a consortium of European weather services of which we're the only ones with any knowledge of and experience with Python. = We're now faced with the task of convincing our partners that Python is worth using. So far, the attitude seems to be that Python is assumed "guilty" until proven "innocent"; the preferences among our collegues are ANSI C and Fortran 77 administered by sh or ksh... :-( There is a lot of good information at python.org but: = If anyone has written a page or two (or more) previously on the benefits of using Python when building a large distributed system, would you please share it with me? Please help! We don't want to be caught in a time warp and forced to use old, slow methods! best, -d +------------------+--------------------------------------------------+ | | Daniel B. Michelson | | /\ |\ /| | | | | Swedish Meteorological and Hydrological Institute| | \ | \/ | |__| | | S-601 76 Norrk=F6ping, Sweden |= | \ | | | | | | Telephone: +46 11 - 15 84 94 | | \/ | | | | | | Telefax: +46 11 - 17 02 07 | | | Internet: Daniel.Michelson@smhi.se | +------------------+--------------------------------------------------+ _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ From arw@dante.mh.lucent.com Tue Jul 29 16:05:28 1997 From: arw@dante.mh.lucent.com (Aaron Watters) Date: Tue, 29 Jul 1997 11:05:28 -0400 Subject: [MATRIX-SIG] propaganda Message-ID: <199707291501.LAA08323@dante.mh.lucent.com> Is there any chance you can convince them that Python is just another C library with which to build tools? Maybe they'd be more receptive if it isn't a black and white situation. If you do, of course, I expect the other tools will be able to sit back and let Python do most of the work, however. I may send you some email addresses privately. The following come to mind: infoseek worldnet (att) industrial light and magic (ILM) ILU (xerox PARC) gstein@microsoft.com (this one I'll send publicly :) ). There is a "who uses python" page at http://www.python.org too. -- Aaron Watters ---------- From: Daniel Michelson To: guido@CNRI.Reston.Va.US Cc: matrix-sig@python.org; image-sig@python.org Subject: [MATRIX-SIG] propaganda Date: Tuesday, July 29, 1997 9:37 AM Greetings! Python is already being used at NASA and we want it to get a foot in the door at EUMETSAT too! We're involved in a very large project designed to prototype the next generation methods and systems for processing meteorological satellite data for all EUMETSAT members. We're part of a consortium of European weather services of which we're the only ones with any knowledge of and experience with Python. We're now faced with the task of convincing our partners that Python is worth using. So far, the attitude seems to be that Python is assumed "guilty" until proven "innocent"; the preferences among our collegues are ANSI C and Fortran 77 administered by sh or ksh... :-( There is a lot of good information at python.org but: If anyone has written a page or two (or more) previously on the benefits of using Python when building a large distributed system, would you please share it with me? Please help! We don't want to be caught in a time warp and forced to use old, slow methods! best, -d +------------------+--------------------------------------------------+ | | Daniel B. Michelson | | /\ |\ /| | | | | Swedish Meteorological and Hydrological Institute| | \ | \/ | |__| | | S-601 76 Norrköping, Sweden | | \ | | | | | | Telephone: +46 11 - 15 84 94 | | \/ | | | | | | Telefax: +46 11 - 17 02 07 | | | Internet: Daniel.Michelson@smhi.se | +------------------+--------------------------------------------------+ _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________ ---------- _______________ MATRIX-SIG - SIG on Matrix Math for Python send messages to: matrix-sig@python.org administrivia to: matrix-sig-request@python.org _______________