From Oliphant.Travis@mayo.edu Thu Sep 2 00:34:33 1999 From: Oliphant.Travis@mayo.edu (Travis Oliphant) Date: Wed, 1 Sep 1999 18:34:33 -0500 (CDT) Subject: [Matrix-SIG] Sparse matrices (again?) Message-ID: Well, it's been a few months since I first talked about sparse matrices and like many of you I've been busy cleaning up some other issues on my plate (and taking a vacation). Well, now I really need sparse matrix support so I'm spending some time on it. There were some excellent comments back in May and quite a bit of interest which is why I'm mailing this list a progress-report-of-sorts so that I can avoid unnecessary duplication of effort if possible and so I can benefit from the more knowledgeable people out there (and solicit any help by interested people). I've wrapped up most of sparsekit2 and some of the sparse blas (from NIST) after making changes to them so they will support single and double precision for real and complex matrices. These packages will provide the core (fast) functionality of sparse matrix operations and conversions from one format to another. Sparse kit contains conversion routines to convert between 15 different formats but only contains basic matrix operations for compressed sparse row format. The sparse blas contains matrix-vector operations for many different formats also but I have only wrapped up compressed sparse row and compressed sparse column (as these were the only ones I converted from double-precision-only to all-precisions as a start). Now I'm setting up the Python class which will call these underlying routines to do the operations and this is where I need some advice. I'd thought to name the class spMatrix and subclass it from the Array base class (thanks to the ExtensionClass foundation for NumPy 1.12), but I wonder what the benefit is of doing that since I will need to redefine most operations anyway. Any ideas? When I get a fleshed out sparse class I'll post what I've got. Alternatively if someone as a server they where CVS development could take place that they'd be willing to donate I would place it there. Just letting those interested know that work is progressing. Travis, From Oliphant.Travis@mayo.edu Thu Sep 2 03:34:59 1999 From: Oliphant.Travis@mayo.edu (Travis Oliphant) Date: Wed, 1 Sep 1999 21:34:59 -0500 (CDT) Subject: [Matrix-SIG] An area where the C-API for NumPy 1.12 is not 100% compatible Message-ID: I just got bit by an area where NumPy 1.12 is not 100% compatible with older versions when using it in C. If you use the O! syntax in PyArg_ParseTuple to check for array object arguments (by using &PyArray_Type as the extra argument) you will have trouble with your extension module with the NumPy 1.12 release since the object passed is now a subclass of the extension class. I don't know of an easy fix other than using O& with a converter function that uses PyArray_Check to check or going to the old check-each-argument-manually approach. If anyone knows of a better approach let me know. Thanks Travis From da@ski.org Thu Sep 2 05:01:23 1999 From: da@ski.org (David Ascher) Date: Wed, 1 Sep 1999 21:01:23 -0700 (Pacific Daylight Time) Subject: [Matrix-SIG] An area where the C-API for NumPy 1.12 is not 100% compatible In-Reply-To: Message-ID: On Wed, 1 Sep 1999, Travis Oliphant wrote: > I just got bit by an area where NumPy 1.12 is not 100% compatible with > older versions when using it in C. > > If you use the O! syntax in PyArg_ParseTuple to check for array object > arguments (by using &PyArray_Type as the extra argument) you will have > trouble with your extension module with the NumPy 1.12 release since the > object passed is now a subclass of the extension class. > > I don't know of an easy fix other than using O& with a converter function > that uses PyArray_Check to check or going to the old > check-each-argument-manually approach. > > If anyone knows of a better approach let me know. Ouch. I can't think of a solution that doesn't involve patching the core. If you write the converter function, let me know and I'll include it somewhere relevant. --david ascher From pearu@ioc.ee Thu Sep 2 07:59:45 1999 From: pearu@ioc.ee (Pearu Peterson) Date: Thu, 2 Sep 1999 09:59:45 +0300 (EETDST) Subject: [Matrix-SIG] Sparse matrices (again?) In-Reply-To: Message-ID: On Wed, 1 Sep 1999, Travis Oliphant wrote: > > I've wrapped up most of sparsekit2 and some of the sparse blas (from NIST) > after making changes to them so they will support single and double > precision for real and complex matrices. These packages will provide the > core (fast) functionality of sparse matrix operations and conversions from > one format to another. Are you using f2py.py for wrapping? I think that I should then keep f2py.py project alive as people may already use it and its serves raw interfacing idea perfectly (I am not sure that PyFort ever will, or at least in near future). Pearu From hinsen@cnrs-orleans.fr Fri Sep 3 15:03:18 1999 From: hinsen@cnrs-orleans.fr (Konrad Hinsen) Date: Fri, 3 Sep 1999 16:03:18 +0200 Subject: [Matrix-SIG] An area where the C-API for NumPy 1.12 is not 100% compatible In-Reply-To: (message from Travis Oliphant on Wed, 1 Sep 1999 21:34:59 -0500 (CDT)) References: Message-ID: <199909031403.QAA31181@chinon.cnrs-orleans.fr> > If you use the O! syntax in PyArg_ParseTuple to check for array object > arguments (by using &PyArray_Type as the extra argument) you will have > trouble with your extension module with the NumPy 1.12 release since the > object passed is now a subclass of the extension class. I'd prefer a fix in NumPy, if possible. Without looking at any code at all, could one rename the type of the "fundamental" array object to something else and use PyArray_Type for the subclass? Assuming of course that subclasses have proper type objects. > I don't know of an easy fix other than using O& with a converter function > that uses PyArray_Check to check or going to the old > check-each-argument-manually approach. I wouldn't want to do either to my code, I have tons of such checks... Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.55.69 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From Oliphant.Travis@mayo.edu Fri Sep 3 17:18:50 1999 From: Oliphant.Travis@mayo.edu (Travis Oliphant) Date: Fri, 3 Sep 1999 11:18:50 -0500 (CDT) Subject: [Matrix-SIG] A sparse matrix implementation. Message-ID: I've got a sparse matrix implementation working based on the SPARSKIT module by Yousef Saad. It's available currently only through cvs. If you are interested in helping with this project (and have the time) let me know and I can get access to the cvs server for you or just send you a copy of the current source code. If you have a cvs client then run export CVSROOT=:pserver:anonymous@koer.ioc.ee:/usr/local/cvsroot (or use setenv CVSROOT ****** if you run csh) cvs login # the password is guest cvs checkout Sparse Once you have a version checked out run cvs update -Pd to get the latest version. (If you have full access then you can modify the files and commit your changes using cvs commit -m "Some comment" ) (You can checkout multipack from the same server in the same way if you want (that is quite large though.)) Organization: It is a base Python class with a helper extension module to perform rapid calculations. The helper module is a wrapping of most of the SPARSKIT toolbox. I have toyed with the idea of building a sparse matrix Extension class which may theoretical improve the speed of some operations and allow for fast and fancy operations, but I don't know if it is worth the effort or not. I'm interested in feedback on this issue. Currently the internal format is Compressed Sparse Row but the structure is set up to allow multiple formats. Supported so far are matrix addition and subtraction, initialization from coordinate format, and multiplication of a sparse matrix by a dense vector. Since sparse matrices inherently store indices there are a couple of C-index style versus Fortran-index style questions to work out. The SPARSEKIT library is in Fortran and uses that style. Right now this is also the internal format of the class. Check it out if you are interested. Thanks, Travis From da@ski.org Fri Sep 3 18:23:48 1999 From: da@ski.org (David Ascher) Date: Fri, 3 Sep 1999 10:23:48 -0700 (Pacific Daylight Time) Subject: [Matrix-SIG] An area where the C-API for NumPy 1.12 is not 100% compatible In-Reply-To: <199909031403.QAA31181@chinon.cnrs-orleans.fr> Message-ID: > I'd prefer a fix in NumPy, if possible. Without looking at any code > at all, could one rename the type of the "fundamental" array object > to something else and use PyArray_Type for the subclass? Assuming > of course that subclasses have proper type objects. Hmm. The problem is that currently all of the internal (C) code creates objects of the fundamental data type, and all the .py code creates objects of a subclass of said type. I think having this duality is in fact a very good thing, as it means we can add features to the standard data type which Python users manipulate w/o messing w/ C code. Whether the fundamental data type (currently PyArray_Type) or the subclass (Array) is used for a 'type equality' test, some code will break, depending on whether it gets its arguments from a 'core' function or a python function. I think the proper check is to do a subclass test instead, which is what PyArray_Check() does. Alas, "O!" is unable to do that right now. I suppose we could slip a PyArg_ParseTuple() which did it in arrayobject.h. Nasty hack, though. (the 'right' solution will have to wait for subclassability of Python types in the Python core, which isn't planned until 2.0). Any other ideas? --david From hinsen@cnrs-orleans.fr Fri Sep 3 18:42:47 1999 From: hinsen@cnrs-orleans.fr (Konrad Hinsen) Date: Fri, 3 Sep 1999 19:42:47 +0200 Subject: [Matrix-SIG] An area where the C-API for NumPy 1.12 is not 100% compatible In-Reply-To: (message from David Ascher on Fri, 3 Sep 1999 10:23:48 -0700 (Pacific Daylight Time)) References: Message-ID: <199909031742.TAA19514@chinon.cnrs-orleans.fr> > Hmm. The problem is that currently all of the internal (C) code creates > objects of the fundamental data type, and all the .py code creates objects > of a subclass of said type. I think having this duality is in fact a very > good thing, as it means we can add features to the standard data type > which Python users manipulate w/o messing w/ C code. Meaning that there are two array types which behave differently, depending on where the array was created? Sounds like a developer's nightmare to me. One of the strong points of Python is being able to do the same thing in Python or C without making any difference to the outside world. I'd like to keep that for arrays as well. Is there any reason why C modules can't return the subclass as well? > python function. I think the proper check is to do a subclass test > instead, which is what PyArray_Check() does. Alas, "O!" is unable to do But comparing type objects is a common approach in C extension modules; I wouldn't be surprised if there was some code out there that doesn't use the type check macro but a direct comparison. I am not saying that I support this kind of style, but it was never explicitly declared unsafe, and used work perfectly well. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.55.69 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From da@ski.org Fri Sep 3 19:15:23 1999 From: da@ski.org (David Ascher) Date: Fri, 3 Sep 1999 11:15:23 -0700 (Pacific Daylight Time) Subject: [Matrix-SIG] An area where the C-API for NumPy 1.12 is not 100% compatible In-Reply-To: <199909031742.TAA19514@chinon.cnrs-orleans.fr> Message-ID: On Fri, 3 Sep 1999, Konrad Hinsen wrote: > > Hmm. The problem is that currently all of the internal (C) code creates > > objects of the fundamental data type, and all the .py code creates objects > > of a subclass of said type. I think having this duality is in fact a very > > good thing, as it means we can add features to the standard data type > > which Python users manipulate w/o messing w/ C code. > > Meaning that there are two array types which behave differently, > depending on where the array was created? Sounds like a developer's > nightmare to me. One of the strong points of Python is being able > to do the same thing in Python or C without making any difference to > the outside world. I'd like to keep that for arrays as well. Understood. > Is there any reason why C modules can't return the subclass as well? I don't know. Given the way the CAPI stuff works, it might be nontrivial (the subclass is defined at Python runtime, so the CAPI struct has to be modified then). It's worth investigating. Alternatively, we could eliminate the subclass -- this should return things to a backwards-compatible state. Currently, all the subclass does is provide a place for the 'setting' of the array w/ a mask array. It's not enough currently to justify breaking backwards compatibility. It was my hope however that we could expand on that feature to provide the gather/scatter functionality in Python, which I believe would be a very significant feature, which is hard to do in C. Whether we make the C modules return an 'official' subclass or remove that subclass altogether, we still have a situation where there is no way to define a subclass the instances of which are usable in C modules, if a type equality test is used. This means that any such subclass will have to be 'cast' before any call to a C extension module is made. Ugh. > But comparing type objects is a common approach in C extension > modules; I wouldn't be surprised if there was some code out there that > doesn't use the type check macro but a direct comparison. I am not > saying that I support this kind of style, but it was never explicitly > declared unsafe, and used work perfectly well. Indeed. I agree it's a problem. I'm not sure how big of a problem it is. --david From Oliphant.Travis@mayo.edu Fri Sep 3 23:50:14 1999 From: Oliphant.Travis@mayo.edu (Travis Oliphant) Date: Fri, 3 Sep 1999 17:50:14 -0500 (CDT) Subject: [Matrix-SIG] New Behavior for NumPy scalars? Message-ID: O.K. I thought I had just gotten used to the old way of having NumPy return Python scalars whenever a routine ends up with a NumPy scalar (not necessarily a good idea when trying to keep everything as Floats). Now, it looks like the math operations on NumPy scalars (rank-0 arrays) result in NumPy scalars but being upcast to what a Python scalar of the appropriate type would be. I don't think this is what is wanted. If we are going to keep NumPy scalars as NumPy scalars after math results (I'm all for this as it would alleviate many of my upcasting problems) we should keep them the same type. Example: say b = array(3.0,'f') a = array(2.0,'f') It used to be that b-a returned a Python Float with value 1.0 Now b-a returns a NumPy Array of rank 0, with value 1.0 but with type 'd'. I don't think anyone likes this idea. The problem is that the __sub__ method of Array returns an Array all of the time but subtract method returns a Python object for rank-0 results. My personal solution preference is to not use PyArray_Return for the ufuncs and let users who really need a scalar do the indexing to get it from the rank-0 array. Using PyArray_Return seems like trying to second-guess the user and leads to these kinds of issues. Anyone else having any fun with the new NumPy 1.12 beta release? --Travis From Oliphant.Travis@mayo.edu Sat Sep 4 00:22:25 1999 From: Oliphant.Travis@mayo.edu (Travis Oliphant) Date: Fri, 3 Sep 1999 18:22:25 -0500 (CDT) Subject: [Matrix-SIG] A fix to recover old behavior for 1.12beta Message-ID: This fix recovers the old behavior for rank-0 arrays (i.e. operations returning rank-0 arrays are converted to Python scalars---again, I don't think I like this behavior, though). Added: def _arrayret(self,val,copy=0): if isarray(val): return self.__class__(val,copy) else: return val to the Array class and replaced self.__class__ with self._arrayret in the class. --Travis From Oliphant.Travis@mayo.edu Sat Sep 4 00:48:00 1999 From: Oliphant.Travis@mayo.edu (Travis Oliphant) Date: Fri, 3 Sep 1999 18:48:00 -0500 (CDT) Subject: [Matrix-SIG] Previous solution dumps core due to typo and long-standing bug. Message-ID: The previous solution I just posted dumped core when used on arrays with rank > 0 (I had just tried it with rank-0 arrays). The mistake is that copy should be copy=copy in the call to self.__class__ in _arrayret. The reason it dumps core is due to a long-standing bug that has still not been fixed the solution to which was posted by Charles Waldman May 26. The C-code uses the typecode argument as a string without first making sure it is a string: The following (indicated with +) needs to be added to both array_array in multiarraymodule.c and now array_init in arraymodule.c: if (tpo == Py_None) { type = PyArray_NOTYPE; } else { + if (!PyString_Check(tpo)){ + PyErr_SetString(PyExc_TypeError, + "typecode must be a character"); + return NULL; + } tp = PyString_AsString(tpo); if (tp[0] == 0) type = PyArray_NOTYPE; else type = tp[0]; Travis From frank@ned.dem.csiro.au Mon Sep 6 07:32:03 1999 From: frank@ned.dem.csiro.au (Frank Horowitz) Date: Mon, 6 Sep 1999 14:32:03 +0800 Subject: [Matrix-SIG] ASCI Red Pentium BLAS linked with NumPy? Message-ID: G'Day folks, After spending the last couple of days strategizing on how to fool the RPM system and chasing unresolved symbols in the linker on an SMP RedHat Linux 6.0 box, I'm nearly ready to give up on building NumPy (the LinearAlgebra.py contents, at least) linked with the optimized BLAS from the ASCI Red project. Has anyone else made this work? What are the tricks you used? :-) TIA for any help! Frank Horowitz -- Frank Horowitz frank@ned.dem.csiro.au Australian Geodynamics Cooperative Research Centre, and CSIRO-Exploration & Mining, PO Box 437, Nedlands, WA 6009, AUSTRALIA Direct: +61 8 9284 8431; FAX: +61 8 9389 1906; Reception: +61 8 9389 8421 From hinsen@cnrs-orleans.fr Mon Sep 6 11:20:29 1999 From: hinsen@cnrs-orleans.fr (Konrad Hinsen) Date: Mon, 6 Sep 1999 12:20:29 +0200 Subject: [Matrix-SIG] An area where the C-API for NumPy 1.12 is not 100% compatible In-Reply-To: (message from David Ascher on Fri, 3 Sep 1999 11:15:23 -0700 (Pacific Daylight Time)) References: Message-ID: <199909061020.MAA01059@chinon.cnrs-orleans.fr> > > Is there any reason why C modules can't return the subclass as well? > > I don't know. Given the way the CAPI stuff works, it might be nontrivial > (the subclass is defined at Python runtime, so the CAPI struct has to be > modified then). It's worth investigating. The C module would have to import the Python module, and that sounds like a nice case of circular imports... Not so simple indeed... > Alternatively, we could eliminate the subclass -- this should return > things to a backwards-compatible state. Currently, all the subclass does That might be the best immediate solution, until we find a better one. > Whether we make the C modules return an 'official' subclass or remove that > subclass altogether, we still have a situation where there is no way to > define a subclass the instances of which are usable in C modules, if a > type equality test is used. This means that any such subclass will have > to be 'cast' before any call to a C extension module is made. Ugh. That should be a general problem with extension classes, and perhaps one should look for a general solution. Ideally, PyArg_ParseTuple() (or a similar function provided by the ExtensionClass module) would offer both direct type comparison and subclass checks. Then of course C modules would have to be adapted - but I'd prefer to impose this only once, when a definite solution is available. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.55.69 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From hinsen@cnrs-orleans.fr Mon Sep 6 14:36:41 1999 From: hinsen@cnrs-orleans.fr (Konrad Hinsen) Date: Mon, 6 Sep 1999 15:36:41 +0200 Subject: [Matrix-SIG] ASCI Red Pentium BLAS linked with NumPy? In-Reply-To: (message from Frank Horowitz on Mon, 6 Sep 1999 14:32:03 +0800) References: Message-ID: <199909061336.PAA21108@chinon.cnrs-orleans.fr> > fool the RPM system and chasing unresolved symbols in the linker on > an SMP RedHat Linux 6.0 box, I'm nearly ready to give up on building > NumPy (the LinearAlgebra.py contents, at least) linked with the > optimized BLAS from the ASCI Red project. I got it linked after some trying, under RedHat 5.2. But the results were not always the same (sometimes very different) as I got from a straight Fortran BLAS, so I dropped it. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.55.69 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From hinsen@cnrs-orleans.fr Mon Sep 6 14:37:08 1999 From: hinsen@cnrs-orleans.fr (Konrad Hinsen) Date: Mon, 6 Sep 1999 15:37:08 +0200 Subject: [Matrix-SIG] ASCI Red Pentium BLAS linked with NumPy? In-Reply-To: (message from Frank Horowitz on Mon, 6 Sep 1999 14:32:03 +0800) References: Message-ID: <199909061337.PAA21112@chinon.cnrs-orleans.fr> > After spending the last couple of days strategizing on how to > fool the RPM system and chasing unresolved symbols in the linker on > an SMP RedHat Linux 6.0 box, I'm nearly ready to give up on building > NumPy (the LinearAlgebra.py contents, at least) linked with the > optimized BLAS from the ASCI Red project. I got it linked after some trying, under RedHat 5.2. But the results were not always the same (sometimes very different) as I got from a straight Fortran BLAS, so I dropped it. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.55.69 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From hinsen@cnrs-orleans.fr Mon Sep 6 14:45:01 1999 From: hinsen@cnrs-orleans.fr (Konrad Hinsen) Date: Mon, 6 Sep 1999 15:45:01 +0200 Subject: [Matrix-SIG] New Behavior for NumPy scalars? In-Reply-To: (message from Travis Oliphant on Fri, 3 Sep 1999 17:50:14 -0500 (CDT)) References: Message-ID: <199909061345.PAA21116@chinon.cnrs-orleans.fr> > My personal solution preference is to not use PyArray_Return for the > ufuncs and let users who really need a scalar do the indexing to get it > from the rank-0 array. Using PyArray_Return seems like trying to > second-guess the user and leads to these kinds of issues. It does what a naive user would expect, and what works well for the vast majority of applications. Anyone who thinks that returning rank-0 arrays is a good idea should volunteer for writing the chapter in the documentation which explains the difference between scalars and rank-0 arrays and why it is necessary to have both! If you want to introduce rank-0 arrays consistently, you also have to return a rank-0 result from indexing, for example. And then code like a = Numeric.zeros((3,)) a[0] = 1 ... if a[0] == 1: ... would do something very unexpected. The best solution would be to integrate the basic array module with the standard Python distribution and make all Python scalars rank-0 arrays. Perhaps this could be proposed seriously for Python 2.0. > Anyone else having any fun with the new NumPy 1.12 beta release? From what you report, it looks as if I could put my time to a better use... Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.55.69 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From strang@NMR.MGH.Harvard.EDU Mon Sep 6 22:09:06 1999 From: strang@NMR.MGH.Harvard.EDU (Gary Strangman) Date: Mon, 6 Sep 1999 17:09:06 -0400 (EDT) Subject: [Matrix-SIG] Creative slicing Message-ID: Hi all, I've recently come across a need that someone on this list might be able to help with. I'm trying to do the following with good speed using numpy ... x = lots_o_data x.shape = (100,40,50) y = indices_into_data y.shape = (40,50) # filled with ints in the 0-99 range I want z where z.shape=(40,50) and the elements of z are chosen from x according to the corresponding index in y. Make sense? That is, if y[0,0] = 21 then I want z[0,0] = x[21,0,0], and so on. take() doesn't seem capable of this. Might indices() be combined with take() or something like that? Maybe there's something hidden down deep inside numpy that could do this for me. I'm looking for a shape-independent solution (i.e., so I can make my function ufunc-like). Or is this just wishful thinking ??? ;-) Gary Strangman -------------------------------------------------------------- Gary Strangman, PhD | Neural Systems Group Office: 617-724-0662 | Massachusetts General Hospital Home: 401-453-3796 | 13th Street, Bldg 149, Room 9103 strang@nmr.mgh.harvard.edu | Charlestown, MA 02129 http://www.nmr.mgh.harvard.edu/Neural_Systems_Group/gary/ From Oliphant.Travis@mayo.edu Mon Sep 6 22:21:09 1999 From: Oliphant.Travis@mayo.edu (Travis Oliphant) Date: Mon, 6 Sep 1999 16:21:09 -0500 (CDT) Subject: [Matrix-SIG] New Behavior for NumPy scalars? In-Reply-To: <199909061345.PAA21116@chinon.cnrs-orleans.fr> Message-ID: > > My personal solution preference is to not use PyArray_Return for the > > ufuncs and let users who really need a scalar do the indexing to get it > > from the rank-0 array. Using PyArray_Return seems like trying to > > second-guess the user and leads to these kinds of issues. > > It does what a naive user would expect, > Perhaps, perhaps, not. When I was new to Python I remember the thing that kept getting me was the difference between lists and multiarrays. This caused a couple of minor hangups until I finally got it in my head that Python had multiple objects and not just one (like the MATLAB I was used to.) Once somebody realizes that it's not such a big leap to have Python scalars and Numeric scalars. > and what works well for the vast majority of applications. Hmm. Unless you are trying to actually use the single precision datatypes. Then, you have to be extremely careful not to lose precision because your NumPy arrays are always getting converted to Python scalars. If operations with NumPy arrays always returned NumPy arrays the precision upcasting problem wouldn't really be the problem it is, one would just have to define all of there constant scalars. Not a bad programming practice at that. > Anyone who thinks that returning > rank-0 arrays is a good idea should volunteer for writing the > chapter in the documentation which explains the difference between > scalars and rank-0 arrays and why it is necessary to have both! I'd do it. I think the reason is the same as why is it necessary to have both lists and NumPy arrays for the beginning user. > If you want to introduce rank-0 arrays consistently, you also have > to return a rank-0 result from indexing, for example. And then code > like Yes, probably true. > > a = Numeric.zeros((3,)) > a[0] = 1 > ... > if a[0] == 1: > ... > would do something very unexpected. > This would be confusing for beginners it is agreed. But there are always confusing issues when learning a new system. Changing something now would undoubtedly break all kinds of old code. Still, it would be nice if rank-0 arrays could exist less ephemerally. I know let's introduce a global flag.... :-) > The best solution would be to integrate the basic array module with > the standard Python distribution and make all Python scalars rank-0 > arrays. Perhaps this could be proposed seriously for Python 2.0. I agree this would be the best solution. Now, either way you have pain under some circumstances. The question is which pain is more tolerable. Lately, for me the precision pain has been, well, a pain. Travis From da@ski.org Mon Sep 6 23:28:34 1999 From: da@ski.org (David Ascher) Date: Mon, 6 Sep 1999 15:28:34 -0700 (Pacific Daylight Time) Subject: [Matrix-SIG] An area where the C-API for NumPy 1.12 is not 100% compatible In-Reply-To: <199909061020.MAA01059@chinon.cnrs-orleans.fr> Message-ID: > [konrad] > That should be a general problem with extension classes, and perhaps > one should look for a general solution. Ideally, PyArg_ParseTuple() > (or a similar function provided by the ExtensionClass module) would > offer both direct type comparison and subclass checks. Then of course > C modules would have to be adapted - but I'd prefer to impose this > only once, when a definite solution is available. I like that idea. We could write a PyArg_ParseTupleExt (or some other more useful name) which, if it finds an O= with a corresponding argument which in an EC, does the class equality (not the type equality, btw, since that is true for all EC's!), and e.g. when it sees a O<, does the subclassing test as well. I'll send a summary email to Guido and Jim Fulton, and pick their brains on the topic. --david From da@ski.org Mon Sep 6 23:44:51 1999 From: da@ski.org (David Ascher) Date: Mon, 6 Sep 1999 15:44:51 -0700 (Pacific Daylight Time) Subject: [Matrix-SIG] PyArg_ParseTuple & ExtensionClasses Message-ID: Release 12beta of NumPy makes NumPy arrays ExtensionClasses. As an experiment, the 'base class' (written in C) is subclassed in the Numeric.py file, which is what most users use to create arrays in Python. Extension modules create arrays from the C modules, hence get the 'base class'. It's been interesting seeing all the implications of moving from a type to a class, given legacy code. Problem: While PyArray_Check() was rewritten to do an ExtensionClass-aware subclassing check (whereas any instance of the array type/class or of a subclass thereof is considered an array for the purpose of the check), there is no way to plug that check into the PyArg_ParseTuple() behavior when the "O!" identifier is sent, along with the PyArray_Type as the argument. The short-term solution proposed by Travis Oliphant is to use O& and to supply a converter function which does that checking. A long-term solution proposed by Konrad Hinsen would be to have a new version of PyArg_ParseTuple which accepts two new typecodes, where one does 'isinstance' checking and the other does 'is-instance-of-a-subclass-of' checking. Off the top of my head, "O=" and "O<" could be used for identifying those two new checks. The new function could be called PyArg_ParseTupleEC or any other name. The reason I'm writing you two is that 1) it's an ExtensionClass-generic issue, and if there are other ways of doing PyArg_ParseTuple checks with EC's, we'd love to hear about it, and 2) when Python merges types and classes, these issues will crop up, so I thought Guido should hear about it. Note that O! does type equality checking and will return true for any ExtensionClass object, not just PyArray_Type. --david From hinsen@cnrs-orleans.fr Tue Sep 7 11:05:31 1999 From: hinsen@cnrs-orleans.fr (Konrad Hinsen) Date: Tue, 7 Sep 1999 12:05:31 +0200 Subject: [Matrix-SIG] New Behavior for NumPy scalars? In-Reply-To: (message from Travis Oliphant on Mon, 6 Sep 1999 16:21:09 -0500 (CDT)) References: Message-ID: <199909071005.MAA00519@chinon.cnrs-orleans.fr> > > and what works well for the vast majority of applications. > > Hmm. Unless you are trying to actually use the single precision > datatypes. Then, you have to be extremely careful not to lose precision > because your NumPy arrays are always getting converted to Python scalars. Right. And I wouldn't mind disabling the conversion to scalars for single-precision rank-0 arrays. After all, there are no single-precision scalars, so there's a good reason for not doing that conversion. And beginners are unlikely to fall into this trap. Does this sound like a useful compromise? > I'd do it. I think the reason is the same as why is it necessary to have > both lists and NumPy arrays for the beginning user. But that's easy, they have different properties. Scalars and rank-0 arrays, at least those that correspond to Python scalar types, are expected to behave absolutely identically. > > a = Numeric.zeros((3,)) > > a[0] = 1 > > ... > > if a[0] == 1: > > ... > > would do something very unexpected. > > > > This would be confusing for beginners it is agreed. But there are always And not only that: the correct version of the test in the example above looks awful. And a test that would work for any sequence object a would become somewhere between incredibly messy and impossible. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.55.69 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From Oliphant.Travis@mayo.edu Tue Sep 7 15:53:24 1999 From: Oliphant.Travis@mayo.edu (Travis Oliphant) Date: Tue, 7 Sep 1999 09:53:24 -0500 (CDT) Subject: [Matrix-SIG] New Behavior for NumPy scalars? In-Reply-To: <199909071005.MAA00519@chinon.cnrs-orleans.fr> Message-ID: > > > > Hmm. Unless you are trying to actually use the single precision > > datatypes. Then, you have to be extremely careful not to lose precision > > because your NumPy arrays are always getting converted to Python scalars. > > Right. And I wouldn't mind disabling the conversion to scalars for > single-precision rank-0 arrays. After all, there are no single-precision > scalars, so there's a good reason for not doing that conversion. > And beginners are unlikely to fall into this trap. > > Does this sound like a useful compromise? Yes, I think that would be quite a useful compromise. Good idea. > > And not only that: the correct version of the test in the example above > looks awful. And a test that would work for any sequence object a would > become somewhere between incredibly messy and impossible. True. Thanks Konrad. It's always good to get advice from the voice of reason before implementing any impulsive born-of-frustration design decision... Travis From Oliphant.Travis@mayo.edu Tue Sep 7 16:49:19 1999 From: Oliphant.Travis@mayo.edu (Travis Oliphant) Date: Tue, 7 Sep 1999 10:49:19 -0500 (CDT) Subject: [Matrix-SIG] A Patch to only convert rank-0 arrays if they match Python Objects Message-ID: Following up on the compromise suggested by Konrad for conversion of rank-0 arrays, I propose the following minor patch so that single Aprecision rank-0 arrays do not get automatically converted to Python scalars. *** LLNLDistribution12beta.orig/Numerical/Src/arrayobject.c Mon Aug 9 13:36:24 1999 --- LLNLDistribution12beta/Numerical/Src/arrayobject.c Tue Sep 7 10:42:51 1999 *************** *** 443,449 **** if (mp != NULL) Py_DECREF(mp); return NULL; } ! if (mp->nd == 0) { op = array_item(mp, 0); Py_DECREF(mp); return op; --- 443,449 ---- if (mp != NULL) Py_DECREF(mp); return NULL; } ! if (mp->nd == 0 && (mp->type_num==PyArray_LONG || mp->type_num==PyArray_DOUBLE || mp->type_num==PyArray_CDOUBLE || mp->type_num==PyArray_OBJECT) ) { op = array_item(mp, 0); Py_DECREF(mp); return op; -Travis From Oliphant.Travis@mayo.edu Tue Sep 7 17:03:35 1999 From: Oliphant.Travis@mayo.edu (Travis Oliphant) Date: Tue, 7 Sep 1999 11:03:35 -0500 (CDT) Subject: [Matrix-SIG] (no subject) Message-ID: There is a function called array_set in the arrayfnsmodule of the gist package that may help you accomplish your goal. I have not used it much but here is the calling syntax. array_set(vals1, indices, vals2) it operates like vals1 [indices] = vals2 It may only work with 1-D arrays (i.e. flattened arrays) though. I'm not sure. This is a common operation and it would be a nice addition to Numeric. I've worked around it most of the time by changing the way I think about assinging and using the where command with the logical operaters. Travis From amullhau@zen-pharaohs.com Tue Sep 7 20:15:20 1999 From: amullhau@zen-pharaohs.com (Andrew P. Mullhaupt) Date: Tue, 7 Sep 1999 15:15:20 -0400 Subject: [Matrix-SIG] Re: [Matrix-SIG] References: Message-ID: <042001bef965$5a0f0720$a602550a@amullhau> > vals1 [indices] = vals2 > > This is a common operation and it would be a nice addition to Numeric. It is an essential addition to Numeric. Later, Andrew Mullhaupt From se6y095@public.uni-hamburg.de Wed Sep 8 09:31:24 1999 From: se6y095@public.uni-hamburg.de (Berthold Höllmann) Date: 08 Sep 1999 10:31:24 +0200 Subject: [Matrix-SIG] ASCI Red Pentium BLAS linked with NumPy? In-Reply-To: Frank Horowitz's message of "Mon, 6 Sep 1999 14:32:03 +0800" References: Message-ID: Frank Horowitz writes: > G'Day folks, > > After spending the last couple of days strategizing on how to > fool the RPM system and chasing unresolved symbols in the linker on > an SMP RedHat Linux 6.0 box, I'm nearly ready to give up on building > NumPy (the LinearAlgebra.py contents, at least) linked with the > optimized BLAS from the ASCI Red project. > > Has anyone else made this work? What are the tricks you used? :-) > The line lapack_lite -I./Include Src/lapack_litemodule.c -llapack -lblas -lg2c -DEXTENSIONCLASS -DCOMPILING_NUMPY in Numerical/Setup.in did it for me. I linked lsblasppro1.1o_08.99.a to libblas.a and lapack-Linux-i686.a to liblapack.a, lapack beeing LAPACK 3.0. Note that this is from using LLNLDistribution12beta and gcc 2.95.1. Greetings Berthold -- bhoel@starship.python.net / http://starship.python.net/crew/bhoel/ It is unlawful to use this email address for unsolicited ads (USC Title 47 Sec.227). I will assess a US$500 charge for reviewing and deleting each unsolicited ad. From da@ski.org Thu Sep 9 01:11:58 1999 From: da@ski.org (David Ascher) Date: Wed, 8 Sep 1999 17:11:58 -0700 (Pacific Daylight Time) Subject: [Matrix-SIG] New bug reporting system for NumPy Message-ID: With gracious help from Greg Stein, we now have an online bug reporting system for NumPy. We welcome additions to the bug database from all users. The URL for the system is: http://www.pythonpros.com/NumPy/bugs/ Please login (this creates an account for you to use in the future), and enter all the bug reports you wish to. Please view the existing problem reports to avoid entering duplicates. We are in the process of entering previously submitted bug reports, but if you submitted a bug report in the past which isn't already in the database, feel free to add it yourself. Questions and feedback about this system should go to support@icf.llnl.gov. Thanks for your help, --David Ascher & Paul Dubois From jhauser@ifm.uni-kiel.de Thu Sep 9 08:11:19 1999 From: jhauser@ifm.uni-kiel.de (Janko Hauser) Date: Thu, 9 Sep 1999 09:11:19 +0200 (CEST) Subject: [Matrix-SIG] Creative slicing In-Reply-To: References: Message-ID: <14295.23959.889135.630357@ifm.uni-kiel.de> Try this code. It is not general for the number of dimensions, but general for the length of the dimensions. Perhaps it is a start... HTH, __Janko ####### cut here from Numeric import * x=arange(10*4*5) x.shape = (10,4,5) y = ones((4,5))*2 y[0,0]=3 y[0,1]=4 y[1,0]=5 y.shape = (4,5) # filled with ints in the 0-9 range # These lines need to be adapted for more dimensions yi,xi=indices(y.shape) linindex = y*product(x.shape[1:])+yi*y.shape[-2]+xi z=take(ravel(x),ravel(linindex)) z.shape=(4,5) From Gary Strangman Fri Sep 10 00:35:06 1999 From: Gary Strangman (Gary Strangman) Date: Thu, 9 Sep 1999 19:35:06 -0400 (EDT) Subject: [Matrix-SIG] Creative slicing In-Reply-To: <14295.23959.889135.630357@ifm.uni-kiel.de> Message-ID: > Try this code. It is not general for the number of dimensions, but > general for the length of the dimensions. Perhaps it is a start... A good start indeed. Thanks! I just thought I'd pass along the following code to any others who might be interested. (In the FWIW category, I agree that this kind of slicing would be an excellent addition to the NumPy core.) > x=arange(10*4*5) > x.shape = (10,4,5) > y = ones((4,5))*2 > y[0,0]=3 > y[0,1]=4 > y[1,0]=5 > y.shape = (4,5) # filled with ints in the 0-9 range > > # These lines need to be adapted for more dimensions > yi,xi=indices(y.shape) > linindex = y*product(x.shape[1:])+yi*y.shape[-2]+xi This wasn't quite right ... but very close (an off-by-one error, probably a mail typo). Instead use ... linindex = y*(product(x.shape[1:]))+yi*y.shape[1]+xi > z=take(ravel(x),ravel(linindex)) > z.shape=(4,5) Starting from your idea, I modified the code to work in the N-D case. It's not exactly pretty (and I didn't bother to find a way around the for-loop) but it seems to work. This general type of workaround is not going to be fast for large arrays ... which is what would make it nice to have such a function ... a[indices] = vals ... in the numpy core. ;-) Gary <------CUT BELOW------> def elementslice(a, indexarray, dimension=0): """ Usage: elementslice(a, indexarray, dimension=0) Returns: a[indices] along 'dimension', resulting shape = indices.shape """ if dimension <> 0: # for convenience, put operating dimension first newdims = range(len(a.shape)) del newdims[dimension] newdims = [dimension] + newdims a = N.transpose(a,newdims) else: newdims = a.shape ind = N.indices(indexarray.shape) linindex = indexarray*N.product(a.shape[1:]) # get offsets for 1st dim for i in range(len(ind)-1): # for remaining dims but one, add in offsets multindices = N.product(indexarray.shape[i+1:]) linindex = linindex + ind[i]*multindices linindex = linindex + ind[-1] # add in last dim's offsets slice = N.take(N.ravel(a),N.ravel(linindex)) # get vals from ravel'd array return N.reshape(slice,indexarray.shape) # reshape properly -------------------------------------------------------------- Gary Strangman, PhD | Neural Systems Group Office: 617-724-0662 | Massachusetts General Hospital Home: xxx-xxx-xxxx | 13th Street, Bldg 149, Room 9103 strang at nmr.mgh.harvard.edu | Charlestown, MA 02129 http://www.nmr.mgh.harvard.edu/Neural_Systems_Group/gary/ From dubois1@llnl.gov Fri Sep 10 16:33:17 1999 From: dubois1@llnl.gov (Paul F. Dubois) Date: Fri, 10 Sep 1999 08:33:17 -0700 Subject: [Matrix-SIG] Pyfort Version 2 now available Message-ID: <000901befba1$cec55d60$3c810b18@plstn1.sfba.home.com> Version 2 of Pyfort, a tool that lets you extend Python with Fortran, is now available. See http://xfiles.llnl.gov/python.htm. To use Pyfort, you construct an input file in a Fortran-90 like syntax. This input file contains interfaces for the functions in Fortran that are to be called. Pyfort then constructs a C-language extension module that contains the "glue" to turn Python calls into the calls to your Fortran. Example: Fortran: function sum (n, x) ! return sum of elements of x integer n real x(n) real sum ....implementation... end function sum Pyfort input: module modsum function sum (n, x) ! return sum of elements of x integer n = size(x) real x(n) real sum end function sum end module modsum Calling sum in Python: from modsum import sum x = [1., 2., 3.] # or a Numeric array print sum(x) print sum.__doc__ #prints "return sum of elements of x" pyfort -c compiler modsum.pyf produces modsummodule.c and modsum.txt; the former is used to construct the Python extension and the latter documents the Python calling sequence. If no comment is given, the doc string is the calling sequence. Users can declare output quantities by using the intent attribute: real, intent(out):: y(n) Such an argument becomes an output of the function. If there are multiple return values, a tuple is returned. Intent "temporary" requests that an array be constructed and passed to the Fortran as a work array. Persons wanting a "raw" interface that is exactly like the Fortran routine and which does as little as possible can achieve this by declaring any array arguments or scalar outputs as intent(inout). New for version 2: Module attributes NONE, TRANSPOSE (default), MIRROR control behavior of multiple dimension intent(in) and intent(output) arrays. NONE -- pass argument as is; return array has transposed strides to "look right" in Python. TRANSPOSE -- transpose the data area on input; return array has transposed strides. A returned array subsequently used in another call as input will not in fact ever be transposed. MIRROR -- the Fortran has been written with all the dimensions reversed; check shapes backward and create output arrays with a backward shape. No data movement ever required. Method set_pyfort_option available to set the default option. Each generated wrapper can take an optional extra argument of one of these three attributes to override the default. Scalar input arguments can be given an initial value as an expression involving constants, the Fortran 90 size operator, other scalar input arguments that were not given an initial value, and scalar input arguments that occur earlier in the argument list. See argument "n" above, for example. The size operator can be size(a) or size(a, dimensionnumber) (Fortran style, 1-based). Example: subroutine doit (n, m, x, k, y, z) integer n = size(x, 1), m = size(x, 2) , k = n * m real x(n, m) real, intent(out):: y(k) complex, intent(temporary):: z(n, m) Python call: y = doit(x) The purpose of this facility is to enable the construction of "nice" interfaces without needing an additional Python layer in many cases. Pyfort has been tested under Linux using the Portland Group (-c pgf77) and g77 (-c g77) compilers, under Solaris (choose -c solaris) and SGI (-c solaris works). Potential users should note that both a Fortran and C compiler are needed. Pyfort has not yet been heavily used and bugs are possible. Documentation is in the file pyfort.pdf that comes with the package. Pyfort has been released for free redistribution. See Legal.htm. Questions to support@pcmdi.llnl.gov Paul F. Dubois Program for Climate Model Diagnosis and Intercomparison Lawrence Livermore National Laboratory

Pyfort 2 - Python / Fortran connection tool. (9 - Sep - 99) From dubois1@llnl.gov Fri Sep 10 18:12:20 1999 From: dubois1@llnl.gov (Paul F. Dubois) Date: Fri, 10 Sep 1999 10:12:20 -0700 Subject: [Matrix-SIG] Numerical plans Message-ID: <000a01befbaf$a4df51a0$3c810b18@plstn1.sfba.home.com> This is a multi-part message in MIME format. ------=_NextPart_000_0007_01BEFB74.F8142340 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable After some consideration, David and I have decided to back out the = changes related to Extension Class. We intend to release a version 12 = that has the bug fixes since version 11 but not the Extension Class = work.=20 Practical experience has given us a different idea for solving the = problem of wanting to inherit from an array class. We want to explore = this idea for a while. While the extension class approach basically = works, it doesn't really turn out to ease the developer's task and it = does raise a number of incompatibility issues that we hadn't forseen.=20 Older and wiser, we will try again. Periodically the issue is raised of=20 x[indices] =3D values=20 x[indices] as an r-value being desireable, as it was recently. As I pointed out before, everyone = agrees that the above line is desireable, but when x has rank > 1, there = are strongly differing views of what this ought to mean. Multiple = intelligent, experienced developers have told me that (a) I am an idiot, = because (b) there is only one obvious and highly desireable solution, = namely theirs Unfortunately the latter does not turn out to be the same = in all cases. The issue is complicated by the fact that the coding for setindex is = already totally incomprehensible and so changing it requires a great = deal of care and testing. It is time for proposals. Concrete, detailed. Discussion to follow. If = we can come to a consensus it would be nice. Do not forget things like NewAxis, Ellipsis, and the colon. How do these = things mix and match? ------=_NextPart_000_0007_01BEFB74.F8142340 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable

After some consideration, David and I have decided = to back out=20 the changes related to Extension Class. We intend to release a version = 12 that=20 has the bug fixes since version 11 but not the Extension Class work.=20
 
Practical experience has given us a different idea = for solving=20 the problem of wanting to inherit from an array class. We want to = explore this=20 idea for a while. While the extension class approach basically works, it = doesn't=20 really turn out to ease the developer's task and it does raise a number = of=20 incompatibility issues that we hadn't forseen.
 
Older and wiser, we will try again.
 
Periodically the issue is raised of
 
x[indices] =3D values
x[indices] as an r-value
 
being desireable, as it was recently. As I pointed = out before,=20 everyone agrees that the above line is desireable, but when x has rank = > 1,=20 there are strongly differing views of what this ought to mean. Multiple=20 intelligent, experienced developers have told me that (a) I am an idiot, = because=20 (b) there is only one obvious and highly desireable solution, namely = theirs=20 Unfortunately the latter does not turn out to be the same in all=20 cases.
 
The issue is complicated by the fact that the coding = for=20 setindex is already totally incomprehensible and so changing it requires = a great=20 deal of care and testing.
 
It is time for proposals. Concrete, detailed. = Discussion to=20 follow. If we can come to a consensus it would be nice.
Do not forget things like NewAxis, Ellipsis, and the = colon.=20 How do these things mix and match?
 
 
------=_NextPart_000_0007_01BEFB74.F8142340-- From godzilla@netmeg.net (Les Schaffer) Fri Sep 10 18:35:34 1999 From: godzilla@netmeg.net (Les Schaffer) (Les Schaffer) Date: Fri, 10 Sep 1999 13:35:34 -0400 (EDT) Subject: [Matrix-SIG] Numerical plans In-Reply-To: <000a01befbaf$a4df51a0$3c810b18@plstn1.sfba.home.com> References: <000a01befbaf$a4df51a0$3c810b18@plstn1.sfba.home.com> Message-ID: <14297.16742.754718.207527@gargle.gargle.HOWL> Paul DuBois said: > Older and wiser, we will try again. since you seem to be in reflection mode, i was wondering if you could explain to the dummies among us (me, myself and i) what was the purpose of this attempt at using extension classes for NumPy? I never caught on, from the discussions here, what the value was supposed to be to the -- ahem -- average user. -- ____ Les Schaffer ___| --->> Engineering R&D <<--- Theoretical & Applied Mechanics | Designspring, Inc. Center for Radiophysics & Space Research | http://www.designspring.com/ Cornell Univ. schaffer@tam.cornell.edu | les@designspring.com From amullhau@zen-pharaohs.com Fri Sep 10 22:26:41 1999 From: amullhau@zen-pharaohs.com (Andrew P. Mullhaupt) Date: Fri, 10 Sep 1999 17:26:41 -0400 Subject: [Matrix-SIG] Numerical plans References: <000a01befbaf$a4df51a0$3c810b18@plstn1.sfba.home.com> Message-ID: <033301befbd6$e9cf4280$a602550a@amullhau> > Periodically the issue is raised of > x[indices] = values > x[indices] as an r-value > being desireable, as it was recently. And it will be, until it's in there or another more attractive language comes along. > As I pointed out before, everyone agrees that the above line is desireable, but when > x has rank > 1, there are strongly differing views of what this ought to mean. Multiple > intelligent, experienced developers have told me that (a) I am an idiot, because (b) > there is only one obvious and highly desireable solution, namely theirs Unfortunately > the latter does not turn out to be the same in all cases. I think these developers don't have much experience. There are clearly multiple useful meanings that this can have. Especially depending on what range indices can have in your language. The one that probably makes the most sense is to coerce x to a rank-1 array, and index that, return the rank-1 result. A lot of languages do that, and nobody is banging on them to change. > The issue is complicated by the fact that the coding for setindex is already totally > incomprehensible and so changing it requires a great deal of care and testing. Time for a replacement. There are some methods for indexing which are both simpler and can be substantially faster. > It is time for proposals. Concrete, detailed. Discussion to follow. If we can come to a > consensus it would be nice. Steal the S style, except you probably can't get away with the special treatment of nonpositive index values and sparse indexing by high rank arrays. The APL style is better thought out in terms of language design, but then you need to add more functions for the stuff cyclic extension does for you, and the code you have to write isn't as fast. Experience has shown that the APL approach was probably too rigid. Later, Andrew Mullhaupt From se6y095@public.uni-hamburg.de Sat Sep 11 12:59:39 1999 From: se6y095@public.uni-hamburg.de (Berthold Höllmann) Date: 11 Sep 1999 13:59:39 +0200 Subject: [Matrix-SIG] Pyfort Version 2 now available In-Reply-To: "Paul F. Dubois"'s message of "Fri, 10 Sep 1999 08:33:17 -0700" References: <000901befba1$cec55d60$3c810b18@plstn1.sfba.home.com> Message-ID: "Paul F. Dubois" writes: > Version 2 of Pyfort, a tool that lets you extend Python with Fortran, is now > available. See http://xfiles.llnl.gov/python.htm. Hello, I downloaded and installed pyfort2. Checking the test subdirectory I get: # python testit.py itimes [1 2 3] [2 3 4] [ 2 6 12] times [ 1. 2. 3.] [ 2. 3. 4.] [ 2. 6. 12.] bang 1.2 [0 1 2 3 4 5 6 7 8 9] [ 3.+0.j 4.+2.j] Speicherzugriffsfehler (Speicherzugriffsfehler is german for "segmentation fault") So I started gdb: # gdb python GNU gdb 4.17.0.11 with Linux support Copyright 1998 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i686-pc-linux-gnu"... (gdb) run testit.py Starting program: /usr/local/bin/python testit.py itimes [1 2 3] [2 3 4] [ 2 6 12] times [ 1. 2. 3.] [ 2. 3. 4.] [ 2. 6. 12.] bang 1.2 [0 1 2 3 4 5 6 7 8 9] [ 3.+0.j 4.+2.j] Program received signal SIGSEGV, Segmentation fault. 0x4013730b in do_array_in (rname=0x4013a0da "bang", vname=0x4013a0d8 "R", v=0x80cdcd4, python_array_type=PyArray_FLOAT) at ./testpyfmodule.c:172 172 if (t->descr->type_num != python_array_type) { (gdb) where #0 0x4013730b in do_array_in (rname=0x4013a0da "bang", vname=0x4013a0d8 "R", v=0x80cdcd4, python_array_type=PyArray_FLOAT) at ./testpyfmodule.c:172 #1 0x40138103 in testpyf_bang (unused=0x0, args=0x80d4f50) at ./testpyfmodule.c:562 #2 0x80539e0 in call_builtin (func=0x80cebf8, arg=0x80d4f50, kw=0x0) at ceval.c:2359 #3 0x80538f8 in PyEval_CallObjectWithKeywords (func=0x80cebf8, arg=0x80d4f50, kw=0x0) at ceval.c:2324 #4 0x8052ac5 in eval_code2 (co=0x80cd7e0, globals=0x80a34d8, locals=0x80a34d8, args=0x0, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, owner=0x0) at ceval.c:1654 #5 0x8050dd2 in PyEval_EvalCode (co=0x80cd7e0, globals=0x80a34d8, locals=0x80a34d8) at ceval.c:324 #6 0x805f8ae in run_node (n=0x80bd658, filename=0xbffff8a4 "testit.py", globals=0x80a34d8, locals=0x80a34d8) at pythonrun.c:887 #7 0x805f868 in run_err_node (n=0x80bd658, filename=0xbffff8a4 "testit.py", globals=0x80a34d8, locals=0x80a34d8) at pythonrun.c:872 #8 0x805f840 in PyRun_File (fp=0x80a1148, filename=0xbffff8a4 "testit.py", start=257, globals=0x80a34d8, locals=0x80a34d8) at pythonrun.c:860 #9 0x805efce in PyRun_SimpleFile (fp=0x80a1148, filename=0xbffff8a4 "testit.py") at pythonrun.c:570 #10 0x805ec61 in PyRun_AnyFile (fp=0x80a1148, filename=0xbffff8a4 "testit.py") at pythonrun.c:451 #11 0x804f5eb in Py_Main (argc=2, argv=0xbffff724) at main.c:287 #12 0x804f0e8 in main (argc=2, argv=0xbffff724) at python.c:12 (gdb) print t->descr->type_num Cannot access memory at address 0x3ff00034. (gdb) print t->descr $1 = (PyArray_Descr *) 0x3ff00000 (gdb) print t->descr->type Cannot access memory at address 0x3ff00044. (gdb) print t->descr->data There is no member named data. (gdb) print t->ob_type $2 = (struct _typeobject *) 0x8094cc0 (gdb) print t->ob_type->tp_name $3 = 0x808c3a0 "float" (gdb) I'm using LLNLDistribution12beta which seems to return Floats instead of array(, Float) from PyArray_ContiguousFromObject for single values. (This is my understanding of the recent dicussion on [Matrix-SIG]) Cheers Berthold -- bhoel@starship.python.net / http://starship.python.net/crew/bhoel/ It is unlawful to use this email address for unsolicited ads (USC Title 47 Sec.227). I will assess a US$500 charge for reviewing and deleting each unsolicited ad. From se6y095@public.uni-hamburg.de Sat Sep 11 14:32:16 1999 From: se6y095@public.uni-hamburg.de (Berthold Höllmann) Date: 11 Sep 1999 15:32:16 +0200 Subject: [Matrix-SIG] Pyfort Version 2 now available In-Reply-To: se6y095@public.uni-hamburg.de's message of "11 Sep 1999 13:59:39 +0200" References: <000901befba1$cec55d60$3c810b18@plstn1.sfba.home.com> Message-ID: se6y095@public.uni-hamburg.de (Berthold Höllmann) writes: > "Paul F. Dubois" writes: > > > Version 2 of Pyfort, a tool that lets you extend Python with Fortran, is now > > available. See http://xfiles.llnl.gov/python.htm. > > Hello, > > I downloaded and installed pyfort2. Checking the test subdirectory I > get: ... > I'm using LLNLDistribution12beta which seems to return Floats instead > of array(, Float) from PyArray_ContiguousFromObject for single > values. (This is my understanding of the recent dicussion on > [Matrix-SIG]) Oops, I reinstalled LLNLDistribution11 and pyfort 1 and still get the segmentation fault. BTW, this is python 1.5.2. Cheers Berthold -- bhoel@starship.python.net / http://starship.python.net/crew/bhoel/ It is unlawful to use this email address for unsolicited ads (USC Title 47 Sec.227). I will assess a US$500 charge for reviewing and deleting each unsolicited ad. From dubois1@llnl.gov Sat Sep 11 16:20:59 1999 From: dubois1@llnl.gov (Paul F. Dubois) Date: Sat, 11 Sep 1999 08:20:59 -0700 Subject: [Matrix-SIG] Pyfort Version 2 now available References: <000901befba1$cec55d60$3c810b18@plstn1.sfba.home.com> Message-ID: <000d01befc69$41259420$3c810b18@plstn1.sfba.home.com> Thanks for the bug report. I had not tested Pyfort with the Version 12 beta. As I mentioned yesterday, we are abandoning that approach and your note is a good reminder for me to go remove the beta distribution. Everyone should reinstall version 11 for now. I also need to mention that if you use Pyfort with LLNL 11, you may want to edit the place in generator.py that says #include "Numeric/arrayboject.h" to make the N lower case, before installation. In the future the installation is going to Numeric instead of numeric, but 11 still does it the old way. From turner@blueskystudios.com Mon Sep 13 21:27:05 1999 From: turner@blueskystudios.com (John A. Turner) Date: Mon, 13 Sep 1999 16:27:05 -0400 (EDT) Subject: [Matrix-SIG] Pickling Message-ID: <14301.24089.837561.510054@waterloo.blueskyprod.com> There was a short thread on NumPy and Pickling back in June, started by this msg: http://www.python.org/pipermail/matrix-sig/1999-June/002909.html with followups by Hinson and Ascher. Just wondering where this stands at the moment. Thanks, -- John A. Turner Senior Research Associate Blue Sky Studios, One South Rd, Harrison, NY 10528 http://www.blueskystudios.com/ (914) 381-8400 From da@ski.org Tue Sep 14 04:54:02 1999 From: da@ski.org (David Ascher) Date: Mon, 13 Sep 1999 20:54:02 -0700 (Pacific Daylight Time) Subject: [Matrix-SIG] Pickling In-Reply-To: <14301.24089.837561.510054@waterloo.blueskyprod.com> Message-ID: > There was a short thread on NumPy and Pickling back in June, started > by this msg: > > http://www.python.org/pipermail/matrix-sig/1999-June/002909.html > > with followups by Hinson and Ascher. Just wondering where this stands > at the moment. Release 12beta has fixed pickling code (which could be stolen to 'fix' version 11). We haven't investigated high-performance pickling much further, due to the unsatisfactory state of understanding of the buffer interface and general lack of time. --david ascher From dubois1@llnl.gov Tue Sep 14 15:28:31 1999 From: dubois1@llnl.gov (Paul F. Dubois) Date: Tue, 14 Sep 1999 07:28:31 -0700 Subject: [Matrix-SIG] Re: Numerical Python References: Message-ID: <002301befebd$6b697520$3c810b18@plstn1.sfba.home.com> Dear Matrix-Sigers who have Macs, Mr. Nataf seeks a Mac version of NumPy 11, and advice about a choice of graphics package. Thanks, Paul ----- Original Message ----- From: Frederic Nataf To: Sent: Tuesday, September 14, 1999 5:39 AM Subject: Numerical Python > Dear Sir, > > I have heard about Numerical Python by a colleague of mine who spent > one year at > LLNL. I am very interested in working with it. I work both on Unix > workstations (gcc) and on Macintosh (CodeWarrior). I have seen on your web > tutorial > http://xfiles.llnl.gov/NumDoc4.html#18177 > that python is available on the Macintosh along with a not so uptodate > version of your code. So, I have two questions: > > 1) is it difficult or a problem to have the last version work on a > Macintosh > > 2) what about the graphic package. Can it work on a Macintosh? > > > Sincerly, > > Frederic Nataf > > PS. How MacOsX will change the situation? > > From da@ski.org Tue Sep 14 17:14:26 1999 From: da@ski.org (David Ascher) Date: Tue, 14 Sep 1999 09:14:26 -0700 (Pacific Daylight Time) Subject: [Matrix-SIG] Numerical plans (fwd) Message-ID: On Fri, 10 Sep 1999, Les Schaffer wrote: > Paul DuBois said: > > > Older and wiser, we will try again. > > since you seem to be in reflection mode, i was wondering if you could > explain to the dummies among us (me, myself and i) what was the > purpose of this attempt at using extension classes for NumPy? > > I never caught on, from the discussions here, what the value was > supposed to be to the -- ahem -- average user. Well, since I think I was the main force behind that, I'll say that the main idea was that I wanted to be able to continue work on the functionality of array objects in Python, not in C. The C code is a mess, and I felt it much easier to do things like the indexing that Paul was talking about (called many things my many people) in Python. ExtensionClasses was supposed to allow this sort of subclassing easily. Other ideas included things like being able to specify (again, in Python) different coercion behaviors. Etc. The issues we hit were: 1) ExtensionClass has some bugs, which are hard to tackle, and the author (Jim Fulton) is too busy to track them down. 2) ExtensionClass is hard to reconcile with the variety of ways NumPy is compiled (the CAPI, static/dynamic linking, etc.) 3) Making a type a class simply isn't doable in a backwards compatible way while keeping the advantages of a class. Instead, what Paul and I are planning to do is to revert to the old type, but to do UserArray *right*. The new features that I mention above would then be implemented in new Python classes, which would use array objects as member data (I mean, instance attributes). --david From se6y095@public.uni-hamburg.de Wed Sep 15 11:25:45 1999 From: se6y095@public.uni-hamburg.de (Berthold Höllmann) Date: 15 Sep 1999 12:25:45 +0200 Subject: [Matrix-SIG] Problem with derived array dimensions in pyfort 2 Message-ID: Hello, The following ".pyf" file gives me an error: --- tst.pyf ---- module BUG subroutine AAA(kx, ky, nxest, nyest, c) integer kx,ky,nxest,nyest real c ( ( ( nxest - kx - 1 ) * ( nyest - ky - 1 ) ) ) end subroutine surfit end module BUG --- tst.pyf ---- I get: >pyfort -c g77 tst.pyf Error concerning token '(' ( File 'tst.pyf', line 4 ) according to pyfort.pdf, p. 11, this syntax should be allowed. Cheers Berthold -- bhoel@starship.python.net / http://starship.python.net/crew/bhoel/ It is unlawful to use this email address for unsolicited ads (USC Title 47 Sec.227). I will assess a US$500 charge for reviewing and deleting each unsolicited ad. From andy@robanal.demon.co.uk Wed Sep 15 22:49:59 1999 From: andy@robanal.demon.co.uk (Andy Robinson) Date: Wed, 15 Sep 1999 21:49:59 GMT Subject: [Matrix-SIG] Numeric tutorials at next conference? Message-ID: <37e312d0.1408915@post.demon.co.uk> I'm trying to organize a few financial talks and events at the next Python conference. It is apparent that Python is being used very successfully in a number of investment banks. Furthermore, I have the impression that a few of them would send people down to learn more if we had the right agenda. This as it is very good advertising - how many other languages beginning with P are used to make money? I'm hoping to persuade a few people to submit papers related to finance, economics or accounting by the end of this month. The icing on the cake would be for a tutorial on the Numeric extensions (and probably for other stuff - I could pitch in and help out on accounting). Is there anyone here who feels interested in giving all or part of a tutorial on this? Ideally this would have a strong applications focus, with realistic examples of things you might acually want to do with NumPy (though not necessarily on financial applications). There is also a VERY high level of interest in the financial community in an extensible event-simulation package and a time series library. I'd like to hear about anyone working in this area with Python. Volunteers, suggestions, ideas and flames all welcome. Regards, Andy Robinson Robinson Analytics Ltd. From godzilla@netmeg.net (Les Schaffer) Wed Sep 15 23:27:42 1999 From: godzilla@netmeg.net (Les Schaffer) (Les Schaffer) Date: Wed, 15 Sep 1999 18:27:42 -0400 (EDT) Subject: [Matrix-SIG] Numeric tutorials at next conference? In-Reply-To: <37e312d0.1408915@post.demon.co.uk> References: <37e312d0.1408915@post.demon.co.uk> Message-ID: <14304.7518.148972.100717@gargle.gargle.HOWL> andy robinson said: > There is also a VERY high level of interest in the financial > community in an extensible event-simulation package and a time > series library. i'd like to hear more about the specifics of whats needed in this area. event-simulation means what? > Volunteers, suggestions, ideas and flames all welcome. flames? lets see? ... down with capitalism? les From HYoon@exchange.ml.com Thu Sep 16 15:05:22 1999 From: HYoon@exchange.ml.com (Yoon, Hoon (CICG - NY Program Trading)) Date: Thu, 16 Sep 1999 10:05:22 -0400 Subject: [Matrix-SIG] Numeric tutorials at next conference? Message-ID: Les, In my def, event simulation often means, how an event like interest rate change, earnings announcement, yen exch rate, Indonesian crisis effects a particular index or bench mark by how much and how long. Monte Carlo type of simulation is not unusual. Some combined with genetic type of AI stuff. This has close association with times seris. It take long time for one of those effects to be work itself out. This is not my specialty, but I dabbled in surprise unexpected earnings (SUE) model in the past, so I know little about it. Obvisouly, SUE is a lot easier to measure than Indonesian crisis. I am glad that's not my job - I can try the economics office upstairs for that. I use Python for many projects that requires data structures. I am in love with the dictionary structure. Although I am pretty much required to use VB for implementation, I still keep many stuff in Python. Best thing for realtime data manipulation and no data gets harder than RT data. How about this for flame?: GODzilla sucks, King Kong rules. Even 1984, I see cheap rubber fins. ************************************************************** S. Hoon Yoon (Quant) Merrill Lynch Equity Trading yelled@yahoo.com hoon@bigfoot.com(w) "Miracle is always only few standard deviations away, but so is catastrophe." * Expressed opinions are often my own, but NOT my employer's. "I feel like a fugitive from the law of averages." Mauldin ************************************************************** > -----Original Message----- > From: Les Schaffer [SMTP:godzilla@netmeg.net] > Sent: Wednesday, September 15, 1999 6:28 PM > To: matrix-sig@python.org > Subject: [Matrix-SIG] Numeric tutorials at next conference? > > andy robinson said: > > > There is also a VERY high level of interest in the financial > > community in an extensible event-simulation package and a time > > series library. > > i'd like to hear more about the specifics of whats needed in this > area. event-simulation means what? > > > Volunteers, suggestions, ideas and flames all welcome. > > flames? lets see? ... down with capitalism? > > les > > _______________________________________________ > Matrix-SIG maillist - Matrix-SIG@python.org > http://www.python.org/mailman/listinfo/matrix-sig From hinsen@cnrs-orleans.fr Thu Sep 16 15:38:38 1999 From: hinsen@cnrs-orleans.fr (Konrad Hinsen) Date: Thu, 16 Sep 1999 16:38:38 +0200 Subject: [Matrix-SIG] Numerical plans (fwd) In-Reply-To: (message from David Ascher on Tue, 14 Sep 1999 09:14:26 -0700 (Pacific Daylight Time)) References: Message-ID: <199909161438.QAA20791@chinon.cnrs-orleans.fr> > Well, since I think I was the main force behind that, I'll say that the > main idea was that I wanted to be able to continue work on the > functionality of array objects in Python, not in C. The C code is a mess, It's a good idea to be able to implement functionality in Python. But if the C code is a mess (and I certainly agree), it ought to be cleaned up; there's no substitute for that, unfortunately. > Instead, what Paul and I are planning to do is to revert to the old type, > but to do UserArray *right*. The new features that I mention above would > then be implemented in new Python classes, which would use array objects > as member data (I mean, instance attributes). But any attempt to implement significant functionality in Python will lead to the same problem: Most users will end up using some Python object, which C modules won't accept as an array. My personal convention in MMTK is that all high-level "array-like" object have an attribute "array" that stores the array object. C modules then check for an array, and if the type check fails they get the attribute "array" and continue with that. Maybe something similar would be a good convention in general. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.55.69 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From da@ski.org Thu Sep 16 17:17:52 1999 From: da@ski.org (David Ascher) Date: Thu, 16 Sep 1999 09:17:52 -0700 (Pacific Daylight Time) Subject: [Matrix-SIG] Numerical plans (fwd) In-Reply-To: <199909161438.QAA20791@chinon.cnrs-orleans.fr> Message-ID: > > Instead, what Paul and I are planning to do is to revert to the old type, > > but to do UserArray *right*. The new features that I mention above would > > then be implemented in new Python classes, which would use array objects > > as member data (I mean, instance attributes). > > But any attempt to implement significant functionality in Python > will lead to the same problem: Most users will end up using some > Python object, which C modules won't accept as an array. > > My personal convention in MMTK is that all high-level "array-like" > object have an attribute "array" that stores the array object. C > modules then check for an array, and if the type check fails they get > the attribute "array" and continue with that. Maybe something similar > would be a good convention in general. That's exactly what I had in mind. We'd just be using UserArray as the prototype for others to follow. --david From amullhau@zen-pharaohs.com Thu Sep 16 17:26:12 1999 From: amullhau@zen-pharaohs.com (Andrew P. Mullhaupt) Date: Thu, 16 Sep 1999 12:26:12 -0400 Subject: [Matrix-SIG] Numeric tutorials at next conference? References: <37e312d0.1408915@post.demon.co.uk> <14304.7518.148972.100717@gargle.gargle.HOWL> Message-ID: <016301bf0066$b1fbda00$a602550a@amullhau> > andy robinson said: > > > There is also a VERY high level of interest in the financial > > community in an extensible event-simulation package and a time > > series library. > > i'd like to hear more about the specifics of whats needed in this > area. event-simulation means what? Event simulation is basically a term for simulation of processes in which there are contingent events. For example, if you wanted to simulate a grocery store, you might have events like "customer enters aisle 6" and "box of Fruit Loops taken from shelf and put in customer's cart", etc. Event simulation support usually means a language in which you can write the events and dependencies in a reasonably simple way and then a simulation can be run. Simulation languages go back a long way (Simula is an example) and people who use them love them. It makes sense if you don't care about high performance, but not really otherwise. Really large scale simulations usually can be written, and have to be written, outside the event simulation paradigm. Later, Andrew Mullhaupt From guido@CNRI.Reston.VA.US Thu Sep 16 19:53:52 1999 From: guido@CNRI.Reston.VA.US (Guido van Rossum) Date: Thu, 16 Sep 1999 14:53:52 -0400 Subject: [Matrix-SIG] Reminder: Python Conference papers due soon! Message-ID: <199909161853.OAA01738@eric.cnri.reston.va.us> The deadline for paper submissions for the next Python conference is nearing! Make sure that if you are considering submitting a paper, the program chair sees your paper by SEPTEMBER 30. Papers can be on any Python, JPython or Zope related subject. Papers will be reviewed and the best papers will be selected for presentation. You will hear about selection by October 22. For more information, see the official call for papers: http://www.python.org/workshops/2000-01/cfp.html ** We are planning a significant Zope presence at this conference! ** The conference will be held from January 24-27 in Alexandria, VA (just across the Potomac from DC). Make Python the first conference in the new millennium you visit! For more info, see the conference home page: http://www.python.org/workshops/2000-01/ Send all email questions about papers to: ipc8papers@cs.uchicago.edu --Guido van Rossum (home page: http://www.python.org/~guido/) (And sorry for the broad spamming.) From se6y095@public.uni-hamburg.de Sun Sep 19 19:38:25 1999 From: se6y095@public.uni-hamburg.de (Berthold Höllmann) Date: 19 Sep 1999 20:38:25 +0200 Subject: [Matrix-SIG] Pyfort: Intent inout must be an array.? / Please make meaningfull exit values. Message-ID: Hello, I try to wrap a FORTRAN function which uses one integer parameter as input as well as as output value, so I tried: integer, intent(inout):: n but pyfort quits with the message: ('Intent inout must be an array.',) is there a technical reason for this, or is this only not implemented. Another point I have is, that if an error occurs in pyfort it uses raise SystemExit in some places, I would prefer to make it raise SystemExit, 1 becaus this would allow make to stop at this point and not trying to proceed. Cheers Berthold -- bhoel@starship.python.net / http://starship.python.net/crew/bhoel/ It is unlawful to use this email address for unsolicited ads (USC Title 47 Sec.227). I will assess a US$500 charge for reviewing and deleting each unsolicited ad. From dubois1@llnl.gov Sun Sep 19 20:29:13 1999 From: dubois1@llnl.gov (Paul F. Dubois) Date: Sun, 19 Sep 1999 12:29:13 -0700 Subject: [Matrix-SIG] Pyfort: Intent inout must be an array.? / Please make meaningfull exit values. References: Message-ID: <000f01bf02d5$41b16e60$3c810b18@plstn1.sfba.home.com> ----- Original Message ----- From: Berthold Höllmann To: Cc: > > I try to wrap a FORTRAN function which uses one integer parameter as > input as well as as output value, so I tried: > > integer, intent(inout):: n > > but pyfort quits with the message: > > ('Intent inout must be an array.',) > > is there a technical reason for this, or is this only not implemented. > The user must pass an array since Python integers are immutable. It would of course be possible to write the wrapper and then object at runtime if the argument passed wan't an array; but then the documentation would not match what the user had to do. I chose to do this to make the point crystal clear. The fix is to change it to n(1) in your Pyfort input. You need not touch the Fortran, of course. I could be talked out of this decision by someone with a forceful personality; it was an arbitrary decision. > Another point I have is, that if an error occurs in pyfort it uses > > raise SystemExit > > in some places, I would prefer to make it > > raise SystemExit, 1 > > becaus this would allow make to stop at this point and not trying to > proceed. > Will fix. From minxu@sci.ccny.cuny.edu Sun Sep 19 20:34:32 1999 From: minxu@sci.ccny.cuny.edu (minxu@sci.ccny.cuny.edu) Date: Sun, 19 Sep 1999 15:34:32 -0400 Subject: [Matrix-SIG] HELP! How to access a large array Message-ID: <199909191934.PAA21699@star.future.com> Hi, I am currently implementing a image reconstruction code with Python. It goes smoothly but I am stuck by one problem. One part of my code calculates some large matrices around 24M+ and stored in netcdf file. The second part wants to read in some data from the created netcdf file and do processing accordingly. But is turns out that it takes minutes for the second part to read the data from netcdf file. It is very sad since I want to increase the size of data further. Is there any way that I can read some subsets of one array stored in netcdf file without loading the whole array? Does f.variables['var'][x, y, ...] avoid loading the whole 'var' array? If netcdf can not do that, any other storage can do that? Thanks. -- Min Xu City College of NY, CUNY Email: mxu1@email.gc.cuny.edu minxu@sci.ccny.cuny.edu Tel: (O) (212) 650-6865 (O) (212) 650-5046 (H) (212) 690-2119 From jhauser@ifm.uni-kiel.de Mon Sep 20 13:15:38 1999 From: jhauser@ifm.uni-kiel.de (Janko Hauser) Date: Mon, 20 Sep 1999 14:15:38 +0200 (CEST) Subject: [Matrix-SIG] HELP! How to access a large array In-Reply-To: <199909191934.PAA21699@star.future.com> References: <199909191934.PAA21699@star.future.com> Message-ID: <14310.9578.229015.356053@ifm.uni-kiel.de> minxu@sci.ccny.cuny.edu writes: > But is turns out that it takes minutes for the second part to read the data > from netcdf file. It is very sad since I want to increase the size of data > further. > Along which dimension are you doing the subsampling. To read subsamples along the first (unlimited) dimension is very timeconsuming but not memory consuming. Do you see, that all the data is read in at the memory footprint? Or do you asume this because of the long time it took. It should not read in all the data. Which version of the netcdf-library are you using? The 2.? is slower then the newer ones. > Is there any way that I can read some subsets of one array stored in netcdf > file without loading the whole array? Does f.variables['var'][x, y, ...] avoid > loading the whole 'var' array? > This should already be possible? I deal with NetCDF-files of ca. 1Gb size, so this is working. __Janko From jhauser@ifm.uni-kiel.de Mon Sep 20 13:21:32 1999 From: jhauser@ifm.uni-kiel.de (Janko Hauser) Date: Mon, 20 Sep 1999 14:21:32 +0200 (CEST) Subject: [Matrix-SIG] PyFort: Other function names. Message-ID: <14310.9932.8635.147745@ifm.uni-kiel.de> Used for the first time the PyFort module. This is very slick and elegant and working out of the box. I want to wrap a function, where an integer parameter defines if the interpolation is done linear or cubic. This affects also the length of the working array, which I give as a temporary. Is it possible to give a new name for the wrapped function and thus to wrap the same function with different interfaces? This would also allow to give sometimes better names to cryptic Fortran names. Alos the use of a temporary could then be more flexibel (have a fast one, where always the same temporary is used, but which needs to be provided by the interpreter) Thanks for this tool, __Janko From minxu@scisun.sci.ccny.cuny.edu Mon Sep 20 16:57:41 1999 From: minxu@scisun.sci.ccny.cuny.edu (Min Xu) Date: Mon, 20 Sep 1999 11:57:41 -0400 Subject: [Matrix-SIG] HELP! How to access a large array In-Reply-To: Message from Janko Hauser of "Mon, 20 Sep 1999 14:15:38 +0200." <14310.9578.229015.356053@ifm.uni-kiel.de> Message-ID: <199909201557.LAA01533@star.future.com> > minxu@sci.ccny.cuny.edu writes: > > But is turns out that it takes minutes for the second part to read the data > > from netcdf file. It is very sad since I want to increase the size of data > > further. > > > > Along which dimension are you doing the subsampling. To read > subsamples along the first (unlimited) dimension is very timeconsuming > but not memory consuming. Do you see, that all the data is read in at > the memory footprint? Or do you asume this because of the long time it > took. It should not read in all the data. Which version of the > netcdf-library are you using? The 2.? is slower then the newer ones. > I try to load the *whole* netcdf array in the second part as my first attempt. The netcdf array is: ['w', 'mf_i', 'mf_r'], double and their sizes are: >>> f.mf_r.shape (13, 18, 18, 38, 38) >>> f.mf_i.shape (13, 18, 18, 38, 38) >>> f.wf.shape (10, 13, 10, 10, 20, 20) The codelet is here: def getdata(self): f = netcdf.NetCDFFile("gkxy.nc", "r") for m in self.c.MEASTYPE: print 'Allocating for', m start = time() self.m[m] = zeros(f.variables[self.mmap[(m, self.c.SAMPTYPE[0])]+'_r'].shape, 'D') print 'time elapsed:', time() - start for s in self.c.SAMPTYPE: print 'Reading meas for', m, s start = time() self.m[m].real = self.m[m].real + \ f.variables[self.mmap[(m, s)]+'_r'][:] self.m[m].imaginary = self.m[m].imaginary + \ f.variables[self.mmap[(m, s)]+'_i'][:] print 'time elasped:', time() - start print 'Reading weight for', m, s start = time() self.w[(m, s)] = f.variables[self.wmap[(m, s)]][:] print 'time elapsed:', time() - start f.close() What I got is: >>> i.getdata() Allocating for 1 time elapsed: 1.53462100029 Reading meas for 1 1 time elasped: 185.918686032 Reading weight for 1 1 time elapsed: 3.38862001896 >>> Although the size of meas and weight is comparable, the time ratio is about *60*. The memory of my SGI200 machine is 256M but CPU is idle (<1% busy) and waiting for SWAP (>97% ratio) most of time when reading meas. Some 20M free memory is reported at the same time. netcdf library is 3.4 and the python module is Hinsen's. Can you tell me some reasons for that? > > Is there any way that I can read some subsets of one array stored in netcdf > > file without loading the whole array? Does f.variables['var'][x, y, ...] avoid > > loading the whole 'var' array? > > > This should already be possible? I deal with NetCDF-files of ca. 1Gb > size, so this is working. > I agree. Since the second part has to read hundred of times of different subsets of the netcdf array and almost covers the whole array, I wonder it may be better to read the whole array at once. Do you have a good method to handle complex data in netcdf? -- Min Xu City College of NY, CUNY Email: mxu1@email.gc.cuny.edu minxu@sci.ccny.cuny.edu Tel: (O) (212) 650-6865 (O) (212) 650-5046 (H) (212) 690-2119 From hinsen@cnrs-orleans.fr Mon Sep 20 20:17:44 1999 From: hinsen@cnrs-orleans.fr (Konrad Hinsen) Date: Mon, 20 Sep 1999 21:17:44 +0200 Subject: [Matrix-SIG] HELP! How to access a large array In-Reply-To: <199909191934.PAA21699@star.future.com> (minxu@sci.ccny.cuny.edu) References: <199909191934.PAA21699@star.future.com> Message-ID: <199909201917.VAA13937@chinon.cnrs-orleans.fr> > Is there any way that I can read some subsets of one array stored in netcdf > file without loading the whole array? Does f.variables['var'][x, y, ...] avoid > loading the whole 'var' array? Yes. Since many netCDF files store arrays that are larger than physical memory, this is in fact often the only way to deal with data in netCDF files. Any access to a subarray will read precisely that subarray. However, the time required for such an operation also depends on the layout of the data in the file. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.55.69 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From perry@stsci.edu Tue Sep 21 20:07:31 1999 From: perry@stsci.edu (Perry Greenfield) Date: Tue, 21 Sep 1999 15:07:31 -0400 (EDT) Subject: [Matrix-SIG] Re: Numeric plans Message-ID: <199909211907.PAA06724@eclipse.stsci.edu> We have a few comments on some of the points made in the past week or so regarding future work on Numeric. 1) "[Numeric] C code is a mess." (David Asher) Could David, Paul or someone elaborate a bit on what this means? Is this meant to say that it is just a mess to modify, or (worse) that it is unreliable as is? If the latter, it would seem to argue for a concerted effort to clean up soon; if the former, it would still argue for clean up, but perhaps on a more relaxed time scale. What scale of effort would be involved? 2) Paul Dubois mentioned the problems he was having settling on a consensus for how the "inverse of take" (the old scatter/gather issue) should be implemented when the array being indexed has rank > 1. > (Periodically the issue is raised of > > x[indices] = values > x[indices] as an r-value > > being desireable, as it was recently. As I pointed out before, everyone > agrees that the above line is desireable, but when x has rank > 1, there > are strongly differing views of what this ought to mean. Multiple > intelligent, experienced developers have told me that (a) I am an idiot, > because (b) there is only one obvious and highly desireable solution, > namely theirs Unfortunately the latter does not turn out to be the same > in all cases. > > The issue is complicated by the fact that the coding for setindex is > already totally incomprehensible and so changing it requires a great > deal of care and testing. > > It is time for proposals. Concrete, detailed. Discussion to follow. If > we can come to a consensus it would be nice. > Do not forget things like NewAxis, Ellipsis, and the colon. How do these > things mix and match? We think (as we've said before) that a less general mechanism would be better than none if consensus can't be achieved. In other words, do what Andrew Mullhaupt suggested and either require x to be rank 1 or coerce x to rank 1, producing a rank 1 result. The advantage of the former is that if we eventually can agree on what should be done with rank>1, then anything we decide will be backward compatible. Assuming that everyone agrees on what this means for a rank-1 array x, that behavior could be added immediately. Perhaps Paul (or someone) could summarize the competing views on this issue to spur some discussion on the more general approach. There is a fairly large number of past messages about this spread over years. A couple page summary of the competing proposals with the pros and cons of each (that we are aware of) would help focus the discussion. Maybe there will be consensus (or a one that a clear majority favors). Paul, is it possible to summarize this? 3) Are there aspects of this work (UserArray, general code clean-ups, or inverse of take) that others can help contribute to? If not... 4) A means of inheriting Numeric arrays is important to us also and we are very interested in how that issue is going to be solved. Is there any way for those of us that are interested to keep abreast of the ongoing design and implementation discussions? Perry Greenfield Rick White Paul Barrett From dubois1@llnl.gov Wed Sep 22 00:07:38 1999 From: dubois1@llnl.gov (Paul F. Dubois) Date: Tue, 21 Sep 1999 16:07:38 -0700 Subject: [Matrix-SIG] Re: Numeric plans References: <199909211907.PAA06724@eclipse.stsci.edu> Message-ID: <001701bf0486$34862ac0$3c810b18@plstn1.sfba.home.com> This is a multi-part message in MIME format. ------=_NextPart_000_000B_01BF044B.6D03FF20 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Perry et. al asked me for comments on the following: > 1) "[Numeric] C code is a mess." (David Asher) I invite you to peruse the source. Imagine you want to make the change requested in #2. I believe you will then hold this truth to be self-evident. > > 2) Paul Dubois mentioned the problems he was having settling on a > consensus for how the "inverse of take" (the old scatter/gather issue) > should be implemented when the array being indexed has rank > 1. ... > Paul, > is it possible to summarize this? Paul has a day job. That is why he asked for proposals. If your proposal is to implement x[indices] as both an r- and l-value for x one-dimensional only, then by golly I do know what to do. But I don't necessarily know how to do it, see #1. Egads, a circular reference, I will never be garbage collected. > > 3) Are there aspects of this work (UserArray, general code clean-ups, > or inverse of take) that others can help contribute to? If not... People have been contributing when the spirit moved them by sending in patches. David or I review the patches and as long as it doesn't offend our sensibilities we put them in. We'd like to get #12 fixed after our screwup and then would welcome things such as #2 being done for us. We both have a lot on our plates so we are in the annoying position of locking out the volunteer help for the moment because we are too busy. > > 4) A means of inheriting Numeric arrays is important to us also and we > are very interested in how that issue is going to be solved. Is there > any way for those of us that are interested to keep abreast of the ongoing > design and implementation discussions? David and I finally got a chance to talk together in person for a couple of hours and we had some ideas about how to proceed. That is the sum total of the "design". The general idea is to make a decent UserArray; this will require adding some things to the core. We got a lot of experience during the recent wild goose chase. I've been working on a facility for what I call Masked Arrays; that is, arrays that have an optional mask that indicates missing values. This *is* part of my day job so it is getting into pretty good shape now. I was going to back-fit UserArray from it. I attach the current version for your comments. This incarnation tries to let Numeric arrays and these MA arrays coexist. > > Perry Greenfield > Rick White > Paul Barrett > Supposing there were a full-featured UserArray module, similar to MA but without the mask, would it be helpful to you? Or perhaps, would it be helpful as a reuse-by-editor form of inheritance? I guess what I think I found out is that it is almost not really possible to inherit from Numeric without fairly massive redefinitions. There are some tricks one might be able to play to make a few extensions possible, such as using self.__class__ as a constructor where possible, but maybe even that isn't a good idea since it assumes you can make a valid Foo using just data as an argument, clearly not true in general. For the longer term, it would be nice to get a do-over. However, there are many different communities here with different needs and few social skills with which to resolve the differences. (:->. This fact is easiest to see if you think about x[i, :]. The speed burners want this to be a reference, the usabilities want this to be a copy. Best regards, Paul ------=_NextPart_000_000B_01BF044B.6D03FF20 Content-Type: text/plain; name="MA.py" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="MA.py" """MA facility Copyright 1999 Regents of the University of California. Released for unlimited redistribution; see file Legal.htm in Numerical = distribution.""" __version__ =3D 2.0 #This version gives up on the Extension Class stuff import Numeric from Precision import * import string, types class MAError (Exception): def __init__ (self, args=3DNone): self.args =3D args def __str__(self): return str(self.args) __MaxElements =3D 300 #Maximum size for printing def setMaxElements (m): "Set the maximum # of elements for printing arrays. " global __MaxElements n =3D int(m) if n <=3D 0: raise ValueError, "setMaxElements requires positive = argument." __MaxElements =3D n def getMaxElements (): "Get the maximum # of elements for printing arrays. " global __MaxElements return __MaxElements def getmask (a): "Mask of values in a, if any; None otherwise." if isMA(a):=20 return a.mask() else:=20 return None def getmaskarray (a): "Mask of values in a, ones if none." m =3D getmask(a) if m is None: return Numeric.ones(shape(a)) else: return m def mask_or (m1, m2): "Logical or of the masks m1 and m2, treating None as false." if m1 is None: return m2 if m2 is None: return m1 return Numeric.logical_or(m1, m2) def filled (a, value =3D None): """ a as a Numeric array with any masked areas replaced by value" No data copied if no masked area and data is already an array. """ if isMA(a): return a.filled(value) else: return Numeric.array(a, copy=3D0) class masked_unary_operation: def __init__ (self, aufunc, fill=3D0): self.f =3D aufunc self.fill =3D fill def __call__ (self, a): m =3D getmask(a) d1 =3D filled(a) if m is None: return self.f(d1) else: return MA(self.f(d1), m) class masked_binary_operation: def __init__ (self, aufunc, fill=3D0): self.f =3D aufunc self.fill =3D fill def __call__ (self, a, b): d1 =3D filled(a, self.fill) d2 =3D filled(b, self.fill) m =3D mask_or(getmask(a), getmask(b)) if m is None: return self.f(d1, d2) else: return MA(self.f(d1,d2), m) def reduce (self, target, axis=3D0): m =3D getmask(target) if m is None: return self.f.reduce (target, axis) else: t =3D target.filled(self.fill) tr =3D self.f.reduce(t, axis) m =3D Numeric.logical_and.reduce(m, axis) return MA(tr, m) class comparison: "A comparison function that returns a plain mask after doing the = comparison" def __init__ (self, f, fill1, fill2): self.f =3D f self.fill1 =3D fill1 self.fill2 =3D fill2 def __call__ (self, a, b): return self.f(filled(a, self.fill1), filled(b, self.fill2)) # class MA =20 # the use of filled in the constructor # makes sure we don't get MA's in data or mask=20 # This prevents recursion # and we can assume these are array objects (or None for mask). # Note current implementation assumes mask is 1's and 0's # so that we can use Numeric.choose in filled. class MA: "Arrays with possibly masked values." default_real_fill_value =3D 0.0 default_complex_fill_value =3D 0.0 + 0.0j default_character_fill_value =3D ' ' default_integer_fill_value =3D 0 default_unsigned_integer_fill_value =3D 0 default_object_fill_value =3D None def __init__(self, data, mask=3DNone, **keys): """MA(data, mask=3DNone)""" self.__data =3D filled(data) if mask is None: self.__mask =3D None else: self.__mask =3D filled(mask, 1) def __getattr__ (self, name): if name =3D=3D 'shape': return self.get_shape() else: return self.__dict__[name] def __setattr__ (self, name, value): if name =3D=3D 'shape': self.set_shape(value) else: self.__dict__[name] =3D value def __str__(self): return str(self.filled()) def __repr__(self): if self.__mask is None: return "MA(" + repr(filled(self)) + ")" else: return "MA(" + repr(filled(self)) + ", \n " + \ repr(self.__mask) + ")" def __float__(self): return MA (float(self.filled(0.0)), self.__mask) def __int__(self): return MA(int(self.filled(0)), self.__mask) # Note copy semantics here differ from Numeric =20 def __getitem__(self, i):=20 m =3D self.__mask if m is None: return Numeric.array(self.__data[i]) else: return MA(Numeric.array(self.__data[i]), = Numeric.array(m[i])) def __getslice__(self, i, j):=20 m =3D self.__mask if m is None: return Numeric.array(self.__data[i:j]) else: return MA(Numeric.array(self.__data[i:j]), = Numeric.array(m[i:j])) # -------- def __setitem__(self, index, value): self.__data[index] =3D filled(value) self.__mask =3D mask_or(self.__mask, getmask(value)) =20 def __setslice__(self, i, j, value): self.__data[i:j] =3D filled(value) m =3D getmask(value) if m is not None: sm =3D getmaskarray(self) sm[i:j] =3D mask_or(sm[i:j], m) self.__mask =3D sm def __len__ (self): "Return total number of elements in array." return self.size() def __cmp__ (self, other): raise MAError, "Cannot use comparison operators on MA = instances." =20 def __abs__(self):=20 return masked_unary_operation(Numeric.absolute)(self) def __neg__(self):=20 return masked_unary_operation(Numeric.negative)(self) def __add__(self, other):=20 return masked_binary_operation(Numeric.add)(self, other) =20 __radd__ =3D __add__ def __sub__(self, other):=20 return masked_binary_operation(Numeric.subtract,0)(self, other) def __rsub__(self, other):=20 return masked_binary_operation(Numeric.subtract,0)(other, self) def __mul__(self, other):=20 return masked_binary_operation(Numeric.multiply,0)(self, other) =20 __rmul__ =3D __mul__ def __div__(self, other):=20 return divide(self, other) def __rdiv__(self, other):=20 return divide(other, self) def __pow__(self,other, third=3DNone):=20 if third is not None:=20 raise MAException, "3 arg power unspoorted" return masked_binary_operation(Numeric.power, 1)(self, other) def __sqrt__(self):=20 return masked_unary_operation(Numeric.sqrt,0)(self) def default_fill_value (self): "Function to calculate default fill value for an array." x =3D self.typecode() if x in typecodes['Float']: return MA.default_real_fill_value if x in typecodes['Integer']: return MA.default_integer_fill_value if x in typecodes['Complex']: return MA.default_complex_fill_value if x in typecodes['Character']: return MA.default_character_fill_value if x in typecodes['UnsignedInteger']: return Numeric.absolute(MA.default_integer_fill_value) else: return MA.default_object_fill_value def set_fill_value (self, v): "Set the fill value to v." self.fill_value =3D v def filled (self, value=3DNone): """ Self with masked data replaced by value. If value is None, self.fill_value used. self.default_fill_value () used if value is None. Not a copy if no masked data. Result is romised to be a Numeric.array. """ if self.__mask is None: return self.__data else: if value is None: if hasattr(self, 'fill_value'): v =3D self.fill_value else: v =3D self.default_fill_value () self.fill_value =3D v else: v =3D value return Numeric.choose(self.__mask, (self.__data, v)) def get_shape(self): "Return the tuple giving the current shape." return self.__data.shape def ids (self): """Return the ids of the data and mask areas""" return (id(self.__data), id(self.__mask)) def mask(self): "Return the data mask, or None." return self.__mask =20 def set_shape (self, *sizes): """shape(n, m, ...) sets the shape.""" self.__data.shape =3D sizes if self.__mask is not None: self.__mask.shape =3D sizes =20 def size (self, axis =3D None): "Number of elements in array, or in a particular axis." s =3D self.shape if axis is None: return reduce(Numeric.multiply, s, 1) else: return s[axis] =20 def typecode(self): return self.__data.typecode() # these are disgusting def is_contiguous (self): return self.data.is_contiguous() def byte_swapped(self): return self.data.byte_swapped() =20 def isMA (x): "Is x an instance of MA?" return isinstance(x, MA) def masked_array (data, masked_value, typecode=3DNone, copy=3D1, = atol=3D1.e-8, rtol=3D1.e-8, notest=3D0): """Create a masked array for numerical data; no mask unless = necessary or if notest !=3D 0 """ abs =3D Numeric.absolute d =3D Numeric.array(data, typecode=3Dtypecode, copy=3Dcopy) dm =3D Numeric.less(abs(d-masked_value), atol+rtol*abs(d)) if notest or Numeric.sometrue(Numeric.ravel(dm)): result =3D MA(d, dm) else: result =3D MA(d) result.set_fill_value(masked_value) return result =20 def masked_object (data, masked_value, copy=3D1): "Create a masked array of places where exactly equal to masked = value" dm =3D Numeric.equal(data, masked_value, copy=3Dcopy) return MA(data, dm) =20 def rank (a): "Get the rank of a" return len(shape(a)) =20 def shape (a): "Get the shape of a" if isMA (a): return a.get_shape() else: return Numeric.shape(a) def size (a, axis=3DNone): "Get the number of elements in a, or along a certain axis." if isMA (a): return a.size(axis) else: s =3D Numeric.shape(a) if axis is None: return reduce(Numeric.multiply, s, 1) else: return s[axis] def count (a, axis =3D None): "Count of the non-masked elements." =20 m =3D getmask(a) if m is None: return size(a, axis) else: n1 =3D size(a, axis) n2 =3D sum (m, axis) return n1 - n2 def sum (a, axis =3D None): "Sum of all elements, or along a certain axis" if axis is None: return Numeric.add.reduce(Numeric.ravel(filled(a, 0))) else: return Numeric.add.reduce(filled(a, 0), axis) =20 def product (a, axis =3D None): "Product of all elements, or along a certain axis." if axis is None: return Numeric.add.reduce(Numeric.ravel(filled(a, 0))) else: return Numeric.add.reduce(filled(a, 0), axis) def average(a, axis=3DNone): "Average of the nonmasked elements of a" c =3D 1.0 * count(a, axis) a =3D sum(a, axis)=20 return divide(a, c) =20 NewAxis =3D Numeric.NewAxis arange =3D Numeric.arange ones =3D Numeric.ones zeros =3D Numeric.zeros array =3D Numeric.array #minimum, maximum? #searchsorted? sqrt =3D masked_unary_operation(Numeric.sqrt, 0.0) sin =3D masked_unary_operation(Numeric.sin, 0.0) cos =3D masked_unary_operation(Numeric.cos, 0.0) absolute =3D masked_unary_operation(Numeric.absolute, 0) negative =3D masked_unary_operation(Numeric.negative, 0) absolute =3D masked_unary_operation(Numeric.absolute, 0) nonzero =3D masked_unary_operation(Numeric.nonzero, 0) add =3D masked_binary_operation(Numeric.add, 0) subtract =3D masked_binary_operation(Numeric.subtract, 0) multiply =3D masked_binary_operation(Numeric.multiply, 1) power =3D masked_binary_operation(Numeric.power, 1) sometrue =3D masked_unary_operation(Numeric.sometrue, 0) alltrue =3D masked_unary_operation(Numeric.alltrue, 1) # all these comparisons return false where masks are true =20 equal =3D comparison(Numeric.equal, 0, 1) not_equal =3D comparison(Numeric.not_equal, 0, 0) less_equal =3D comparison(Numeric.less_equal, 1, 0) greater_equal =3D comparison(Numeric.greater_equal, 0, 1) less =3D comparison(Numeric.less, 1, 0) greater =3D comparison(Numeric.greater, 0, 1) def choose (condition, t): "Shaped like condition, values t[0] where condition true, t[1] = otherwise" d =3D Numeric.choose(filled(condition,0), map(filled, t)) m =3D Numeric.choose(filled(condition,0), map(getmaskarray, t)) return MA(d, m) def where (condition, x, y): return choose(Numeric.not_equal(condition, 0), (y, x)) def reshape (a, newshape): "Copy of a with a new shape." m =3D getmask(a) d =3D Numeric.reshape(filled(a), newshape) if m is None: return d else: return MA(d, Numeric.reshape(m, newshape)) def ravel (a): "a as one-dimensional, may share data and mask" m =3D getmask(a) d =3D Numeric.ravel(filled(a)) =20 if m is None: return d else: return MA(d, Numeric.ravel(m)) def concatenate (somearrays, axis=3D0): "concatenate the arrays along the given axis" d =3D Numeric.concatenate(map(filled, somearrays), axis) dm =3D Numeric.concatenate(map(getmaskarray, somearrays), axis) return MA(d, dm) def take (a, indices, axis=3D0): m =3D getmask(a) d =3D filled(a) if m is None: return Numeric.take(d, indices, axis) else: return MA(Numeric.take(d, indices, axis),=20 Numeric.take(m, indices, axis)) =20 def divide (x, y): "a/b, masked where b was masked or zero" a =3D filled(x, 0) b =3D filled(y, 1) bad_elements =3D equal(b, 0) c_mask =3D mask_or(mask_or(getmask(a), getmask(b)), bad_elements) return MA(Numeric.divide(a, b), c_mask) =20 # This section is stolen from a post about how to limit array printing. def limitedArrayRepr(a, max_line_width =3D None, precision =3D None, = suppress_small =3D None): global __MaxElements s =3D a.shape elems =3D Numeric.multiply.reduce(s) if elems > __MaxElements: if len(s) > 1: return 'array (%s) , type =3D %s, has %d elements' % \ (string.join(map(str, s), ","), a.typecode(), elems) else: return Numeric.array2string (a[:__MaxElements], = max_line_width, precision, suppress_small,',',0) + \ ('\n + %d more elements' % (elems - __MaxElements)) else: return Numeric.array2string (a, max_line_width, precision,=20 suppress_small,',',0) # replace the default with a smarter function for huge arrays original_array_repr =3D Numeric.array_repr original_array_str =3D Numeric.array_str Numeric.array_repr =3D limitedArrayRepr Numeric.array_str =3D limitedArrayRepr # fixup multiarray, as well. import multiarray multiarray.set_string_function(limitedArrayRepr, 0) multiarray.set_string_function(limitedArrayRepr, 1) if __name__ =3D=3D '__main__': import MAtest ------=_NextPart_000_000B_01BF044B.6D03FF20 Content-Type: text/plain; name="MAtest.py" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="MAtest.py" from MA import * def put(*stuff): for x in stuff: print x, print def test (va, mv): ua =3D masked_array(va, mv) put("shape(ua), ua.shape, ua.get_shape()", shape(ua), ua.shape, = ua.get_shape()) put("rank(ua), size(ua), size(ua,0), size(ua,1)\n",=20 rank(ua), size(ua), size(ua,0), size(ua,1)) put("mask(ua)\n", getmask(ua)) put("repr(ua)\n", repr(ua)) put("va\n", va) put("ua\n", ua) put ("ua.filled(-1)\n", ua.filled(-1)) put("isMA(ua)", isMA(ua)) put("isMA(va)", isMA(va)) put("ua * va", type(ua*va), '\n', ua*va) put("va * ua", type(va*ua), '\n', va*ua) put("ua[:,2]\n", ua[:,2]) u1 =3D ravel(ua) put("u1", u1.filled(), u1.mask()) =20 put("u1[7]\n", u1[7]) put("u1[4:12]\n", u1[4:12]) put("cos(ua)\n", cos(ua)) put("abssolute(-ua)\n", absolute(-ua)) put("ua+ua\n", ua+ua) put("ua**2\n", ua**2) put("sum(ua)\n", sum(ua)) put("sum(ua, 1)\n", sum(ua, 1)) put("equal(ua, va)\n", equal(ua, va)) put("average(ua)\n", average(ua)) put("average(ua, 1)\n", average(ua, 1)) put("take(ua, (1,2))\n", take(ua, (1,2))) put("take(ua, (1,2), 1)\n", take(ua, (1,2), 1)) va=3Dreshape(arange(12), (4,3)) mv =3D 99 va[0,1]=3Dmv va[2,2] =3D mv test (va, mv) va =3D reshape(arange(12)*1.0, (4,3)) mv =3D 1.e20 va[0,1] =3D mv va[2,2] =3D mv test (va, mv) # now try pathology va=3Dreshape(arange(12)*1.0, (4,3)) mv =3D 1.e20 va[0,1] =3D mv va[2,2] =3D mv va[:, 2] =3D mv test (va, mv) a =3D arange(4, typecode=3DFloat) b =3D arange(4, typecode=3DFloat) -2.0 am =3D MA(a, [1,0,0,0]) bm =3D MA(b, [0,0,0,1]) put("am\n", am) put("bm\n", bm) put("a\n", a) put("b\n", b) put("divide(a, b)\n", divide(a, b)) put("am / b\n", am / b) put("a / bm\n", a / bm) put("am / bm\n", am / bm) ua =3D masked_array(va, mv, copy=3D0) ub =3D MA(va, ua.mask()) #shares a mask and data ua[1,0] =3D 22.0 uc =3D MA(ua, ua.mask()) #shares only the mask ua.mask()[0,0] =3D 1 put("ids\n", map(lambda x: x.ids(), (ua,ub,uc))) put("3 tests of sharing, should be all zeros\n") put("ub[1,0] - ua[1,0]\n", ub[1,0] - ua[1, 0]) put("ua.filled(10.) - ub.filled(10.)\n", ua.filled(10.) - = ub.filled(10.)) put("ua - uc\n", ua - uc) wo =3D where([1,1,0,0], am, am*10.) wo.missing_value =3D 1.e20 put("wo\n", wo) ns =3D arange(12, typecode=3DFloat) ms =3D masked_array(ns, 5.0) qs =3D masked_array(ns, 100.0) ns =3D reshape(ns, (4,3)) ms.set_shape(4,3) qs.set_shape(4,3) put("Should be (4,3)\n", shape(ns)) put("Should be (4,3)\n", shape(ms)) put("Should be (4,3)\n", shape(qs)) ------=_NextPart_000_000B_01BF044B.6D03FF20-- From amullhau@zen-pharaohs.com Wed Sep 22 02:45:01 1999 From: amullhau@zen-pharaohs.com (Andrew P. Mullhaupt) Date: Tue, 21 Sep 1999 21:45:01 -0400 Subject: [Matrix-SIG] Re: Numeric plans References: <199909211907.PAA06724@eclipse.stsci.edu> Message-ID: <10ce01bf049c$169472e0$a602550a@amullhau> > 1) "[Numeric] C code is a mess." (David Asher) > > Could David, Paul or someone elaborate a bit on what this means? Have a look at the source. If you don't shudder, you've either been programming not long enough or way too long. Later, Andrew Mullhaupt From mhagger@blizzard.harvard.edu Fri Sep 24 01:52:31 1999 From: mhagger@blizzard.harvard.edu (Michael Haggerty) Date: 24 Sep 1999 00:52:31 -0000 Subject: [Matrix-SIG] ANNOUNCE: Gnuplot.py 1.3 Message-ID: <19990924005231.13350.qmail@cyclone.harvard.edu> This is to announce the release of version 1.3 of Gnuplot.py. Gnuplot.py is a Python [1] package that allows you to create graphs from within Python using the gnuplot [2] plotting program. This version fixes plotting of function objects and moves to a package structure. Gnuplot.py can be obtained from http://monsoon.harvard.edu/~mhagger/Gnuplot/Gnuplot.html Prerequisites (see footnotes): the Python interpreter [1] the Python Numeric module [3] the gnuplot program [2] Some ways this package can be used: 1. Interactive data processing: Use Python's excellent Numeric package to create and manipulate arrays of numbers, and use Gnuplot.py to visualize the results. 2. Web graphics: write CGI scripts in Python that use gnuplot to output plots in GIF format and return them to the client. 3. Glue for numerical applications (this is my favorite): wrap your C++/C/Fortran subroutines so that they are callable from Python, then you can perform numerical computations interactively from scripts or from the command line and use Gnuplot.py to plot the output on the fly. 4. Compute a series of datasets in Python and plot them one after the other using Gnuplot.py to produce a crude animation. Features added in version 1.3: * Converted to package format. The main file is now called __init__.py, which can be loaded by typing 'import Gnuplot'. * Passing GridData a callable function was basically broken because of the kludgey way of overloading the argument. Instead of trying to fix it, I moved that functionality to a new type of PlotItem called 'GridFunc'. Features already present in older versions: + Two and three-dimensional plots. + Plot data from memory, from a file, or from an expression. + Support for multiple simultaneous gnuplot sessions. + Can pass arbitrary commands to the gnuplot program. + Object oriented, extensible design with several built-in types of plot items. + Portable and easy to install (nothing to compile except on Windows). + Support for MS Windows, using the `pgnuplot.exe' program. + Support for sending data to gnuplot as `inline' or `binary' data. These are optimizations that also remove the need for temporary files. Temporary files are also still supported. Footnotes: ---------- [1] Python is an excellent object-oriented scripting/rapid development language that is also especially good at gluing programs together. [2] gnuplot is a free, popular, very portable plotting program with a command-line interface. It can make 2-d and 3-d plots and can output to myriad printers and graphics terminals. [3] The Numeric Python extension is a Python module that adds fast and convenient array manipulations to the Python language. Please pass along any comments, suggestions, or complaints! Yours, Michael -- Michael Haggerty mhagger@blizzard.harvard.edu From ryszard@arqule.com Fri Sep 24 12:42:02 1999 From: ryszard@arqule.com (Czerminski, Ryszard) Date: Fri, 24 Sep 1999 07:42:02 -0400 Subject: [Matrix-SIG] ANNOUNCE: Gnuplot.py 1.3 Message-ID: <6B00FB949906D211A88800104B8AE520F419F7@WIREHEAD> I am looking for a function which could compute combinations given lexicographical index e.g. C(1;3,2) -> 1,2 C(2;3,2) -> 1,3 C(3;3,2) -> 2,3 I have found function which does this ( http://math.nist.gov/GAMS.html ) but it is limited to small numbers since it is using regular 4 bytes representation for integers and therefore index range is severly limited ( < 2^32 ). Any pointers to the software which does this for integers of arbitrary length would be very much appreciated. Ryszard Czerminski phone: (781)395-1269 x 479 ArQule, Inc. e-mail: ryszard@arqule.com 200 Boston Avenue http://www.arqule.com Medford, MA 02155 From p.s.craig@durham.ac.uk Fri Sep 24 16:49:57 1999 From: p.s.craig@durham.ac.uk (Peter Craig) Date: Fri, 24 Sep 1999 16:49:57 +0100 Subject: [Matrix-SIG] Bug(s) in Numeric Message-ID: <37EB9DA5.7F0FDC04@durham.ac.uk> This is a multi-part message in MIME format. --------------67510CF3F1534029D986A5E6 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Greetings all, I have finally found two bugs in the Numeric extension which for some time have been causing occasional segmentation faults in a fairly heavily used piece of software in my department. I attach a patch for arrayobject.c from LLNLDistribution11 which fixes both bugs. Note that these bugs affect only programs using the facility to process arrays of arbitrary python objects. Arrays of integers, floating point and strings are not affected. The bug in both cases is that a pointer from malloc is being incremented before the call to free! Cheers, Peter Craig --------------67510CF3F1534029D986A5E6 Content-Type: text/plain; charset=us-ascii; name="patch" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="patch" --- arrayobject.c.dist Fri Apr 2 01:14:45 1999 +++ arrayobject.c Fri Sep 24 16:15:28 1999 @@ -159,7 +159,7 @@ /* every python object in the array. */ extern int PyArray_INCREF(PyArrayObject *mp) { int i, n; - PyObject **data; + PyObject **data, **data2; if (mp->descr->type_num != PyArray_OBJECT) return 0; @@ -170,16 +170,17 @@ } n = SIZE(mp); + data2 = data; for(i=0; idescr->type_num != PyArray_OBJECT) return 0; @@ -190,9 +191,10 @@ } n = SIZE(mp); + data2 = data; for(i=0; i Message-ID: <000001bf0726$0ea96f40$582d153f@tim> [Czerminski, Ryszard] > I am looking for a function which could compute > combinations given lexicographical index e.g. > > C(1;3,2) -> 1,2 > C(2;3,2) -> 1,3 > C(3;3,2) -> 2,3 > > I have found function which does this > ( http://math.nist.gov/GAMS.html ) but it is limited > to small numbers since it is using regular 4 bytes > representation for integers and therefore index > range is severly limited ( < 2^32 ). As is often the case, relaxing such limits creates other problems; in this case, the Fortran TOMS 515 algorithm recomputes the number of combinations from scratch each time thru the inner loop, which is very expensive if choosing many elements from a very large set. > Any pointers to the software which does this for > integers of arbitrary length would be very much > appreciated. One is attached. Note that since this is Python instead of Fortran, everything is 0-based. That is, when asking for the i'th combination of m things taken n at a time, i ranges from 0 up to but not including C(m, n), and the "canonical set" is taken to be range(m). The inner loop avoids the problem above, so this is reasonably fast even for multi-hundred digit indices. more-combinations-than-a-reasonable-person-could-want-ly y'rs - tim def _chop(n): """n -> int if it fits, else long.""" try: return int(n) except OverflowError: return n def comb(m, n): """m, n -> number of combinations of m items, n at a time. m >= n >= 0 required. """ if not m >= n >= 0: raise ValueError("m >= n >= 0 required: " + `m, n`) if n > (m >> 1): n = m-n if n == 0: return 1 result = long(m) i = 2 m, n = m-1, n-1 while n: # assert (result * m) % i == 0 result = result * m / i i = i+1 n = n-1 m = m-1 return _chop(result) def combatindex(m, n, i): """m, n, i -> i'th combination of m items taken n at a time. m >= n >= 1 and 0 <= i < comb(m, n) required. Return the i'th combination in lexicographic order, as a list of n elements taken from range(m). The index (i) is 0-based. Example: >>> for i in range(6): ... print combatindex(4, 2, i) [0, 1] [0, 2] [0, 3] [1, 2] [1, 3] [2, 3] """ if not m >= n >= 1: raise ValueError("m >= n >= 1 required: " + `m, n`) c = long(comb(m, n)) if not 0 <= i < c: raise ValueError("0 <= i < comb(m,n) required: " + `i, c`) result = [] # have c == comb(m, n), want comb(m-1,n-1) c = c * n / m # invariant: c == comb(m-1, n-1) for element in xrange(m): if i < c: # take this element, and n-1 from the remaining result.append(element) n = n-1 if n == 0: break # have c == comb(m-1,n), want comb(m-2,n-1) c = c * n / (m-1) else: # skip this element, and take all from the remaining i = i-c # have c == comb(m-1,n-1), want comb(m-2,n-1) c = c * (m-n) / (m-1) m = m-1 assert i == 0 return result def _test(): failures = 0 m, n = 10, 6 c = comb(m, n) last = [0] * n for i in xrange(c): this = combatindex(m, n, i) if len(this) != n: print "*** length error:", m, n, i, this failures = failures + 1 if not last < this: print "*** lexicographic error:", m, n, i, last, this failures = failures + 1 last = this if this != range(m-n, m): print "*** last value wrong:", m, n, c-1, this failures = failures + 1 # c has more than 1400 digits in the next case -- may take a # few seconds m, n = 5000, 2000 c = comb(m, n) this = combatindex(m, n, 0) if this != range(n): print "*** first value wrong:", m, n, 0, this failures = failures + 1 this = combatindex(m, n, c-1) if this != range(m-n, m): print "*** last value wrong:", m, n, c-1, this failures = failures + 1 if failures: print "*********** TEST FAILED *************" if __name__ == "__main__": _test() From ryszard@arqule.com Wed Sep 29 19:08:07 1999 From: ryszard@arqule.com (Czerminski, Ryszard) Date: Wed, 29 Sep 1999 14:08:07 -0400 Subject: [Matrix-SIG] Re: combinations (summary) Message-ID: <6B00FB949906D211A88800104B8AE520F41A0E@WIREHEAD> This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_000_01BF0AA5.94F84666 Content-Type: text/plain; charset="iso-8859-1" I would like to express my appreciation to the people who took time to help me with combinations problem: Tim Peters - for Python code which does EXACTLY what I wanted, for advise and for pointers to GNU's GMP package Mirek Gorski - for pointer to GNU's GMP package http://www.math.iastate.edu/cbergman/crypto/gmp/gmp_toc.html Pat Walters - for C++ class for generating combinations Emili Besalu - for pointer to his web page dealing with similar problems http://iqc.udg.es/~emili/emili_nc.htm Thank you all very much! -- Ryszard ORIGINAL QUESTION: I am looking for a function which could compute combinations given lexicographical index e.g. C(1;3,2) -> 1,2 C(2;3,2) -> 1,3 C(3;3,2) -> 2,3 I have found function which does this ( http://math.nist.gov/GAMS.html ) but it is limited to small numbers since it is using regular 4 bytes representation for integers and therefore index range is severly limited ( < 2^32 ). Any pointers to the software which does this for integers of arbitrary length would be very much appreciated. Ryszard Czerminski phone: (781)395-1269 x 479 ArQule, Inc. e-mail: ryszard@arqule.com 200 Boston Avenue http://www.arqule.com Medford, MA 02155 P.S. from Tim Peters: > I am looking for a function which could compute > combinations given lexicographical index e.g. > > C(1;3,2) -> 1,2 > C(2;3,2) -> 1,3 > C(3;3,2) -> 2,3 > > I have found function which does this > ( http://math.nist.gov/GAMS.html ) but it is limited > to small numbers since it is using regular 4 bytes > representation for integers and therefore index > range is severly limited ( < 2^32 ). As is often the case, relaxing such limits creates other problems; in this case, the Fortran TOMS 515 algorithm recomputes the number of combinations from scratch each time thru the inner loop, which is very expensive if choosing many elements from a very large set. > Any pointers to the software which does this for > integers of arbitrary length would be very much > appreciated. One is attached. Note that since this is Python instead of Fortran, everything is 0-based. That is, when asking for the i'th combination of m things taken n at a time, i ranges from 0 up to but not including C(m, n), and the "canonical set" is taken to be range(m). The inner loop avoids the problem above, so this is reasonably fast even for multi-hundred digit indices. more-combinations-than-a-reasonable-person-could-want-ly y'rs - tim def _chop(n): """n -> int if it fits, else long.""" try: return int(n) except OverflowError: return n def comb(m, n): """m, n -> number of combinations of m items, n at a time. m >= n >= 0 required. """ if not m >= n >= 0: raise ValueError("m >= n >= 0 required: " + `m, n`) if n > (m >> 1): n = m-n if n == 0: return 1 result = long(m) i = 2 m, n = m-1, n-1 while n: # assert (result * m) % i == 0 result = result * m / i i = i+1 n = n-1 m = m-1 return _chop(result) def combatindex(m, n, i): """m, n, i -> i'th combination of m items taken n at a time. m >= n >= 1 and 0 <= i < comb(m, n) required. Return the i'th combination in lexicographic order, as a list of n elements taken from range(m). The index (i) is 0-based. Example: >>> for i in range(6): ... print combatindex(4, 2, i) [0, 1] [0, 2] [0, 3] [1, 2] [1, 3] [2, 3] """ if not m >= n >= 1: raise ValueError("m >= n >= 1 required: " + `m, n`) c = long(comb(m, n)) if not 0 <= i < c: raise ValueError("0 <= i < comb(m,n) required: " + `i, c`) result = [] # have c == comb(m, n), want comb(m-1,n-1) c = c * n / m # invariant: c == comb(m-1, n-1) for element in xrange(m): if i < c: # take this element, and n-1 from the remaining result.append(element) n = n-1 if n == 0: break # have c == comb(m-1,n), want comb(m-2,n-1) c = c * n / (m-1) else: # skip this element, and take all from the remaining i = i-c # have c == comb(m-1,n-1), want comb(m-2,n-1) c = c * (m-n) / (m-1) m = m-1 assert i == 0 return result def _test(): failures = 0 m, n = 10, 6 c = comb(m, n) last = [0] * n for i in xrange(c): this = combatindex(m, n, i) if len(this) != n: print "*** length error:", m, n, i, this failures = failures + 1 if not last < this: print "*** lexicographic error:", m, n, i, last, this failures = failures + 1 last = this if this != range(m-n, m): print "*** last value wrong:", m, n, c-1, this failures = failures + 1 # c has more than 1400 digits in the next case -- may take a # few seconds m, n = 5000, 2000 c = comb(m, n) this = combatindex(m, n, 0) if this != range(n): print "*** first value wrong:", m, n, 0, this failures = failures + 1 this = combatindex(m, n, c-1) if this != range(m-n, m): print "*** last value wrong:", m, n, c-1, this failures = failures + 1 if failures: print "*********** TEST FAILED *************" if __name__ == "__main__": _test() ========================================================= from Pat Walters: ------_=_NextPart_000_01BF0AA5.94F84666 Content-Type: application/octet-stream; name="combo.cpp" Content-Disposition: attachment; filename="combo.cpp" #include #include #include #include using namespace std; #include "combo.h" // Combination generator class // Takes a vector of integers as input and generates // all combinations /* int main(int argc, char *argv[]) { vector b(3); b[0] = 2; b[1] = 3; b[2] = 2; combo c(b); while (!c.done()) { print_v(c.get_current()); c.next(); } return(0); } Produces 0 0 0 0 0 1 0 1 0 0 1 1 0 2 0 0 2 1 1 0 0 1 0 1 1 1 0 1 1 1 1 2 0 1 2 1 */ // support function to print a vector void print_v(const vector &v) { for (int i = 0; i < v.size(); i++) printf("%4d ",v[i]); printf("\n"); fflush(stdout); } // constructor - takes a vector of ints // each int represents the maximum number of possibilities for // that position. combo::combo(const vector &x) { enough = false; high.resize(x.size()); curr.resize(x.size()); fill (curr.begin(),curr.end(),0); for ( int i = 0; i < x.size(); i++) high[i] = x[i] - 1; len = high.size(); limit = len-1; } // debuging function to show a combination void combo::show() { for (int i = 0; i < curr.size(); i++) printf("%4d ",curr[i]); printf("\n"); fflush(stdout); } // generate the next combination void combo::next() { bool raised; if (curr[limit] high; bool enough; int len; int limit; vector curr; public : combo(const vector &high); void next(); vector &get_current() {return(curr);}; bool done() {return(enough);} void show(); }; void print_v(const vector &v); ------_=_NextPart_000_01BF0AA5.94F84666-- From acornet@emory.edu Wed Sep 29 23:03:30 1999 From: acornet@emory.edu (Ben Cornett) Date: Wed, 29 Sep 1999 18:03:30 -0400 Subject: [Matrix-SIG] Numerical compile problems Message-ID: <37F28CB2.F4B3420B@emory.edu> I feel like this question has probably come up before, but since I couldn't find anything that seemed pertinent in the archives, I post it here, hoping someone might lend me a hand. I'm having problems installing Numerical on an SGI Octane R10000 under Irix6.5.4, using the 7.3 compilers. The compilation seems to go ok, but I get one unhappy python interpreter (core dump) when I attempt to import Numerical. I've tried various permutations in the 'Setup' file, both with and without the sgi blas libraries installed. I can't really think of any other relevant info at the moment. If anyone can help me, I would greatly appreciate it. Best, Ben From hinsen@cnrs-orleans.fr Thu Sep 30 16:40:13 1999 From: hinsen@cnrs-orleans.fr (Konrad Hinsen) Date: Thu, 30 Sep 1999 17:40:13 +0200 Subject: [Matrix-SIG] Numerical compile problems In-Reply-To: <37F28CB2.F4B3420B@emory.edu> (message from Ben Cornett on Wed, 29 Sep 1999 18:03:30 -0400) References: <37F28CB2.F4B3420B@emory.edu> Message-ID: <199909301540.RAA06726@chinon.cnrs-orleans.fr> > couldn't find anything that seemed pertinent in the archives, I post it > here, hoping someone might lend me a hand. I'm having problems > installing Numerical on an SGI Octane R10000 under Irix6.5.4, using the > 7.3 compilers. The compilation seems to go ok, but I get one unhappy > python interpreter (core dump) when I attempt to import Numerical. I've Did you try this under a debugger in order to find out *where* it crashes? Something else you might try is linking the NumPy C modules statically; that gives you better checking for missing or wrong symbols, which might well be the cause. I have compiled NumPy on an Octane system without any problems a while ago, but I don't remember the Irix and compiler version numbers. Konrad. -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.55.69 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From heather@v1.wustl.edu Thu Sep 30 23:24:52 1999 From: heather@v1.wustl.edu (Heather Drury) Date: Thu, 30 Sep 1999 17:24:52 -0500 (CDT) Subject: [Matrix-SIG] performance hit in c extensions Message-ID: <199909302224.RAA04282@v1.wustl.edu> Hi, I have sucessfully written wrappers for most of my C code so I can call the modules directly from python. Unfortunately, it *appears* that the code executes about 50% slower (I haven't actually timed it) when called from python versus being called from the command line. I would not have thought there would be any performance impact. Does that seem reasonable? Thanks, Heather -- Heather Drury heather@v1.wustl.edu Washington University School of Medicine http://v1.wustl.edu Department of Anatomy & Neurobiology Phone: 314-362-4325 660 S. Euclid, MS 8108 FAX: 314-747-4370 St. Louis, MO 63110-1093