From jdhunter at ace.bsd.uchicago.edu Wed Oct 2 11:36:13 2002 From: jdhunter at ace.bsd.uchicago.edu (John Hunter) Date: Wed Oct 2 11:36:13 2002 Subject: [Numpy-discussion] calling multiarraymodule.c funcs from c api Message-ID: I am new to numpy extension writing, and am trying to call some of the functions defined in Numeric-21.3/Src/multiarraymodule.c, eg extern PyObject *PyArray_Correlate(PyObject *op1, PyObject *op2, int mode) in a Numeric extension module that I am writing. Are these functions available via the C API, and if so, how should I go about accessing them? Or are the only array functions defined in Include/arrayobject.h available in the C API? Below is my prototype module. I am defining a function 'xcorr' that just does what cross_correlate does, as a test case to see if I can access the multiarray C API: /* jdhscipy.c -- Wed Oct 2 2002 */ #include "Python.h" #include "Numeric/arrayobject.h" #include "stdio.h" static PyObject *ErrorObject; extern PyObject *PyArray_Correlate(PyObject*, PyObject*, int); static char xcorr__doc__[] = ""; static PyObject * xcorr(PyObject *self, PyObject *args) { PyObject *shape, *a0; int mode=0; if (!PyArg_ParseTuple(args, "OO|i", &a0, &shape, &mode)) return NULL; return PyArray_Correlate(a0, shape, mode); } static struct PyMethodDef jdhscipy_methods[] = { {"xcorr", xcorr, 1, xcorr__doc__}, {NULL, NULL} /* sentinel */ }; /* Initialization function for the module (*must* be called initjdhscipy) */ static char jdhscipy_module_documentation[] = "My first scipy extension" ; DL_EXPORT(void) initjdhscipy(void) { PyObject *m, *d; /* Create the module and add the functions */ m = Py_InitModule4("jdhscipy", jdhscipy_methods, jdhscipy_module_documentation, (PyObject*)NULL,PYTHON_API_VERSION); /* Import the array object */ import_array(); /* Add some symbolic constants to the module */ d = PyModule_GetDict(m); ErrorObject = PyString_FromString("jdhscipy.error"); PyDict_SetItemString(d, "error", ErrorObject); /* XXXX Add constants here */ /* Check for errors */ if (PyErr_Occurred()) Py_FatalError("can't initialize module jdhscipy"); } From nwagner at mecha.uni-stuttgart.de Fri Oct 4 03:52:01 2002 From: nwagner at mecha.uni-stuttgart.de (Nils Wagner) Date: Fri Oct 4 03:52:01 2002 Subject: [Numpy-discussion] numarray via CVS Message-ID: <3D9D745C.2E29986B@mecha.uni-stuttgart.de> Hi, I tried to install numarray via latest CVS. However python setup.py install failed This is the output Src/_ndarraymodule.c:240: warning: `_ndarray_bytestride_set' defined but not use d Src/_ndarraymodule.c:260: warning: `_ndarray_byteoffset_get' defined but not use d Src/_ndarraymodule.c:266: warning: `_ndarray_byteoffset_set' defined but not use d Src/_ndarraymodule.c:288: warning: `_ndarray_aligned_get' defined but not used Src/_ndarraymodule.c:294: warning: `_ndarray_aligned_set' defined but not used Src/_ndarraymodule.c:323: warning: `_ndarray_contiguous_get' defined but not use d Src/_ndarraymodule.c:329: warning: `_ndarray_contiguous_set' defined but not use d error: command 'gcc' failed with exit status 1 Any suggestion ? Nils Nils From jmiller at stsci.edu Fri Oct 4 06:26:05 2002 From: jmiller at stsci.edu (Todd Miller) Date: Fri Oct 4 06:26:05 2002 Subject: [Numpy-discussion] numarray via CVS Message-ID: <3D9D96BB.6070401@stsci.edu> Nils Wagner wrote: >Hi, > >I tried to install numarray via latest CVS. However >python setup.py install failed > >This is the output > >Src/_ndarraymodule.c:240: warning: `_ndarray_bytestride_set' defined but >not use d >Src/_ndarraymodule.c:260: warning: `_ndarray_byteoffset_get' defined but >not use d >Src/_ndarraymodule.c:266: warning: `_ndarray_byteoffset_set' defined but >not use d >Src/_ndarraymodule.c:288: warning: `_ndarray_aligned_get' defined but >not used >Src/_ndarraymodule.c:294: warning: `_ndarray_aligned_set' defined but >not used >Src/_ndarraymodule.c:323: warning: `_ndarray_contiguous_get' defined but >not use d >Src/_ndarraymodule.c:329: warning: `_ndarray_contiguous_set' defined but >not use d >error: command 'gcc' failed with exit status 1 > > >Any suggestion ? > >Nils > >Nils > I'm in the process of merging some new C basetypes into the head of CVS so I guess the HEAD has to be considered unstable for the next few days. I work hard to keep the CVS in working order, but you've caught me in the middle of "something big". For now, I have this advice: 1. Use numarray-0.3.6 until numarray-0.4 is released. 2. When upgrading numarray, always remove the old version. 3. If you're intereseted in writing extensions, stick with numarray's Numeric compatability API. Todd -- Todd Miller jmiller at stsci.edu STSCI / SSB From bsder at mail.allcaps.org Sun Oct 6 19:18:03 2002 From: bsder at mail.allcaps.org (Andrew P. Lentvorski) Date: Sun Oct 6 19:18:03 2002 Subject: [Numpy-discussion] int and float ufunc? Message-ID: <20021006190227.O41304-100000@mail.allcaps.org> Could someone explain the reason why int() and float() don't work as ufuncs? Is it just that someone needs to write the code, or is there some subtlety at work that I am missing? It took a bit of digging to go find the fact that what I wanted is newarray = oldarray.astype(MagicNewTypeCode) Thanks, -a From oliphant at ee.byu.edu Sun Oct 6 23:30:01 2002 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Sun Oct 6 23:30:01 2002 Subject: [Numpy-discussion] int and float ufunc? In-Reply-To: <20021006190227.O41304-100000@mail.allcaps.org> Message-ID: > Could someone explain the reason why int() and float() don't work as > ufuncs? Is it just that someone needs to write the code, or is there some > subtlety at work that I am missing? What would you have them do? Their current definition works for returning suitable arrays as integers and floats respectively, just as the Python documentation says these two functions should. It would be an easy thing to map int() to oldarray.astype(Int) and so forth but is this a good policy. I submit it is not. -Travis O. From bsder at mail.allcaps.org Mon Oct 7 01:43:04 2002 From: bsder at mail.allcaps.org (Andrew P. Lentvorski) Date: Mon Oct 7 01:43:04 2002 Subject: [Numpy-discussion] int and float ufunc? In-Reply-To: Message-ID: <20021007011036.P41789-100000@mail.allcaps.org> On Mon, 7 Oct 2002, Travis Oliphant wrote: > > Could someone explain the reason why int() and float() don't work as > > ufuncs? Is it just that someone needs to write the code, or is there some > > subtlety at work that I am missing? > > What would you have them do? Their current definition works for returning > suitable arrays as integers and floats respectively, just as the Python > documentation says these two functions should. Okay, now I'm really confused ... >>> import Numeric >>> a = Numeric.array([1.5, 2.5, 3.5], Numeric.Float64) >>> a array([ 1.5, 2.5, 3.5]) >>> int(a) Traceback (most recent call last): File "", line 1, in ? TypeError: Only rank-0 arrays can be converted to Python scalars. I guess I'm really dense, but how does that result constitute a definition that "works"? Has this behavior changed recently such that I should update my version of Numeric? If so, that would be a wonderful solution to my problem (explained below). > It would be an easy thing to map int() to oldarray.astype(Int) and so > forth but is this a good policy. I submit it is not. I could be persuaded either way. I was simply wondering what the arguments were. My usage is in converting vectors of data before passing them to the screen for display. The conversion function is a bunch of ufuncs and then the resulting vector needs to get converted to int before being passed to X. The same code works for Python floats, scalars, or Numeric arrays with the exception of the int function. If the problem was simply a lack of someone to write the code, I thought I might do it. I thought it might be a little faster than me writing array versions of all of my conversion functions, although I haven't looked at the complexity of the Numeric code. So, I could be completely wrong about that. If the traceback is "designed" behavior, I guess I'll just have to suck it up and write array versions of my functions. Thanks, -a From jmiller at stsci.edu Mon Oct 7 04:31:02 2002 From: jmiller at stsci.edu (Todd Miller) Date: Mon Oct 7 04:31:02 2002 Subject: [Numpy-discussion] arange(-10) Message-ID: <3DA17273.3020707@stsci.edu> The subject line contains what I consider to be an invalid range expression. Numarray, now and always, reacts badly to it. Currently, numarray tries to allocate a multi-gigabyte array. In the past, it has dumped core. The question is, what should it do? 1. raise ValueError, "Invalid negative range expression" (my +1) 2. zeros((0,), 'l') (Numeric does this) Is there a good justification to keep the existing Numeric behavior? Any other suggestions? Todd From pearu at cens.ioc.ee Mon Oct 7 05:26:05 2002 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Mon Oct 7 05:26:05 2002 Subject: [Numpy-discussion] arange(-10) In-Reply-To: <3DA17273.3020707@stsci.edu> Message-ID: On Mon, 7 Oct 2002, Todd Miller wrote: > The subject line contains what I consider to be an invalid range > expression. Numarray, now and always, reacts badly to it. Currently, > numarray tries to allocate a multi-gigabyte array. In the past, it has > dumped core. > > The question is, what should it do? > > 1. raise ValueError, "Invalid negative range expression" (my +1) > 2. zeros((0,), 'l') > (Numeric does this) > > Is there a good justification to keep the existing Numeric behavior? > Any other suggestions? Does Numarray support empty arrays? If yes, I'd vote for Python behavior: >>> range(-10) [] Otherwise, the case 1. Hmm, according to Numeric.arange docs, "arange is just like range() except it returns an array whose type can be specified ...". But given example shows that there are other (undocumented?) exceptions when comparing arange and range. Pearu From pearu at cens.ioc.ee Mon Oct 7 05:43:02 2002 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Mon Oct 7 05:43:02 2002 Subject: [Numpy-discussion] int and float ufunc? In-Reply-To: <20021007011036.P41789-100000@mail.allcaps.org> Message-ID: On Mon, 7 Oct 2002, Andrew P. Lentvorski wrote: > On Mon, 7 Oct 2002, Travis Oliphant wrote: > > > What would you have them do? Their current definition works for returning > > suitable arrays as integers and floats respectively, just as the Python ^^^^^^^^^^^^^^^ > > documentation says these two functions should. > > Okay, now I'm really confused ... > > >>> import Numeric > >>> a = Numeric.array([1.5, 2.5, 3.5], Numeric.Float64) > >>> a > array([ 1.5, 2.5, 3.5]) > >>> int(a) > Traceback (most recent call last): > File "", line 1, in ? > TypeError: Only rank-0 arrays can be converted to Python scalars. ^^^^^^^^^^^^^ > I guess I'm really dense, but how does that result constitute a definition > that "works"? See ^^^ above. In this case, suitable arrays are rank-0 arrays. So, the definition works (see also below). > > It would be an easy thing to map int() to oldarray.astype(Int) and so > > forth but is this a good policy. I submit it is not. > > I could be persuaded either way. I was simply wondering what the > arguments were. According to int.__doc__, int(..) should *always* return a Python integer. But you are asking for an integer array as a result of int(array(..)); and that would be a contradiction with the Python definition for int(anyobj). Pearu From oliphant at ee.byu.edu Mon Oct 7 08:52:03 2002 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Oct 7 08:52:03 2002 Subject: [Numpy-discussion] int and float ufunc? In-Reply-To: <20021007011036.P41789-100000@mail.allcaps.org> Message-ID: On Mon, 7 Oct 2002, Andrew P. Lentvorski wrote: > On Mon, 7 Oct 2002, Travis Oliphant wrote: > > > > Could someone explain the reason why int() and float() don't work as > > > ufuncs? Is it just that someone needs to write the code, or is there some > > > subtlety at work that I am missing? > > > > What would you have them do? Their current definition works for returning > > suitable arrays as integers and floats respectively, just as the Python > > documentation says these two functions should. > > Okay, now I'm really confused ... > > >>> import Numeric > >>> a = Numeric.array([1.5, 2.5, 3.5], Numeric.Float64) > >>> a > array([ 1.5, 2.5, 3.5]) > >>> int(a) > Traceback (most recent call last): > File "", line 1, in ? > TypeError: Only rank-0 arrays can be converted to Python scalars. > I didn't mean to confuse you. I said "suitable" arrays. Returning a single Python integer from an array doesn't really make sense unless that array is 0-dimensional (rank-0). Now, one could argue that an array of length 1 could also be converted unambiguously, but that discussion has never taken place. Python uses the int() function to return a Python integer type whenever it is explicitly needed. > I could be persuaded either way. I was simply wondering what the > arguments were. > The main argument is that int() should return a Python integer type representation of your object. There is no way to unambiguously return a Python integer object from a general array unless that array contains only one element. Currently the int() function only converts rank-0 arrays (which are Numeric scalars with array headers). > My usage is in converting vectors of data before passing them to the > screen for display. The conversion function is a bunch of ufuncs and then > the resulting vector needs to get converted to int before being passed to > X. The same code works for Python floats, scalars, or Numeric arrays with > the exception of the int function. > I don't understand what the problem is. Why does X.astype() not work for you here? What do you mean "passed to X?" Is this in C, in Python? The int() function was never intended to work on general Numeric arrays. It could be made to (and would be a very simple job), but that discussion has never taken place. instead of int() why don't you use asarray(x).astype('l') for your code > If the problem was simply a lack of someone to write the code, I thought I > might do it. I thought it might be a little faster than me writing array > versions of all of my conversion functions, although I haven't looked at > the complexity of the Numeric code. So, I could be completely wrong about > that. If you look in SciPy (www.scipy.org), we have defined a dictionary of cast functions for each of the types that works the way you apparently wanted int() to work. Using this command you would write scipy.cast[Numeric.Int]() I don't know what your functions are, but I doubt array versions would be that difficult to write. I guess I just always write array versions of my functions. Almost every routine I write starts with asarray(x) to convert inputs to array objects Good luck, -Travis O. From jmiller at stsci.edu Mon Oct 7 08:53:04 2002 From: jmiller at stsci.edu (Todd Miller) Date: Mon Oct 7 08:53:04 2002 Subject: [Numpy-discussion] arange(-10) References: Message-ID: <3DA1AFC3.6080008@stsci.edu> Pearu Peterson wrote: >On Mon, 7 Oct 2002, Todd Miller wrote: > >>The subject line contains what I consider to be an invalid range >>expression. Numarray, now and always, reacts badly to it. Currently, >>numarray tries to allocate a multi-gigabyte array. In the past, it has >>dumped core. >> >>The question is, what should it do? >> >>1. raise ValueError, "Invalid negative range expression" (my +1) >>2. zeros((0,), 'l') >> (Numeric does this) >> >>Is there a good justification to keep the existing Numeric behavior? >> Any other suggestions? >> > >Does Numarray support empty arrays? If yes, I'd vote for Python behavior: > Numarray has no problem with empty arrays so both choices are easy to implement. It apparently has a different repr (than Numeric) for "nothing" which is: array([]) Currently the tally is +2 exception, +1 Numeric-compatible. I'll probably just implement it at COB today, before I forget. Thanks for voting Todd From oliphant at ee.byu.edu Mon Oct 7 08:54:02 2002 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Oct 7 08:54:02 2002 Subject: [Numpy-discussion] arange(-10) In-Reply-To: <3DA17273.3020707@stsci.edu> Message-ID: On Mon, 7 Oct 2002, Todd Miller wrote: > The subject line contains what I consider to be an invalid range > expression. Numarray, now and always, reacts badly to it. Currently, > numarray tries to allocate a multi-gigabyte array. In the past, it has > dumped core. > > The question is, what should it do? > > 1. raise ValueError, "Invalid negative range expression" (my +1) +1 I didn't realize Numeric behaved badly in this area. -Travis From jmiller at stsci.edu Mon Oct 7 11:50:07 2002 From: jmiller at stsci.edu (Todd Miller) Date: Mon Oct 7 11:50:07 2002 Subject: [Numpy-discussion] arange(-10) References: Message-ID: <3DA1D966.2060609@stsci.edu> Travis Oliphant wrote: >On Mon, 7 Oct 2002, Todd Miller wrote: > >>The subject line contains what I consider to be an invalid range >>expression. Numarray, now and always, reacts badly to it. Currently, >>numarray tries to allocate a multi-gigabyte array. In the past, it has >>dumped core. >> >>The question is, what should it do? >> >>1. raise ValueError, "Invalid negative range expression" (my +1) >> >+1 > >I didn't realize Numeric behaved badly in this area. > >-Travis > > I replied to this earlier, but I may have forgotten to reply-all, because I never saw my response. Anyway, I don't think Numeric has a problem with arange(), but numarray does. The vote is currently +3 "raise", +2 "compatible". All votes appreciated Todd From bsder at mail.allcaps.org Mon Oct 7 11:57:10 2002 From: bsder at mail.allcaps.org (Andrew P. Lentvorski) Date: Mon Oct 7 11:57:10 2002 Subject: [Numpy-discussion] int and float ufunc? In-Reply-To: Message-ID: <20021007114938.F42968-100000@mail.allcaps.org> On Mon, 7 Oct 2002, Travis Oliphant wrote: > The main argument is that int() should return a Python integer type > representation of your object. There is no way to unambiguously return a > Python integer object from a general array unless that array contains only > one element. Okay. > I don't understand what the problem is. Why does X.astype() not work for > you here? I guess I wasn't clear here. It does. However, I have *lots* of these functions scattered around the code (inherited from someone else), so I will have to hunt them all down and change all the int(thingtoconvert) to thingtoconvert.astype('l'). Since there are good arguments for int behaving the way it does, I'll make the changes required to my code. No big deal, just lots of tedium. Thanks for taking the time to explain this, -a From kern at caltech.edu Mon Oct 7 13:02:05 2002 From: kern at caltech.edu (Robert Kern) Date: Mon Oct 7 13:02:05 2002 Subject: [Numpy-discussion] int and float ufunc? In-Reply-To: <20021007114938.F42968-100000@mail.allcaps.org> References: <20021007114938.F42968-100000@mail.allcaps.org> Message-ID: <20021007200156.GA9813@taliesen.caltech.edu> On Mon, Oct 07, 2002 at 11:56:48AM -0700, Andrew P. Lentvorski wrote: > However, I have *lots* of these > functions scattered around the code (inherited from someone else), so I > will have to hunt them all down and change all the int(thingtoconvert) to > thingtoconvert.astype('l'). How about defining a function like so: def int(object, **kwds): if type(object) is Numeric.ArrayType: return object.astype(Numeric.Integer) else: return __builtins__.int(object, **kwds) That should probably take care of all uses of int() in your code. -- Robert Kern Ruddock House President kern at caltech.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From fgibbons at hms.harvard.edu Mon Oct 7 13:54:06 2002 From: fgibbons at hms.harvard.edu (Frank Gibbons) Date: Mon Oct 7 13:54:06 2002 Subject: [Numpy-discussion] bug fix for exp() Message-ID: <5.0.0.25.2.20021007163622.00b0ed98@hms.harvard.edu> Hi, I've been lurking for a while, and using Numeric for about 5-6 months. I submitted a bug report about two months ago (https://sourceforge.net/tracker/index.php?func=detail&aid=594761&group_id=1369&atid=101369) regarding incorrect underflow behavior in exp(). I'd really like to get that fixed, nobody else seems to be working on it, and so I checked out the CVS tree. From reading the API section of the manual, I can see that unary ufuncs are driven by PyUFunc_FromFuncAndData() and that exp already has check_return set to 1, so that it should clean up rank-0 arrays, and raise the appropriate Python exception when 'errno' is set by a math error. I'd like to take a closer look into things, and one way of doing that would be to use a unit-test suite, but it's not obvious where that is (if at all) in the source distribution. Is there one? Another way to look into it would be if I could write some C code, then drop into Numeric's code and track what's going on when I call exp() for arguments that underflow. Is there a harness out there that I could just adapt to my own needs. Surely that's how other developers do it. If not, how *do* you fix bugs? I've already added some features to JNumeric, and introduced a unit-test suite there. I have so few problems using Numeric, I'd really like to nail this one involving exp(), since it's holding me back. Thanks, -Frank Gibbons PhD, Computational Biologist, Harvard Medical School BCMP/SGM-322, 250 Longwood Ave, Boston MA 02115, USA. Tel: 617-432-3555 Fax: 617-432-3557 http://llama.med.harvard.edu/~fgibbons From pearu at cens.ioc.ee Mon Oct 7 14:42:09 2002 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Mon Oct 7 14:42:09 2002 Subject: [Numpy-discussion] int and float ufunc? In-Reply-To: <20021007200156.GA9813@taliesen.caltech.edu> Message-ID: On Mon, 7 Oct 2002, Robert Kern wrote: > On Mon, Oct 07, 2002 at 11:56:48AM -0700, Andrew P. Lentvorski wrote: > > However, I have *lots* of these > > functions scattered around the code (inherited from someone else), so I > > will have to hunt them all down and change all the int(thingtoconvert) to > > thingtoconvert.astype('l'). > > How about defining a function like so: > > def int(object, **kwds): > if type(object) is Numeric.ArrayType: > return object.astype(Numeric.Integer) > else: > return __builtins__.int(object, **kwds) > > That should probably take care of all uses of int() in your code. I would rename this function to `aint' and still change all the int(x) to aint(x). When messing with Python built-ins, sooner or later you'll find a bullet in your leg ... Pearu From oliphant at ee.byu.edu Mon Oct 7 16:49:09 2002 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Oct 7 16:49:09 2002 Subject: [Numpy-discussion] bug fix for exp() In-Reply-To: <5.0.0.25.2.20021007163622.00b0ed98@hms.harvard.edu> Message-ID: > Hi, > > I've been lurking for a while, and using Numeric for about 5-6 months. I > submitted a bug report about two months ago > (https://sourceforge.net/tracker/index.php?func=detail&aid=594761&group_id=1369&atid=101369) > regarding incorrect underflow behavior in exp(). My understanding is that this bug went away with a new version of Python. It came about due to some change in Python and has not gone away. My advice is to start using scipy (but of course that's my advice because that's what I do). Or at least start using scipy_base (which is easier to install). SciPy handles NAN's and INFS instead of raising errors and this particular "bug" is not present because the error is not checked for. > Another way to look into it would be if I could write some C code, then > drop into Numeric's code and track what's going on when I call exp() for > arguments that underflow. Is there a harness out there that I could just > adapt to my own needs. Surely that's how other developers do it. If not, > how *do* you fix bugs? What's going on is that the system library for exp() is raising an underflow error and rather than ignore it and return 0.0 like a good exp() function, the error propagates to the top. It's an annoying bug, I agree. I've heard that it disappears with a new version of Python (2.2) but, I'm not entirely sure because I use scipy_base extensively. -Travis O. From peterson at math.utwente.nl Tue Oct 8 03:40:02 2002 From: peterson at math.utwente.nl (Pearu Peterson) Date: Tue Oct 8 03:40:02 2002 Subject: [Numpy-discussion] arange(-10) In-Reply-To: <3DA1AFC3.6080008@stsci.edu> Message-ID: On Mon, 7 Oct 2002, Todd Miller wrote: > Pearu Peterson wrote: > > >On Mon, 7 Oct 2002, Todd Miller wrote: > > > >>The subject line contains what I consider to be an invalid range > >>expression. Numarray, now and always, reacts badly to it. Currently, > >>numarray tries to allocate a multi-gigabyte array. In the past, it has > >>dumped core. > >> > >>The question is, what should it do? > >> > >>1. raise ValueError, "Invalid negative range expression" (my +1) > >>2. zeros((0,), 'l') > >> (Numeric does this) > >> > >>Is there a good justification to keep the existing Numeric behavior? > >> Any other suggestions? > >> > > > >Does Numarray support empty arrays? If yes, I'd vote for Python behavior: > > > Numarray has no problem with empty arrays so both choices are easy to > implement. It apparently has a different repr (than Numeric) for > "nothing" which is: > > array([]) Hmm, so it means that Numeric.arange is consistent with Python range after all. Then I'd like to change my vote from exception to array([]) (if it is not too late). Rationale: sometimes using range(i1) + range(i2) + range(i3) can save lots of code if the indices i1,i2,i3 can also be negative with the meaning that the resulting lists are then empty. If using arange, then the above would be equivalent to concatenate((arange(i1),arange(i2),arange(i3))) or concatenate(map(arange,[i1,i2,i3])) for short. Now, if arange would raise an exception if one of i1,i2,i3 is negative, then the above would not work without checking i1,i2,i3 first. And I find that bad because additional (read: meaningless) code is needed. Pearu From jmiller at stsci.edu Tue Oct 8 05:07:04 2002 From: jmiller at stsci.edu (Todd Miller) Date: Tue Oct 8 05:07:04 2002 Subject: [Numpy-discussion] arange(-10) References: Message-ID: <3DA2CA45.4060004@stsci.edu> Pearu Peterson wrote: > >On Mon, 7 Oct 2002, Todd Miller wrote: > >>Pearu Peterson wrote: >> >>>On Mon, 7 Oct 2002, Todd Miller wrote: >>> >>>>The subject line contains what I consider to be an invalid range >>>>expression. Numarray, now and always, reacts badly to it. Currently, >>>>numarray tries to allocate a multi-gigabyte array. In the past, it has >>>>dumped core. >>>> >>>>The question is, what should it do? >>>> >>>>1. raise ValueError, "Invalid negative range expression" (my +1) >>>>2. zeros((0,), 'l') >>>> (Numeric does this) >>>> >>>>Is there a good justification to keep the existing Numeric behavior? >>>>Any other suggestions? >>>> >>>Does Numarray support empty arrays? If yes, I'd vote for Python behavior: >>> >>Numarray has no problem with empty arrays so both choices are easy to >>implement. It apparently has a different repr (than Numeric) for >>"nothing" which is: >> >>array([]) >> > >Hmm, so it means that Numeric.arange is consistent with Python range after >all. Then I'd like to change my vote from exception to array([]) (if it >is not too late). > >Rationale: sometimes using > > range(i1) + range(i2) + range(i3) > >can save lots of code if the indices i1,i2,i3 can also be negative with >the meaning that the resulting lists are then empty. If using arange, then >the above would be equivalent to > > concatenate((arange(i1),arange(i2),arange(i3))) > >or > > concatenate(map(arange,[i1,i2,i3])) > >for short. Now, if arange would raise an exception if one of i1,i2,i3 is >negative, then the above would not work without checking i1,i2,i3 first. >And I find that bad because additional (read: meaningless) code is >needed. > >Pearu > > Hi Pearu, Well, as fate would have it, your vote was already mis-cast in favor of Numeric compatability. However, you've changed my mind, so I'm changing *my* vote. That leaves the current vote at +2 "raise", +3 "compatible", so barring any new votes, I'll check it in as Numeric compatible (returning an empty range) at COB today. Todd -- Todd Miller jmiller at stsci.edu STSCI / SSB From paustin at eos.ubc.ca Tue Oct 8 07:34:03 2002 From: paustin at eos.ubc.ca (Philip Austin) Date: Tue Oct 8 07:34:03 2002 Subject: [Numpy-discussion] libnumarray.h and NO_IMPORT_ARRAY Message-ID: <15778.60563.584317.60674@gull.eos.ubc.ca> We're currently porting several of our boost-wrapped Numeric classes to numarray. The routines are spread across several cpp files, and **libnumarray_API is needed in each file. In Numeric, we invoke import_array() in the module file, to initialize **PyArray_API, and def NO_IMPORT_ARRAY in the other files; this produces extern void **PyArray_API in those files, and everyone's happy. Currently libnumarray.h doesn't implement this -- is there another way to compile a multiple file project which initializes the API just once? Thanks, Phil Austin From jmiller at stsci.edu Tue Oct 8 07:59:04 2002 From: jmiller at stsci.edu (Todd Miller) Date: Tue Oct 8 07:59:04 2002 Subject: [Numpy-discussion] libnumarray.h and NO_IMPORT_ARRAY References: <15778.60563.584317.60674@gull.eos.ubc.ca> Message-ID: <3DA2F27C.80707@stsci.edu> Philip Austin wrote: >We're currently porting several of our boost-wrapped Numeric classes >to numarray. The routines are spread across several cpp files, and >**libnumarray_API is needed in each file. In Numeric, we invoke >import_array() in the module file, to initialize **PyArray_API, and >def NO_IMPORT_ARRAY in the other files; this produces >extern void **PyArray_API in those files, and everyone's happy. > >Currently libnumarray.h doesn't implement this -- is there >another way to compile a multiple file project which >initializes the API just once? > >Thanks, Phil Austin > > > >------------------------------------------------------- >This sf.net email is sponsored by:ThinkGeek >Welcome to geek heaven. >http://thinkgeek.com/sf >_______________________________________________ >Numpy-discussion mailing list >Numpy-discussion at lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/numpy-discussion > Someone else asked for this recently, and the solution is half-way implemented in CVS now. I'll finish it this morning so that it works the way you're used to, but the next "official" numarray release is still a ways off. So, if you still want to port now, here are the caveats: 1. Numarray CVS was working as of Friday on Windows, Linux, Solaris, and Tru64. 2. Use the "Numeric compatible" portion of the C-API. This API is a subset of the Numeric API which may cause you trouble. Let me know if it does. Don't use the "native" NDInfo based APIs because they're headed for deprecation. 3. The version you'll be working with has not yet been widely used and may have significant bugs I don't know about. It does pass all its unit tests on all our platforms. Todd -- Todd Miller jmiller at stsci.edu STSCI / SSB From jmiller at stsci.edu Tue Oct 8 08:28:06 2002 From: jmiller at stsci.edu (Todd Miller) Date: Tue Oct 8 08:28:06 2002 Subject: [Numpy-discussion] libnumarray.h and NO_IMPORT_ARRAY References: <15778.60563.584317.60674@gull.eos.ubc.ca> Message-ID: <3DA2F969.60300@stsci.edu> Philip Austin wrote: > >Currently libnumarray.h doesn't implement this -- is there >another way to compile a multiple file project which >initializes the API just once? > Hopefully you got my earlier message about this. Since I CC'ed numpy-discussion and never saw it there, I've included it below. But, assuming you saw it, I think what you need is in CVS now. Todd -- Todd Miller jmiller at stsci.edu STSCI / SSB My earlier response: Someone else asked for this recently, and the solution is half-way implemented in CVS now. I'll finish it this morning so that it works the way you're used to, but the next "official" numarray release is still a ways off. So, if you still want to port now, here are the caveats: 1. Numarray CVS was working as of Friday on Windows, Linux, Solaris, and Tru64. 2. Use the "Numeric compatible" portion of the C-API. This API is a subset of the Numeric API which may cause you trouble. Let me know if it does. Don't use the "native" NDInfo based APIs because they're headed for deprecation. 3. The version you'll be working with has not yet been widely used and may have significant bugs I don't know about. It does pass all its unit tests on all our platforms. From paustin at eos.ubc.ca Wed Oct 9 11:04:05 2002 From: paustin at eos.ubc.ca (Philip Austin) Date: Wed Oct 9 11:04:05 2002 Subject: [Numpy-discussion] libnumarray.h and NO_IMPORT_ARRAY In-Reply-To: <3DA2F969.60300@stsci.edu> References: <15778.60563.584317.60674@gull.eos.ubc.ca> <3DA2F969.60300@stsci.edu> Message-ID: <15780.28521.612331.840142@gull.eos.ubc.ca> Todd Miller writes: > Philip Austin wrote: > > > > >Currently libnumarray.h doesn't implement this -- is there > >another way to compile a multiple file project which > >initializes the API just once? > > > Hopefully you got my earlier message about this. Since I CC'ed > numpy-discussion and never saw it there, I've included it below. But, > assuming you saw it, I think what you need is in CVS now. Todd, thanks for the quick response. Regarding your header comments: /* Extensions constructed from seperate compilation units can access the C-API defined here by defining "libnumarray_UNIQUE_SYMBOL" to a global name unique to the extension. Doing this circumvents the requirement to import libnumarray into each compilation unit, but is nevertheless mildly discouraged as "outside the Python norm" and potentially leading to problems. Looking around at "existing Python art", most extension modules are monolithic C files, and likely for good reason. */ I'd agree with this sentiment, but with numarray's NDInfo interface I think we were stuck (we have an implementation file for a bunch of boost numarray helper functions that require the API but have static data initializations, so they can't go into a header). With Numeric, we could avoid import_array altogether, and get everything we needed from PyObject_CallObject calls to Numeric to create arrays using zeros or ones, followed by a cast to PyArrayObject* to get the data pointer and fill the array. With numarray, we need (or needed) getNDInfo to do the same thing, mandating an import_libnumarray. Are the plans for the revised interface to behave in the same way as NDInfo? It's not a big deal to us one way or another, but it would simplify Dave Abraham's boost support for numeric/numarray if much of the functionality could be done through python calls from the C side. Thanks, Phil From jmiller at stsci.edu Wed Oct 9 11:48:02 2002 From: jmiller at stsci.edu (Todd Miller) Date: Wed Oct 9 11:48:02 2002 Subject: [Numpy-discussion] libnumarray.h and NO_IMPORT_ARRAY References: <15778.60563.584317.60674@gull.eos.ubc.ca> <3DA2F969.60300@stsci.edu> <15780.28521.612331.840142@gull.eos.ubc.ca> Message-ID: <3DA479A7.6010409@stsci.edu> Philip Austin wrote: > >Todd, thanks for the quick response. Regarding your header comments: > >/* >Extensions constructed from seperate compilation units can access the >C-API defined here by defining "libnumarray_UNIQUE_SYMBOL" to a global >name unique to the extension. Doing this circumvents the requirement >to import libnumarray into each compilation unit, but is nevertheless >mildly discouraged as "outside the Python norm" and potentially >leading to problems. Looking around at "existing Python art", most >extension modules are monolithic C files, and likely for good reason. >*/ > > >I'd agree with this sentiment, but with numarray's NDInfo interface I >think we were stuck (we have an implementation file for a bunch of >boost numarray helper functions that require the API but have static >data initializations, so they can't go into a header). > Long term, you would be better off ditching NDInfo and working with the Numeric compatability API. Since it's a subset of the Numeric API, it may not support your purposes. However, if it doesn't, I want to extend it until it does. > >With Numeric, we could avoid import_array altogether, and get >everything we needed from PyObject_CallObject calls to Numeric to >create arrays using zeros or ones, followed by a cast to >PyArrayObject* to get the data pointer and fill the array. With >numarray, we need (or needed) getNDInfo to do the same thing, >mandating an import_libnumarray. Are the plans for the revised >interface to behave in the same way as NDInfo? > The current plan is to deprecate the "native" APIs which make use of NDInfo and to make numarray as compatible as possible with Numeric. The bulk of the changes which make this reasonable are in numarray CVS now. In a nutshell, prior to our use of C basetypes, we needed something like NDInfo. Now that we're using C basetypes, NDInfo appears less efficient, less compatible, and more complex than the "Numeric compatability" interface. Its possible for our "basetypes" to appear to be Numeric arrays, so that's what we did. If you guys find that you need missing parts of the Numeric C-API, I'll add them to numarray. So far, I've just added what I needed or knew someone else needed. > It's not a big deal to >us one way or another, but it would simplify Dave Abraham's boost >support for numeric/numarray if much of the functionality could be >done through python calls from the C side. > Can you send me the Numeric wrapper code or let me know where to get it? > >Thanks, Phil > > > >------------------------------------------------------- >This sf.net email is sponsored by:ThinkGeek >Welcome to geek heaven. >http://thinkgeek.com/sf >_______________________________________________ >Numpy-discussion mailing list >Numpy-discussion at lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/numpy-discussion > Thanks, Todd -- Todd Miller jmiller at stsci.edu STSCI / SSB From paustin at eos.ubc.ca Wed Oct 9 18:49:03 2002 From: paustin at eos.ubc.ca (Philip Austin) Date: Wed Oct 9 18:49:03 2002 Subject: [Numpy-discussion] libnumarray.h and NO_IMPORT_ARRAY In-Reply-To: <3DA479A7.6010409@stsci.edu> References: <15778.60563.584317.60674@gull.eos.ubc.ca> <3DA2F969.60300@stsci.edu> <15780.28521.612331.840142@gull.eos.ubc.ca> <3DA479A7.6010409@stsci.edu> Message-ID: <15780.56445.730334.591361@gull.eos.ubc.ca> Todd Miller writes: > Can you send me the Numeric wrapper code or let me know where to > get it? It's now on the cvs main trunk: cvs -d:pserver:anonymous at cvs.boost.sourceforge.net:/cvsroot/boost login cvs -z3 -d:pserver:anonymous at cvs.boost.sourceforge.net:/cvsroot/boost checkout boost You'll need all of boost if you want to try to compile/run the test code. If you haven't worked with boost and boost.build (bjam) before, I could send my install notes. If you'd just like to browse the code, the pertinent files are: boost/libs/python/test/numpy.cpp boost/libs/python/test/numpy.py boost/python/numeric.hpp and the new tutorial (which doesn't cover numpy yet) is quite good: boost/boost/libs/python/doc/tutorial as well as: boost/libs/python/doc/v2/reference.html (also without numpy documentation) Regards, Phil From jmiller at stsci.edu Thu Oct 10 06:26:05 2002 From: jmiller at stsci.edu (Todd Miller) Date: Thu Oct 10 06:26:05 2002 Subject: [Numpy-discussion] libnumarray.h and NO_IMPORT_ARRAY References: <15778.60563.584317.60674@gull.eos.ubc.ca> <3DA2F969.60300@stsci.edu> <15780.28521.612331.840142@gull.eos.ubc.ca> <3DA479A7.6010409@stsci.edu> <15780.56445.730334.591361@gull.eos.ubc.ca> Message-ID: <3DA57FC4.9030208@stsci.edu> Philip Austin wrote: >Todd Miller writes: > > Can you send me the Numeric wrapper code or let me know where to > > get it? > >It's now on the cvs main trunk: > >cvs -d:pserver:anonymous at cvs.boost.sourceforge.net:/cvsroot/boost login >cvs -z3 -d:pserver:anonymous at cvs.boost.sourceforge.net:/cvsroot/boost checkout boost > >You'll need all of boost if you want to try to compile/run the test >code. If you haven't worked with boost and boost.build (bjam) >before, I could send my install notes. If you'd just like to >browse the code, the pertinent files are: > >boost/libs/python/test/numpy.cpp >boost/libs/python/test/numpy.py >boost/python/numeric.hpp > >and the new tutorial (which doesn't cover numpy yet) is >quite good: > >boost/boost/libs/python/doc/tutorial > >as well as: > >boost/libs/python/doc/v2/reference.html >(also without numpy documentation) > >Regards, Phil > I took a look at the files above trying to see what you're doing. What I thought I would see, was some form of direct coupling between a module in boost and either the Numeric or numarray C-API. However, when I search the whole source tree (*.cpp *.hpp) I can find neither PyArrayObject, arrayobject.h, NDInfo, nor libnumarray.h. So I'm not sure what part of any Numeric/numarray C-API you're using now (if any) and in particular, what isn't working for you. Note: numarray and Numeric are not totally compatible at either the C or Python levels, so either area could have been a source of difficulty. Todd -- Todd Miller jmiller at stsci.edu STSCI / SSB From paustin at eos.ubc.ca Thu Oct 10 09:47:07 2002 From: paustin at eos.ubc.ca (Philip Austin) Date: Thu Oct 10 09:47:07 2002 Subject: [Numpy-discussion] libnumarray.h and NO_IMPORT_ARRAY In-Reply-To: <3DA57FC4.9030208@stsci.edu> References: <15778.60563.584317.60674@gull.eos.ubc.ca> <3DA2F969.60300@stsci.edu> <15780.28521.612331.840142@gull.eos.ubc.ca> <3DA479A7.6010409@stsci.edu> <15780.56445.730334.591361@gull.eos.ubc.ca> <3DA57FC4.9030208@stsci.edu> Message-ID: <15781.44767.831692.188152@gull.eos.ubc.ca> > Philip Austin wrote: [snip] > > > >boost/libs/python/test/numpy.cpp > >boost/libs/python/test/numpy.py > >boost/python/numeric.hpp > > Todd Miller writes: > I took a look at the files above trying to see what you're doing. (note that all of this was written by Dave Abrahams, I'm just one of the first Numeric users.) > What I thought I would see, was some form of direct coupling > between a module in boost and either the Numeric or numarray C-API. > However, when I search the whole source tree (*.cpp *.hpp) I can > find neither PyArrayObject, arrayobject.h, NDInfo, nor > libnumarray.h. So I'm not sure what part of any Numeric/numarray > C-API you're using now (if any) The answer is -- no part of the C-API if he can get away with it. Your move towards Numeric compatibility for numarray means that he now can get away with it. Specifically, the extension tries to import the numarray module, and if that fails, imports Numeric (the user can change this with numeric::set_module_and_type("Numeric", "ArrayType"); or: numeric::set_module_and_type("numarray", "NDArray"); ) numeric wraps the module pointer, and calls functions through it -- e.g. (from boost/libs/python/src/numeric.cpp): void array_base::resize(object const& shape) { attr("resize")(shape); } which invokes PyObject_CallObject to call the array's resize method, with the tuple shape (similar to CXX). If the method isn't part of the library (say for Numeric and getflat) then a python exception is thrown. > and in particular, what isn't > working for you. Note: numarray and Numeric are not totally > compatible at either the C or Python levels, so either area could > have been a source of difficulty. The problem was that, as long as a call to getNDInfo was required to access an array's data pointer, a Python-only approach like the above wouldn't work. Since it is now possible to do this via a cast to PyArrayObject* for a numarray, the problem has disappeared. (there's nothing stopping you from using the C-API if you want, though. For example you can create a numarray with data copied from a C array with something like: numeric::array makeNum(int * data, int n=0){ object obj(detail::new_reference(PyArray_FromDims(1, &n, PyArray_INT))); char *arr_data = ((PyArrayObject*) obj.ptr())->data; memcpy(arr_data, data, 4 * n); return numeric::array(obj); } ) Regards, Phil From dave at boost-consulting.com Thu Oct 10 10:29:05 2002 From: dave at boost-consulting.com (David Abrahams) Date: Thu Oct 10 10:29:05 2002 Subject: [Numpy-discussion] libnumarray.h and NO_IMPORT_ARRAY In-Reply-To: <15781.44767.831692.188152@gull.eos.ubc.ca> References: <15778.60563.584317.60674@gull.eos.ubc.ca> <3DA2F969.60300@stsci.edu> <15780.28521.612331.840142@gull.eos.ubc.ca> <3DA479A7.6010409@stsci.edu> <15780.56445.730334.591361@gull.eos.ubc.ca> <3DA57FC4.9030208@stsci.edu> <15781.44767.831692.188152@gull.eos.ubc.ca> Message-ID: Philip Austin writes: > > What I thought I would see, was some form of direct coupling > > between a module in boost and either the Numeric or numarray C-API. > > However, when I search the whole source tree (*.cpp *.hpp) I can > > find neither PyArrayObject, arrayobject.h, NDInfo, nor > > libnumarray.h. So I'm not sure what part of any Numeric/numarray > > C-API you're using now (if any) > > The answer is -- no part of the C-API if he can get away with it. > Your move towards Numeric compatibility for numarray means that he > now can get away with it. I've been getting away with it already. It's easy enough to access all that functionality through the usual Python/C API. > numeric wraps the module pointer, and calls functions > through it -- e.g. (from boost/libs/python/src/numeric.cpp): > > void array_base::resize(object const& shape) > { > attr("resize")(shape); > } > > which invokes PyObject_CallObject to call the array's resize method, > with the tuple shape (similar to CXX). If the method isn't part of the > library (say for Numeric and getflat) then a python exception is > thrown. Slow but effective. If you want to touch the raw PyObject*, of course, you can do this: void f(numeric::array x) { PyObject* danger = x.ptr(); // Cast to whatever you like. } > > and in particular, what isn't > > working for you. Note: numarray and Numeric are not totally > > compatible at either the C or Python levels, so either area could > > have been a source of difficulty. > > The problem was that, as long as a call to getNDInfo was required > to access an array's data pointer, a Python-only approach > like the above wouldn't work. Since it is now possible to > do this via a cast to PyArrayObject* for a numarray, the problem > has disappeared. Not sure what you're saying here. I /do/ remember when I was writing the Numeric/NumArray support that there seemed no publicly-accessible definition of the structure of the underlying array objects. > (there's nothing stopping you from using the C-API if you want, > though. For example you can create a numarray with data > copied from a C array with something like: > > numeric::array makeNum(int * data, int n=0){ > object obj(detail::new_reference(PyArray_FromDims(1, &n, PyArray_INT))); > char *arr_data = ((PyArrayObject*) obj.ptr())->data; > memcpy(arr_data, data, 4 * n); > return numeric::array(obj); > } Well, this uses undocumented implementation details. The supported approach is: handle<> raw_array(PyArray_FromDims(1, &n, PyArray_INT)); object obj(raw_array); Note, though, that you can also write: handle raw_array(PyArray_FromDims(1, &n, PyArray_INT)); Assuming, of course, that PyArray_FromDims returns a PyArrayObject*. And now you have access to the PyArrayObject* through raw_array. Also note that it's probably smarter to use extract(obj) Than numeric::array(obj) See the note about pitfalls at http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/*checkout*/boost/boost/libs/python/doc/tutorial/doc/derived_object_types.html for the reasons why. -- David Abrahams * Boost Consulting dave at boost-consulting.com * http://www.boost-consulting.com From paustin at eos.ubc.ca Thu Oct 10 11:14:11 2002 From: paustin at eos.ubc.ca (Philip Austin) Date: Thu Oct 10 11:14:11 2002 Subject: [Numpy-discussion] libnumarray.h and NO_IMPORT_ARRAY In-Reply-To: References: <15778.60563.584317.60674@gull.eos.ubc.ca> <3DA2F969.60300@stsci.edu> <15780.28521.612331.840142@gull.eos.ubc.ca> <3DA479A7.6010409@stsci.edu> <15780.56445.730334.591361@gull.eos.ubc.ca> <3DA57FC4.9030208@stsci.edu> <15781.44767.831692.188152@gull.eos.ubc.ca> Message-ID: <15781.49982.342376.307126@gull.eos.ubc.ca> David Abrahams writes: > > Not sure what you're saying here. I /do/ remember when I was writing > the Numeric/NumArray support that there seemed no publicly-accessible > definition of the structure of the underlying array objects. As I understand it, the numarray now in CVS deprecates the NDInfo struct (see section 10.1 of numarray.pdf) in favor of PyArrayObject. [snip] > See the note about pitfalls at > http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/*checkout*/boost/boost/libs/python/doc/tutorial/doc/derived_object_types.html > for the reasons why. Thanks for the info -- Phil From jmiller at stsci.edu Thu Oct 10 12:45:01 2002 From: jmiller at stsci.edu (Todd Miller) Date: Thu Oct 10 12:45:01 2002 Subject: [Numpy-discussion] libnumarray.h and NO_IMPORT_ARRAY References: <15778.60563.584317.60674@gull.eos.ubc.ca> <3DA2F969.60300@stsci.edu> <15780.28521.612331.840142@gull.eos.ubc.ca> <3DA479A7.6010409@stsci.edu> <15780.56445.730334.591361@gull.eos.ubc.ca> <3DA57FC4.9030208@stsci.edu> <15781.44767.831692.188152@gull.eos.ubc.ca> <15781.49982.342376.307126@gull.eos.ubc.ca> Message-ID: <3DA5D86F.7030204@stsci.edu> Philip Austin wrote: >David Abrahams writes: > > > > Not sure what you're saying here. I /do/ remember when I was writing > > the Numeric/NumArray support that there seemed no publicly-accessible > > definition of the structure of the underlying array objects. > Numeric and numarray both have "arrayobject.h" which makes PyArrayObject public. Numarray also has "libnumarray.h" which indirectly exposes NDInfo. > >As I understand it, the numarray now in CVS deprecates the NDInfo >struct (see section 10.1 of numarray.pdf) in favor of PyArrayObject. > This is true, but is currently neither released nor documented. On the plus side, numarray is becoming more compatible with Numeric, getting faster, and simpler as well. Todd -- Todd Miller jmiller at stsci.edu STSCI / SSB From karthik at james.hut.fi Fri Oct 11 10:06:07 2002 From: karthik at james.hut.fi (Karthikesh Raju) Date: Fri Oct 11 10:06:07 2002 Subject: [Numpy-discussion] Issues about class initialialization in with large numerical data Message-ID: Hi All, i am writing a class called communicate. Something like this: class Communicate: """ This is the main communicate class : there will be more """ def __init__(self): """This should hopefully read a file and set self's parameters """ self.C = 10 self.K = 6 self.N = 8 self.Na = 3 self.L = 4 self.modulation = "QPSK" 1. Shown is just one method in the class. Now there are lot of self variables (i understand they are something like private variables). Now i want to have an .ini file, so that i call something like this in the __init__ module : def __init__(self, fileName): file = open(fileName, 'r') self = something(file) and all the self's variables are set. i want the ini file to be the interface between the class and the outside world. Some one suggested ConfigParser , couldnt get an idea of what it is and how to use it.. 2. Is this method of having all the data in self variables good.The size of some of these 'self' could be very high 40000x40000 complex. Should i create local variables in the methods (i need a lot of variables, but their def life is too small .....) for example one of the method could be: def generateNoise(self): """ Generates AWGN """ power = Numeric.sqrt(RandomArray.random()*10) Noise = Numeric.zeros(....) Noise.real = Random ...... Noise.imag = RandomArray.... (Boared to type the whole thing that is why there are ......'s ) Any way power is a local variable, which i guess dies after generateNoise() method is called in the object. Am i right.. Can i have variables like 'power' 3. Thirdly, i want an easy method like '.mat' that exists in matlab. Can i use pickle (cPickle or Numeric.pickle) to store all the variables that i want. These matrices could be 40000x40000 entries long. Assuming that all the variables are in 'self' should i write an assignment module before pickling them (in case pickle is the right way to store these variables) def writeToFile(self): Na = self.Na x = self.x pickle the variables Na, x Thankx in advance and in anticipation, karthik PS: slowly moving from Matlab to Python, hence these stupid doubts ----------------------------------------------------------------------- Karthikesh Raju, email: karthik at james.hut.fi Researcher, http://www.cis.hut.fi/karthik Helsinki University of Technology, Tel: +358-9-451 5389 Laboratory of Comp. & Info. Sc., Fax: +358-9-451 3277 Department of Computer Sc., P.O Box 5400, FIN 02015 HUT, Espoo, FINLAND ----------------------------------------------------------------------- From rob at pythonemproject.com Sat Oct 12 15:16:03 2002 From: rob at pythonemproject.com (Rob) Date: Sat Oct 12 15:16:03 2002 Subject: [Numpy-discussion] Dyadic Python Message-ID: <3DA89E75.51168D7F@pythonemproject.com> As a help for me learning Dyadic vector analysis, I have been working on a Numpy class for manipulating Dyads. I am wondering if anyone else has any similar interests. Most of the impetus for the work comes from "Methods for Electromagnetic Field Analysis" by Ismo Lindell. In numerous searches I found no computer code dealing with Dyads. My biggest stumbling block so far is integrating the symbolism of Dyadic analysis into something understandable. Operator overloading isn't going to help me. For example in my program: A=dyad(a,b) where a and b are complex vectors B=dyad(c,d) " A.dmmd(B) is equivalent to double dyadic multiplication of the dyads A and B. :) In a book this would be written something like A xx B except that the x's would be aligned vertically. Ok, its an insane project. Rob. -- ----------------------------- The Numeric Python EM Project www.pythonemproject.com From paul at pfdubois.com Sat Oct 12 21:33:05 2002 From: paul at pfdubois.com (Paul F Dubois) Date: Sat Oct 12 21:33:05 2002 Subject: [Numpy-discussion] Dyadic Python In-Reply-To: <3DA89E75.51168D7F@pythonemproject.com> Message-ID: <000601c27271$86e6d1a0$6701a8c0@NICKLEBY> What took me a long while to appreciate is that inheriting from Numeric wasn't going to do me any good precisely because any operation such as a+b was going to produce an array not one of my new foos, unless I overrode it. So in fact you end up having to override darn near everything. I can only suggest MA as an example. People have succeeded in similar tasks by starting with the MA source and changing it to suit. > -----Original Message----- > From: numpy-discussion-admin at lists.sourceforge.net > [mailto:numpy-discussion-admin at lists.sourceforge.net] On Behalf Of Rob > Sent: Saturday, October 12, 2002 3:13 PM > To: numpy-discussion at lists.sourceforge.net > Subject: [Numpy-discussion] Dyadic Python > > > As a help for me learning Dyadic vector analysis, I have been > working on a Numpy class for manipulating Dyads. I am > wondering if anyone else has any similar interests. Most of > the impetus for the work comes from "Methods for > Electromagnetic Field Analysis" by Ismo Lindell. In numerous > searches I found no computer code dealing with Dyads. > > My biggest stumbling block so far is integrating the > symbolism of Dyadic analysis into something understandable. > Operator overloading isn't > going to help me. For example in my program: > > A=dyad(a,b) where a and b are complex vectors > B=dyad(c,d) " > A.dmmd(B) is equivalent to double dyadic multiplication of the dyads A > and B. :) In a book this would be written something like A xx B > except that the x's would be aligned vertically. > > Ok, its an insane project. Rob. > > > -- > ----------------------------- > The Numeric Python EM Project > www.pythonemproject.com ------------------------------------------------------- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf _______________________________________________ Numpy-discussion mailing list Numpy-discussion at lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion From retype at terra.com.br Tue Oct 15 12:24:02 2002 From: retype at terra.com.br (Leonardo Santagada) Date: Tue Oct 15 12:24:02 2002 Subject: [Numpy-discussion] Re: Numpy-discussion -- confirmation of subscription -- request 961098 In-Reply-To: References: Message-ID: <20021015162035.75591153.retype@terra.com.br> On Tue, 15 Oct 2002 12:22:27 -0700 numpy-discussion-request at lists.sourceforge.net wrote: > Numpy-discussion -- confirmation of subscription -- request 961098 > > We have received a request from 200.176.39.98 for subscription of your > email address, , to the > numpy-discussion at lists.sourceforge.net mailing list. To confirm the > request, please send a message to > numpy-discussion-request at lists.sourceforge.net, and either: > > - maintain the subject line as is (the reply's additional "Re:" is > ok), > > - or include the following line - and only the following line - in the > message body: > > confirm 961098 > > (Simply sending a 'reply' to this message should work from most email > interfaces, since that usually leaves the subject line in the right > form.) > > If you do not wish to subscribe to this list, please simply disregard > this message. Send questions to > numpy-discussion-admin at lists.sourceforge.net. > > Esta mensagem foi verificada pelo E-mail Protegido Terra. > Scan engine: VirusScan / Atualizado em 09/10/2002 / Vers?o: 1.3.13 > Proteja o seu e-mail Terra: http://www.emailprotegido.terra.com.br/ > From blackfield1 at llnl.gov Wed Oct 16 13:33:08 2002 From: blackfield1 at llnl.gov (Don Blackfield) Date: Wed Oct 16 13:33:08 2002 Subject: [Numpy-discussion] Fwd: RE: Problem with pyfort Message-ID: <5.0.0.25.2.20021016132723.028c04c0@poptop.llnl.gov> Hello, I sent the following email to Paul Dubois. Since he is not familiar with MS WINDOWS, he suggested that I send my problem to this list. In short, I am unable to get the pyfdemo to work in pyfort_8_.2 using python2.2 because my FORTRAN compiler, COMPAQ/DEC Visual FORTRAN uses NMAKE.EXE rather than make. When I try to run the demo, I get an error message saying that make is not recognized. What file or file(s) should I change. I cannot run under LINUX, it must be WINDOWS2000. Thanks for your assistance, Don >From: paul at pfdubois.com >Date: Wed, 16 Oct 2002 13:26:27 -0700 >Subject: RE: Problem with pyfort >To: "Don Blackfield" > >There is a subdirectory of the source that is devoted to how to do things >on Microsoft. I don't have any experience of it myself. You can try the >numpy-discussion at lists.sf.net list if you want to contact other users. > > >-- Original Message -- > >Date: Wed, 16 Oct 2002 12:10:54 -0700 > >To: "Paul F Dubois" > >From: Don Blackfield > >Subject: Problem with pyfort > > > > > > Paul, > > > > You may be cursing Microsoft by now, I know that I am. I have > >"successfully" installed pyfort. However, > >when I try to run the pyfdemo script by typing > > > >pyfort.bat -i pyfdemo > > > >it begins to work. However it crashes because WINDOWS does not recognize > > > >make. The COMPAQ/DEC/Visual > >Fortran uses a similar routine called NMAKE.EXE . Is there some place in > > > >some file(s) where I need to change > >make to NMAKE.EXE ?? > > > > Sorry for the hassle..... > > > >Don > > ============================================== Donald T. Blackfield Fusion Energy Program New Technologies Engineering Division Lawrence Livermore National Laboratory P.O. Box 808, MailStop L-641 Livermore, CA 94550 (925)422-0496 / (925)422-7390 (fax) dtb at llnl.gov ============================================== From nobody at maui.dnsvault.com Tue Oct 29 17:34:02 2002 From: nobody at maui.dnsvault.com (Nobody) Date: Tue Oct 29 17:34:02 2002 Subject: [Numpy-discussion] A larger gold balance! Message-ID: An HTML attachment was scrubbed... URL: From jdhunter at ace.bsd.uchicago.edu Wed Oct 2 11:36:13 2002 From: jdhunter at ace.bsd.uchicago.edu (John Hunter) Date: Wed Oct 2 11:36:13 2002 Subject: [Numpy-discussion] calling multiarraymodule.c funcs from c api Message-ID: I am new to numpy extension writing, and am trying to call some of the functions defined in Numeric-21.3/Src/multiarraymodule.c, eg extern PyObject *PyArray_Correlate(PyObject *op1, PyObject *op2, int mode) in a Numeric extension module that I am writing. Are these functions available via the C API, and if so, how should I go about accessing them? Or are the only array functions defined in Include/arrayobject.h available in the C API? Below is my prototype module. I am defining a function 'xcorr' that just does what cross_correlate does, as a test case to see if I can access the multiarray C API: /* jdhscipy.c -- Wed Oct 2 2002 */ #include "Python.h" #include "Numeric/arrayobject.h" #include "stdio.h" static PyObject *ErrorObject; extern PyObject *PyArray_Correlate(PyObject*, PyObject*, int); static char xcorr__doc__[] = ""; static PyObject * xcorr(PyObject *self, PyObject *args) { PyObject *shape, *a0; int mode=0; if (!PyArg_ParseTuple(args, "OO|i", &a0, &shape, &mode)) return NULL; return PyArray_Correlate(a0, shape, mode); } static struct PyMethodDef jdhscipy_methods[] = { {"xcorr", xcorr, 1, xcorr__doc__}, {NULL, NULL} /* sentinel */ }; /* Initialization function for the module (*must* be called initjdhscipy) */ static char jdhscipy_module_documentation[] = "My first scipy extension" ; DL_EXPORT(void) initjdhscipy(void) { PyObject *m, *d; /* Create the module and add the functions */ m = Py_InitModule4("jdhscipy", jdhscipy_methods, jdhscipy_module_documentation, (PyObject*)NULL,PYTHON_API_VERSION); /* Import the array object */ import_array(); /* Add some symbolic constants to the module */ d = PyModule_GetDict(m); ErrorObject = PyString_FromString("jdhscipy.error"); PyDict_SetItemString(d, "error", ErrorObject); /* XXXX Add constants here */ /* Check for errors */ if (PyErr_Occurred()) Py_FatalError("can't initialize module jdhscipy"); } From nwagner at mecha.uni-stuttgart.de Fri Oct 4 03:52:01 2002 From: nwagner at mecha.uni-stuttgart.de (Nils Wagner) Date: Fri Oct 4 03:52:01 2002 Subject: [Numpy-discussion] numarray via CVS Message-ID: <3D9D745C.2E29986B@mecha.uni-stuttgart.de> Hi, I tried to install numarray via latest CVS. However python setup.py install failed This is the output Src/_ndarraymodule.c:240: warning: `_ndarray_bytestride_set' defined but not use d Src/_ndarraymodule.c:260: warning: `_ndarray_byteoffset_get' defined but not use d Src/_ndarraymodule.c:266: warning: `_ndarray_byteoffset_set' defined but not use d Src/_ndarraymodule.c:288: warning: `_ndarray_aligned_get' defined but not used Src/_ndarraymodule.c:294: warning: `_ndarray_aligned_set' defined but not used Src/_ndarraymodule.c:323: warning: `_ndarray_contiguous_get' defined but not use d Src/_ndarraymodule.c:329: warning: `_ndarray_contiguous_set' defined but not use d error: command 'gcc' failed with exit status 1 Any suggestion ? Nils Nils From jmiller at stsci.edu Fri Oct 4 06:26:05 2002 From: jmiller at stsci.edu (Todd Miller) Date: Fri Oct 4 06:26:05 2002 Subject: [Numpy-discussion] numarray via CVS Message-ID: <3D9D96BB.6070401@stsci.edu> Nils Wagner wrote: >Hi, > >I tried to install numarray via latest CVS. However >python setup.py install failed > >This is the output > >Src/_ndarraymodule.c:240: warning: `_ndarray_bytestride_set' defined but >not use d >Src/_ndarraymodule.c:260: warning: `_ndarray_byteoffset_get' defined but >not use d >Src/_ndarraymodule.c:266: warning: `_ndarray_byteoffset_set' defined but >not use d >Src/_ndarraymodule.c:288: warning: `_ndarray_aligned_get' defined but >not used >Src/_ndarraymodule.c:294: warning: `_ndarray_aligned_set' defined but >not used >Src/_ndarraymodule.c:323: warning: `_ndarray_contiguous_get' defined but >not use d >Src/_ndarraymodule.c:329: warning: `_ndarray_contiguous_set' defined but >not use d >error: command 'gcc' failed with exit status 1 > > >Any suggestion ? > >Nils > >Nils > I'm in the process of merging some new C basetypes into the head of CVS so I guess the HEAD has to be considered unstable for the next few days. I work hard to keep the CVS in working order, but you've caught me in the middle of "something big". For now, I have this advice: 1. Use numarray-0.3.6 until numarray-0.4 is released. 2. When upgrading numarray, always remove the old version. 3. If you're intereseted in writing extensions, stick with numarray's Numeric compatability API. Todd -- Todd Miller jmiller at stsci.edu STSCI / SSB From bsder at mail.allcaps.org Sun Oct 6 19:18:03 2002 From: bsder at mail.allcaps.org (Andrew P. Lentvorski) Date: Sun Oct 6 19:18:03 2002 Subject: [Numpy-discussion] int and float ufunc? Message-ID: <20021006190227.O41304-100000@mail.allcaps.org> Could someone explain the reason why int() and float() don't work as ufuncs? Is it just that someone needs to write the code, or is there some subtlety at work that I am missing? It took a bit of digging to go find the fact that what I wanted is newarray = oldarray.astype(MagicNewTypeCode) Thanks, -a From oliphant at ee.byu.edu Sun Oct 6 23:30:01 2002 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Sun Oct 6 23:30:01 2002 Subject: [Numpy-discussion] int and float ufunc? In-Reply-To: <20021006190227.O41304-100000@mail.allcaps.org> Message-ID: > Could someone explain the reason why int() and float() don't work as > ufuncs? Is it just that someone needs to write the code, or is there some > subtlety at work that I am missing? What would you have them do? Their current definition works for returning suitable arrays as integers and floats respectively, just as the Python documentation says these two functions should. It would be an easy thing to map int() to oldarray.astype(Int) and so forth but is this a good policy. I submit it is not. -Travis O. From bsder at mail.allcaps.org Mon Oct 7 01:43:04 2002 From: bsder at mail.allcaps.org (Andrew P. Lentvorski) Date: Mon Oct 7 01:43:04 2002 Subject: [Numpy-discussion] int and float ufunc? In-Reply-To: Message-ID: <20021007011036.P41789-100000@mail.allcaps.org> On Mon, 7 Oct 2002, Travis Oliphant wrote: > > Could someone explain the reason why int() and float() don't work as > > ufuncs? Is it just that someone needs to write the code, or is there some > > subtlety at work that I am missing? > > What would you have them do? Their current definition works for returning > suitable arrays as integers and floats respectively, just as the Python > documentation says these two functions should. Okay, now I'm really confused ... >>> import Numeric >>> a = Numeric.array([1.5, 2.5, 3.5], Numeric.Float64) >>> a array([ 1.5, 2.5, 3.5]) >>> int(a) Traceback (most recent call last): File "", line 1, in ? TypeError: Only rank-0 arrays can be converted to Python scalars. I guess I'm really dense, but how does that result constitute a definition that "works"? Has this behavior changed recently such that I should update my version of Numeric? If so, that would be a wonderful solution to my problem (explained below). > It would be an easy thing to map int() to oldarray.astype(Int) and so > forth but is this a good policy. I submit it is not. I could be persuaded either way. I was simply wondering what the arguments were. My usage is in converting vectors of data before passing them to the screen for display. The conversion function is a bunch of ufuncs and then the resulting vector needs to get converted to int before being passed to X. The same code works for Python floats, scalars, or Numeric arrays with the exception of the int function. If the problem was simply a lack of someone to write the code, I thought I might do it. I thought it might be a little faster than me writing array versions of all of my conversion functions, although I haven't looked at the complexity of the Numeric code. So, I could be completely wrong about that. If the traceback is "designed" behavior, I guess I'll just have to suck it up and write array versions of my functions. Thanks, -a From jmiller at stsci.edu Mon Oct 7 04:31:02 2002 From: jmiller at stsci.edu (Todd Miller) Date: Mon Oct 7 04:31:02 2002 Subject: [Numpy-discussion] arange(-10) Message-ID: <3DA17273.3020707@stsci.edu> The subject line contains what I consider to be an invalid range expression. Numarray, now and always, reacts badly to it. Currently, numarray tries to allocate a multi-gigabyte array. In the past, it has dumped core. The question is, what should it do? 1. raise ValueError, "Invalid negative range expression" (my +1) 2. zeros((0,), 'l') (Numeric does this) Is there a good justification to keep the existing Numeric behavior? Any other suggestions? Todd From pearu at cens.ioc.ee Mon Oct 7 05:26:05 2002 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Mon Oct 7 05:26:05 2002 Subject: [Numpy-discussion] arange(-10) In-Reply-To: <3DA17273.3020707@stsci.edu> Message-ID: On Mon, 7 Oct 2002, Todd Miller wrote: > The subject line contains what I consider to be an invalid range > expression. Numarray, now and always, reacts badly to it. Currently, > numarray tries to allocate a multi-gigabyte array. In the past, it has > dumped core. > > The question is, what should it do? > > 1. raise ValueError, "Invalid negative range expression" (my +1) > 2. zeros((0,), 'l') > (Numeric does this) > > Is there a good justification to keep the existing Numeric behavior? > Any other suggestions? Does Numarray support empty arrays? If yes, I'd vote for Python behavior: >>> range(-10) [] Otherwise, the case 1. Hmm, according to Numeric.arange docs, "arange is just like range() except it returns an array whose type can be specified ...". But given example shows that there are other (undocumented?) exceptions when comparing arange and range. Pearu From pearu at cens.ioc.ee Mon Oct 7 05:43:02 2002 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Mon Oct 7 05:43:02 2002 Subject: [Numpy-discussion] int and float ufunc? In-Reply-To: <20021007011036.P41789-100000@mail.allcaps.org> Message-ID: On Mon, 7 Oct 2002, Andrew P. Lentvorski wrote: > On Mon, 7 Oct 2002, Travis Oliphant wrote: > > > What would you have them do? Their current definition works for returning > > suitable arrays as integers and floats respectively, just as the Python ^^^^^^^^^^^^^^^ > > documentation says these two functions should. > > Okay, now I'm really confused ... > > >>> import Numeric > >>> a = Numeric.array([1.5, 2.5, 3.5], Numeric.Float64) > >>> a > array([ 1.5, 2.5, 3.5]) > >>> int(a) > Traceback (most recent call last): > File "", line 1, in ? > TypeError: Only rank-0 arrays can be converted to Python scalars. ^^^^^^^^^^^^^ > I guess I'm really dense, but how does that result constitute a definition > that "works"? See ^^^ above. In this case, suitable arrays are rank-0 arrays. So, the definition works (see also below). > > It would be an easy thing to map int() to oldarray.astype(Int) and so > > forth but is this a good policy. I submit it is not. > > I could be persuaded either way. I was simply wondering what the > arguments were. According to int.__doc__, int(..) should *always* return a Python integer. But you are asking for an integer array as a result of int(array(..)); and that would be a contradiction with the Python definition for int(anyobj). Pearu From oliphant at ee.byu.edu Mon Oct 7 08:52:03 2002 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Oct 7 08:52:03 2002 Subject: [Numpy-discussion] int and float ufunc? In-Reply-To: <20021007011036.P41789-100000@mail.allcaps.org> Message-ID: On Mon, 7 Oct 2002, Andrew P. Lentvorski wrote: > On Mon, 7 Oct 2002, Travis Oliphant wrote: > > > > Could someone explain the reason why int() and float() don't work as > > > ufuncs? Is it just that someone needs to write the code, or is there some > > > subtlety at work that I am missing? > > > > What would you have them do? Their current definition works for returning > > suitable arrays as integers and floats respectively, just as the Python > > documentation says these two functions should. > > Okay, now I'm really confused ... > > >>> import Numeric > >>> a = Numeric.array([1.5, 2.5, 3.5], Numeric.Float64) > >>> a > array([ 1.5, 2.5, 3.5]) > >>> int(a) > Traceback (most recent call last): > File "", line 1, in ? > TypeError: Only rank-0 arrays can be converted to Python scalars. > I didn't mean to confuse you. I said "suitable" arrays. Returning a single Python integer from an array doesn't really make sense unless that array is 0-dimensional (rank-0). Now, one could argue that an array of length 1 could also be converted unambiguously, but that discussion has never taken place. Python uses the int() function to return a Python integer type whenever it is explicitly needed. > I could be persuaded either way. I was simply wondering what the > arguments were. > The main argument is that int() should return a Python integer type representation of your object. There is no way to unambiguously return a Python integer object from a general array unless that array contains only one element. Currently the int() function only converts rank-0 arrays (which are Numeric scalars with array headers). > My usage is in converting vectors of data before passing them to the > screen for display. The conversion function is a bunch of ufuncs and then > the resulting vector needs to get converted to int before being passed to > X. The same code works for Python floats, scalars, or Numeric arrays with > the exception of the int function. > I don't understand what the problem is. Why does X.astype() not work for you here? What do you mean "passed to X?" Is this in C, in Python? The int() function was never intended to work on general Numeric arrays. It could be made to (and would be a very simple job), but that discussion has never taken place. instead of int() why don't you use asarray(x).astype('l') for your code > If the problem was simply a lack of someone to write the code, I thought I > might do it. I thought it might be a little faster than me writing array > versions of all of my conversion functions, although I haven't looked at > the complexity of the Numeric code. So, I could be completely wrong about > that. If you look in SciPy (www.scipy.org), we have defined a dictionary of cast functions for each of the types that works the way you apparently wanted int() to work. Using this command you would write scipy.cast[Numeric.Int]() I don't know what your functions are, but I doubt array versions would be that difficult to write. I guess I just always write array versions of my functions. Almost every routine I write starts with asarray(x) to convert inputs to array objects Good luck, -Travis O. From jmiller at stsci.edu Mon Oct 7 08:53:04 2002 From: jmiller at stsci.edu (Todd Miller) Date: Mon Oct 7 08:53:04 2002 Subject: [Numpy-discussion] arange(-10) References: Message-ID: <3DA1AFC3.6080008@stsci.edu> Pearu Peterson wrote: >On Mon, 7 Oct 2002, Todd Miller wrote: > >>The subject line contains what I consider to be an invalid range >>expression. Numarray, now and always, reacts badly to it. Currently, >>numarray tries to allocate a multi-gigabyte array. In the past, it has >>dumped core. >> >>The question is, what should it do? >> >>1. raise ValueError, "Invalid negative range expression" (my +1) >>2. zeros((0,), 'l') >> (Numeric does this) >> >>Is there a good justification to keep the existing Numeric behavior? >> Any other suggestions? >> > >Does Numarray support empty arrays? If yes, I'd vote for Python behavior: > Numarray has no problem with empty arrays so both choices are easy to implement. It apparently has a different repr (than Numeric) for "nothing" which is: array([]) Currently the tally is +2 exception, +1 Numeric-compatible. I'll probably just implement it at COB today, before I forget. Thanks for voting Todd From oliphant at ee.byu.edu Mon Oct 7 08:54:02 2002 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Oct 7 08:54:02 2002 Subject: [Numpy-discussion] arange(-10) In-Reply-To: <3DA17273.3020707@stsci.edu> Message-ID: On Mon, 7 Oct 2002, Todd Miller wrote: > The subject line contains what I consider to be an invalid range > expression. Numarray, now and always, reacts badly to it. Currently, > numarray tries to allocate a multi-gigabyte array. In the past, it has > dumped core. > > The question is, what should it do? > > 1. raise ValueError, "Invalid negative range expression" (my +1) +1 I didn't realize Numeric behaved badly in this area. -Travis From jmiller at stsci.edu Mon Oct 7 11:50:07 2002 From: jmiller at stsci.edu (Todd Miller) Date: Mon Oct 7 11:50:07 2002 Subject: [Numpy-discussion] arange(-10) References: Message-ID: <3DA1D966.2060609@stsci.edu> Travis Oliphant wrote: >On Mon, 7 Oct 2002, Todd Miller wrote: > >>The subject line contains what I consider to be an invalid range >>expression. Numarray, now and always, reacts badly to it. Currently, >>numarray tries to allocate a multi-gigabyte array. In the past, it has >>dumped core. >> >>The question is, what should it do? >> >>1. raise ValueError, "Invalid negative range expression" (my +1) >> >+1 > >I didn't realize Numeric behaved badly in this area. > >-Travis > > I replied to this earlier, but I may have forgotten to reply-all, because I never saw my response. Anyway, I don't think Numeric has a problem with arange(), but numarray does. The vote is currently +3 "raise", +2 "compatible". All votes appreciated Todd From bsder at mail.allcaps.org Mon Oct 7 11:57:10 2002 From: bsder at mail.allcaps.org (Andrew P. Lentvorski) Date: Mon Oct 7 11:57:10 2002 Subject: [Numpy-discussion] int and float ufunc? In-Reply-To: Message-ID: <20021007114938.F42968-100000@mail.allcaps.org> On Mon, 7 Oct 2002, Travis Oliphant wrote: > The main argument is that int() should return a Python integer type > representation of your object. There is no way to unambiguously return a > Python integer object from a general array unless that array contains only > one element. Okay. > I don't understand what the problem is. Why does X.astype() not work for > you here? I guess I wasn't clear here. It does. However, I have *lots* of these functions scattered around the code (inherited from someone else), so I will have to hunt them all down and change all the int(thingtoconvert) to thingtoconvert.astype('l'). Since there are good arguments for int behaving the way it does, I'll make the changes required to my code. No big deal, just lots of tedium. Thanks for taking the time to explain this, -a From kern at caltech.edu Mon Oct 7 13:02:05 2002 From: kern at caltech.edu (Robert Kern) Date: Mon Oct 7 13:02:05 2002 Subject: [Numpy-discussion] int and float ufunc? In-Reply-To: <20021007114938.F42968-100000@mail.allcaps.org> References: <20021007114938.F42968-100000@mail.allcaps.org> Message-ID: <20021007200156.GA9813@taliesen.caltech.edu> On Mon, Oct 07, 2002 at 11:56:48AM -0700, Andrew P. Lentvorski wrote: > However, I have *lots* of these > functions scattered around the code (inherited from someone else), so I > will have to hunt them all down and change all the int(thingtoconvert) to > thingtoconvert.astype('l'). How about defining a function like so: def int(object, **kwds): if type(object) is Numeric.ArrayType: return object.astype(Numeric.Integer) else: return __builtins__.int(object, **kwds) That should probably take care of all uses of int() in your code. -- Robert Kern Ruddock House President kern at caltech.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From fgibbons at hms.harvard.edu Mon Oct 7 13:54:06 2002 From: fgibbons at hms.harvard.edu (Frank Gibbons) Date: Mon Oct 7 13:54:06 2002 Subject: [Numpy-discussion] bug fix for exp() Message-ID: <5.0.0.25.2.20021007163622.00b0ed98@hms.harvard.edu> Hi, I've been lurking for a while, and using Numeric for about 5-6 months. I submitted a bug report about two months ago (https://sourceforge.net/tracker/index.php?func=detail&aid=594761&group_id=1369&atid=101369) regarding incorrect underflow behavior in exp(). I'd really like to get that fixed, nobody else seems to be working on it, and so I checked out the CVS tree. From reading the API section of the manual, I can see that unary ufuncs are driven by PyUFunc_FromFuncAndData() and that exp already has check_return set to 1, so that it should clean up rank-0 arrays, and raise the appropriate Python exception when 'errno' is set by a math error. I'd like to take a closer look into things, and one way of doing that would be to use a unit-test suite, but it's not obvious where that is (if at all) in the source distribution. Is there one? Another way to look into it would be if I could write some C code, then drop into Numeric's code and track what's going on when I call exp() for arguments that underflow. Is there a harness out there that I could just adapt to my own needs. Surely that's how other developers do it. If not, how *do* you fix bugs? I've already added some features to JNumeric, and introduced a unit-test suite there. I have so few problems using Numeric, I'd really like to nail this one involving exp(), since it's holding me back. Thanks, -Frank Gibbons PhD, Computational Biologist, Harvard Medical School BCMP/SGM-322, 250 Longwood Ave, Boston MA 02115, USA. Tel: 617-432-3555 Fax: 617-432-3557 http://llama.med.harvard.edu/~fgibbons From pearu at cens.ioc.ee Mon Oct 7 14:42:09 2002 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Mon Oct 7 14:42:09 2002 Subject: [Numpy-discussion] int and float ufunc? In-Reply-To: <20021007200156.GA9813@taliesen.caltech.edu> Message-ID: On Mon, 7 Oct 2002, Robert Kern wrote: > On Mon, Oct 07, 2002 at 11:56:48AM -0700, Andrew P. Lentvorski wrote: > > However, I have *lots* of these > > functions scattered around the code (inherited from someone else), so I > > will have to hunt them all down and change all the int(thingtoconvert) to > > thingtoconvert.astype('l'). > > How about defining a function like so: > > def int(object, **kwds): > if type(object) is Numeric.ArrayType: > return object.astype(Numeric.Integer) > else: > return __builtins__.int(object, **kwds) > > That should probably take care of all uses of int() in your code. I would rename this function to `aint' and still change all the int(x) to aint(x). When messing with Python built-ins, sooner or later you'll find a bullet in your leg ... Pearu From oliphant at ee.byu.edu Mon Oct 7 16:49:09 2002 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Oct 7 16:49:09 2002 Subject: [Numpy-discussion] bug fix for exp() In-Reply-To: <5.0.0.25.2.20021007163622.00b0ed98@hms.harvard.edu> Message-ID: > Hi, > > I've been lurking for a while, and using Numeric for about 5-6 months. I > submitted a bug report about two months ago > (https://sourceforge.net/tracker/index.php?func=detail&aid=594761&group_id=1369&atid=101369) > regarding incorrect underflow behavior in exp(). My understanding is that this bug went away with a new version of Python. It came about due to some change in Python and has not gone away. My advice is to start using scipy (but of course that's my advice because that's what I do). Or at least start using scipy_base (which is easier to install). SciPy handles NAN's and INFS instead of raising errors and this particular "bug" is not present because the error is not checked for. > Another way to look into it would be if I could write some C code, then > drop into Numeric's code and track what's going on when I call exp() for > arguments that underflow. Is there a harness out there that I could just > adapt to my own needs. Surely that's how other developers do it. If not, > how *do* you fix bugs? What's going on is that the system library for exp() is raising an underflow error and rather than ignore it and return 0.0 like a good exp() function, the error propagates to the top. It's an annoying bug, I agree. I've heard that it disappears with a new version of Python (2.2) but, I'm not entirely sure because I use scipy_base extensively. -Travis O. From peterson at math.utwente.nl Tue Oct 8 03:40:02 2002 From: peterson at math.utwente.nl (Pearu Peterson) Date: Tue Oct 8 03:40:02 2002 Subject: [Numpy-discussion] arange(-10) In-Reply-To: <3DA1AFC3.6080008@stsci.edu> Message-ID: On Mon, 7 Oct 2002, Todd Miller wrote: > Pearu Peterson wrote: > > >On Mon, 7 Oct 2002, Todd Miller wrote: > > > >>The subject line contains what I consider to be an invalid range > >>expression. Numarray, now and always, reacts badly to it. Currently, > >>numarray tries to allocate a multi-gigabyte array. In the past, it has > >>dumped core. > >> > >>The question is, what should it do? > >> > >>1. raise ValueError, "Invalid negative range expression" (my +1) > >>2. zeros((0,), 'l') > >> (Numeric does this) > >> > >>Is there a good justification to keep the existing Numeric behavior? > >> Any other suggestions? > >> > > > >Does Numarray support empty arrays? If yes, I'd vote for Python behavior: > > > Numarray has no problem with empty arrays so both choices are easy to > implement. It apparently has a different repr (than Numeric) for > "nothing" which is: > > array([]) Hmm, so it means that Numeric.arange is consistent with Python range after all. Then I'd like to change my vote from exception to array([]) (if it is not too late). Rationale: sometimes using range(i1) + range(i2) + range(i3) can save lots of code if the indices i1,i2,i3 can also be negative with the meaning that the resulting lists are then empty. If using arange, then the above would be equivalent to concatenate((arange(i1),arange(i2),arange(i3))) or concatenate(map(arange,[i1,i2,i3])) for short. Now, if arange would raise an exception if one of i1,i2,i3 is negative, then the above would not work without checking i1,i2,i3 first. And I find that bad because additional (read: meaningless) code is needed. Pearu From jmiller at stsci.edu Tue Oct 8 05:07:04 2002 From: jmiller at stsci.edu (Todd Miller) Date: Tue Oct 8 05:07:04 2002 Subject: [Numpy-discussion] arange(-10) References: Message-ID: <3DA2CA45.4060004@stsci.edu> Pearu Peterson wrote: > >On Mon, 7 Oct 2002, Todd Miller wrote: > >>Pearu Peterson wrote: >> >>>On Mon, 7 Oct 2002, Todd Miller wrote: >>> >>>>The subject line contains what I consider to be an invalid range >>>>expression. Numarray, now and always, reacts badly to it. Currently, >>>>numarray tries to allocate a multi-gigabyte array. In the past, it has >>>>dumped core. >>>> >>>>The question is, what should it do? >>>> >>>>1. raise ValueError, "Invalid negative range expression" (my +1) >>>>2. zeros((0,), 'l') >>>> (Numeric does this) >>>> >>>>Is there a good justification to keep the existing Numeric behavior? >>>>Any other suggestions? >>>> >>>Does Numarray support empty arrays? If yes, I'd vote for Python behavior: >>> >>Numarray has no problem with empty arrays so both choices are easy to >>implement. It apparently has a different repr (than Numeric) for >>"nothing" which is: >> >>array([]) >> > >Hmm, so it means that Numeric.arange is consistent with Python range after >all. Then I'd like to change my vote from exception to array([]) (if it >is not too late). > >Rationale: sometimes using > > range(i1) + range(i2) + range(i3) > >can save lots of code if the indices i1,i2,i3 can also be negative with >the meaning that the resulting lists are then empty. If using arange, then >the above would be equivalent to > > concatenate((arange(i1),arange(i2),arange(i3))) > >or > > concatenate(map(arange,[i1,i2,i3])) > >for short. Now, if arange would raise an exception if one of i1,i2,i3 is >negative, then the above would not work without checking i1,i2,i3 first. >And I find that bad because additional (read: meaningless) code is >needed. > >Pearu > > Hi Pearu, Well, as fate would have it, your vote was already mis-cast in favor of Numeric compatability. However, you've changed my mind, so I'm changing *my* vote. That leaves the current vote at +2 "raise", +3 "compatible", so barring any new votes, I'll check it in as Numeric compatible (returning an empty range) at COB today. Todd -- Todd Miller jmiller at stsci.edu STSCI / SSB From paustin at eos.ubc.ca Tue Oct 8 07:34:03 2002 From: paustin at eos.ubc.ca (Philip Austin) Date: Tue Oct 8 07:34:03 2002 Subject: [Numpy-discussion] libnumarray.h and NO_IMPORT_ARRAY Message-ID: <15778.60563.584317.60674@gull.eos.ubc.ca> We're currently porting several of our boost-wrapped Numeric classes to numarray. The routines are spread across several cpp files, and **libnumarray_API is needed in each file. In Numeric, we invoke import_array() in the module file, to initialize **PyArray_API, and def NO_IMPORT_ARRAY in the other files; this produces extern void **PyArray_API in those files, and everyone's happy. Currently libnumarray.h doesn't implement this -- is there another way to compile a multiple file project which initializes the API just once? Thanks, Phil Austin From jmiller at stsci.edu Tue Oct 8 07:59:04 2002 From: jmiller at stsci.edu (Todd Miller) Date: Tue Oct 8 07:59:04 2002 Subject: [Numpy-discussion] libnumarray.h and NO_IMPORT_ARRAY References: <15778.60563.584317.60674@gull.eos.ubc.ca> Message-ID: <3DA2F27C.80707@stsci.edu> Philip Austin wrote: >We're currently porting several of our boost-wrapped Numeric classes >to numarray. The routines are spread across several cpp files, and >**libnumarray_API is needed in each file. In Numeric, we invoke >import_array() in the module file, to initialize **PyArray_API, and >def NO_IMPORT_ARRAY in the other files; this produces >extern void **PyArray_API in those files, and everyone's happy. > >Currently libnumarray.h doesn't implement this -- is there >another way to compile a multiple file project which >initializes the API just once? > >Thanks, Phil Austin > > > >------------------------------------------------------- >This sf.net email is sponsored by:ThinkGeek >Welcome to geek heaven. >http://thinkgeek.com/sf >_______________________________________________ >Numpy-discussion mailing list >Numpy-discussion at lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/numpy-discussion > Someone else asked for this recently, and the solution is half-way implemented in CVS now. I'll finish it this morning so that it works the way you're used to, but the next "official" numarray release is still a ways off. So, if you still want to port now, here are the caveats: 1. Numarray CVS was working as of Friday on Windows, Linux, Solaris, and Tru64. 2. Use the "Numeric compatible" portion of the C-API. This API is a subset of the Numeric API which may cause you trouble. Let me know if it does. Don't use the "native" NDInfo based APIs because they're headed for deprecation. 3. The version you'll be working with has not yet been widely used and may have significant bugs I don't know about. It does pass all its unit tests on all our platforms. Todd -- Todd Miller jmiller at stsci.edu STSCI / SSB From jmiller at stsci.edu Tue Oct 8 08:28:06 2002 From: jmiller at stsci.edu (Todd Miller) Date: Tue Oct 8 08:28:06 2002 Subject: [Numpy-discussion] libnumarray.h and NO_IMPORT_ARRAY References: <15778.60563.584317.60674@gull.eos.ubc.ca> Message-ID: <3DA2F969.60300@stsci.edu> Philip Austin wrote: > >Currently libnumarray.h doesn't implement this -- is there >another way to compile a multiple file project which >initializes the API just once? > Hopefully you got my earlier message about this. Since I CC'ed numpy-discussion and never saw it there, I've included it below. But, assuming you saw it, I think what you need is in CVS now. Todd -- Todd Miller jmiller at stsci.edu STSCI / SSB My earlier response: Someone else asked for this recently, and the solution is half-way implemented in CVS now. I'll finish it this morning so that it works the way you're used to, but the next "official" numarray release is still a ways off. So, if you still want to port now, here are the caveats: 1. Numarray CVS was working as of Friday on Windows, Linux, Solaris, and Tru64. 2. Use the "Numeric compatible" portion of the C-API. This API is a subset of the Numeric API which may cause you trouble. Let me know if it does. Don't use the "native" NDInfo based APIs because they're headed for deprecation. 3. The version you'll be working with has not yet been widely used and may have significant bugs I don't know about. It does pass all its unit tests on all our platforms. From paustin at eos.ubc.ca Wed Oct 9 11:04:05 2002 From: paustin at eos.ubc.ca (Philip Austin) Date: Wed Oct 9 11:04:05 2002 Subject: [Numpy-discussion] libnumarray.h and NO_IMPORT_ARRAY In-Reply-To: <3DA2F969.60300@stsci.edu> References: <15778.60563.584317.60674@gull.eos.ubc.ca> <3DA2F969.60300@stsci.edu> Message-ID: <15780.28521.612331.840142@gull.eos.ubc.ca> Todd Miller writes: > Philip Austin wrote: > > > > >Currently libnumarray.h doesn't implement this -- is there > >another way to compile a multiple file project which > >initializes the API just once? > > > Hopefully you got my earlier message about this. Since I CC'ed > numpy-discussion and never saw it there, I've included it below. But, > assuming you saw it, I think what you need is in CVS now. Todd, thanks for the quick response. Regarding your header comments: /* Extensions constructed from seperate compilation units can access the C-API defined here by defining "libnumarray_UNIQUE_SYMBOL" to a global name unique to the extension. Doing this circumvents the requirement to import libnumarray into each compilation unit, but is nevertheless mildly discouraged as "outside the Python norm" and potentially leading to problems. Looking around at "existing Python art", most extension modules are monolithic C files, and likely for good reason. */ I'd agree with this sentiment, but with numarray's NDInfo interface I think we were stuck (we have an implementation file for a bunch of boost numarray helper functions that require the API but have static data initializations, so they can't go into a header). With Numeric, we could avoid import_array altogether, and get everything we needed from PyObject_CallObject calls to Numeric to create arrays using zeros or ones, followed by a cast to PyArrayObject* to get the data pointer and fill the array. With numarray, we need (or needed) getNDInfo to do the same thing, mandating an import_libnumarray. Are the plans for the revised interface to behave in the same way as NDInfo? It's not a big deal to us one way or another, but it would simplify Dave Abraham's boost support for numeric/numarray if much of the functionality could be done through python calls from the C side. Thanks, Phil From jmiller at stsci.edu Wed Oct 9 11:48:02 2002 From: jmiller at stsci.edu (Todd Miller) Date: Wed Oct 9 11:48:02 2002 Subject: [Numpy-discussion] libnumarray.h and NO_IMPORT_ARRAY References: <15778.60563.584317.60674@gull.eos.ubc.ca> <3DA2F969.60300@stsci.edu> <15780.28521.612331.840142@gull.eos.ubc.ca> Message-ID: <3DA479A7.6010409@stsci.edu> Philip Austin wrote: > >Todd, thanks for the quick response. Regarding your header comments: > >/* >Extensions constructed from seperate compilation units can access the >C-API defined here by defining "libnumarray_UNIQUE_SYMBOL" to a global >name unique to the extension. Doing this circumvents the requirement >to import libnumarray into each compilation unit, but is nevertheless >mildly discouraged as "outside the Python norm" and potentially >leading to problems. Looking around at "existing Python art", most >extension modules are monolithic C files, and likely for good reason. >*/ > > >I'd agree with this sentiment, but with numarray's NDInfo interface I >think we were stuck (we have an implementation file for a bunch of >boost numarray helper functions that require the API but have static >data initializations, so they can't go into a header). > Long term, you would be better off ditching NDInfo and working with the Numeric compatability API. Since it's a subset of the Numeric API, it may not support your purposes. However, if it doesn't, I want to extend it until it does. > >With Numeric, we could avoid import_array altogether, and get >everything we needed from PyObject_CallObject calls to Numeric to >create arrays using zeros or ones, followed by a cast to >PyArrayObject* to get the data pointer and fill the array. With >numarray, we need (or needed) getNDInfo to do the same thing, >mandating an import_libnumarray. Are the plans for the revised >interface to behave in the same way as NDInfo? > The current plan is to deprecate the "native" APIs which make use of NDInfo and to make numarray as compatible as possible with Numeric. The bulk of the changes which make this reasonable are in numarray CVS now. In a nutshell, prior to our use of C basetypes, we needed something like NDInfo. Now that we're using C basetypes, NDInfo appears less efficient, less compatible, and more complex than the "Numeric compatability" interface. Its possible for our "basetypes" to appear to be Numeric arrays, so that's what we did. If you guys find that you need missing parts of the Numeric C-API, I'll add them to numarray. So far, I've just added what I needed or knew someone else needed. > It's not a big deal to >us one way or another, but it would simplify Dave Abraham's boost >support for numeric/numarray if much of the functionality could be >done through python calls from the C side. > Can you send me the Numeric wrapper code or let me know where to get it? > >Thanks, Phil > > > >------------------------------------------------------- >This sf.net email is sponsored by:ThinkGeek >Welcome to geek heaven. >http://thinkgeek.com/sf >_______________________________________________ >Numpy-discussion mailing list >Numpy-discussion at lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/numpy-discussion > Thanks, Todd -- Todd Miller jmiller at stsci.edu STSCI / SSB From paustin at eos.ubc.ca Wed Oct 9 18:49:03 2002 From: paustin at eos.ubc.ca (Philip Austin) Date: Wed Oct 9 18:49:03 2002 Subject: [Numpy-discussion] libnumarray.h and NO_IMPORT_ARRAY In-Reply-To: <3DA479A7.6010409@stsci.edu> References: <15778.60563.584317.60674@gull.eos.ubc.ca> <3DA2F969.60300@stsci.edu> <15780.28521.612331.840142@gull.eos.ubc.ca> <3DA479A7.6010409@stsci.edu> Message-ID: <15780.56445.730334.591361@gull.eos.ubc.ca> Todd Miller writes: > Can you send me the Numeric wrapper code or let me know where to > get it? It's now on the cvs main trunk: cvs -d:pserver:anonymous at cvs.boost.sourceforge.net:/cvsroot/boost login cvs -z3 -d:pserver:anonymous at cvs.boost.sourceforge.net:/cvsroot/boost checkout boost You'll need all of boost if you want to try to compile/run the test code. If you haven't worked with boost and boost.build (bjam) before, I could send my install notes. If you'd just like to browse the code, the pertinent files are: boost/libs/python/test/numpy.cpp boost/libs/python/test/numpy.py boost/python/numeric.hpp and the new tutorial (which doesn't cover numpy yet) is quite good: boost/boost/libs/python/doc/tutorial as well as: boost/libs/python/doc/v2/reference.html (also without numpy documentation) Regards, Phil From jmiller at stsci.edu Thu Oct 10 06:26:05 2002 From: jmiller at stsci.edu (Todd Miller) Date: Thu Oct 10 06:26:05 2002 Subject: [Numpy-discussion] libnumarray.h and NO_IMPORT_ARRAY References: <15778.60563.584317.60674@gull.eos.ubc.ca> <3DA2F969.60300@stsci.edu> <15780.28521.612331.840142@gull.eos.ubc.ca> <3DA479A7.6010409@stsci.edu> <15780.56445.730334.591361@gull.eos.ubc.ca> Message-ID: <3DA57FC4.9030208@stsci.edu> Philip Austin wrote: >Todd Miller writes: > > Can you send me the Numeric wrapper code or let me know where to > > get it? > >It's now on the cvs main trunk: > >cvs -d:pserver:anonymous at cvs.boost.sourceforge.net:/cvsroot/boost login >cvs -z3 -d:pserver:anonymous at cvs.boost.sourceforge.net:/cvsroot/boost checkout boost > >You'll need all of boost if you want to try to compile/run the test >code. If you haven't worked with boost and boost.build (bjam) >before, I could send my install notes. If you'd just like to >browse the code, the pertinent files are: > >boost/libs/python/test/numpy.cpp >boost/libs/python/test/numpy.py >boost/python/numeric.hpp > >and the new tutorial (which doesn't cover numpy yet) is >quite good: > >boost/boost/libs/python/doc/tutorial > >as well as: > >boost/libs/python/doc/v2/reference.html >(also without numpy documentation) > >Regards, Phil > I took a look at the files above trying to see what you're doing. What I thought I would see, was some form of direct coupling between a module in boost and either the Numeric or numarray C-API. However, when I search the whole source tree (*.cpp *.hpp) I can find neither PyArrayObject, arrayobject.h, NDInfo, nor libnumarray.h. So I'm not sure what part of any Numeric/numarray C-API you're using now (if any) and in particular, what isn't working for you. Note: numarray and Numeric are not totally compatible at either the C or Python levels, so either area could have been a source of difficulty. Todd -- Todd Miller jmiller at stsci.edu STSCI / SSB From paustin at eos.ubc.ca Thu Oct 10 09:47:07 2002 From: paustin at eos.ubc.ca (Philip Austin) Date: Thu Oct 10 09:47:07 2002 Subject: [Numpy-discussion] libnumarray.h and NO_IMPORT_ARRAY In-Reply-To: <3DA57FC4.9030208@stsci.edu> References: <15778.60563.584317.60674@gull.eos.ubc.ca> <3DA2F969.60300@stsci.edu> <15780.28521.612331.840142@gull.eos.ubc.ca> <3DA479A7.6010409@stsci.edu> <15780.56445.730334.591361@gull.eos.ubc.ca> <3DA57FC4.9030208@stsci.edu> Message-ID: <15781.44767.831692.188152@gull.eos.ubc.ca> > Philip Austin wrote: [snip] > > > >boost/libs/python/test/numpy.cpp > >boost/libs/python/test/numpy.py > >boost/python/numeric.hpp > > Todd Miller writes: > I took a look at the files above trying to see what you're doing. (note that all of this was written by Dave Abrahams, I'm just one of the first Numeric users.) > What I thought I would see, was some form of direct coupling > between a module in boost and either the Numeric or numarray C-API. > However, when I search the whole source tree (*.cpp *.hpp) I can > find neither PyArrayObject, arrayobject.h, NDInfo, nor > libnumarray.h. So I'm not sure what part of any Numeric/numarray > C-API you're using now (if any) The answer is -- no part of the C-API if he can get away with it. Your move towards Numeric compatibility for numarray means that he now can get away with it. Specifically, the extension tries to import the numarray module, and if that fails, imports Numeric (the user can change this with numeric::set_module_and_type("Numeric", "ArrayType"); or: numeric::set_module_and_type("numarray", "NDArray"); ) numeric wraps the module pointer, and calls functions through it -- e.g. (from boost/libs/python/src/numeric.cpp): void array_base::resize(object const& shape) { attr("resize")(shape); } which invokes PyObject_CallObject to call the array's resize method, with the tuple shape (similar to CXX). If the method isn't part of the library (say for Numeric and getflat) then a python exception is thrown. > and in particular, what isn't > working for you. Note: numarray and Numeric are not totally > compatible at either the C or Python levels, so either area could > have been a source of difficulty. The problem was that, as long as a call to getNDInfo was required to access an array's data pointer, a Python-only approach like the above wouldn't work. Since it is now possible to do this via a cast to PyArrayObject* for a numarray, the problem has disappeared. (there's nothing stopping you from using the C-API if you want, though. For example you can create a numarray with data copied from a C array with something like: numeric::array makeNum(int * data, int n=0){ object obj(detail::new_reference(PyArray_FromDims(1, &n, PyArray_INT))); char *arr_data = ((PyArrayObject*) obj.ptr())->data; memcpy(arr_data, data, 4 * n); return numeric::array(obj); } ) Regards, Phil From dave at boost-consulting.com Thu Oct 10 10:29:05 2002 From: dave at boost-consulting.com (David Abrahams) Date: Thu Oct 10 10:29:05 2002 Subject: [Numpy-discussion] libnumarray.h and NO_IMPORT_ARRAY In-Reply-To: <15781.44767.831692.188152@gull.eos.ubc.ca> References: <15778.60563.584317.60674@gull.eos.ubc.ca> <3DA2F969.60300@stsci.edu> <15780.28521.612331.840142@gull.eos.ubc.ca> <3DA479A7.6010409@stsci.edu> <15780.56445.730334.591361@gull.eos.ubc.ca> <3DA57FC4.9030208@stsci.edu> <15781.44767.831692.188152@gull.eos.ubc.ca> Message-ID: Philip Austin writes: > > What I thought I would see, was some form of direct coupling > > between a module in boost and either the Numeric or numarray C-API. > > However, when I search the whole source tree (*.cpp *.hpp) I can > > find neither PyArrayObject, arrayobject.h, NDInfo, nor > > libnumarray.h. So I'm not sure what part of any Numeric/numarray > > C-API you're using now (if any) > > The answer is -- no part of the C-API if he can get away with it. > Your move towards Numeric compatibility for numarray means that he > now can get away with it. I've been getting away with it already. It's easy enough to access all that functionality through the usual Python/C API. > numeric wraps the module pointer, and calls functions > through it -- e.g. (from boost/libs/python/src/numeric.cpp): > > void array_base::resize(object const& shape) > { > attr("resize")(shape); > } > > which invokes PyObject_CallObject to call the array's resize method, > with the tuple shape (similar to CXX). If the method isn't part of the > library (say for Numeric and getflat) then a python exception is > thrown. Slow but effective. If you want to touch the raw PyObject*, of course, you can do this: void f(numeric::array x) { PyObject* danger = x.ptr(); // Cast to whatever you like. } > > and in particular, what isn't > > working for you. Note: numarray and Numeric are not totally > > compatible at either the C or Python levels, so either area could > > have been a source of difficulty. > > The problem was that, as long as a call to getNDInfo was required > to access an array's data pointer, a Python-only approach > like the above wouldn't work. Since it is now possible to > do this via a cast to PyArrayObject* for a numarray, the problem > has disappeared. Not sure what you're saying here. I /do/ remember when I was writing the Numeric/NumArray support that there seemed no publicly-accessible definition of the structure of the underlying array objects. > (there's nothing stopping you from using the C-API if you want, > though. For example you can create a numarray with data > copied from a C array with something like: > > numeric::array makeNum(int * data, int n=0){ > object obj(detail::new_reference(PyArray_FromDims(1, &n, PyArray_INT))); > char *arr_data = ((PyArrayObject*) obj.ptr())->data; > memcpy(arr_data, data, 4 * n); > return numeric::array(obj); > } Well, this uses undocumented implementation details. The supported approach is: handle<> raw_array(PyArray_FromDims(1, &n, PyArray_INT)); object obj(raw_array); Note, though, that you can also write: handle raw_array(PyArray_FromDims(1, &n, PyArray_INT)); Assuming, of course, that PyArray_FromDims returns a PyArrayObject*. And now you have access to the PyArrayObject* through raw_array. Also note that it's probably smarter to use extract(obj) Than numeric::array(obj) See the note about pitfalls at http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/*checkout*/boost/boost/libs/python/doc/tutorial/doc/derived_object_types.html for the reasons why. -- David Abrahams * Boost Consulting dave at boost-consulting.com * http://www.boost-consulting.com From paustin at eos.ubc.ca Thu Oct 10 11:14:11 2002 From: paustin at eos.ubc.ca (Philip Austin) Date: Thu Oct 10 11:14:11 2002 Subject: [Numpy-discussion] libnumarray.h and NO_IMPORT_ARRAY In-Reply-To: References: <15778.60563.584317.60674@gull.eos.ubc.ca> <3DA2F969.60300@stsci.edu> <15780.28521.612331.840142@gull.eos.ubc.ca> <3DA479A7.6010409@stsci.edu> <15780.56445.730334.591361@gull.eos.ubc.ca> <3DA57FC4.9030208@stsci.edu> <15781.44767.831692.188152@gull.eos.ubc.ca> Message-ID: <15781.49982.342376.307126@gull.eos.ubc.ca> David Abrahams writes: > > Not sure what you're saying here. I /do/ remember when I was writing > the Numeric/NumArray support that there seemed no publicly-accessible > definition of the structure of the underlying array objects. As I understand it, the numarray now in CVS deprecates the NDInfo struct (see section 10.1 of numarray.pdf) in favor of PyArrayObject. [snip] > See the note about pitfalls at > http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/*checkout*/boost/boost/libs/python/doc/tutorial/doc/derived_object_types.html > for the reasons why. Thanks for the info -- Phil From jmiller at stsci.edu Thu Oct 10 12:45:01 2002 From: jmiller at stsci.edu (Todd Miller) Date: Thu Oct 10 12:45:01 2002 Subject: [Numpy-discussion] libnumarray.h and NO_IMPORT_ARRAY References: <15778.60563.584317.60674@gull.eos.ubc.ca> <3DA2F969.60300@stsci.edu> <15780.28521.612331.840142@gull.eos.ubc.ca> <3DA479A7.6010409@stsci.edu> <15780.56445.730334.591361@gull.eos.ubc.ca> <3DA57FC4.9030208@stsci.edu> <15781.44767.831692.188152@gull.eos.ubc.ca> <15781.49982.342376.307126@gull.eos.ubc.ca> Message-ID: <3DA5D86F.7030204@stsci.edu> Philip Austin wrote: >David Abrahams writes: > > > > Not sure what you're saying here. I /do/ remember when I was writing > > the Numeric/NumArray support that there seemed no publicly-accessible > > definition of the structure of the underlying array objects. > Numeric and numarray both have "arrayobject.h" which makes PyArrayObject public. Numarray also has "libnumarray.h" which indirectly exposes NDInfo. > >As I understand it, the numarray now in CVS deprecates the NDInfo >struct (see section 10.1 of numarray.pdf) in favor of PyArrayObject. > This is true, but is currently neither released nor documented. On the plus side, numarray is becoming more compatible with Numeric, getting faster, and simpler as well. Todd -- Todd Miller jmiller at stsci.edu STSCI / SSB From karthik at james.hut.fi Fri Oct 11 10:06:07 2002 From: karthik at james.hut.fi (Karthikesh Raju) Date: Fri Oct 11 10:06:07 2002 Subject: [Numpy-discussion] Issues about class initialialization in with large numerical data Message-ID: Hi All, i am writing a class called communicate. Something like this: class Communicate: """ This is the main communicate class : there will be more """ def __init__(self): """This should hopefully read a file and set self's parameters """ self.C = 10 self.K = 6 self.N = 8 self.Na = 3 self.L = 4 self.modulation = "QPSK" 1. Shown is just one method in the class. Now there are lot of self variables (i understand they are something like private variables). Now i want to have an .ini file, so that i call something like this in the __init__ module : def __init__(self, fileName): file = open(fileName, 'r') self = something(file) and all the self's variables are set. i want the ini file to be the interface between the class and the outside world. Some one suggested ConfigParser , couldnt get an idea of what it is and how to use it.. 2. Is this method of having all the data in self variables good.The size of some of these 'self' could be very high 40000x40000 complex. Should i create local variables in the methods (i need a lot of variables, but their def life is too small .....) for example one of the method could be: def generateNoise(self): """ Generates AWGN """ power = Numeric.sqrt(RandomArray.random()*10) Noise = Numeric.zeros(....) Noise.real = Random ...... Noise.imag = RandomArray.... (Boared to type the whole thing that is why there are ......'s ) Any way power is a local variable, which i guess dies after generateNoise() method is called in the object. Am i right.. Can i have variables like 'power' 3. Thirdly, i want an easy method like '.mat' that exists in matlab. Can i use pickle (cPickle or Numeric.pickle) to store all the variables that i want. These matrices could be 40000x40000 entries long. Assuming that all the variables are in 'self' should i write an assignment module before pickling them (in case pickle is the right way to store these variables) def writeToFile(self): Na = self.Na x = self.x pickle the variables Na, x Thankx in advance and in anticipation, karthik PS: slowly moving from Matlab to Python, hence these stupid doubts ----------------------------------------------------------------------- Karthikesh Raju, email: karthik at james.hut.fi Researcher, http://www.cis.hut.fi/karthik Helsinki University of Technology, Tel: +358-9-451 5389 Laboratory of Comp. & Info. Sc., Fax: +358-9-451 3277 Department of Computer Sc., P.O Box 5400, FIN 02015 HUT, Espoo, FINLAND ----------------------------------------------------------------------- From rob at pythonemproject.com Sat Oct 12 15:16:03 2002 From: rob at pythonemproject.com (Rob) Date: Sat Oct 12 15:16:03 2002 Subject: [Numpy-discussion] Dyadic Python Message-ID: <3DA89E75.51168D7F@pythonemproject.com> As a help for me learning Dyadic vector analysis, I have been working on a Numpy class for manipulating Dyads. I am wondering if anyone else has any similar interests. Most of the impetus for the work comes from "Methods for Electromagnetic Field Analysis" by Ismo Lindell. In numerous searches I found no computer code dealing with Dyads. My biggest stumbling block so far is integrating the symbolism of Dyadic analysis into something understandable. Operator overloading isn't going to help me. For example in my program: A=dyad(a,b) where a and b are complex vectors B=dyad(c,d) " A.dmmd(B) is equivalent to double dyadic multiplication of the dyads A and B. :) In a book this would be written something like A xx B except that the x's would be aligned vertically. Ok, its an insane project. Rob. -- ----------------------------- The Numeric Python EM Project www.pythonemproject.com From paul at pfdubois.com Sat Oct 12 21:33:05 2002 From: paul at pfdubois.com (Paul F Dubois) Date: Sat Oct 12 21:33:05 2002 Subject: [Numpy-discussion] Dyadic Python In-Reply-To: <3DA89E75.51168D7F@pythonemproject.com> Message-ID: <000601c27271$86e6d1a0$6701a8c0@NICKLEBY> What took me a long while to appreciate is that inheriting from Numeric wasn't going to do me any good precisely because any operation such as a+b was going to produce an array not one of my new foos, unless I overrode it. So in fact you end up having to override darn near everything. I can only suggest MA as an example. People have succeeded in similar tasks by starting with the MA source and changing it to suit. > -----Original Message----- > From: numpy-discussion-admin at lists.sourceforge.net > [mailto:numpy-discussion-admin at lists.sourceforge.net] On Behalf Of Rob > Sent: Saturday, October 12, 2002 3:13 PM > To: numpy-discussion at lists.sourceforge.net > Subject: [Numpy-discussion] Dyadic Python > > > As a help for me learning Dyadic vector analysis, I have been > working on a Numpy class for manipulating Dyads. I am > wondering if anyone else has any similar interests. Most of > the impetus for the work comes from "Methods for > Electromagnetic Field Analysis" by Ismo Lindell. In numerous > searches I found no computer code dealing with Dyads. > > My biggest stumbling block so far is integrating the > symbolism of Dyadic analysis into something understandable. > Operator overloading isn't > going to help me. For example in my program: > > A=dyad(a,b) where a and b are complex vectors > B=dyad(c,d) " > A.dmmd(B) is equivalent to double dyadic multiplication of the dyads A > and B. :) In a book this would be written something like A xx B > except that the x's would be aligned vertically. > > Ok, its an insane project. Rob. > > > -- > ----------------------------- > The Numeric Python EM Project > www.pythonemproject.com ------------------------------------------------------- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf _______________________________________________ Numpy-discussion mailing list Numpy-discussion at lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion From retype at terra.com.br Tue Oct 15 12:24:02 2002 From: retype at terra.com.br (Leonardo Santagada) Date: Tue Oct 15 12:24:02 2002 Subject: [Numpy-discussion] Re: Numpy-discussion -- confirmation of subscription -- request 961098 In-Reply-To: References: Message-ID: <20021015162035.75591153.retype@terra.com.br> On Tue, 15 Oct 2002 12:22:27 -0700 numpy-discussion-request at lists.sourceforge.net wrote: > Numpy-discussion -- confirmation of subscription -- request 961098 > > We have received a request from 200.176.39.98 for subscription of your > email address, , to the > numpy-discussion at lists.sourceforge.net mailing list. To confirm the > request, please send a message to > numpy-discussion-request at lists.sourceforge.net, and either: > > - maintain the subject line as is (the reply's additional "Re:" is > ok), > > - or include the following line - and only the following line - in the > message body: > > confirm 961098 > > (Simply sending a 'reply' to this message should work from most email > interfaces, since that usually leaves the subject line in the right > form.) > > If you do not wish to subscribe to this list, please simply disregard > this message. Send questions to > numpy-discussion-admin at lists.sourceforge.net. > > Esta mensagem foi verificada pelo E-mail Protegido Terra. > Scan engine: VirusScan / Atualizado em 09/10/2002 / Vers?o: 1.3.13 > Proteja o seu e-mail Terra: http://www.emailprotegido.terra.com.br/ > From blackfield1 at llnl.gov Wed Oct 16 13:33:08 2002 From: blackfield1 at llnl.gov (Don Blackfield) Date: Wed Oct 16 13:33:08 2002 Subject: [Numpy-discussion] Fwd: RE: Problem with pyfort Message-ID: <5.0.0.25.2.20021016132723.028c04c0@poptop.llnl.gov> Hello, I sent the following email to Paul Dubois. Since he is not familiar with MS WINDOWS, he suggested that I send my problem to this list. In short, I am unable to get the pyfdemo to work in pyfort_8_.2 using python2.2 because my FORTRAN compiler, COMPAQ/DEC Visual FORTRAN uses NMAKE.EXE rather than make. When I try to run the demo, I get an error message saying that make is not recognized. What file or file(s) should I change. I cannot run under LINUX, it must be WINDOWS2000. Thanks for your assistance, Don >From: paul at pfdubois.com >Date: Wed, 16 Oct 2002 13:26:27 -0700 >Subject: RE: Problem with pyfort >To: "Don Blackfield" > >There is a subdirectory of the source that is devoted to how to do things >on Microsoft. I don't have any experience of it myself. You can try the >numpy-discussion at lists.sf.net list if you want to contact other users. > > >-- Original Message -- > >Date: Wed, 16 Oct 2002 12:10:54 -0700 > >To: "Paul F Dubois" > >From: Don Blackfield > >Subject: Problem with pyfort > > > > > > Paul, > > > > You may be cursing Microsoft by now, I know that I am. I have > >"successfully" installed pyfort. However, > >when I try to run the pyfdemo script by typing > > > >pyfort.bat -i pyfdemo > > > >it begins to work. However it crashes because WINDOWS does not recognize > > > >make. The COMPAQ/DEC/Visual > >Fortran uses a similar routine called NMAKE.EXE . Is there some place in > > > >some file(s) where I need to change > >make to NMAKE.EXE ?? > > > > Sorry for the hassle..... > > > >Don > > ============================================== Donald T. Blackfield Fusion Energy Program New Technologies Engineering Division Lawrence Livermore National Laboratory P.O. Box 808, MailStop L-641 Livermore, CA 94550 (925)422-0496 / (925)422-7390 (fax) dtb at llnl.gov ============================================== From nobody at maui.dnsvault.com Tue Oct 29 17:34:02 2002 From: nobody at maui.dnsvault.com (Nobody) Date: Tue Oct 29 17:34:02 2002 Subject: [Numpy-discussion] A larger gold balance! Message-ID: An HTML attachment was scrubbed... URL: