From joostvanevert at gmail.com Fri Jul 1 04:44:58 2005 From: joostvanevert at gmail.com (Joost van Evert) Date: Fri Jul 1 04:44:58 2005 Subject: [Numpy-discussion] numarray problem creating large memmap In-Reply-To: <42B721E1.2040503@cs.utk.edu> References: <42B437AD.3040805@sympatico.ca> <42B71450.6090102@cs.utk.edu> <42B721E1.2040503@cs.utk.edu> Message-ID: <1120217998.20139.10.camel@inpc93.et.tudelft.nl> Hello, does anyone know why it is not possible to create a memmap beyond a certain length. For example: m = MM.open('/tmp/test','write',len=500000000) works, while the same command using a length of 600000000 gives the following error message: /shannon/joost/lib/python/numarray/memmap.py in __init__(self, filename, mode, len) 299 else: 300 acc = mmap.ACCESS_WRITE --> 301 self._mmap = mmap.mmap(file.fileno(), len, access=acc) 302 else: 303 self._mmap = None EnvironmentError: [Errno 22] Invalid argument I am using: - numarray version 1.3.1 - python 2.4.1 - linux Greetings, Joost From jmiller at stsci.edu Fri Jul 1 09:52:26 2005 From: jmiller at stsci.edu (Todd Miller) Date: Fri Jul 1 09:52:26 2005 Subject: [Numpy-discussion] numarray problem creating large memmap In-Reply-To: <1120217998.20139.10.camel@inpc93.et.tudelft.nl> References: <42B437AD.3040805@sympatico.ca> <42B71450.6090102@cs.utk.edu> <42B721E1.2040503@cs.utk.edu> <1120217998.20139.10.camel@inpc93.et.tudelft.nl> Message-ID: <1120236614.30306.126.camel@halloween.stsci.edu> On Fri, 2005-07-01 at 07:39, Joost van Evert wrote: > Hello, > > does anyone know why it is not possible to create a memmap beyond a > certain length. For example: m = > MM.open('/tmp/test','write',len=500000000) works, while the same command > using a length of 600000000 gives the following error message: > > /shannon/joost/lib/python/numarray/memmap.py in __init__(self, filename, > mode, len) > 299 else: > 300 acc = mmap.ACCESS_WRITE > --> 301 self._mmap = mmap.mmap(file.fileno(), len, > access=acc) > 302 else: > 303 self._mmap = None > > EnvironmentError: [Errno 22] Invalid argument > > I am using: > - numarray version 1.3.1 > - python 2.4.1 > - linux I'm not sure it matters, but how much RAM and swap do you have? How much free space do you have on /tmp? Regards, Todd From rays at blue-cove.com Fri Jul 1 14:01:38 2005 From: rays at blue-cove.com (Ray Schumacher) Date: Fri Jul 1 14:01:38 2005 Subject: [Numpy-discussion] a Numeric.where re-coded for weave.inline: very fast... Message-ID: <5.2.0.4.2.20050701134209.09aecff8@blue-cove.com> I tried weave for executing a C routine over a Numeric array. The Numeric data array is c_int16 with 14 significant bits (bit 13 is the sign) and I wanted it as "normal" Int16. So: t1=clock() m = where(greater(dataArray[:1000], 8191), dataArray[:1000]-16383, dataArray[:1000]) t2 = clock() print 'where', round((t2-t1)*1000000, 1), 'us' Usually gives: where 634.3 us PyInline has about the same calling overhead as ctypes (I had tried a DLL and ctypes: http://sourceforge.net/mailarchive/forum.php?thread_id=7630224&forum_id=24606 ), apparently, about 240us for a simple C statement. It also requires some extra hoops to digest the pointer to short I need ( void b2int(short *f, int N) ), so I didn't fully try it. weave.inline testing was very good, with different issues. I found I had to patch msvccompiler.py for MS C 7.1 .NET ( http://www.vrplumber.com/programming/mstoolkit/ ). After the free but massive download/install... import weave inline = weave.inline # for speed N = 1000 code="int i; for(i = 0; i < N; i++){if (dataArray[i]>8191){dataArray[i]-=16383;}}" inline(code, ['dataArray', 'N']) # just pass Python object's names This created sc_019a1cf36209cb2dfc688820080541ef0.pyd in C:\Documents and Settings\rays\Local Settings\Temp\rays\python24_compiled\ Using the above code is slow; ~32000 us, as the compiler checks or runs each time. However, after much fiddling and dir()s, I copied the long-name.pyd to C:\Python24\DLLs and just did import sc_019a1cf36209cb2dfc688820080541ef0 b2iw = sc_019a1cf36209cb2dfc688820080541ef0.compiled_func # note the exposed function! N = 1 t1=clock() b2iw({'dataArray':dataArray}, {'N':N}) # note the dicts! t2 = clock() print 'weave', round((t2-t1)*1000000, 1), 'us' 25us! (~300us for N=10000) I think that's as close to C speed as I can expect, although I'm looking at the compiler options in msvccompiler.py for P4 optimization... I still get "Missing compiler_cxx fix for MSVCCompiler" on the initial compile, but apparently to no harm. As a final note, I also found that psyco _slows down_ both ctypes and weave calls. I did psyco.full() at the app's start. Without pysco: b2i 210.8 us weave 53.7 us with: b2i 250.0 us weave 234.7 us Comment/criticism ? Ray From rowen at cesmail.net Fri Jul 1 15:05:15 2005 From: rowen at cesmail.net (Russell E. Owen) Date: Fri Jul 1 15:05:15 2005 Subject: [Numpy-discussion] Trouble installing numarray on Solaris Message-ID: A user of my application reports that installing numarray 1.3.2 on Solaris using Python 2.3.1 has run into trouble: prompt> python Python 2.3.1 (#1, Sep 30 2003, 20:29:58) [C] on sunos5 Type "help", "copyright", "credits" or "license" for more information. >>> import numarray Fatal Python error: Call to API function without first calling import_libnumarray() in Src/_convmodule.c Abort (core dumped) I've seen a one or two recent reports of this error when running extension modules, but I've never heard of it happening when simply trying to import numarray. Any suggestions? -- Russell P.S. I do realize that python 2.3.1 is old and should be updated, and I have already suggested it. But it seems a bit unlikely to fix this particular problem. From nicopernetty at yahoo.fr Sat Jul 2 03:35:23 2005 From: nicopernetty at yahoo.fr (Nicolas Pernetty) Date: Sat Jul 2 03:35:23 2005 Subject: [Numpy-discussion] Trouble installing numarray on Solaris In-Reply-To: References: Message-ID: <20050702123323.05f93476@linuxcestcomplique.fr> On Fri, 01 Jul 2005 15:02:44 -0700, "Russell E. Owen" wrote : > A user of my application reports that installing numarray 1.3.2 on > Solaris using Python 2.3.1 has run into trouble: > > prompt> python > Python 2.3.1 (#1, Sep 30 2003, 20:29:58) [C] on sunos5 > Type "help", "copyright", "credits" or "license" for more > information. > >>> import numarray > Fatal Python error: Call to API function without first calling > import_libnumarray() in Src/_convmodule.c > Abort (core dumped) > > I've seen a one or two recent reports of this error when running > extension modules, but I've never heard of it happening when simply > trying to import numarray. > > Any suggestions? > > -- Russell > > P.S. I do realize that python 2.3.1 is old and should be updated, and > I have already suggested it. But it seems a bit unlikely to fix this > particular problem. > Hello, I'm happily using Python and numarray on Solaris 5.8 without any problems. But it's Python 2.4.1 and numarray 1.3.2. Maybe it's some sort of incompatibility ? Then you should try to install older version of numarray... From jmiller at stsci.edu Sat Jul 2 03:44:31 2005 From: jmiller at stsci.edu (Todd Miller) Date: Sat Jul 2 03:44:31 2005 Subject: [Numpy-discussion] Trouble installing numarray on Solaris In-Reply-To: References: Message-ID: <1120300999.5263.40.camel@localhost.localdomain> On Fri, 2005-07-01 at 15:02 -0700, Russell E. Owen wrote: > A user of my application reports that installing numarray 1.3.2 on > Solaris using Python 2.3.1 has run into trouble: > > prompt> python > Python 2.3.1 (#1, Sep 30 2003, 20:29:58) [C] on sunos5 > Type "help", "copyright", "credits" or "license" for more information. > >>> import numarray > Fatal Python error: Call to API function without first calling > import_libnumarray() in Src/_convmodule.c > Abort (core dumped) It looks vaguely like an binary API mismatch of some kind... or one of you numarray modules is stale or broken. To restate the obvious, it's likely that numarray won't import (in C!) because import_libnumarray() is very defintely in _convmodule.c. Since it's there and the API pointer was not initialized, the libnumarray import failed. I just tried: Python-2.3.4 (non-debug) numarray-1.3.2 (should be irrelevant that it's not 1.3.1) % uname -a SunOS 5.8 Generic_117350-18 sun4u sparc SUNW,Ultra-5_10 It checked out fine. I recommend finding your old numarray install and deleting it and re-installing 1.3.2. Please let me know if that doesn't solve the problem... I realize it's an unsatisfying answer. Regards, Todd > I've seen a one or two recent reports of this error when running > extension modules, but I've never heard of it happening when simply > trying to import numarray. > > Any suggestions? > > -- Russell > > P.S. I do realize that python 2.3.1 is old and should be updated, and I > have already suggested it. But it seems a bit unlikely to fix this > particular problem. > > > > ------------------------------------------------------- > SF.Net email is sponsored by: Discover Easy Linux Migration Strategies > from IBM. Find simple to follow Roadmaps, straightforward articles, > informative Webcasts and more! Get everything you need to get up to > speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From accounts at eBay.com Sat Jul 2 07:28:18 2005 From: accounts at eBay.com (eBay) Date: Sat Jul 2 07:28:18 2005 Subject: [Numpy-discussion] You have successfully added a new email address Message-ID: <1120314405.124551.qmail@paypal.com> An HTML attachment was scrubbed... URL: From schaffer at optonline.net Sat Jul 2 19:40:22 2005 From: schaffer at optonline.net (Les Schaffer) Date: Sat Jul 2 19:40:22 2005 Subject: [Numpy-discussion] subclassing RecordArrays Message-ID: <42C74FDD.9030908@optonline.net> Is there a simple way to subclass RecordArrays? it seems not. suppose i want to have a class RecArrayD(RecArray) that returns a list of its fields, much like a Dict does: recarrayD.keys() the problem is that the fromXXX functions in records.py know only about the one RecArray. and RecArrayD._copyFrom() seems to be of little help, since the new style RecArray has to be framed out first, so to speak. which leaves me to write a new fromRecArray() function that uses the new class. all to get a list of the fields. is this by design, or am i missing something? yes, i can get the keys like so: recarray._names, but i assume that the '_' in front of '_names' is there for a reason, like maybe don't count on it??? or would the designers be willing to add a keys()-like method in for the next release? les schaffer From rowen at u.washington.edu Sat Jul 2 20:31:48 2005 From: rowen at u.washington.edu (Russell E Owen) Date: Sat Jul 2 20:31:48 2005 Subject: [Numpy-discussion] Trouble installing numarray on Solaris In-Reply-To: <1120300999.5263.40.camel@localhost.localdomain> References: <1120300999.5263.40.camel@localhost.localdomain> Message-ID: At 6:43 AM -0400 7/2/05, Todd Miller wrote: >On Fri, 2005-07-01 at 15:02 -0700, Russell E. Owen wrote: >> A user of my application reports that installing numarray 1.3.2 on >> Solaris using Python 2.3.1 has run into trouble: >> >> prompt> python >> Python 2.3.1 (#1, Sep 30 2003, 20:29:58) [C] on sunos5 >> Type "help", "copyright", "credits" or "license" for more information. >> >>> import numarray >> Fatal Python error: Call to API function without first calling >> import_libnumarray() in Src/_convmodule.c >> Abort (core dumped) > >It looks vaguely like an binary API mismatch of some kind... or one of >you numarray modules is stale or broken. To restate the obvious, it's >likely that numarray won't import (in C!) because import_libnumarray() >is very defintely in _convmodule.c. Since it's there and the API >pointer was not initialized, the libnumarray import failed. Here's what the person had to say about it. I found the solution rather puzzling, but perhaps it makes sense to somebody who understands more about such things: Got it working; it was indeed that the shared object modules weren't being linked correctly at build time, so when python tried to load them, they were missing symbols (eg, that are defined in the C math library, for example). Probably the result of compiling numarray w/Solaris's cc rather than gcc. I manually linked the modules for now, and numarry can now be imported... It even passed all its tests. -- Russell From theller at python.net Mon Jul 4 04:43:22 2005 From: theller at python.net (Thomas Heller) Date: Mon Jul 4 04:43:22 2005 Subject: [Numpy-discussion] Re: Trouble installing numarray on Solaris References: <1120300999.5263.40.camel@localhost.localdomain> Message-ID: Todd Miller writes: > On Fri, 2005-07-01 at 15:02 -0700, Russell E. Owen wrote: >> A user of my application reports that installing numarray 1.3.2 on >> Solaris using Python 2.3.1 has run into trouble: >> >> prompt> python >> Python 2.3.1 (#1, Sep 30 2003, 20:29:58) [C] on sunos5 >> Type "help", "copyright", "credits" or "license" for more information. >> >>> import numarray >> Fatal Python error: Call to API function without first calling >> import_libnumarray() in Src/_convmodule.c >> Abort (core dumped) > > It looks vaguely like an binary API mismatch of some kind... or one of > you numarray modules is stale or broken. To restate the obvious, it's > likely that numarray won't import (in C!) because import_libnumarray() > is very defintely in _convmodule.c. Since it's there and the API > pointer was not initialized, the libnumarray import failed. To the numarray authors/maintainers: Isn't it a bit harsh to call Py_FatalError() in the init function? Thomas From jmiller at stsci.edu Mon Jul 4 05:58:32 2005 From: jmiller at stsci.edu (Todd Miller) Date: Mon Jul 4 05:58:32 2005 Subject: [Numpy-discussion] Re: Trouble installing numarray on Solaris In-Reply-To: References: <1120300999.5263.40.camel@localhost.localdomain> Message-ID: <1120481830.5222.23.camel@localhost.localdomain> On Mon, 2005-07-04 at 13:40 +0200, Thomas Heller wrote: > Todd Miller writes: > > > On Fri, 2005-07-01 at 15:02 -0700, Russell E. Owen wrote: > >> A user of my application reports that installing numarray 1.3.2 on > >> Solaris using Python 2.3.1 has run into trouble: > >> > >> prompt> python > >> Python 2.3.1 (#1, Sep 30 2003, 20:29:58) [C] on sunos5 > >> Type "help", "copyright", "credits" or "license" for more information. > >> >>> import numarray > >> Fatal Python error: Call to API function without first calling > >> import_libnumarray() in Src/_convmodule.c > >> Abort (core dumped) > > > > It looks vaguely like an binary API mismatch of some kind... or one of > > you numarray modules is stale or broken. To restate the obvious, it's > > likely that numarray won't import (in C!) because import_libnumarray() > > is very defintely in _convmodule.c. Since it's there and the API > > pointer was not initialized, the libnumarray import failed. > > To the numarray authors/maintainers: Isn't it a bit harsh to call > Py_FatalError() in the init function? That's an interesting observation... the old recourse was to dump core by calling through a NULL function pointer. :-) Since this error can occur in the context of any API function, possibly called from any sub-function within an extension function, I'm not sure there is a soft & general way to handle it: getting it implies the function must fail but that can't always be reported. I'm open to suggestions... it's just not obvious to me how to handle it better. I've concluded that the message is overly specific and may shorten it to just the facts: "call through NULL numarray C-API pointer in ." Regards, Todd From theller at python.net Mon Jul 4 10:01:17 2005 From: theller at python.net (Thomas Heller) Date: Mon Jul 4 10:01:17 2005 Subject: [Numpy-discussion] Re: Trouble installing numarray on Solaris References: <1120300999.5263.40.camel@localhost.localdomain> <1120481830.5222.23.camel@localhost.localdomain> Message-ID: Todd Miller writes: > On Mon, 2005-07-04 at 13:40 +0200, Thomas Heller wrote: >> Todd Miller writes: >> >> > On Fri, 2005-07-01 at 15:02 -0700, Russell E. Owen wrote: >> >> A user of my application reports that installing numarray 1.3.2 on >> >> Solaris using Python 2.3.1 has run into trouble: >> >> >> >> prompt> python >> >> Python 2.3.1 (#1, Sep 30 2003, 20:29:58) [C] on sunos5 >> >> Type "help", "copyright", "credits" or "license" for more information. >> >> >>> import numarray >> >> Fatal Python error: Call to API function without first calling >> >> import_libnumarray() in Src/_convmodule.c >> >> Abort (core dumped) >> > >> > It looks vaguely like an binary API mismatch of some kind... or one of >> > you numarray modules is stale or broken. To restate the obvious, it's >> > likely that numarray won't import (in C!) because import_libnumarray() >> > is very defintely in _convmodule.c. Since it's there and the API >> > pointer was not initialized, the libnumarray import failed. >> >> To the numarray authors/maintainers: Isn't it a bit harsh to call >> Py_FatalError() in the init function? > > That's an interesting observation... the old recourse was to dump core > by calling through a NULL function pointer. :-) > > Since this error can occur in the context of any API function, possibly > called from any sub-function within an extension function, I'm not sure > there is a soft & general way to handle it: getting it implies the > function must fail but that can't always be reported. I'm open to > suggestions... it's just not obvious to me how to handle it better. Maybe I was confusing numarray and numeric. In numeric's _numpymodule.c, function init_numpy(), there is this [...] if (PyErr_Occurred()) goto err; return; err: Py_FatalError("can't initialize module _numpy"); } Sorry for the confusion. > I've concluded that the message is overly specific and may shorten it to > just the facts: "call through NULL numarray C-API pointer in ." Wouldn't it be possible to raise an exception - that would make it easier to find the bug? Thomas From jmiller at stsci.edu Mon Jul 4 17:27:54 2005 From: jmiller at stsci.edu (Todd Miller) Date: Mon Jul 4 17:27:54 2005 Subject: [Numpy-discussion] Re: Trouble installing numarray on Solaris In-Reply-To: References: <1120300999.5263.40.camel@localhost.localdomain> <1120481830.5222.23.camel@localhost.localdomain> Message-ID: <1120523113.4725.17.camel@localhost.localdomain> On Mon, 2005-07-04 at 18:59 +0200, Thomas Heller wrote: > Todd Miller writes: > > > On Mon, 2005-07-04 at 13:40 +0200, Thomas Heller wrote: > >> Todd Miller writes: > >> > >> > On Fri, 2005-07-01 at 15:02 -0700, Russell E. Owen wrote: > >> >> A user of my application reports that installing numarray 1.3.2 on > >> >> Solaris using Python 2.3.1 has run into trouble: > >> >> > >> >> prompt> python > >> >> Python 2.3.1 (#1, Sep 30 2003, 20:29:58) [C] on sunos5 > >> >> Type "help", "copyright", "credits" or "license" for more information. > >> >> >>> import numarray > >> >> Fatal Python error: Call to API function without first calling > >> >> import_libnumarray() in Src/_convmodule.c > >> >> Abort (core dumped) > >> > > >> > It looks vaguely like an binary API mismatch of some kind... or one of > >> > you numarray modules is stale or broken. To restate the obvious, it's > >> > likely that numarray won't import (in C!) because import_libnumarray() > >> > is very defintely in _convmodule.c. Since it's there and the API > >> > pointer was not initialized, the libnumarray import failed. > >> > >> To the numarray authors/maintainers: Isn't it a bit harsh to call > >> Py_FatalError() in the init function? > > > > That's an interesting observation... the old recourse was to dump core > > by calling through a NULL function pointer. :-) > > > > Since this error can occur in the context of any API function, possibly > > called from any sub-function within an extension function, I'm not sure > > there is a soft & general way to handle it: getting it implies the > > function must fail but that can't always be reported. I'm open to > > suggestions... it's just not obvious to me how to handle it better. > > Maybe I was confusing numarray and numeric. In numeric's > _numpymodule.c, function init_numpy(), there is this > > [...] > if (PyErr_Occurred()) goto err; > return; > err: > Py_FatalError("can't initialize module _numpy"); > } > > Sorry for the confusion. > > > I've concluded that the message is overly specific and may shorten it to > > just the facts: "call through NULL numarray C-API pointer in ." > > Wouldn't it be possible to raise an exception - that would make it > easier to find the bug? The two cases where this error have been triggered are: 1. A bug in a C extension module which an author will catch before release. An exception is not worth much there. 2. A corrupted numarray installation which will no longer import. I agree it would be nice there. But... it's hard to raise an exception because the failure macro would have to handle a variety of different return values, even no return value. I don't see how to do it without too much extra complexity for a relatively rare problem. Regards, Todd From faltet at carabos.com Tue Jul 5 09:58:23 2005 From: faltet at carabos.com (Francesc Altet) Date: Tue Jul 5 09:58:23 2005 Subject: [Numpy-discussion] subclassing RecordArrays In-Reply-To: <42C74FDD.9030908@optonline.net> References: <42C74FDD.9030908@optonline.net> Message-ID: <200507051856.28581.faltet@carabos.com> A Sunday 03 July 2005 04:39, Les Schaffer va escriure: > suppose i want to have a class RecArrayD(RecArray) that returns a list > of its fields, much like a Dict does: recarrayD.keys() > > the problem is that the fromXXX functions in records.py know only about > the one RecArray. and RecArrayD._copyFrom() seems to be of little help, > since the new style RecArray has to be framed out first, so to speak. I don't know exactly what do you mean here. If RecArrayD inherits from RecArray, then RecArrayD should be a RecArray object (and also a RecArrayD, of course!). So I would say that the fromXXX fucntions should work just fine for these objects. May you put some example on what are you trying to do?. > yes, i can get the keys like so: recarray._names, but i assume that the > '_' in front of '_names' is there for a reason, like maybe don't count > on it??? Maybe. However I guess that "_names" would not change for some time so it might be safe using it for a while :-P. Nevertheless I agree that "_names" attribute should be promoted to "names" because it's a first-class attribute for RecArray object IMO. Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From schaffer at optonline.net Tue Jul 5 10:36:49 2005 From: schaffer at optonline.net (Les Schaffer) Date: Tue Jul 5 10:36:49 2005 Subject: [Numpy-discussion] subclassing RecordArrays In-Reply-To: <200507051856.28581.faltet@carabos.com> References: <42C74FDD.9030908@optonline.net> <200507051856.28581.faltet@carabos.com> Message-ID: <42CAC4EB.1020304@optonline.net> Francesc Altet wrote: >I don't know exactly what do you mean here. If RecArrayD inherits from >RecArray, then RecArrayD should be a RecArray object (and also a >RecArrayD, of course!). So I would say that the fromXXX fucntions >should work just fine for these objects. May you put some example on >what are you trying to do?. > > consider the calling sequence: <<<<<<<<<<<<<<<<< def array(buffer=None, formats=None, shape=0, names=None, byteorder=sys.byteorder, aligned=0): ... [code to decide which fromXXXX to use] def fromrecords(recList, formats=None, names=None, shape=0, byteorder=sys.byteorder, aligned=0): ... _array = fromarrays(arrlist, formats=formats, shape=_shape, names=names, byteorder=byteorder, aligned=aligned) ... def fromarrays(arrayList, formats=None, names=None, shape=0, byteorder=sys.byteorder, aligned=0): ... _array = RecArray(None, formats=formats, shape=_shape, names=names, byteorder=byteorder, aligned=aligned) >>>>>>>>>>>>>>>>>>>>> there is no way to tell fromXXXX to use my new class. one would need something like: >>>>>>>>>>>>>>>>>>>>>>>>>>>>> def array(buffer=None, formats=None, shape=0, names=None, byteorder=sys.byteorder, aligned=0,rec_class = RecArray ): def fromrecords(recList, formats=None, names=None, shape=0, byteorder=sys.byteorder, aligned=0, rec_class): _array = fromarrays(arrlist, formats=formats, shape=_shape, names=names, byteorder=byteorder, aligned=aligned, rec_class) def fromarrays(arrayList, formats=None, names=None, shape=0, byteorder=sys.byteorder, aligned=0, rec_class): ... _array = rec_class(None, formats=formats, shape=_shape, names=names, byteorder=byteorder, aligned=aligned) <<<<<<<<<<<<<<<<<<<<<<<<<<<<<< THEN i could use my MyRecArrayWithDict class to my hearts content. all that though to have a RecArray.keys() or .names() or .RemindMeWhatWasTheNamesOfTheFields() ???? >Maybe. However I guess that "_names" would not change for some time so >it might be safe using it for a while :-P. Nevertheless I agree that >"_names" attribute should be promoted to "names" because it's a >first-class attribute for RecArray object IMO. > agreed. seems the safest design tho would be to GRAB the "names" entered when instantiating the RecArray (names='XXX, YYY, ...') plus whatever col_N's are added for un-named columns and keep it in an attribute accessible by keys() (or whatever name you like). les schaffer From faltet at carabos.com Tue Jul 5 11:31:03 2005 From: faltet at carabos.com (Francesc Altet) Date: Tue Jul 5 11:31:03 2005 Subject: [Numpy-discussion] subclassing RecordArrays In-Reply-To: <42CAC4EB.1020304@optonline.net> References: <42C74FDD.9030908@optonline.net> <200507051856.28581.faltet@carabos.com> <42CAC4EB.1020304@optonline.net> Message-ID: <200507052028.57782.faltet@carabos.com> A Tuesday 05 July 2005 19:35, Les Schaffer va escriure: > there is no way to tell fromXXXX to use my new class. one would need > something like: I see. Yep, you are right. [snip...] > THEN i could use my MyRecArrayWithDict class to my hearts content. Yes. We encountered similar problems to this that you reported in a subclass of RecArray to support nested columns. We finally ended implementing specific versions of array() and fromarray() factories. Sorry, as I was not the main author of the module, I did not remember that issue. Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From edcjones at comcast.net Tue Jul 5 11:47:29 2005 From: edcjones at comcast.net (Edward C. Jones) Date: Tue Jul 5 11:47:29 2005 Subject: [Numpy-discussion] numarray: Bug printing an object array after unpickling it Message-ID: <42CAD56A.4070707@comcast.net> #! /usr/bin/env python # I have a PC with up-to-date Debian unstable Linux. I have installed # Python 2.4.1 and numarray 1.3.2 from source (not from a Debian # package). The following program fails for both pickle and cPickle: import pickle import numarray, numarray.objects obj = numarray.objects.array([0,0,0,0], (2,2)) print obj s = pickle.dumps(obj, pickle.HIGHEST_PROTOCOL) open('mess2', 'w').write(s) s = open('mess2', 'r').read() obj = pickle.loads(s) print obj # Fails here From faltet at carabos.com Tue Jul 5 11:49:17 2005 From: faltet at carabos.com (Francesc Altet) Date: Tue Jul 5 11:49:17 2005 Subject: [Numpy-discussion] nestedrecords.py module Message-ID: <200507052047.17356.faltet@carabos.com> Hi, In our company we had the need for an extension of RecArray objects so that they would be able to keep nested fields (that is, fields that can have other subfields). The outcome was a module that is intended to do that. We have tested it quite extensively and we think it is quite free of bugs. The conversation with Les has reminded me about it, and thought that it would interesting for the numarray community. I'm attaching a tarball with the next files: - nestedrecords.py -- Where the NestedRecArray lives - nriterators.py -- A suite of iterator utilities for nestedrecords.py - nestedrecords.txt -- Documentation in restructured text The nestedrecords module is mainly a job of Vicent Mas (vmas at carabos.com) and is offered with a BSD license. Of course, we would be glad if this stuff ends in numarray sometime :-) -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" -------------- next part -------------- A non-text attachment was scrubbed... Name: nestedrecords.tar.gz Type: application/x-tgz Size: 12901 bytes Desc: not available URL: From faltet at carabos.com Tue Jul 5 12:01:07 2005 From: faltet at carabos.com (Francesc Altet) Date: Tue Jul 5 12:01:07 2005 Subject: [Numpy-discussion] nestedrecords.py module In-Reply-To: <200507052047.17356.faltet@carabos.com> References: <200507052047.17356.faltet@carabos.com> Message-ID: <200507052058.57780.faltet@carabos.com> Ops, forgot to include the test units in the tarball. Hopefully they are there now. -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" -------------- next part -------------- A non-text attachment was scrubbed... Name: nestedrecords.tar.gz Type: application/x-tgz Size: 16801 bytes Desc: not available URL: From haase at msg.ucsf.edu Tue Jul 5 16:06:27 2005 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue Jul 5 16:06:27 2005 Subject: [Numpy-discussion] patch numarray.fromfuntion with optional type argument In-Reply-To: <200506161112.49144.haase@msg.ucsf.edu> References: <200506161112.49144.haase@msg.ucsf.edu> Message-ID: <200507051604.34642.haase@msg.ucsf.edu> Hi, (this is regarding my post from 16 June 2005) Todd, maybe you can confirm that this patch is a good idea and that you will put it into upcoming numarray versions !? I would like to streamline my code - but only of course if you OK this. Sebastian On Thursday 16 June 2005 11:12, Sebastian Haase wrote: > Hi, > Please tell if this patch is a good idea. (Use: For large image data we > always use Float32 and I thought it would be extra overhead if I first > create everything in Float64 and then have to convert to Float32 > afterwards, myself) > > haase at gorilla:~: diff -p ~/myCVS/numarray/Lib/generic.py > ~/myCVS/numarray/Lib/generic.py.~1.74.~ > *** /home/haase/myCVS/numarray/Lib/generic.py Thu Jun 16 11:04:10 2005 > --- /home/haase/myCVS/numarray/Lib/generic.py.~1.74.~ Fri Apr 22 13:35:26 > 2005 > *************** def indices(shape, type=None): > *** 1167,1179 **** > a = a.astype(type) > return a > > ! def fromfunction(function, dimensions, type=None): # from Numeric > """fromfunction() returns an array constructed by calling function > on a tuple of number grids. The function should accept as many > arguments as there are dimensions which is a list of numbers > indicating the length of the desired output for each axis. > """ > ! return apply(function, tuple(indices(dimensions,type))) > > def _broadcast_or_resize(a, b): > try: > --- 1167,1179 ---- > a = a.astype(type) > return a > > ! def fromfunction(function, dimensions): # from Numeric > """fromfunction() returns an array constructed by calling function > on a tuple of number grids. The function should accept as many > arguments as there are dimensions which is a list of numbers > indicating the length of the desired output for each axis. > """ > ! return apply(function, tuple(indices(dimensions))) > > def _broadcast_or_resize(a, b): > try: > > Thanks, > - Sebastian Haase > > > ------------------------------------------------------- > SF.Net email is sponsored by: Discover Easy Linux Migration Strategies > from IBM. Find simple to follow Roadmaps, straightforward articles, > informative Webcasts and more! Get everything you need to get up to > speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From haase at msg.ucsf.edu Tue Jul 5 17:37:01 2005 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue Jul 5 17:37:01 2005 Subject: [Numpy-discussion] unexpected behavior of na.where (no short-circuiting) Message-ID: <200507051734.59020.haase@msg.ucsf.edu> Hi, I was very surprised when I got this warning: >>> a = na.arange(4)-2 >>> na.where(a != 0,1./a, 999) Warning: Encountered divide by zero(s) in divide [ -0.5 -1. 999. 1. ] Then I realized that this is generally referred to as (not) "short circuiting" (e.g. in the case of the '?:'-C-operator, the middle part never gets evaluated at all if the first part evals to 0 ) Especially annoying was this because (for debugging) I had set this error-mode: >>> na.Error.setMode(dividebyzero="error") My questions are: a) Did other people encounter this problem ? b) What is the general feeling about this actually being a "problem" ? c) Could this (at all possible) be implemented differently ? Thanks, Sebastian Haase From jmiller at stsci.edu Wed Jul 6 05:58:42 2005 From: jmiller at stsci.edu (Todd Miller) Date: Wed Jul 6 05:58:42 2005 Subject: [Numpy-discussion] patch numarray.fromfuntion with optional type argument In-Reply-To: <200507051604.34642.haase@msg.ucsf.edu> References: <200506161112.49144.haase@msg.ucsf.edu> <200507051604.34642.haase@msg.ucsf.edu> Message-ID: <1120654644.5565.15.camel@localhost.localdomain> On Tue, 2005-07-05 at 16:04 -0700, Sebastian Haase wrote: > Hi, (this is regarding my post from 16 June 2005) > Todd, maybe you can confirm that this patch is a good idea and that you will > put it into upcoming numarray versions !? > I would like to streamline my code - but only of course if you OK this. > > Sebastian I added a 'type' parameter to fromfunction() and committed it to CVS this morning. It'll be in 1.3.3. Regards, Todd From jmiller at stsci.edu Wed Jul 6 06:10:44 2005 From: jmiller at stsci.edu (Todd Miller) Date: Wed Jul 6 06:10:44 2005 Subject: [Numpy-discussion] Trouble installing numarray on Solaris In-Reply-To: References: <1120300999.5263.40.camel@localhost.localdomain> Message-ID: <1120655414.5565.26.camel@localhost.localdomain> On Sat, 2005-07-02 at 20:30 -0700, Russell E Owen wrote: > At 6:43 AM -0400 7/2/05, Todd Miller wrote: > >On Fri, 2005-07-01 at 15:02 -0700, Russell E. Owen wrote: > >> A user of my application reports that installing numarray 1.3.2 on > >> Solaris using Python 2.3.1 has run into trouble: > >> > >> prompt> python > >> Python 2.3.1 (#1, Sep 30 2003, 20:29:58) [C] on sunos5 > >> Type "help", "copyright", "credits" or "license" for more information. > >> >>> import numarray > >> Fatal Python error: Call to API function without first calling > >> import_libnumarray() in Src/_convmodule.c > >> Abort (core dumped) > > > >It looks vaguely like an binary API mismatch of some kind... or one of > >you numarray modules is stale or broken. To restate the obvious, it's > >likely that numarray won't import (in C!) because import_libnumarray() > >is very defintely in _convmodule.c. Since it's there and the API > >pointer was not initialized, the libnumarray import failed. > > Here's what the person had to say about it. I found the solution > rather puzzling, but perhaps it makes sense to somebody who > understands more about such things: > > Got it working; it was indeed that the shared object modules weren't > being linked correctly at build time, so when python tried to load > them, they were missing symbols (eg, that are defined in the C math > library, for example). Probably the result of compiling numarray > w/Solaris's cc rather than gcc. I manually linked the modules for > now, and numarry can now be imported... It even passed all its tests. One thing I should have mentioned earlier is that numarray has been built at STScI using Solaris's cc from the beginning. Although our primary development platform is now x86 Linux, we still ultimately deploy to Solaris so I think Solaris cc per-say is probably not the problem here. Here are the tools I use personally w/o problems: basil> uname -a SunOS anysun.stsci.edu 5.8 Generic_117350-18 sun4u sparc SUNW,Ultra-5_10 basil> cc -V cc: Sun WorkShop 6 2000/06/19 C 5.1 Patch 109491-02 Regards, Todd From cjw at sympatico.ca Wed Jul 6 08:58:38 2005 From: cjw at sympatico.ca (Colin J. Williams) Date: Wed Jul 6 08:58:38 2005 Subject: [Numpy-discussion] _fastCopyAndTranpose Message-ID: <42CBFF51.4030606@sympatico.ca> numarray.LinearAlgebra2.py has # _fastCopyAndTranpose is an optimized version of _castCopyAndTranspose. # It assumes the input is 2D (as all the calls in here are). However the following code is commented out. Is this considered a satisactory replacement for _castCopyAndTranspose? Colin W From rowen at u.washington.edu Wed Jul 6 16:24:36 2005 From: rowen at u.washington.edu (Russell E Owen) Date: Wed Jul 6 16:24:36 2005 Subject: [Numpy-discussion] Trouble installing numarray on Solaris In-Reply-To: <1120655414.5565.26.camel@localhost.localdomain> References: <1120300999.5263.40.camel@localhost.localdomain> <1120655414.5565.26.camel@localhost.localdomain> Message-ID: At 9:10 AM -0400 2005-07-06, Todd Miller wrote: >One thing I should have mentioned earlier is that numarray has been >built at STScI using Solaris's cc from the beginning. Although our >primary development platform is now x86 Linux, we still ultimately >deploy to Solaris so I think Solaris cc per-say is probably not the >problem here. > >Here are the tools I use personally w/o problems: > >basil> uname -a >SunOS anysun.stsci.edu 5.8 Generic_117350-18 sun4u sparc SUNW,Ultra-5_10 >basil> cc -V >cc: Sun WorkShop 6 2000/06/19 C 5.1 Patch 109491-02 I asked John for more information and got the following: --- begin quote --- First: The numarray modules (shared library/object files) weren't being "linked" against any libraries when built, and as a result, any symbols (functions) that they referenced which are in shared libraries not already being loaded by python itself would go unresolved whenever python tried to load the modules. In particular, my dynamically linked copy of python (which I compiled from source, btw) isn't linked against the C math library, so any math functions being used by numarray were unresolved when the modules were loaded. The fix was to add a "-lm" to the command being used to link the module, eg, change: cc -G build/temp.solaris-2.7-sun4u-2.3/Src/_ufuncFloat32module.o -o \ build/lib.solaris-2.7-sun4u-2.3/numarray/_ufuncFloat32.so to cc -G build/temp.solaris-2.7-sun4u-2.3/Src/_ufuncFloat32module.o -o \ build/lib.solaris-2.7-sun4u-2.3/numarray/_ufuncFloat32.so -lm With that, now whenever this module gets loaded, the dynamic loader routines will automatically pull in any required routines from the math library (libm.so). Given the time constraints and my lack of python experience, I couldn't figure out how to feed the standard "python setup.py build" command a list of libraries to link against, so instead I just let setup.py build it as is, logged the commands used to create the shared objects, and then turned the log into a script to relink the objects after manually added a "-lm" to the end of each link command. Looking through distutils now, I think I can feed setup.py a "--libraries" option with the name of the math library, and with that, it will then build/link the modules correctly. > Before I...pass it along, though, I just thought of something: you > did specify the --gencode argument, didn't you? The need for that > argument is an unwelcome numarray oddity. Yes I did, although I wasn't sure why that was needed. BTW, PIL built OK, and likewise, its modules link against libraries which python doesn't automatically pull in itself (eg, libjpeg, zlib, etc), so something in its setup.py is automatically including these libraries when it builds/links the shared objects. Another datapoint: I *think* (or at least it used to be the case) that gcc always includes the math library when linking, whereas with Sun's (and may other vendors') compilers, -lm needs to be given explicitly. In general, I usually try to compile things with an OS's/vendor's native compilers instead of gcc, just to keep developers honest (ie, in the interest of weeding out compiler dependencies/bugs that inhibit source portability). :) I'm guessing that if I had compiled w/ gcc, I wouldn't have had this problem. However, I had compiled python and the other python modules I have installed with Sun's cc (from Sun's Studio 8 compiler collection, btw), so for consistency, I wouldn't have wanted to use gcc anyway. --- end quote --- Unless you have any questions, that's the last I'll say on the subject. I hope something in here may help track down a bug. Regards, -- Russell From jmiller at stsci.edu Thu Jul 7 04:34:52 2005 From: jmiller at stsci.edu (Todd Miller) Date: Thu Jul 7 04:34:52 2005 Subject: [Numpy-discussion] Trouble installing numarray on Solaris In-Reply-To: References: <1120300999.5263.40.camel@localhost.localdomain> <1120655414.5565.26.camel@localhost.localdomain> Message-ID: <1120736002.4694.25.camel@localhost.localdomain> On Wed, 2005-07-06 at 16:22 -0700, Russell E Owen wrote: > Unless you have any questions, that's the last I'll say on the > subject. I hope something in here may help track down a bug. > > Regards, > > -- Russell I added "-lm" globally to the extra link arguments... hopefully that should take care of it. Regards, Todd From jmiller at stsci.edu Sat Jul 9 10:26:15 2005 From: jmiller at stsci.edu (Todd Miller) Date: Sat Jul 9 10:26:15 2005 Subject: [Numpy-discussion] ANN: numarray-1.3.3 Message-ID: <1120929968.4884.2.camel@localhost.localdomain> Numarray is an array processing package designed to efficiently manipulate large multi-dimensional arrays. Numarray is modelled after Numeric and features c-code generated from python template scripts, the capacity to operate directly on arrays in files, arrays of heterogeneous records, string arrays, and in-place operation on memory mapped files. ========================================================================= Release Notes for numarray-1.3.3 I. ENHANCEMENTS None. 1.3.3 fixes a fatal problem with embedding numarray and loading it more than once. It also fixes some numarray memory leaks associated with restarting the Python interpreter. II. BUGS FIXED / CLOSED 1233431 Add 'type' to fromfunction() 1230490 Numarray: integer overflow in sum() is not detected 1216688 ufunc's ignoring complex operand 1213675 Problems embedding after 1.1.1 1209946 Mac OS-X --use_lapack 1212975 numarray.fromfunction(...) misbehavior 1209929 config.h and Python-2.2 1235278 Add -lm to numarray link lists See http://sourceforge.net/tracker/?atid=450446&group_id=1369&func=browse for more details. ========================================================================= Release Notes for numarray-1.3.2 I. ENHANCEMENTS None. 1.3.2 fixes the problem with gcc4 on Mac OS-X 10.4 Tiger II. BUGS FIXED / CLOSED 1193125 Build failure on Mac OS 10.4 (Tiger) 1195370 numarray1.3.1 setup.py broken See http://sourceforge.net/tracker/?atid=450446&group_id=1369&func=browse for more details. ========================================================================= Release Notes for numarray-1.3.1 I. ENHANCEMENTS None. 1.3.1 fixes the problem with gcc-3.4.3 II. BUGS FIXED / CLOSED 1152323 /usr/include/fenv.h:96: error: conflicting types for 'fegete 1185024 numarray-1.2.3 fails to compile with gcc-3.4.3 1187162 Numarray 1.3.0 installation failure See http://sourceforge.net/tracker/?atid=450446&group_id=1369&func=browse for more details. ========================================================================= Release Notes for numarray-1.3.0 Numarray is an array processing package designed to efficiently manipulate large multi-dimensional arrays. Numarray is modelled after Numeric and features c-code generated from python template scripts, the capacity to operate directly on arrays in files, arrays of heterogeneous records, string arrays, and in-place operation on memory mapped files. I. ENHANCEMENTS 1. Migration of NumArray.__del__ to C (tp_dealloc). Overall performance. 2. Removal of dictionary update from array view creation improves performance of view/slice/subarray creation. This should e.g. improve the performance of wxPython sequence protocol access to Nx2 arrays. Subclasses now need to do a.flags |= numarray.generic._UPDATEDICT to ensure that dictionary based attributes are inherited by views. NumArrays no longer do this by default. 2. Modifications to support scipy.special. 3. Removal of an unnecessary getattr() from ufunc calling sequence. Ufunc performance. II. BUGS FIXED / CLOSED 1179355 average() broken in numarray 1.2.3 1167184 Floating point exception in numarray's dot() 1151892 Bug in matrixmultiply with zero size arrays 1160184 RecArray reversal 1156172 Incorect error message for shape incompatability 1155538 Incorrect error message when multiplying arrays See http://sourceforge.net/tracker/?atid=450446&group_id=1369&func=browse for more details. III. CAUTIONS This release should be backward binary compatible with numarray-1.1.1 and 1.2.3. WHERE ----------- Numarray-1.3.0 windows executable installers, source code, and manual is here: http://sourceforge.net/project/showfiles.php?group_id=1369 Numarray is hosted by Source Forge in the same project which hosts Numeric: http://sourceforge.net/projects/numpy/ The web page for Numarray information is at: http://stsdas.stsci.edu/numarray/index.html Trackers for Numarray Bugs, Feature Requests, Support, and Patches are at the Source Forge project for NumPy at: http://sourceforge.net/tracker/?group_id=1369 REQUIREMENTS ------------------------------ numarray-1.3.0 requires Python 2.2.2 or greater. AUTHORS, LICENSE ------------------------------ Numarray was written by Perry Greenfield, Rick White, Todd Miller, JC Hsu, Paul Barrett, Phil Hodge at the Space Telescope Science Institute. We'd like to acknowledge the assitance of Francesc Alted, Paul Dubois, Sebastian Haase, Chuck Harris, Tim Hochberg, Nadav Horesh, Edward C. Jones, Eric Jones, Jochen Kuepper, Travis Oliphant, Pearu Peterson, Peter Verveer, Colin Williams, Rory Yorke, and everyone else who has contributed with comments and feedback. Numarray is made available under a BSD-style License. See LICENSE.txt in the source distribution for details. -- Todd Miller jmiller at stsci.edu From jmiller at stsci.edu Sun Jul 10 18:21:58 2005 From: jmiller at stsci.edu (Todd Miller) Date: Sun Jul 10 18:21:58 2005 Subject: [Numpy-discussion] ANN: numarray-1.3.3 In-Reply-To: <42D14226.4050701@optonline.net> References: <1120929968.4884.2.camel@localhost.localdomain> <42D00C20.6010402@optonline.net> <1120998167.4921.52.camel@localhost.localdomain> <42D14226.4050701@optonline.net> Message-ID: <1121044171.4694.31.camel@localhost.localdomain> On Sun, 2005-07-10 at 11:43 -0400, Les Schaffer wrote: > Todd Miller wrote: > > >numarray-1.3.3 is a "maintenance release" which solves a couple esoteric > >(embedded Python + numarray) but serious (core dumping) problems. > > > > > > > cool > > >I think there's a good chance of adding something for the next release > >but I don't think it will be called keys() because that's a dictionary > >cue and RecArrays aren't dictionaries. > > > > > > tho as column spaces, they essentially act as dicts. > > >I'm in favor of adding .names to make _names public. > > > > > that would work, at least it tells outsiders .names is public and stable > attribute. > > and even better is help us subclass easily by making the constructor > functions fromarray, fromrecord, and array take an optional argument > which is the RecArray class to use. as it stands RecArray cannot be > subclasses without re-doing all those constructor functions. > > i would be happy to contribute code if i knew it would be used. What you want to do sounds reasonable so chances are good a patch would be used. I would: 1. Make .names another reference to ._names 2. Either convert the functions you want to be subclassable into "class functions" (and add backward compatible globals) or else add a "kind" parameter for passing in the subclass (see strings.py) to the current functions. I'm not sure which approach is better. Regards, Todd From gr at grrrr.org Mon Jul 11 01:51:01 2005 From: gr at grrrr.org (Thomas Grill) Date: Mon Jul 11 01:51:01 2005 Subject: [Numpy-discussion] optimizations In-Reply-To: <1120929968.4884.2.camel@localhost.localdomain> References: <1120929968.4884.2.camel@localhost.localdomain> Message-ID: <78581171a9e444def1705b4a1eb5067c@grrrr.org> Hi all, i'm using numarrays with audio and video processing with the real-time framework pure data (http://www.puredata.org). I realized that the ufuncs in numarrays are not as fast as possible because they are coded very straightforward. Is there any particular reason why this is the case, like cleanness of code, confidence in the compiler or because the code was automatically generated? I would like to contribute assembly SIMD codelets (SSE and Altivec), i have been successfully using in my projects for quite some time. Is it feasible to submit patches, so that these go into the main numarray distribution, or should i rather implement this as separate ufuncs? best greetings, Thomas From onlinebanking at chevychasebank.com Mon Jul 11 06:17:11 2005 From: onlinebanking at chevychasebank.com (Chevy Chase Bank) Date: Mon Jul 11 06:17:11 2005 Subject: [Numpy-discussion] Online Banking - Verify Your Account Information Message-ID: An HTML attachment was scrubbed... URL: From curzio.basso at gmail.com Mon Jul 11 10:40:18 2005 From: curzio.basso at gmail.com (Curzio Basso) Date: Mon Jul 11 10:40:18 2005 Subject: [Numpy-discussion] bug or feature? Message-ID: Hi all, I noticed the following (unexpected for me) behaviour: >>> a = NA.arange(12) >>> a.shape = (3,4) >>> a[[0,2]][:,:2] = a[[0,2]][:,2:] >>> print a array([[ 0, 1, 2, 3], [ 4, 5, 6, 7], [ 8, 9, 10, 11]]) rather than array([[ 2, 3, 2, 3], [ 4, 5, 6, 7], [ 10, 11, 10, 11]]) as I was expecting. This happens only if I try to combine indices lists with slices. Is it by design or I ran into a bug? curzio From perry at stsci.edu Mon Jul 11 10:53:35 2005 From: perry at stsci.edu (Perry Greenfield) Date: Mon Jul 11 10:53:35 2005 Subject: [Numpy-discussion] bug or feature? In-Reply-To: References: Message-ID: <5deb92a4eb50a0ed681dbf3e8d8e3a50@stsci.edu> On Jul 11, 2005, at 1:38 PM, Curzio Basso wrote: > Hi all, > > I noticed the following (unexpected for me) behaviour: > >>>> a = NA.arange(12) >>>> a.shape = (3,4) >>>> a[[0,2]][:,:2] = a[[0,2]][:,2:] >>>> print a > array([[ 0, 1, 2, 3], > [ 4, 5, 6, 7], > [ 8, 9, 10, 11]]) > > rather than > > array([[ 2, 3, 2, 3], > [ 4, 5, 6, 7], > [ 10, 11, 10, 11]]) > > as I was expecting. This happens only if I try to combine indices > lists with slices. > Is it by design or I ran into a bug? > > curzio > An array used as an index produces a new copy (if you think about it, given the possibilities, there isn't much else that can be done). So the entity on the left of your expression is modified, but since nothing refers to it, it is discarded immediately thereafter. The array 'a' itself was never changed. I'd say that it isn't either a bug or feature. Perry Greenfield From perry at stsci.edu Mon Jul 11 12:24:24 2005 From: perry at stsci.edu (Perry Greenfield) Date: Mon Jul 11 12:24:24 2005 Subject: [Numpy-discussion] bug or feature? In-Reply-To: References: <5deb92a4eb50a0ed681dbf3e8d8e3a50@stsci.edu> Message-ID: <23974d4bfbd55596f8212bbfd6ff6723@stsci.edu> On Jul 11, 2005, at 2:38 PM, Curzio Basso wrote: >> An array used as an index produces a new copy (if you think about it, >> given the possibilities, there isn't much else that can be done). So >> the entity on the left of your expression is modified, but since >> nothing refers to it, it is discarded immediately thereafter. The >> array >> 'a' itself was never changed. I'd say that it isn't either a bug or >> feature. > > ok. on the other hand: > >>>> a[[0,2],0] = 1 >>>> print a > array([[ 1, 1, 2, 3], > [ 4, 5, 6, 7], > [ 1, 9, 10, 11]]) > > I have some problem understanding the difference at the implementation > level between the two cases. I would have assumed that at end they > would operate on a similar way... > > curzio I was sloppy in my description. For array indexing, assignment does modify the original array (as your case above illustrates), but once you sliced the array-indexed array, it changed the context of the indexing. I.e., (to take a simpler, 1-d array x) x[[0,2]] = 1 # modifies x This form results in __setitem__ being called on x x[[0,2]][:4] = 1 # doesn't modify x This form results in __getitem__ being called on x (and thus producing a new copy of the array) for the array indexing and __setitem__ being called for the slice. At least that's what I think is happening. Todd or someone more recently familiar with how Python handles this can correct me if I'm wrong. Perry From fccoelho at gmail.com Tue Jul 12 13:34:21 2005 From: fccoelho at gmail.com (Flavio Coelho) Date: Tue Jul 12 13:34:21 2005 Subject: [Numpy-discussion] problem with poisson generators Message-ID: Hi, I am having problems with the poisson random number generators of both Numarray and Numeric. I can't replicate it when calling the function from the python cosonle, but it has consistently malfunctioned when used within one of my scripts. What happens is that it frequently return a value greater than zero when called with zero mean: poisson(0.0) Unfortunately My program is too big to send attached but I have confirmed the malfunction by printing both the mean and the result whenever it spits out a wrong result. This is very weird indeed, I have run poisson millions of times by itsel on the python console, without any problems... I hope it is some stupid mistake, but when I replace the poisson function call within my program by the R equivalent command (rpois) via the rpy wrapper, everything works just fine... any Ideas? -- Fl?vio Code?o Coelho registered Linux user # 386432 --------------------------- Great spirits have always found violent opposition from mediocrities. The latter cannot understand it when a man does not thoughtlessly submit to hereditary prejudices but honestly and courageously uses his intelligence. Albert Einstein -------------- next part -------------- An HTML attachment was scrubbed... URL: From jdhunter at ace.bsd.uchicago.edu Tue Jul 12 13:54:47 2005 From: jdhunter at ace.bsd.uchicago.edu (John Hunter) Date: Tue Jul 12 13:54:47 2005 Subject: [Numpy-discussion] problem with poisson generators In-Reply-To: (Flavio Coelho's message of "Tue, 12 Jul 2005 17:32:25 -0300") References: Message-ID: <87eka39281.fsf@peds-pc311.bsd.uchicago.edu> >>>>> "Flavio" == Flavio Coelho writes: Flavio> This is very weird indeed, I have run poisson millions of Flavio> times by itsel on the python console, without any Flavio> problems... Does your code include any custom or non-standard extension code? If so, you could have a pointer error somewhere which is fouling up the ranlib memory layout and causing the error. If you can't reproduce the error with a pure python + Numeric script, this may be the explanation. JDH From haase at msg.ucsf.edu Tue Jul 12 14:51:55 2005 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue Jul 12 14:51:55 2005 Subject: [Numpy-discussion] problem with poisson generators In-Reply-To: References: Message-ID: <200507121447.01356.haase@msg.ucsf.edu> Hi Flavio! I had reported this long time ago and this list (about numarray). Somehow this got more or less dismissed. If I recall correctly the argument was that nobody could reproduce it (I ran this on Debian Woody ,py2.2, (with CVS numarray at the time). I ended up writting my own wrapper(s): def poissonArr(shape=defshape, mean=1): from numarray import random_array as ra if mean == 0: return zeroArrF(shape) elif mean < 0: raise "poisson not defined for mean < 0" else: return ra.poisson(mean, shape).astype(na.UInt16) def poissonize(arr): from numarray import random_array as ra return na.where(arr<=0, 0, ra.poisson(arr)).astype(na.UInt16) (I use the astype(na.UInt16) because of some OpenGL code) Just last week had this problem on a windows98 computer (python2.4). This should get sorted out ... Thanks for reporting this problem. Sebastian Haase On Tuesday 12 July 2005 13:32, Flavio Coelho wrote: > Hi, > > I am having problems with the poisson random number generators of both > Numarray and Numeric. > I can't replicate it when calling the function from the python cosonle, but > it has consistently malfunctioned when used within one of my scripts. > > What happens is that it frequently return a value greater than zero when > called with zero mean: poisson(0.0) > > Unfortunately My program is too big to send attached but I have confirmed > the malfunction by printing both the mean and the result whenever it spits > out a wrong result. > > This is very weird indeed, I have run poisson millions of times by itsel on > the python console, without any problems... > > I hope it is some stupid mistake, but when I replace the poisson function > call within my program by the R equivalent command (rpois) via the rpy > wrapper, everything works just fine... > > any Ideas? From fccoelho at gmail.com Wed Jul 13 04:06:16 2005 From: fccoelho at gmail.com (Flavio Coelho) Date: Wed Jul 13 04:06:16 2005 Subject: [Numpy-discussion] problem with poisson generators In-Reply-To: <87eka39281.fsf@peds-pc311.bsd.uchicago.edu> References: <87eka39281.fsf@peds-pc311.bsd.uchicago.edu> Message-ID: No, its a pure python code... 2005/7/12, John Hunter : > > >>>>> "Flavio" == Flavio Coelho writes: > > Flavio> This is very weird indeed, I have run poisson millions of > Flavio> times by itsel on the python console, without any > Flavio> problems... > > Does your code include any custom or non-standard extension code? If > so, you could have a pointer error somewhere which is fouling up the > ranlib memory layout and causing the error. If you can't reproduce > the error with a pure python + Numeric script, this may be the > explanation. > > > > JDH > -- Fl?vio Code?o Coelho registered Linux user # 386432 --------------------------- Great spirits have always found violent opposition from mediocrities. The latter cannot understand it when a man does not thoughtlessly submit to hereditary prejudices but honestly and courageously uses his intelligence. Albert Einstein -------------- next part -------------- An HTML attachment was scrubbed... URL: From fccoelho at gmail.com Wed Jul 13 04:22:05 2005 From: fccoelho at gmail.com (Flavio Coelho) Date: Wed Jul 13 04:22:05 2005 Subject: [Numpy-discussion] problem with poisson generators In-Reply-To: <200507121447.01356.haase@msg.ucsf.edu> References: <200507121447.01356.haase@msg.ucsf.edu> Message-ID: Well, I am definetly glad that someone has also stumbled onto the same problem. But it is nevertheless intriguing, that you can run poisson a million times with mean zero or negative(it assumes zero mean inthis case) without any problems by itself. But when I call it within my code, the rate of error is very high (I would say it returns a wrong result every time, but I haven't checked every answer). Meanwhile, my solution will be: import rpy n = rpy.r.rpois(n,mean) I don't feel I can trust poisson while this "funny" behavior is still there... If someone has any Idea of how I could trace this bug please tell me, and I'll hunt it down. At least I can reproduce it in a very specific context. thanks, Fl?vio 2005/7/12, Sebastian Haase : > > Hi Flavio! > I had reported this long time ago and this list (about numarray). > Somehow this got more or less dismissed. If I recall correctly the > argument > was that nobody could reproduce it (I ran this on Debian Woody ,py2.2, > (with > CVS numarray at the time). > > I ended up writting my own wrapper(s): > def poissonArr(shape=defshape, mean=1): > from numarray import random_array as ra > if mean == 0: > return zeroArrF(shape) > elif mean < 0: > raise "poisson not defined for mean < 0" > else: > return ra.poisson(mean, shape).astype(na.UInt16) > > def poissonize(arr): > from numarray import random_array as ra > return na.where(arr<=0, 0, ra.poisson(arr)).astype(na.UInt16) > > (I use the astype(na.UInt16) because of some OpenGL code) > > Just last week had this problem on a windows98 computer (python2.4). > > This should get sorted out ... > > Thanks for reporting this problem. > Sebastian Haase > > > > > On Tuesday 12 July 2005 13:32, Flavio Coelho wrote: > > Hi, > > > > I am having problems with the poisson random number generators of both > > Numarray and Numeric. > > I can't replicate it when calling the function from the python cosonle, > but > > it has consistently malfunctioned when used within one of my scripts. > > > > What happens is that it frequently return a value greater than zero when > > called with zero mean: poisson(0.0) > > > > Unfortunately My program is too big to send attached but I have > confirmed > > the malfunction by printing both the mean and the result whenever it > spits > > out a wrong result. > > > > This is very weird indeed, I have run poisson millions of times by itsel > on > > the python console, without any problems... > > > > I hope it is some stupid mistake, but when I replace the poisson > function > > call within my program by the R equivalent command (rpois) via the rpy > > wrapper, everything works just fine... > > > > any Ideas? > -- Fl?vio Code?o Coelho registered Linux user # 386432 --------------------------- Great spirits have always found violent opposition from mediocrities. The latter cannot understand it when a man does not thoughtlessly submit to hereditary prejudices but honestly and courageously uses his intelligence. Albert Einstein -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsouthey at gmail.com Wed Jul 13 06:20:42 2005 From: bsouthey at gmail.com (Bruce Southey) Date: Wed Jul 13 06:20:42 2005 Subject: [Numpy-discussion] problem with poisson generators In-Reply-To: References: <200507121447.01356.haase@msg.ucsf.edu> Message-ID: Hi, What is the seed used? It is really important that you set the seed? Did you build Python and numarray from source? Can you reduce your code to a few lines that have the problem? Ranlib uses static floats so it may relate to numerical precision (as well as the random uniform random number generator). Really the only way is for you to debug the ranlib.c file Note that R provides a standalone math library in C that includes the random number generators so you could you those instead. SWIG handles it rather well. Regards Bruce On 7/13/05, Flavio Coelho wrote: > Well, > I am definetly glad that someone has also stumbled onto the same problem. > > But it is nevertheless intriguing, that you can run poisson a million times > with mean zero or negative(it assumes zero mean inthis case) without any > problems by itself. But when I call it within my code, the rate of error is > very high (I would say it returns a wrong result every time, but I haven't > checked every answer). > > Meanwhile, my solution will be: > > import rpy > > n = rpy.r.rpois(n,mean) > > I don't feel I can trust poisson while this "funny" behavior is still > there... > If someone has any Idea of how I could trace this bug please tell me, and > I'll hunt it down. At least I can reproduce it in a very specific context. > > thanks, > > Fl?vio > > 2005/7/12, Sebastian Haase : > > Hi Flavio! > > I had reported this long time ago and this list (about numarray). > > Somehow this got more or less dismissed. If I recall correctly the > argument > > was that nobody could reproduce it (I ran this on Debian Woody , py2.2, > (with > > CVS numarray at the time). > > > > I ended up writting my own wrapper(s): > > def poissonArr(shape=defshape, mean=1): > > from numarray import random_array as ra > > if mean == 0: > > return zeroArrF(shape) > > elif mean < 0: > > raise "poisson not defined for mean < 0" > > else: > > return ra.poisson(mean, shape).astype(na.UInt16) > > > > def poissonize(arr): > > from numarray import random_array as ra > > return na.where(arr<=0, 0, ra.poisson(arr)).astype(na.UInt16) > > > > (I use the astype( na.UInt16) because of some OpenGL code) > > > > Just last week had this problem on a windows98 computer (python2.4). > > > > This should get sorted out ... > > > > Thanks for reporting this problem. > > Sebastian Haase > > > > > > > > > > On Tuesday 12 July 2005 13:32, Flavio Coelho wrote: > > > Hi, > > > > > > I am having problems with the poisson random number generators of both > > > Numarray and Numeric. > > > I can't replicate it when calling the function from the python cosonle, > but > > > it has consistently malfunctioned when used within one of my scripts. > > > > > > What happens is that it frequently return a value greater than zero when > > > called with zero mean: poisson(0.0) > > > > > > Unfortunately My program is too big to send attached but I have > confirmed > > > the malfunction by printing both the mean and the result whenever it > spits > > > out a wrong result. > > > > > > This is very weird indeed, I have run poisson millions of times by itsel > on > > > the python console, without any problems... > > > > > > I hope it is some stupid mistake, but when I replace the poisson > function > > > call within my program by the R equivalent command (rpois) via the rpy > > > wrapper, everything works just fine... > > > > > > any Ideas? > > > > > > -- > > Fl?vio Code?o Coelho > registered Linux user # 386432 > --------------------------- > Great spirits have always found violent opposition from mediocrities. The > latter cannot understand it when a man does not thoughtlessly submit to > hereditary prejudices but honestly and courageously uses his intelligence. > Albert Einstein > From fccoelho at gmail.com Wed Jul 13 08:17:21 2005 From: fccoelho at gmail.com (Flavio Coelho) Date: Wed Jul 13 08:17:21 2005 Subject: [Numpy-discussion] problem with poisson generators In-Reply-To: References: <200507121447.01356.haase@msg.ucsf.edu> Message-ID: 2005/7/13, Bruce Southey : > > Hi, > What is the seed used? I am not setting the seed. It is really important that you set the seed? No. Did you build Python and numarray from source? Yes, I use Gentoo. I build everything from source. Can you reduce your code to a few lines that have the problem? Not really, poisson and binomial are the only two functions from Numeric that I use but they are called inside a rather complex object oriented code environment (objects within objetcs, being called recursively...) Bellow is the function within which poisson is called: def stepSEIR_s(self,inits=(0,0,0),par=(0.001,1,1,0.5,1,0,0,0),theta=0, npass=0,dist='poisson'): """ Defines an stochastic model SEIR: - inits = (E,I,S) - par = (Beta, alpha, E,r,delta,B,w,p) see docs. - theta = infectious individuals from neighbor sites """ E,I,S = inits N = self.parentSite.totpop beta,alpha,e,r,delta,B,w,p = par Lpos_esp = beta*S*((I+theta)/(N+npass))**alpha #Number of new cases if dist == 'poisson': Lpos = poisson(Lpos_esp) ## if theta == 0 and Lpos_esp == 0 and Lpos > 0: ## print Lpos,Lpos_esp,S,I,theta,N,self.parentSite.sitename elif dist == 'negbin': prob = I/(I+Lpos_esp) #convertin between parameterizations Lpos = negative_binomial(I,prob) self.parentSite.totalcases += Lpos #update number of cases Epos = (1-e)*E + Lpos Ipos = e*E + (1-r)*I Spos = S + B - Lpos Rpos = N-(Spos+Epos+Ipos) #self.parentSite.totpop = Spos+Epos+Ipos+Rpos self.parentSite.incidence.append(Lpos) if not self.parentSite.infected: if Lpos > 0: self.parentSite.infected = self.parentSite.parentGraph.simstep self.parentSite.parentGraph.epipath.append(( self.parentSite.parentGraph.simstep,self.parentSite,self.parentSite.infector )) return [Epos,Ipos,Spos] I wonder if called by itself it would trigger the problem... The commented Lines Is where I catch the errors: when poisson returns a greater than zero number, when called with mean 0. Ranlib uses static floats so it may relate to numerical precision (as > well as the random uniform random number generator). Really the only > way is for you to debug the ranlib.c file I'll see what I can do. Note that R provides a standalone math library in C that includes the > random number generators so you could you those instead. SWIG handles > it rather well. I think thats what Rpy already does...is it not? thanks Bruce, Fl?vio Regards > Bruce > > On 7/13/05, Flavio Coelho wrote: > > Well, > > I am definetly glad that someone has also stumbled onto the same > problem. > > > > But it is nevertheless intriguing, that you can run poisson a million > times > > with mean zero or negative(it assumes zero mean inthis case) without any > > problems by itself. But when I call it within my code, the rate of error > is > > very high (I would say it returns a wrong result every time, but I > haven't > > checked every answer). > > > > Meanwhile, my solution will be: > > > > import rpy > > > > n = rpy.r.rpois(n,mean) > > > > I don't feel I can trust poisson while this "funny" behavior is still > > there... > > If someone has any Idea of how I could trace this bug please tell me, > and > > I'll hunt it down. At least I can reproduce it in a very specific > context. > > > > thanks, > > > > Fl?vio > > > > 2005/7/12, Sebastian Haase : > > > Hi Flavio! > > > I had reported this long time ago and this list (about numarray). > > > Somehow this got more or less dismissed. If I recall correctly the > > argument > > > was that nobody could reproduce it (I ran this on Debian Woody , py2.2 > , > > (with > > > CVS numarray at the time). > > > > > > I ended up writting my own wrapper(s): > > > def poissonArr(shape=defshape, mean=1): > > > from numarray import random_array as ra > > > if mean == 0: > > > return zeroArrF(shape) > > > elif mean < 0: > > > raise "poisson not defined for mean < 0" > > > else: > > > return ra.poisson(mean, shape).astype(na.UInt16) > > > > > > def poissonize(arr): > > > from numarray import random_array as ra > > > return na.where(arr<=0, 0, ra.poisson(arr)).astype(na.UInt16) > > > > > > (I use the astype( na.UInt16) because of some OpenGL code) > > > > > > Just last week had this problem on a windows98 computer (python2.4). > > > > > > This should get sorted out ... > > > > > > Thanks for reporting this problem. > > > Sebastian Haase > > > > > > > > > > > > > > > On Tuesday 12 July 2005 13:32, Flavio Coelho wrote: > > > > Hi, > > > > > > > > I am having problems with the poisson random number generators of > both > > > > Numarray and Numeric. > > > > I can't replicate it when calling the function from the python > cosonle, > > but > > > > it has consistently malfunctioned when used within one of my > scripts. > > > > > > > > What happens is that it frequently return a value greater than zero > when > > > > called with zero mean: poisson(0.0) > > > > > > > > Unfortunately My program is too big to send attached but I have > > confirmed > > > > the malfunction by printing both the mean and the result whenever it > > spits > > > > out a wrong result. > > > > > > > > This is very weird indeed, I have run poisson millions of times by > itsel > > on > > > > the python console, without any problems... > > > > > > > > I hope it is some stupid mistake, but when I replace the poisson > > function > > > > call within my program by the R equivalent command (rpois) via the > rpy > > > > wrapper, everything works just fine... > > > > > > > > any Ideas? > > > > > > > > > > > -- > > > > Fl?vio Code?o Coelho > > registered Linux user # 386432 > > --------------------------- > > Great spirits have always found violent opposition from mediocrities. > The > > latter cannot understand it when a man does not thoughtlessly submit to > > hereditary prejudices but honestly and courageously uses his > intelligence. > > Albert Einstein > > > -- Fl?vio Code?o Coelho registered Linux user # 386432 --------------------------- Great spirits have always found violent opposition from mediocrities. The latter cannot understand it when a man does not thoughtlessly submit to hereditary prejudices but honestly and courageously uses his intelligence. Albert Einstein -------------- next part -------------- An HTML attachment was scrubbed... URL: From cookedm at physics.mcmaster.ca Wed Jul 13 08:28:41 2005 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Wed Jul 13 08:28:41 2005 Subject: [Numpy-discussion] problem with poisson generators In-Reply-To: References: Message-ID: <20050713152444.GA31450@arbutus.physics.mcmaster.ca> On Tue, Jul 12, 2005 at 05:32:25PM -0300, Flavio Coelho wrote: > Hi, > > I am having problems with the poisson random number generators of both > Numarray and Numeric. > I can't replicate it when calling the function from the python cosonle, but > it has consistently malfunctioned when used within one of my scripts. > > What happens is that it frequently return a value greater than zero when > called with zero mean: poisson(0.0) > > Unfortunately My program is too big to send attached but I have confirmed > the malfunction by printing both the mean and the result whenever it spits > out a wrong result. > > This is very weird indeed, I have run poisson millions of times by itsel on > the python console, without any problems... > > I hope it is some stupid mistake, but when I replace the poisson function > call within my program by the R equivalent command (rpois) via the rpy > wrapper, everything works just fine... > > any Ideas? This looks like bug #1123145 in Numeric: http://sourceforge.net/tracker/index.php?func=detail&aid=1123145&group_id=1369&atid=101369 which was fixed a few months ago. numarray, I believe, originally took ranlib.c from Numeric, so it doesn't have this bug fix. Try replacing numarray's ranlib.c with the version from Numeric 24.0b2 (or CVS). -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From rowen at cesmail.net Wed Jul 13 10:15:40 2005 From: rowen at cesmail.net (Russell E. Owen) Date: Wed Jul 13 10:15:40 2005 Subject: [Numpy-discussion] numarray 1.3.3 build question Message-ID: I'm on MacOS X 10.3.9 and am getting mysterious warnings when building numarray 1.3.3 which I've never seen before. Building for the unix/x11 version yields: [dhcp-254:~/Archives\342\200\242/PythonPackages/numarray-1.3.3] rowen% python setup.py config build --gencode Running sitecustomize.pyc Using EXTRA_COMPILE_ARGS = ['-Ddarwin'] Running sitecustomize.pyc Running sitecustomize.pyc generating new API module 'libnumarray' .c & .h generating new API module 'libteacup' .c & .h generating new API module 'libnumeric' .c & .h Using external BLAS and LAPACK running config Wrote config.h ... Packages/LinearAlgebra2/Src/lapack_litemodule.c:851: warning: function declaration isn't a prototype Packages/LinearAlgebra2/Src/lapack_litemodule.c:848: warning: `lapack_liteError' defined but not used gcc: -framework: linker input file unused because linking not done gcc: vecLib: linker input file unused because linking not done Should I worry? I haven't figured out how to run the self-tests w/out installing and I'm reluctant to install w/out knowing if the warnings are a problem. also: - why do we need --gencode? Is there some way to make that the default so we can leave it off? - why do we need "config"? That's new to 1.3.3 isn't it? -- Russell P.S. I have two completely different versions of python installed: - the normal built-in Python 2.3.0, which I run using 'pythonw' - a pure unix/x11 python that I installed from source using an x11 version of tcl/tk. This is run using "python" (which runs /usr/local/python). However, the errors occur while building with either version (much to my surprised -- I expected the framework complain to go away when building with pythonw, the framework python). From perry at stsci.edu Wed Jul 13 10:53:11 2005 From: perry at stsci.edu (Perry Greenfield) Date: Wed Jul 13 10:53:11 2005 Subject: [Numpy-discussion] problem with poisson generators In-Reply-To: <20050713152444.GA31450@arbutus.physics.mcmaster.ca> References: <20050713152444.GA31450@arbutus.physics.mcmaster.ca> Message-ID: <0a6666184dc9fa774a8a6dcecb6a771c@stsci.edu> On Jul 13, 2005, at 11:24 AM, David M. Cooke wrote: > > This looks like bug #1123145 in Numeric: > > http://sourceforge.net/tracker/index.php? > func=detail&aid=1123145&group_id=1369&atid=101369 > > which was fixed a few months ago. numarray, I believe, originally took > ranlib.c from Numeric, so it doesn't have this bug fix. Try replacing > numarray's ranlib.c with the version from Numeric 24.0b2 (or CVS). > Thanks for noticing this. By the way, Todd is away this week so numarray issues won't likely be responded to this week by him. Perry From fccoelho at gmail.com Thu Jul 14 00:19:26 2005 From: fccoelho at gmail.com (Flavio Coelho) Date: Thu Jul 14 00:19:26 2005 Subject: [Numpy-discussion] problem with poisson generators In-Reply-To: <20050713152444.GA31450@arbutus.physics.mcmaster.ca> References: <20050713152444.GA31450@arbutus.physics.mcmaster.ca> Message-ID: 2005/7/13, David M. Cooke : > > On Tue, Jul 12, 2005 at 05:32:25PM -0300, Flavio Coelho wrote: > > Hi, > > > > I am having problems with the poisson random number generators of both > > Numarray and Numeric. > > I can't replicate it when calling the function from the python cosonle, > but > > it has consistently malfunctioned when used within one of my scripts. > > > > What happens is that it frequently return a value greater than zero when > > called with zero mean: poisson(0.0) > > > > Unfortunately My program is too big to send attached but I have > confirmed > > the malfunction by printing both the mean and the result whenever it > spits > > out a wrong result. > > > > This is very weird indeed, I have run poisson millions of times by itsel > on > > the python console, without any problems... > > > > I hope it is some stupid mistake, but when I replace the poisson > function > > call within my program by the R equivalent command (rpois) via the rpy > > wrapper, everything works just fine... > > > > any Ideas? > > This looks like bug #1123145 in Numeric: > > > http://sourceforge.net/tracker/index.php?func=detail&aid=1123145&group_id=1369&atid=101369 > > which was fixed a few months ago. numarray, I believe, originally took > ranlib.c from Numeric, so it doesn't have this bug fix. Try replacing > numarray's ranlib.c with the version from Numeric 24.0b2 (or CVS). I have both numeric 23.7 and numarray 1.3.1 installed and on neither of them I could reproduce the bug when I called them directly from the python interpreter. However they fail on mean 0.0 every time when called within my code. So it appears to me that the Bug you mentioned is not what is causing my problem. It seems to stem from interaction with the way its being called, maybe some carryover from previous calls... this is the test I ran on the interpreter: [(poisson(i),i) for i in uniform(-20,20,1000) if i<=0] I also ran: sum(poisson(0,100000)) they both worked flawlessly. In the first test I wanted to see if there was some carry over from previous runs when called with various means (which is closer to the way it is used within my code), but it returned zero every time. (I don't use negative means in my code, but I wanted to try it here just in case) -- > |>|\/|< > > /--------------------------------------------------------------------------\ > |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ > |cookedm at physics.mcmaster.ca > -- Fl?vio Code?o Coelho registered Linux user # 386432 --------------------------- Great spirits have always found violent opposition from mediocrities. The latter cannot understand it when a man does not thoughtlessly submit to hereditary prejudices but honestly and courageously uses his intelligence. Albert Einstein -------------- next part -------------- An HTML attachment was scrubbed... URL: From rkern at ucsd.edu Thu Jul 14 00:28:06 2005 From: rkern at ucsd.edu (Robert Kern) Date: Thu Jul 14 00:28:06 2005 Subject: [Numpy-discussion] problem with poisson generators In-Reply-To: References: <20050713152444.GA31450@arbutus.physics.mcmaster.ca> Message-ID: <42D61349.5060700@ucsd.edu> Flavio Coelho wrote: > I have both numeric 23.7 and numarray 1.3.1 installed and on neither > of them I could reproduce the bug when I called them directly from the > python interpreter. However they fail on mean 0.0 every time when called > within my code. So it appears to me that the Bug you mentioned is not > what is causing my problem. It seems to stem from interaction with the > way its being called, maybe some carryover from previous calls... > this is the test I ran on the interpreter: > [(poisson(i),i) for i in uniform(-20,20,1000) if i<=0] > > I also ran: > > sum(poisson(0,100000)) > > they both worked flawlessly. In the first test I wanted to see if there > was some carry over from previous runs when called with various means > (which is closer to the way it is used within my code), but it returned > zero every time. (I don't use negative means in my code, but I wanted to > try it here just in case) Are you sure that you don't have other versions of Numeric or numarray installed that might be getting picked up by your code? -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From jmiller at stsci.edu Thu Jul 14 02:44:59 2005 From: jmiller at stsci.edu (Todd Miller) Date: Thu Jul 14 02:44:59 2005 Subject: [Numpy-discussion] bug or feature? In-Reply-To: <23974d4bfbd55596f8212bbfd6ff6723@stsci.edu> References: <5deb92a4eb50a0ed681dbf3e8d8e3a50@stsci.edu> <23974d4bfbd55596f8212bbfd6ff6723@stsci.edu> Message-ID: <1121334126.4697.5.camel@localhost.localdomain> On Mon, 2005-07-11 at 15:22 -0400, Perry Greenfield wrote: > For array indexing, assignment does > modify the original array (as your case above illustrates), but once > you sliced the array-indexed array, it changed the context of the > indexing. I.e., (to take a simpler, 1-d array x) > > x[[0,2]] = 1 # modifies x > > This form results in __setitem__ being called on x > > x[[0,2]][:4] = 1 # doesn't modify x > > This form results in __getitem__ being called on x (and thus producing > a new copy of the array) for the array indexing and __setitem__ being > called for the slice. At least that's what I think is happening. Todd > or someone more recently familiar with how Python handles this can > correct me if I'm wrong. > > Perry > This is indeed what happens. A temporary is created by the first sub- expression x[[0,2]], modified by the slice subexpression [:4] = 1, and then discarded. I think the array indexing "grammar" could be changed to support x[[0,2],:4] as a single expression which would create the temporary and then copy-back at the end... but numarray's array indexing is not currently that fancy. Todd From jmiller at stsci.edu Thu Jul 14 03:13:10 2005 From: jmiller at stsci.edu (Todd Miller) Date: Thu Jul 14 03:13:10 2005 Subject: [Numpy-discussion] numarray 1.3.3 build question In-Reply-To: References: Message-ID: <1121335822.4697.25.camel@localhost.localdomain> On Wed, 2005-07-13 at 10:09 -0700, Russell E. Owen wrote: > I'm on MacOS X 10.3.9 and am getting mysterious warnings when building > numarray 1.3.3 which I've never seen before. > > Building for the unix/x11 version yields: > [dhcp-254:~/Archives\342\200\242/PythonPackages/numarray-1.3.3] rowen% > python setup.py config build --gencode > Running sitecustomize.pyc > Using EXTRA_COMPILE_ARGS = ['-Ddarwin'] > Running sitecustomize.pyc > Running sitecustomize.pyc > generating new API module 'libnumarray' .c & .h > generating new API module 'libteacup' .c & .h > generating new API module 'libnumeric' .c & .h > Using external BLAS and LAPACK > running config > Wrote config.h > ... > Packages/LinearAlgebra2/Src/lapack_litemodule.c:851: warning: function > declaration isn't a prototype > Packages/LinearAlgebra2/Src/lapack_litemodule.c:848: warning: > `lapack_liteError' defined but not used > gcc: -framework: linker input file unused because linking not done > gcc: vecLib: linker input file unused because linking not done > > Should I worry? > I haven't figured out how to run the self-tests w/out > installing and I'm reluctant to install w/out knowing if the warnings > are a problem. > > also: > - why do we need --gencode? You don't need it, I do. In general, it's not necessary to use but it's there because the code generation needs to remain "live" in order to be an asset but is also time consuming for people doing repetitive installs to fix problems. Prior to Python-2.4 and VC.NET, --gen_code had meaning on win32 because VC6.0 didn't fully support UInt64. Everywhere else, the CVS'ed C source just works. > Is there some way to make that the default > so we can leave it off? Sure. I would say just don't worry about it unless you're building on win32 from source using VC6.0. > - why do we need "config"? That's new to 1.3.3 isn't it? No, I think that's been there since 1.2.1 or so. It's an artifact of user submitted fixes to the linear algebra package which I accepted as- is because they were submitted by someone who both knew enough technically and had enough idealism to make the effort. Regards, Todd From fccoelho at gmail.com Thu Jul 14 07:12:35 2005 From: fccoelho at gmail.com (Flavio Coelho) Date: Thu Jul 14 07:12:35 2005 Subject: [Numpy-discussion] problem with poisson generators In-Reply-To: <42D61349.5060700@ucsd.edu> References: <20050713152444.GA31450@arbutus.physics.mcmaster.ca> <42D61349.5060700@ucsd.edu> Message-ID: 2005/7/14, Robert Kern : > > Flavio Coelho wrote: > > > I have both numeric 23.7 and numarray 1.3.1 installed and on neither > > of them I could reproduce the bug when I called them directly from the > > python interpreter. However they fail on mean 0.0 every time when called > > within my code. So it appears to me that the Bug you mentioned is not > > what is causing my problem. It seems to stem from interaction with the > > way its being called, maybe some carryover from previous calls... > > this is the test I ran on the interpreter: > > [(poisson(i),i) for i in uniform(-20,20,1000) if i<=0] > > > > I also ran: > > > > sum(poisson(0,100000)) > > > > they both worked flawlessly. In the first test I wanted to see if there > > was some carry over from previous runs when called with various means > > (which is closer to the way it is used within my code), but it returned > > zero every time. (I don't use negative means in my code, but I wanted to > > try it here just in case) > > Are you sure that you don't have other versions of Numeric or numarray > installed that might be getting picked up by your code? I have just double checked. And the answer is no, there is a single version of each of the modules. -- > Robert Kern > rkern at ucsd.edu > > "In the fields of hell where the grass grows high > Are the graves of dreams allowed to die." > -- Richard Harter > > > ------------------------------------------------------- > This SF.Net email is sponsored by the 'Do More With Dual!' > webinar happening > July 14 at 8am PDT/11am EDT. We invite you to explore the latest in dual > core and dual graphics technology at this free one hour event hosted by > HP, > AMD, and NVIDIA. To register visit http://www.hp.com/go/dualwebinar > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -- Fl?vio Code?o Coelho registered Linux user # 386432 --------------------------- Great spirits have always found violent opposition from mediocrities. The latter cannot understand it when a man does not thoughtlessly submit to hereditary prejudices but honestly and courageously uses his intelligence. Albert Einstein -------------- next part -------------- An HTML attachment was scrubbed... URL: From jmiller at stsci.edu Fri Jul 15 03:57:02 2005 From: jmiller at stsci.edu (Todd Miller) Date: Fri Jul 15 03:57:02 2005 Subject: [Numpy-discussion] problem with poisson generators In-Reply-To: References: <200507121447.01356.haase@msg.ucsf.edu> Message-ID: <1121424865.4707.16.camel@localhost.localdomain> On Wed, 2005-07-13 at 12:13 -0300, Flavio Coelho wrote: > > > 2005/7/13, Bruce Southey : > Hi, > What is the seed used? > > I am not setting the seed. > > > It is really important that you set the seed? > > No. > > > Did you build Python and numarray from source? > > > Yes, I use Gentoo. I build everything from source. > > > Can you reduce your code to a few lines that have the problem? > > > Not really, poisson and binomial are the only two functions from > Numeric that I use but they are called inside a rather complex object > oriented code environment (objects within objetcs, being called > recursively...) Bellow is the function within which poisson is called: > > def stepSEIR_s(self,inits=(0,0,0),par=(0.001,1,1,0.5,1,0,0,0),theta=0, > npass=0,dist='poisson'): > """ > Defines an stochastic model SEIR: > - inits = (E,I,S) > - par = (Beta, alpha, E,r,delta,B,w,p) see docs. > - theta = infectious individuals from neighbor sites > """ > E,I,S = inits > N = self.parentSite.totpop > beta,alpha,e,r,delta,B,w,p = par > Lpos_esp = beta*S*((I+theta)/(N+npass))**alpha #Number of new > cases > > if dist == 'poisson': > Lpos = poisson(Lpos_esp) > ## if theta == 0 and Lpos_esp == 0 and Lpos > 0: > ## print > Lpos,Lpos_esp,S,I,theta,N,self.parentSite.sitename > elif dist == 'negbin': > prob = I/(I+Lpos_esp) #convertin between parameterizations > Lpos = negative_binomial(I,prob) > self.parentSite.totalcases += Lpos #update number of cases > Epos = (1-e)*E + Lpos > Ipos = e*E + (1-r)*I > Spos = S + B - Lpos > Rpos = N-(Spos+Epos+Ipos) > #self.parentSite.totpop = Spos+Epos+Ipos+Rpos > self.parentSite.incidence.append(Lpos) > if not self.parentSite.infected: > if Lpos > 0: > self.parentSite.infected = > self.parentSite.parentGraph.simstep > self.parentSite.parentGraph.epipath.append > ((self.parentSite.parentGraph.simstep,self.parentSite,self.parentSite.infector)) > > > return [Epos,Ipos,Spos] > Having this code is a good start to solving the problem. I think the next step is to simplify your example to make it runnable and provide known inputs for all the parameters which lead to a failure for you. Being really literal (spelling out every single damn thing) cuts down on speculation and inefficiency on our end. It would also be good to express the condition (or it's inverse) you think is an error as an assertion, so something like this might be what we need: from numarray.random_array import poisson E, I, S = (0,0,0) beta,alpha,e,r,delta,B,w,p = (0.001,1,1,0.5,1,0,0,0) theta = 0 npass = 0 N = 100 # total guess here for me Lpos_esp = beta*S*((I+theta)/(N+npass))**alpha #Number of new cases Lpos = poisson(Lpos_esp) assert not (theta == 0 and Lpos_esp == 0 and Lpos > 0) The problem is, the above works for me. Make it sane and get it to expose the error for you and I'll look into this some more. Regards, Todd > > I wonder if called by itself it would trigger the problem... The > commented Lines Is where I catch the errors: when poisson returns a > greater than zero number, when called with mean 0. > > > > Ranlib uses static floats so it may relate to numerical > precision (as > well as the random uniform random number generator). Really > the only > way is for you to debug the ranlib.c file > > I'll see what I can do. > > > Note that R provides a standalone math library in C that > includes the > random number generators so you could you those instead. SWIG > handles > it rather well. > > > I think thats what Rpy already does...is it not? > > thanks Bruce, > > Fl?vio > > > Regards > Bruce > > On 7/13/05, Flavio Coelho wrote: > > Well, > > I am definetly glad that someone has also stumbled onto > the same problem. > > > > But it is nevertheless intriguing, that you can run poisson > a million times > > with mean zero or negative(it assumes zero mean inthis > case) without any > > problems by itself. But when I call it within my code, the > rate of error is > > very high (I would say it returns a wrong result every time, > but I haven't > > checked every answer). > > > > Meanwhile, my solution will be: > > > > import rpy > > > > n = rpy.r.rpois(n,mean) > > > > I don't feel I can trust poisson while this "funny" > behavior is still > > there... > > If someone has any Idea of how I could trace this bug > please tell me, and > > I'll hunt it down. At least I can reproduce it in a very > specific context. > > > > thanks, > > > > Fl?vio > > > > 2005/7/12, Sebastian Haase : > > > Hi Flavio! > > > I had reported this long time ago and this list (about > numarray). > > > Somehow this got more or less dismissed. If I recall > correctly the > > argument > > > was that nobody could reproduce it (I ran this on Debian > Woody , py2.2, > > (with > > > CVS numarray at the time). > > > > > > I ended up writting my own wrapper(s): > > > def poissonArr(shape=defshape, mean=1): > > > from numarray import random_array as ra > > > if mean == 0: > > > return zeroArrF(shape) > > > elif mean < 0: > > > raise "poisson not defined for mean < 0" > > > else: > > > return ra.poisson(mean, shape).astype > (na.UInt16) > > > > > > def poissonize(arr): > > > from numarray import random_array as ra > > > return na.where(arr<=0, 0, ra.poisson(arr)).astype > ( na.UInt16) > > > > > > (I use the astype( na.UInt16) because of some OpenGL code) > > > > > > Just last week had this problem on a windows98 computer > (python2.4). > > > > > > This should get sorted out ... > > > > > > Thanks for reporting this problem. > > > Sebastian Haase > > > > > > > > > > > > > > > On Tuesday 12 July 2005 13:32, Flavio Coelho wrote: > > > > Hi, > > > > > > > > I am having problems with the poisson random number > generators of both > > > > Numarray and Numeric. > > > > I can't replicate it when calling the function from the > python cosonle, > > but > > > > it has consistently malfunctioned when used within one > of my scripts. > > > > > > > > What happens is that it frequently return a value > greater than zero when > > > > called with zero mean: poisson( 0.0) > > > > > > > > Unfortunately My program is too big to send attached but > I have > > confirmed > > > > the malfunction by printing both the mean and the result > whenever it > > spits > > > > out a wrong result. > > > > > > > > This is very weird indeed, I have run poisson millions > of times by itsel > > on > > > > the python console, without any problems... > > > > > > > > I hope it is some stupid mistake, but when I replace the > poisson > > function > > > > call within my program by the R equivalent command > (rpois) via the rpy > > > > wrapper, everything works just fine... > > > > > > > > any Ideas? > > > > > > > > > > > -- > > > > Fl?vio Code?o Coelho > > registered Linux user # 386432 > > --------------------------- > > Great spirits have always found violent opposition from > mediocrities. The > > latter cannot understand it when a man does not > thoughtlessly submit to > > hereditary prejudices but honestly and courageously uses his > intelligence. > > Albert Einstein > > > > > > -- > Fl?vio Code?o Coelho > registered Linux user # 386432 > --------------------------- > Great spirits have always found violent opposition from mediocrities. > The > latter cannot understand it when a man does not thoughtlessly submit > to > hereditary prejudices but honestly and courageously uses his > intelligence. > Albert Einstein From faltet at carabos.com Fri Jul 15 04:01:00 2005 From: faltet at carabos.com (Francesc Altet) Date: Fri Jul 15 04:01:00 2005 Subject: [Numpy-discussion] ANN: PyTables 1.1 released Message-ID: <200507151258.28250.faltet@carabos.com> ========================= Announcing PyTables 1.1 ========================= The PyTables development team is happy to announce the availability of a new version of PyTables package. On this version you will find support for a nice set of new features, like nested datatypes, enumerated datatypes, nested iterators (for reading only), support for native HDF5 multidimensional attributes, a new object for dealing with compressed, non-enlargeable arrays (CArray), bzip2 compression support and more. Many bugs has been addressed as well. Go to the PyTables web site for downloading the beast: http://pytables.sourceforge.net/ Or keep reading for more info about the new features and bugs fixed. Changes more in depth ===================== Improvements: - Support for nested datatypes is in place. You can now made columns of tables that host another columns for an unlimited depth (well, theoretically, in practice until the python recursive limit would be reached). Convenient NestedRecArray objects has been implemented as data containers. Cols and Description accessors has been improved so you can navigate on the type hierarchy very easily (natural naming is has been implemented for the task). - ``Table``, ``EArray`` and ``VLArray`` objects now support enumerated types. ``Array`` objects support opening existing HDF5 enumerated arrays. Enumerated types are restricted sets of ``(name, value)`` pairs. Use the ``Enum`` class to easily define new enumerations that will be saved along with your data. - Now, the HDF5 library is responsible to do data conversions when the datasets are written in a machine with different byte-ordering than the machine that reads the dataset. With this, all the data is converted on-the-fly and you always get native datatypes in memory. I think this approach to be more convenient in terms of CPU consumption when using these datasets. Right now, this only works for tables, though. - Added support for native HDF5 multidimensional attributes. Now, you can load native HDF5 files that contains fully multidimensional attributes; these attributes will be mapped to NumArray objects. Also, when you save NumArray objects as attributes, they get saved as native HDF5 attributes (before, NumArray attributes where pickled). - A brand-new class, called CArray, has been introduced. It's mainly like an Array class (i.e. non-enlargeable), but with compression capabilities enabled. The existence of CArray also allows PyTables to read native HDF5 chunked, non-enlargeable datasets. - Bzip2 compressor is supported. Such a support was already in PyTables 1.0, but forgot to announce it. - New LZO2 (http://www.oberhumer.com/opensource/lzo/lzonews.php) compressor is supported. The installer now recognizes whether LZO1 or LZO2 is installed, and adapts automatically to it. If both are installed in your system, then LZO2 is chosen. LZO2 claims to be fully compatible (both backward and forward) with LZO1, so you should not experience any problem during this transition. - The old limit of 256 columns in a table has been released. Now, you can have tables with any number of columns, although if you try to use a too high number (i.e. > 1024), you will start to consume a lot of system resources. You have been warned!. - The limit in the length of column names has been released also. - Nested iterators for reading in tables are supported now. - A new section in tutorial about how to modify values in tables and arrays has been added to the User's Manual. Backward-incompatible changes: - None. Bug fixes: - VLArray now correctly updates the number of rows internal counter when opening an existing VLArray object. Now you can add new rows to existing VLA's without problems. - Tuple flavor for VLArrays now works as intended, i.e. reading VLArray objects will always return tuples even in the case of multidimensional Atoms. Before, this operations returned a mix of tuples and lists. - If a column was not able to be indexed because it has too few entries, then _whereInRange is called instead of _whereIndexed. Fixes #1203202. - You can call now Row.append() in the middle of Table iterators without resetting loop counters. Fixes #1205588. - PyTables used to give a segmentation fault when removing the last row out of a table with the table.removeRows() method. This is due to a limitation in the HDF5 library. Until this get fixed in HDF5, a NotImplemented error is raised when trying to do that. Address #1201023. - You can safely break a loop over an iterator returned by Table.where(). Fixes #1234637. - When removing a Group with hidden child groups, those are effectively closed now. - Now, there is a distinction between shapes 1 and (1,) in tables. The former represents a scalar, and the later a 1-D array with just one element. That follows the numarray convention for records, and makes more sense as well. Before 1.1, shapes 1 and (1,) were represented by an scalar on disk. Known bugs: - Classes inheriting from IsDescription subclasses do not inherit columns defined in the super-class. See SF bug #1207732 for more info. - Time datatypes are non-portable between big-endian and little-endian architectures. This is ultimately a consequence of a HDF5 limitation. See SF bug #1234709 for more info. Known issues: - UCL compressor seems to work badly on MacOSX platforms. Until the problem would be isolated and eventually solved, UCL will not be compiled by default on MacOSX platforms, even if the installer finds it in the system. However, if you still want to get UCL support on MacOSX, you can use the --force-ucl flag in setup.py. Important note for Python 2.4 and Windows users =============================================== If you are willing to use PyTables with Python 2.4 in Windows platforms, you will need to get the HDF5 library compiled for MSVC 7.1, aka .NET 2003. It can be found at: ftp://ftp.ncsa.uiuc.edu/HDF/HDF5/current/bin/windows/5-164-win-net.ZIP Users of Python 2.3 on Windows will have to download the version of HDF5 compiled with MSVC 6.0 available in: ftp://ftp.ncsa.uiuc.edu/HDF/HDF5/current/bin/windows/5-164-win.ZIP What it is ========== **PyTables** is a package for managing hierarchical datasets and designed to efficiently cope with extremely large amounts of data (with support for full 64-bit file addressing). It features an object-oriented interface that, combined with C extensions for the performance-critical parts of the code, makes it a very easy-to-use tool for high performance data storage and retrieval. Perhaps its more interesting feature is that it optimizes memory and disk resources so that data take much less space (between a factor 3 to 5, and more if the data is compressible) than other solutions, like for example, relational or object oriented databases. Besides, PyTables I/O for table objects is buffered, implemented in C and carefully tuned so that you can reach much better performance with PyTables than with your own home-grown wrappings to the HDF5 library. PyTables sports indexing capabilities as well, allowing doing selections in tables exceeding one billion of rows in just seconds. Where can PyTables be applied? ============================== PyTables is not designed to work as a relational database competitor, but rather as a teammate. If you want to work with large datasets of multidimensional data (for example, for multidimensional analysis), or just provide a categorized structure for some portions of your cluttered RDBS, then give PyTables a try. It works well for storing data from data acquisition systems (DAS), simulation software, network data monitoring systems (for example, traffic measurements of IP packets on routers), very large XML files, or for creating a centralized repository for system logs, to name only a few possible uses. What is a table? ================ A table is defined as a collection of records whose values are stored in fixed-length fields. All records have the same structure and all values in each field have the same data type. The terms "fixed-length" and "strict data types" seem to be quite a strange requirement for a language like Python that supports dynamic data types, but they serve a useful function if the goal is to save very large quantities of data (such as is generated by many scientific applications, for example) in an efficient manner that reduces demand on CPU time and I/O resources. What is HDF5? ============= For those people who know nothing about HDF5, it is a general purpose library and file format for storing scientific data made at NCSA. HDF5 can store two primary objects: datasets and groups. A dataset is essentially a multidimensional array of data elements, and a group is a structure for organizing objects in an HDF5 file. Using these two basic constructs, one can create and store almost any kind of scientific data structure, such as images, arrays of vectors, and structured and unstructured grids. You can also mix and match them in HDF5 files according to your needs. Platforms ========= We are using Linux on top of Intel32 as the main development platform, but PyTables should be easy to compile/install on other UNIX machines. This package has also been successfully compiled and tested on a FreeBSD 5.4 with Opteron64 processors, a UltraSparc platform with Solaris 7 and Solaris 8, a SGI Origin3000 with Itanium processors running IRIX 6.5 (using the gcc compiler), Microsoft Windows and MacOSX (10.2 although 10.3 should work fine as well). In particular, it has been thoroughly tested on 64-bit platforms, like Linux-64 on top of an Intel Itanium, AMD Opteron (in 64-bit mode) or PowerPC G5 (in 64-bit mode) where all the tests pass successfully. Regarding Windows platforms, PyTables has been tested with Windows 2000 and Windows XP (using the Microsoft Visual C compiler), but it should also work with other flavors as well. Web site ======== Go to the PyTables web site for more details: http://pytables.sourceforge.net/ To know more about the company behind the PyTables development, see: http://www.carabos.com/ Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. ---- **Enjoy data!** -- The PyTables Team From Sebastien.deMentendeHorne at electrabel.com Fri Jul 15 04:13:04 2005 From: Sebastien.deMentendeHorne at electrabel.com (Sebastien.deMentendeHorne at electrabel.com) Date: Fri Jul 15 04:13:04 2005 Subject: [Numpy-discussion] problem with poisson generators Message-ID: <6E48F3D185CF644788F55917A0D50A93017CF4A3@seebex02.eib.electrabel.be> Hi Flavio, Could you give us every call to poisson or negative_binomial (ie all functions related to random numbers) preceeded with the seed ? Adding before your declaration of stepSEIR_s code like: randoutput = file("randoutput.py", "w") old_poisson = poisson def poisson(m): print >> randoutput, "print get_seed(), poisson(%s)"%m result = old_poisson(m) print >> randoutput, "# result = %s"%result return result old_negative_binomial = negative_binomial def negative_binomial(i,p): print >> randoutput, "print get_seed(), negative_binomial(%s,%s)"%(i,p) result = old_negative_binomial(i,p) print >> randoutput, "# result = %s"%result return result will ouput every call. We can then easily try on our machines what it gives. Best, Sebastien > -----Original Message----- > From: numpy-discussion-admin at lists.sourceforge.net > [mailto:numpy-discussion-admin at lists.sourceforge.net]On Behalf Of Todd > Miller > Sent: vendredi 15 juillet 2005 12:54 > To: Flavio Coelho > Cc: Bruce Southey; Sebastian Haase; numpy-discussion > Subject: Re: [Numpy-discussion] problem with poisson generators > > > On Wed, 2005-07-13 at 12:13 -0300, Flavio Coelho wrote: > > > > > > 2005/7/13, Bruce Southey : > > Hi, > > What is the seed used? > > > > I am not setting the seed. > > > > > > It is really important that you set the seed? > > > > No. > > > > > > Did you build Python and numarray from source? > > > > > > Yes, I use Gentoo. I build everything from source. > > > > > > Can you reduce your code to a few lines that have > the problem? > > > > > > Not really, poisson and binomial are the only two functions from > > Numeric that I use but they are called inside a rather > complex object > > oriented code environment (objects within objetcs, being called > > recursively...) Bellow is the function within which poisson > is called: > > > > def > stepSEIR_s(self,inits=(0,0,0),par=(0.001,1,1,0.5,1,0,0,0),theta=0, > > npass=0,dist='poisson'): > > """ > > Defines an stochastic model SEIR: > > - inits = (E,I,S) > > - par = (Beta, alpha, E,r,delta,B,w,p) see docs. > > - theta = infectious individuals from neighbor sites > > """ > > E,I,S = inits > > N = self.parentSite.totpop > > beta,alpha,e,r,delta,B,w,p = par > > Lpos_esp = beta*S*((I+theta)/(N+npass))**alpha > #Number of new > > cases > > > > if dist == 'poisson': > > Lpos = poisson(Lpos_esp) > > ## if theta == 0 and Lpos_esp == 0 and Lpos > 0: > > ## print > > Lpos,Lpos_esp,S,I,theta,N,self.parentSite.sitename > > elif dist == 'negbin': > > prob = I/(I+Lpos_esp) #convertin between > parameterizations > > Lpos = negative_binomial(I,prob) > > self.parentSite.totalcases += Lpos #update number > of cases > > Epos = (1-e)*E + Lpos > > Ipos = e*E + (1-r)*I > > Spos = S + B - Lpos > > Rpos = N-(Spos+Epos+Ipos) > > #self.parentSite.totpop = Spos+Epos+Ipos+Rpos > > self.parentSite.incidence.append(Lpos) > > if not self.parentSite.infected: > > if Lpos > 0: > > self.parentSite.infected = > > self.parentSite.parentGraph.simstep > > self.parentSite.parentGraph.epipath.append > > > ((self.parentSite.parentGraph.simstep,self.parentSite,self.par > entSite.infector)) > > > > > > return [Epos,Ipos,Spos] > > > > Having this code is a good start to solving the problem. I think the > next step is to simplify your example to make it runnable and provide > known inputs for all the parameters which lead to a failure for you. > > Being really literal (spelling out every single damn thing) > cuts down on > speculation and inefficiency on our end. > > It would also be good to express the condition (or it's inverse) you > think is an error as an assertion, so something like this > might be what > we need: > > from numarray.random_array import poisson > > E, I, S = (0,0,0) > beta,alpha,e,r,delta,B,w,p = (0.001,1,1,0.5,1,0,0,0) > theta = 0 > npass = 0 > N = 100 # total guess here for me > Lpos_esp = beta*S*((I+theta)/(N+npass))**alpha #Number of new cases > Lpos = poisson(Lpos_esp) > assert not (theta == 0 and Lpos_esp == 0 and Lpos > 0) > > The problem is, the above works for me. Make it sane and get it to > expose the error for you and I'll look into this some more. > > Regards, > Todd > > > > > I wonder if called by itself it would trigger the problem... The > > commented Lines Is where I catch the errors: when poisson returns a > > greater than zero number, when called with mean 0. > > > > > > > > Ranlib uses static floats so it may relate to numerical > > precision (as > > well as the random uniform random number generator). Really > > the only > > way is for you to debug the ranlib.c file > > > > I'll see what I can do. > > > > > > Note that R provides a standalone math library in C that > > includes the > > random number generators so you could you those > instead. SWIG > > handles > > it rather well. > > > > > > I think thats what Rpy already does...is it not? > > > > thanks Bruce, > > > > Fl?vio > > > > > > Regards > > Bruce > > > > On 7/13/05, Flavio Coelho wrote: > > > Well, > > > I am definetly glad that someone has also stumbled onto > > the same problem. > > > > > > But it is nevertheless intriguing, that you can > run poisson > > a million times > > > with mean zero or negative(it assumes zero mean inthis > > case) without any > > > problems by itself. But when I call it within my code, the > > rate of error is > > > very high (I would say it returns a wrong result > every time, > > but I haven't > > > checked every answer). > > > > > > Meanwhile, my solution will be: > > > > > > import rpy > > > > > > n = rpy.r.rpois(n,mean) > > > > > > I don't feel I can trust poisson while this "funny" > > behavior is still > > > there... > > > If someone has any Idea of how I could trace this bug > > please tell me, and > > > I'll hunt it down. At least I can reproduce it in a very > > specific context. > > > > > > thanks, > > > > > > Fl?vio > > > > > > 2005/7/12, Sebastian Haase : > > > > Hi Flavio! > > > > I had reported this long time ago and this list (about > > numarray). > > > > Somehow this got more or less dismissed. If I recall > > correctly the > > > argument > > > > was that nobody could reproduce it (I ran this on Debian > > Woody , py2.2, > > > (with > > > > CVS numarray at the time). > > > > > > > > I ended up writting my own wrapper(s): > > > > def poissonArr(shape=defshape, mean=1): > > > > from numarray import random_array as ra > > > > if mean == 0: > > > > return zeroArrF(shape) > > > > elif mean < 0: > > > > raise "poisson not defined for mean < 0" > > > > else: > > > > return ra.poisson(mean, shape).astype > > (na.UInt16) > > > > > > > > def poissonize(arr): > > > > from numarray import random_array as ra > > > > return na.where(arr<=0, 0, > ra.poisson(arr)).astype > > ( na.UInt16) > > > > > > > > (I use the astype( na.UInt16) because of some > OpenGL code) > > > > > > > > Just last week had this problem on a windows98 computer > > (python2.4). > > > > > > > > This should get sorted out ... > > > > > > > > Thanks for reporting this problem. > > > > Sebastian Haase > > > > > > > > > > > > > > > > > > > > On Tuesday 12 July 2005 13:32, Flavio Coelho wrote: > > > > > Hi, > > > > > > > > > > I am having problems with the poisson random number > > generators of both > > > > > Numarray and Numeric. > > > > > I can't replicate it when calling the > function from the > > python cosonle, > > > but > > > > > it has consistently malfunctioned when used within one > > of my scripts. > > > > > > > > > > What happens is that it frequently return a value > > greater than zero when > > > > > called with zero mean: poisson( 0.0) > > > > > > > > > > Unfortunately My program is too big to send > attached but > > I have > > > confirmed > > > > > the malfunction by printing both the mean and > the result > > whenever it > > > spits > > > > > out a wrong result. > > > > > > > > > > This is very weird indeed, I have run poisson millions > > of times by itsel > > > on > > > > > the python console, without any problems... > > > > > > > > > > I hope it is some stupid mistake, but when I > replace the > > poisson > > > function > > > > > call within my program by the R equivalent command > > (rpois) via the rpy > > > > > wrapper, everything works just fine... > > > > > > > > > > any Ideas? > > > > > > > > > > > > > > > > -- > > > > > > Fl?vio Code?o Coelho > > > registered Linux user # 386432 > > > --------------------------- > > > Great spirits have always found violent opposition from > > mediocrities. The > > > latter cannot understand it when a man does not > > thoughtlessly submit to > > > hereditary prejudices but honestly and > courageously uses his > > intelligence. > > > Albert Einstein > > > > > > > > > > > -- > > Fl?vio Code?o Coelho > > registered Linux user # 386432 > > --------------------------- > > Great spirits have always found violent opposition from > mediocrities. > > The > > latter cannot understand it when a man does not thoughtlessly submit > > to > > hereditary prejudices but honestly and courageously uses his > > intelligence. > > Albert Einstein > > > > ------------------------------------------------------- > SF.Net email is sponsored by: Discover Easy Linux Migration Strategies > from IBM. Find simple to follow Roadmaps, straightforward articles, > informative Webcasts and more! Get everything you need to get up to > speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > ======================================================= This message is confidential. It may also be privileged or otherwise protected by work product immunity or other legal rules. If you have received it by mistake please let us know by reply and then delete it from your system; you should not copy it or disclose its contents to anyone. All messages sent to and from Electrabel may be monitored to ensure compliance with internal policies and to protect our business. Emails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, lost or destroyed, or contain viruses. Anyone who communicates with us by email is taken to accept these risks. http://www.electrabel.be/homepage/general/disclaimer_EN.asp ======================================================= From fccoelho at gmail.com Fri Jul 15 17:41:29 2005 From: fccoelho at gmail.com (Flavio Coelho) Date: Fri Jul 15 17:41:29 2005 Subject: [Numpy-discussion] problem with poisson generators In-Reply-To: <1121424865.4707.16.camel@localhost.localdomain> References: <200507121447.01356.haase@msg.ucsf.edu> <1121424865.4707.16.camel@localhost.localdomain> Message-ID: 2005/7/15, Todd Miller : Todd, the function within which the problem happens runs fine by itself, as well as the simplified version you sent me. I am running the code on a AMD64 (though all my software and OS is compiled to 686 - Gentoo Linux). Is there any issue I should be aware of, regarding this specific cpu? I can try to run the software on another machine... Having this code is a good start to solving the problem. I think the > next step is to simplify your example to make it runnable and provide > known inputs for all the parameters which lead to a failure for you. > > Being really literal (spelling out every single damn thing) cuts down on > speculation and inefficiency on our end. > > It would also be good to express the condition (or it's inverse) you > think is an error as an assertion, so something like this might be what > we need: > > from numarray.random_array import poisson > > E, I, S = (0,0,0) > beta,alpha,e,r,delta,B,w,p = (0.001,1,1,0.5,1,0,0,0) > theta = 0 > npass = 0 > N = 100 # total guess here for me > Lpos_esp = beta*S*((I+theta)/(N+npass))**alpha #Number of new cases > Lpos = poisson(Lpos_esp) > assert not (theta == 0 and Lpos_esp == 0 and Lpos > 0) > > The problem is, the above works for me. Make it sane and get it to > expose the error for you and I'll look into this some more. > > Regards, > Todd > > > > > I wonder if called by itself it would trigger the problem... The > > commented Lines Is where I catch the errors: when poisson returns a > > greater than zero number, when called with mean 0. > > > > > > > > Ranlib uses static floats so it may relate to numerical > > precision (as > > well as the random uniform random number generator). Really > > the only > > way is for you to debug the ranlib.c file > > > > I'll see what I can do. > > > > > > Note that R provides a standalone math library in C that > > includes the > > random number generators so you could you those instead. SWIG > > handles > > it rather well. > > > > > > I think thats what Rpy already does...is it not? > > > > thanks Bruce, > > > > Fl?vio > > > > > > Regards > > Bruce > > > > On 7/13/05, Flavio Coelho wrote: > > > Well, > > > I am definetly glad that someone has also stumbled onto > > the same problem. > > > > > > But it is nevertheless intriguing, that you can run poisson > > a million times > > > with mean zero or negative(it assumes zero mean inthis > > case) without any > > > problems by itself. But when I call it within my code, the > > rate of error is > > > very high (I would say it returns a wrong result every time, > > but I haven't > > > checked every answer). > > > > > > Meanwhile, my solution will be: > > > > > > import rpy > > > > > > n = rpy.r.rpois(n,mean) > > > > > > I don't feel I can trust poisson while this "funny" > > behavior is still > > > there... > > > If someone has any Idea of how I could trace this bug > > please tell me, and > > > I'll hunt it down. At least I can reproduce it in a very > > specific context. > > > > > > thanks, > > > > > > Fl?vio > > > > > > 2005/7/12, Sebastian Haase : > > > > Hi Flavio! > > > > I had reported this long time ago and this list (about > > numarray). > > > > Somehow this got more or less dismissed. If I recall > > correctly the > > > argument > > > > was that nobody could reproduce it (I ran this on Debian > > Woody , py2.2, > > > (with > > > > CVS numarray at the time). > > > > > > > > I ended up writting my own wrapper(s): > > > > def poissonArr(shape=defshape, mean=1): > > > > from numarray import random_array as ra > > > > if mean == 0: > > > > return zeroArrF(shape) > > > > elif mean < 0: > > > > raise "poisson not defined for mean < 0" > > > > else: > > > > return ra.poisson(mean, shape).astype > > (na.UInt16) > > > > > > > > def poissonize(arr): > > > > from numarray import random_array as ra > > > > return na.where(arr<=0, 0, ra.poisson(arr)).astype > > ( na.UInt16) > > > > > > > > (I use the astype( na.UInt16) because of some OpenGL code) > > > > > > > > Just last week had this problem on a windows98 computer > > (python2.4). > > > > > > > > This should get sorted out ... > > > > > > > > Thanks for reporting this problem. > > > > Sebastian Haase > > > > > > > > > > > > > > > > > > > > On Tuesday 12 July 2005 13:32, Flavio Coelho wrote: > > > > > Hi, > > > > > > > > > > I am having problems with the poisson random number > > generators of both > > > > > Numarray and Numeric. > > > > > I can't replicate it when calling the function from the > > python cosonle, > > > but > > > > > it has consistently malfunctioned when used within one > > of my scripts. > > > > > > > > > > What happens is that it frequently return a value > > greater than zero when > > > > > called with zero mean: poisson( 0.0) > > > > > > > > > > Unfortunately My program is too big to send attached but > > I have > > > confirmed > > > > > the malfunction by printing both the mean and the result > > whenever it > > > spits > > > > > out a wrong result. > > > > > > > > > > This is very weird indeed, I have run poisson millions > > of times by itsel > > > on > > > > > the python console, without any problems... > > > > > > > > > > I hope it is some stupid mistake, but when I replace the > > poisson > > > function > > > > > call within my program by the R equivalent command > > (rpois) via the rpy > > > > > wrapper, everything works just fine... > > > > > > > > > > any Ideas? > > > > > > > > > > > > > > > > -- > > > > > > Fl?vio Code?o Coelho > > > registered Linux user # 386432 > > > --------------------------- > > > Great spirits have always found violent opposition from > > mediocrities. The > > > latter cannot understand it when a man does not > > thoughtlessly submit to > > > hereditary prejudices but honestly and courageously uses his > > intelligence. > > > Albert Einstein > > > > > > > > > > > -- > > Fl?vio Code?o Coelho > > registered Linux user # 386432 > > --------------------------- > > Great spirits have always found violent opposition from mediocrities. > > The > > latter cannot understand it when a man does not thoughtlessly submit > > to > > hereditary prejudices but honestly and courageously uses his > > intelligence. > > Albert Einstein > > -- Fl?vio Code?o Coelho registered Linux user # 386432 --------------------------- Great spirits have always found violent opposition from mediocrities. The latter cannot understand it when a man does not thoughtlessly submit to hereditary prejudices but honestly and courageously uses his intelligence. Albert Einstein -------------- next part -------------- An HTML attachment was scrubbed... URL: From jmiller at stsci.edu Fri Jul 15 18:52:31 2005 From: jmiller at stsci.edu (Todd Miller) Date: Fri Jul 15 18:52:31 2005 Subject: [Numpy-discussion] problem with poisson generators In-Reply-To: References: <200507121447.01356.haase@msg.ucsf.edu> <1121424865.4707.16.camel@localhost.localdomain> Message-ID: <1121478711.4695.14.camel@localhost.localdomain> On Fri, 2005-07-15 at 21:30 -0300, Flavio Coelho wrote: > > > 2005/7/15, Todd Miller : > > Todd, > > the function within which the problem happens runs fine by itself, > as well as the simplified version you sent me. Maybe you could stick in an assert rather than the print and then use pdb.pm() to find the actual parameters leading to the failure. (The parameters may or may not matter, i.e. the problem may be hidden state somewhere your program accumulates in some complicated way, but maybe we'll get lucky...) > I am running the code on a AMD64 (though all my software and OS is > compiled to 686 - Gentoo Linux). Is there any issue I should be aware > of, regarding this specific cpu? Other than the fact that 64-bit addressing functionality isn't where we want it to be, there are no remaining issues I'm aware of for AMD64. But the addressing is currently hamstrung by 32-bit limits in Python's sequence protocol so you should be aware that even with your AMD64 you may run out of head room. > I can try to run the software on another machine... I wouldn't bother: our goal should be to replicate the problem simply and that suggests to me maintaining identical initial conditions. BTW, I'm using AMD64 too so that won't be another variable. Regards, Todd > > Having this code is a good start to solving the problem. I > think the > next step is to simplify your example to make it runnable and > provide > known inputs for all the parameters which lead to a failure > for you. > > Being really literal (spelling out every single damn thing) > cuts down on > speculation and inefficiency on our end. > > It would also be good to express the condition (or it's > inverse) you > think is an error as an assertion, so something like this > might be what > we need: > > from numarray.random_array import poisson > > E, I, S = (0,0,0) > beta,alpha,e,r,delta,B,w,p = (0.001,1,1,0.5,1,0,0,0) > theta = 0 > npass = 0 > N = 100 # total guess here for me > Lpos_esp = beta*S*((I+theta)/(N+npass))**alpha #Number of new > cases > Lpos = poisson(Lpos_esp) > assert not (theta == 0 and Lpos_esp == 0 and Lpos > 0) > > The problem is, the above works for me. Make it sane and get > it to > expose the error for you and I'll look into this some more. > > Regards, > Todd > > > > > I wonder if called by itself it would trigger the > problem... The > > commented Lines Is where I catch the errors: when poisson > returns a > > greater than zero number, when called with mean 0. > > > > > > > > Ranlib uses static floats so it may relate to > numerical > > precision (as > > well as the random uniform random number generator). > Really > > the only > > way is for you to debug the ranlib.c file > > > > I'll see what I can do. > > > > > > Note that R provides a standalone math library in C > that > > includes the > > random number generators so you could you those > instead. SWIG > > handles > > it rather well. > > > > > > I think thats what Rpy already does...is it not? > > > > thanks Bruce, > > > > Fl?vio > > > > > > Regards > > Bruce > > > > On 7/13/05, Flavio Coelho > wrote: > > > Well, > > > I am definetly glad that someone has also > stumbled onto > > the same problem. > > > > > > But it is nevertheless intriguing, that you can > run poisson > > a million times > > > with mean zero or negative(it assumes zero mean > inthis > > case) without any > > > problems by itself. But when I call it within my > code, the > > rate of error is > > > very high (I would say it returns a wrong result > every time, > > but I haven't > > > checked every answer). > > > > > > Meanwhile, my solution will be: > > > > > > import rpy > > > > > > n = rpy.r.rpois(n,mean) > > > > > > I don't feel I can trust poisson while this > "funny" > > behavior is still > > > there... > > > If someone has any Idea of how I could trace this > bug > > please tell me, and > > > I'll hunt it down. At least I can reproduce it in > a very > > specific context. > > > > > > thanks, > > > > > > Fl?vio > > > > > > 2005/7/12, Sebastian Haase : > > > > Hi Flavio! > > > > I had reported this long time ago and this list > (about > > numarray). > > > > Somehow this got more or less dismissed. If I > recall > > correctly the > > > argument > > > > was that nobody could reproduce it (I ran this > on Debian > > Woody , py2.2, > > > (with > > > > CVS numarray at the time). > > > > > > > > I ended up writting my own wrapper(s): > > > > def poissonArr(shape=defshape, mean=1): > > > > from numarray import random_array as ra > > > > if mean == 0: > > > > return zeroArrF(shape) > > > > elif mean < 0: > > > > raise "poisson not defined for > mean < 0" > > > > else: > > > > return ra.poisson(mean, > shape).astype > > (na.UInt16) > > > > > > > > def poissonize(arr): > > > > from numarray import random_array as ra > > > > return na.where(arr<=0, 0, ra.poisson > (arr)).astype > > ( na.UInt16) > > > > > > > > (I use the astype( na.UInt16) because of some > OpenGL code) > > > > > > > > Just last week had this problem on a windows98 > computer > > (python2.4). > > > > > > > > This should get sorted out ... > > > > > > > > Thanks for reporting this problem. > > > > Sebastian Haase > > > > > > > > > > > > > > > > > > > > On Tuesday 12 July 2005 13:32, Flavio Coelho > wrote: > > > > > Hi, > > > > > > > > > > I am having problems with the poisson random > number > > generators of both > > > > > Numarray and Numeric. > > > > > I can't replicate it when calling the function > from the > > python cosonle, > > > but > > > > > it has consistently malfunctioned when used > within one > > of my scripts. > > > > > > > > > > What happens is that it frequently return a > value > > greater than zero when > > > > > called with zero mean: poisson( 0.0) > > > > > > > > > > Unfortunately My program is too big to send > attached but > > I have > > > confirmed > > > > > the malfunction by printing both the mean and > the result > > whenever it > > > spits > > > > > out a wrong result. > > > > > > > > > > This is very weird indeed, I have run poisson > millions > > of times by itsel > > > on > > > > > the python console, without any problems... > > > > > > > > > > I hope it is some stupid mistake, but when I > replace the > > poisson > > > function > > > > > call within my program by the R equivalent > command > > (rpois) via the rpy > > > > > wrapper, everything works just fine... > > > > > > > > > > any Ideas? > > > > > > > > > > > > > > > > -- > > > > > > Fl?vio Code?o Coelho > > > registered Linux user # 386432 > > > --------------------------- > > > Great spirits have always found violent opposition > from > > mediocrities. The > > > latter cannot understand it when a man does not > > thoughtlessly submit to > > > hereditary prejudices but honestly and > courageously uses his > > intelligence. > > > Albert Einstein > > > > > > > > > > > -- > > Fl?vio Code?o Coelho > > registered Linux user # 386432 > > --------------------------- > > Great spirits have always found violent opposition from > mediocrities. > > The > > latter cannot understand it when a man does not > thoughtlessly submit > > to > > hereditary prejudices but honestly and courageously uses his > > intelligence. > > Albert Einstein > > > > > -- > Fl?vio Code?o Coelho > registered Linux user # 386432 > --------------------------- > Great spirits have always found violent opposition from mediocrities. > The > latter cannot understand it when a man does not thoughtlessly submit > to > hereditary prejudices but honestly and courageously uses his > intelligence. > Albert Einstein From bsouthey at gmail.com Sat Jul 16 20:53:32 2005 From: bsouthey at gmail.com (Bruce Southey) Date: Sat Jul 16 20:53:32 2005 Subject: [Numpy-discussion] problem with poisson generators In-Reply-To: References: <200507121447.01356.haase@msg.ucsf.edu> <1121424865.4707.16.camel@localhost.localdomain> Message-ID: Hi, This seems more and more like some coding problem. The fact that everything you have tried seems to work especially replacing the ranlib generator with R's (using the standalone library just avoids the calls to R). I am not an expert in this but it seems that your code may not be correctly calling things correctly. Perhaps you should find the types of the variables like Lpos_esp as this really should be an C int. Also, if it remains a problem I wouldn't mind seeing the values and types of the variables used to compute Lpos_esp. Regards Bruce On 7/15/05, Flavio Coelho wrote: > > > 2005/7/15, Todd Miller : > > Todd, > > the function within which the problem happens runs fine by itself, as > well as the simplified version you sent me. > > I am running the code on a AMD64 (though all my software and OS is compiled > to 686 - Gentoo Linux). Is there any issue I should be aware of, regarding > this specific cpu? I can try to run the software on another machine... > > > > Having this code is a good start to solving the problem. I think the > > next step is to simplify your example to make it runnable and provide > > known inputs for all the parameters which lead to a failure for you. > > > > Being really literal (spelling out every single damn thing) cuts down on > > speculation and inefficiency on our end. > > > > It would also be good to express the condition (or it's inverse) you > > think is an error as an assertion, so something like this might be what > > we need: > > > > from numarray.random_array import poisson > > > > E, I, S = (0,0,0) > > beta,alpha,e,r,delta,B,w,p = (0.001,1,1,0.5,1,0,0,0) > > theta = 0 > > npass = 0 > > N = 100 # total guess here for me > > Lpos_esp = beta*S*((I+theta)/(N+npass))**alpha #Number of > new cases > > Lpos = poisson(Lpos_esp) > > assert not (theta == 0 and Lpos_esp == 0 and Lpos > 0) > > > > The problem is, the above works for me. Make it sane and get it to > > expose the error for you and I'll look into this some more. > > > > Regards, > > Todd > > > > > > > > I wonder if called by itself it would trigger the problem... The > > > commented Lines Is where I catch the errors: when poisson returns a > > > greater than zero number, when called with mean 0. > > > > > > > > > > > > Ranlib uses static floats so it may relate to numerical > > > precision (as > > > well as the random uniform random number generator). Really > > > the only > > > way is for you to debug the ranlib.c file > > > > > > I'll see what I can do. > > > > > > > > > Note that R provides a standalone math library in C that > > > includes the > > > random number generators so you could you those instead. SWIG > > > handles > > > it rather well. > > > > > > > > > I think thats what Rpy already does...is it not? > > > > > > thanks Bruce, > > > > > > Fl?vio > > > > > > > > > Regards > > > Bruce > > > > > > On 7/13/05, Flavio Coelho wrote: > > > > Well, > > > > I am definetly glad that someone has also stumbled onto > > > the same problem. > > > > > > > > But it is nevertheless intriguing, that you can run poisson > > > a million times > > > > with mean zero or negative(it assumes zero mean inthis > > > case) without any > > > > problems by itself. But when I call it within my code, the > > > rate of error is > > > > very high (I would say it returns a wrong result every time, > > > but I haven't > > > > checked every answer). > > > > > > > > Meanwhile, my solution will be: > > > > > > > > import rpy > > > > > > > > n = rpy.r.rpois(n,mean) > > > > > > > > I don't feel I can trust poisson while this "funny" > > > behavior is still > > > > there... > > > > If someone has any Idea of how I could trace this bug > > > please tell me, and > > > > I'll hunt it down. At least I can reproduce it in a very > > > specific context. > > > > > > > > thanks, > > > > > > > > Fl?vio > > > > > > > > 2005/7/12, Sebastian Haase < haase at msg.ucsf.edu>: > > > > > Hi Flavio! > > > > > I had reported this long time ago and this list (about > > > numarray). > > > > > Somehow this got more or less dismissed. If I recall > > > correctly the > > > > argument > > > > > was that nobody could reproduce it (I ran this on Debian > > > Woody , py2.2, > > > > (with > > > > > CVS numarray at the time). > > > > > > > > > > I ended up writting my own wrapper(s): > > > > > def poissonArr(shape=defshape, mean=1): > > > > > from numarray import random_array as ra > > > > > if mean == 0: > > > > > return zeroArrF(shape) > > > > > elif mean < 0: > > > > > raise "poisson not defined for mean < 0" > > > > > else: > > > > > return ra.poisson(mean, shape).astype > > > (na.UInt16) > > > > > > > > > > def poissonize(arr): > > > > > from numarray import random_array as ra > > > > > return na.where(arr<=0, 0, ra.poisson(arr)).astype > > > ( na.UInt16) > > > > > > > > > > (I use the astype( na.UInt16) because of some OpenGL code) > > > > > > > > > > Just last week had this problem on a windows98 computer > > > (python2.4). > > > > > > > > > > This should get sorted out ... > > > > > > > > > > Thanks for reporting this problem. > > > > > Sebastian Haase > > > > > > > > > > > > > > > > > > > > > > > > > On Tuesday 12 July 2005 13:32, Flavio Coelho wrote: > > > > > > Hi, > > > > > > > > > > > > I am having problems with the poisson random number > > > generators of both > > > > > > Numarray and Numeric. > > > > > > I can't replicate it when calling the function from the > > > python cosonle, > > > > but > > > > > > it has consistently malfunctioned when used within one > > > of my scripts. > > > > > > > > > > > > What happens is that it frequently return a value > > > greater than zero when > > > > > > called with zero mean: poisson( 0.0) > > > > > > > > > > > > Unfortunately My program is too big to send attached but > > > I have > > > > confirmed > > > > > > the malfunction by printing both the mean and the result > > > whenever it > > > > spits > > > > > > out a wrong result. > > > > > > > > > > > > This is very weird indeed, I have run poisson millions > > > of times by itsel > > > > on > > > > > > the python console, without any problems... > > > > > > > > > > > > I hope it is some stupid mistake, but when I replace the > > > poisson > > > > function > > > > > > call within my program by the R equivalent command > > > (rpois) via the rpy > > > > > > wrapper, everything works just fine... > > > > > > > > > > > > any Ideas? > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Fl?vio Code?o Coelho > > > > registered Linux user # 386432 > > > > --------------------------- > > > > Great spirits have always found violent opposition from > > > mediocrities. The > > > > latter cannot understand it when a man does not > > > thoughtlessly submit to > > > > hereditary prejudices but honestly and courageously uses his > > > intelligence. > > > > Albert Einstein > > > > > > > > > > > > > > > > -- > > > Fl?vio Code?o Coelho > > > registered Linux user # 386432 > > > --------------------------- > > > Great spirits have always found violent opposition from mediocrities. > > > The > > > latter cannot understand it when a man does not thoughtlessly submit > > > to > > > hereditary prejudices but honestly and courageously uses his > > > intelligence. > > > Albert Einstein > > > > > > > > -- > > Fl?vio Code?o Coelho > registered Linux user # 386432 > --------------------------- > Great spirits have always found violent opposition from mediocrities. The > latter cannot understand it when a man does not thoughtlessly submit to > hereditary prejudices but honestly and courageously uses his intelligence. > Albert Einstein > From gr at grrrr.org Mon Jul 18 01:46:08 2005 From: gr at grrrr.org (Thomas Grill) Date: Mon Jul 18 01:46:08 2005 Subject: [Numpy-discussion] optimizations In-Reply-To: <78581171a9e444def1705b4a1eb5067c@grrrr.org> References: <1120929968.4884.2.camel@localhost.localdomain> <78581171a9e444def1705b4a1eb5067c@grrrr.org> Message-ID: <2df447904c0dc255e826fcffc2c3703d@grrrr.org> Ok, i'm answering myself partly since no-one else did > I realized that the ufuncs in numarrays are not as fast as possible > because they are coded very straightforward. Is there any particular > reason why this is the case, like cleanness of code, confidence in the > compiler or because the code was automatically generated? > sorry for disregarding the manual in this respect... clearly the codelets are generated > I would like to contribute assembly SIMD codelets (SSE and Altivec), i > have been successfully using in my projects for quite some time. Is it > feasible to submit patches, so that these go into the main numarray > distribution, or should i rather implement this as separate ufuncs? > well then, is it an option to provide hand-written ufunc implementations, circumventing the code generation? Is the function nomenclature stable enough that this would work for a longer time? thanks, Thomas From webb.sprague at gmail.com Mon Jul 18 11:14:35 2005 From: webb.sprague at gmail.com (Egg) Date: Mon Jul 18 11:14:35 2005 Subject: [Numpy-discussion] Reversing elements of array Message-ID: Hi all, I would like to reverse the elements of an array (or vector). If it were a regular Python list, I could say L.reverse(). If it were in Matlab (ick), I could say U(4:-1:1). Can anyone give me a hint on doing this in Numeric (not Numarray)? Thx From faltet at carabos.com Mon Jul 18 11:23:07 2005 From: faltet at carabos.com (Francesc Altet) Date: Mon Jul 18 11:23:07 2005 Subject: [Numpy-discussion] Reversing elements of array In-Reply-To: References: Message-ID: <200507182021.18324.faltet@carabos.com> A Monday 18 July 2005 20:00, Egg va escriure: > I would like to reverse the elements of an array (or vector). If it were a > regular Python list, I could say L.reverse(). If it were in Matlab (ick), > I could say U(4:-1:1). Can anyone give me a hint on doing this in Numeric > (not Numarray)? >>> import Numeric >>> a=Numeric.arange(10) >>> a[::-1] array([9, 8, 7, 6, 5, 4, 3, 2, 1, 0]) (the same works for python lists) Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From rkern at ucsd.edu Mon Jul 18 11:50:26 2005 From: rkern at ucsd.edu (Robert Kern) Date: Mon Jul 18 11:50:26 2005 Subject: [Numpy-discussion] Reversing elements of array In-Reply-To: References: Message-ID: <42DBF1FE.6090700@ucsd.edu> Egg wrote: > Hi all, > > I would like to reverse the elements of an array (or vector). If it were a > regular Python list, I could say L.reverse(). If it were in Matlab (ick), I > could say U(4:-1:1). Can anyone give me a hint on doing this in Numeric (not > Numarray)? In [2]: U = arange(10) In [3]: U[::-1] Out[3]: array([9, 8, 7, 6, 5, 4, 3, 2, 1, 0]) -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From haase at msg.ucsf.edu Tue Jul 19 21:12:03 2005 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue Jul 19 21:12:03 2005 Subject: [Numpy-discussion] unexpected behavior of na.where (no short-circuiting) In-Reply-To: <200507051734.59020.haase@msg.ucsf.edu> References: <200507051734.59020.haase@msg.ucsf.edu> Message-ID: <200507191740.43694.haase@msg.ucsf.edu> Hi, any thoughts on this ? I really think it's counter intuitive ? Thanks, Sebastian Haase On Tuesday 05 July 2005 17:34, Sebastian Haase wrote: > Hi, > > I was very surprised when I got this warning: > >>> a = na.arange(4)-2 > >>> na.where(a != 0,1./a, 999) > > Warning: Encountered divide by zero(s) in divide > [ -0.5 -1. 999. 1. ] > > Then I realized that this is generally referred to as (not) "short > circuiting" (e.g. in the case of the '?:'-C-operator, the middle part never > gets evaluated at all if the first part evals to 0 ) > > Especially annoying was this because (for debugging) I had set this > > error-mode: > >>> na.Error.setMode(dividebyzero="error") > > My questions are: > a) Did other people encounter this problem ? > b) What is the general feeling about this actually being a "problem" ? > c) Could this (at all possible) be implemented differently ? > > Thanks, > Sebastian Haase > > > > > > > > ------------------------------------------------------- > SF.Net email is sponsored by: Discover Easy Linux Migration Strategies > from IBM. Find simple to follow Roadmaps, straightforward articles, > informative Webcasts and more! Get everything you need to get up to > speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From focke at slac.stanford.edu Wed Jul 20 10:37:17 2005 From: focke at slac.stanford.edu (Warren Focke) Date: Wed Jul 20 10:37:17 2005 Subject: [Numpy-discussion] unexpected behavior of na.where (no short-circuiting) In-Reply-To: <200507191740.43694.haase@msg.ucsf.edu> References: <200507051734.59020.haase@msg.ucsf.edu> <200507191740.43694.haase@msg.ucsf.edu> Message-ID: Well, that's the way Python works - function arguments are evaluated before the function is called. Even if there were some way to pass the arguments in as expressions, and evaluate them on the fly, you'd need some sort of lazy evaluation, and some pretty deep parsing of the expression - in this case, you could calc 1./a[x,y,z,whatever] for the appropriate values of [x,y,z,whatever], but what if, instead of 1./a, you have divide.outer(b,c)? Or videoCamera.getNextFrame()? I don't think the general problem of figuring out how to iterate over the indices of subexpressions in the argument to get the appropriate index into the evaluated argument is soluble, even when there are indexable subexpressions. In this case, you can pre-mask the array: >>> a = na.arange(4)-2 a[a==0] = 1./999 >>> na.where(a != 0,1./a, 999) Warren Focke On Tue, 19 Jul 2005, Sebastian Haase wrote: > Hi, > any thoughts on this ? I really think it's counter intuitive ? > > Thanks, > Sebastian Haase > > On Tuesday 05 July 2005 17:34, Sebastian Haase wrote: > > Hi, > > > > I was very surprised when I got this warning: > > >>> a = na.arange(4)-2 > > >>> na.where(a != 0,1./a, 999) > > > > Warning: Encountered divide by zero(s) in divide > > [ -0.5 -1. 999. 1. ] > > > > Then I realized that this is generally referred to as (not) "short > > circuiting" (e.g. in the case of the '?:'-C-operator, the middle part never > > gets evaluated at all if the first part evals to 0 ) > > > > Especially annoying was this because (for debugging) I had set this > > > > error-mode: > > >>> na.Error.setMode(dividebyzero="error") > > > > My questions are: > > a) Did other people encounter this problem ? > > b) What is the general feeling about this actually being a "problem" ? > > c) Could this (at all possible) be implemented differently ? > > > > Thanks, > > Sebastian Haase > > > > > > > > > > > > > > > > ------------------------------------------------------- > > SF.Net email is sponsored by: Discover Easy Linux Migration Strategies > > from IBM. Find simple to follow Roadmaps, straightforward articles, > > informative Webcasts and more! Get everything you need to get up to > > speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > ------------------------------------------------------- > SF.Net email is sponsored by: Discover Easy Linux Migration Strategies > from IBM. Find simple to follow Roadmaps, straightforward articles, > informative Webcasts and more! Get everything you need to get up to > speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From Chris.Barker at noaa.gov Wed Jul 20 11:48:14 2005 From: Chris.Barker at noaa.gov (Chris Barker) Date: Wed Jul 20 11:48:14 2005 Subject: [Numpy-discussion] LAPACK/BLAS for Fedora core 4 i386 ? Message-ID: <42DE9BEC.9070902@noaa.gov> Hi all, Can anyone recommend the best way to get a native LAPACK installed on Fedora core 4? I'm quite surprised that I can't find an atlas rpm with yum. While I'm at it, when I was using Gentoo, it came with a nice atlas blas/lapack. Unfortunately, the atlas lapack does not include all of lapack. It has everything Numeric/numarray needs, but not some stuff i need for another project (Banded Matrix solvers). Does anyone have a suggestion for how to add the addional lapack stuff I need, while still using atlas stuff where possible? Frankly, I don't understand why atlas doesn't just include all of lapack, using generic versions for anything they haven't optimized, it would be a lot easier to get one stop shopping. -Chris From dd55 at cornell.edu Wed Jul 20 11:59:20 2005 From: dd55 at cornell.edu (Darren Dale) Date: Wed Jul 20 11:59:20 2005 Subject: [Numpy-discussion] LAPACK/BLAS for Fedora core 4 i386 ? In-Reply-To: <42DE9BEC.9070902@noaa.gov> References: <42DE9BEC.9070902@noaa.gov> Message-ID: <200507201458.39116.dd55@cornell.edu> Hi Chris, On Wednesday 20 July 2005 02:46 pm, Chris Barker wrote: > Hi all, > > Can anyone recommend the best way to get a native LAPACK installed on > Fedora core 4? > > I'm quite surprised that I can't find an atlas rpm with yum. > > While I'm at it, when I was using Gentoo, it came with a nice atlas > blas/lapack. Unfortunately, the atlas lapack does not include all of > lapack. It has everything Numeric/numarray needs, but not some stuff i > need for another project (Banded Matrix solvers). Does anyone have a > suggestion for how to add the addional lapack stuff I need, while still > using atlas stuff where possible? I can't comment on the Fedora question. With Gentoo, you should not use the "atlas" package, because it comes with an incomplete LAPACK, as you pointed out. Instead, Gentoo-ers should use the "blas-atlas" and "lapack-atlas" packages. -- Darren From Chris.Barker at noaa.gov Wed Jul 20 12:11:13 2005 From: Chris.Barker at noaa.gov (Chris Barker) Date: Wed Jul 20 12:11:13 2005 Subject: [Numpy-discussion] LAPACK/BLAS for Fedora core 4 i386 ? In-Reply-To: <200507201458.39116.dd55@cornell.edu> References: <42DE9BEC.9070902@noaa.gov> <200507201458.39116.dd55@cornell.edu> Message-ID: <42DEA1AB.7070200@noaa.gov> Darren Dale wrote: > I can't comment on the Fedora question. With Gentoo, you should not use the > "atlas" package, because it comes with an incomplete LAPACK, as you pointed > out. > > Instead, Gentoo-ers should use the "blas-atlas" and "lapack-atlas" packages. Well, I've moved away from Gentoo, but I just may go back. However, I'm pretty sure I was using the lapack-atlas package, and while it was described as "complete", it did not, in fact, have all of lapack. Maybe I was doing something wrong, however. -thanks, Chris From Chris.Barker at noaa.gov Wed Jul 20 14:57:15 2005 From: Chris.Barker at noaa.gov (Chris Barker) Date: Wed Jul 20 14:57:15 2005 Subject: [Numpy-discussion] Biulding Numarray with atlas. Message-ID: <42DEC894.5030304@noaa.gov> Hi all, I'm working on building numarray with atlas on a Fedora core 4 system. I installed atlas by downloading a binary from the atlas sourceforge site, and copying stuff into /usr/local/... Then in cfc_packages.py, I added: USE_LAPACK = True at the top. That doesn't seem like where I should do it, but it works. then I got: Src/_dotblas.c:15:19: error: cblas.h: No such file or directory So I looked and saw: else: lapack_dirs = ['/usr/local/lib/atlas'] lapack_libs = ['lapack', 'cblas', 'f77blas', 'atlas', 'g2c', 'm'] which makes sense, we haven't added the include directory that cblas.h is in. So I added: lapack_include_dirs += ['/usr/local/include/atlas/'] and now it seems to work. Perhaps an include line should be added to the stock cfc_packages.py -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From jmiller at stsci.edu Thu Jul 21 07:36:27 2005 From: jmiller at stsci.edu (Todd Miller) Date: Thu Jul 21 07:36:27 2005 Subject: [Numpy-discussion] Biulding Numarray with atlas. In-Reply-To: <42DEC894.5030304@noaa.gov> References: <42DEC894.5030304@noaa.gov> Message-ID: <1121954393.12577.0.camel@halloween.stsci.edu> On Wed, 2005-07-20 at 17:56, Chris Barker wrote: > Hi all, > > I'm working on building numarray with atlas on a Fedora core 4 system. I > installed atlas by downloading a binary from the atlas sourceforge site, > and copying stuff into /usr/local/... > > Then in cfc_packages.py, I added: > > USE_LAPACK = True > > at the top. That doesn't seem like where I should do it, but it works. > > then I got: > > Src/_dotblas.c:15:19: error: cblas.h: No such file or directory > > So I looked and saw: > > else: > lapack_dirs = ['/usr/local/lib/atlas'] > lapack_libs = ['lapack', 'cblas', 'f77blas', 'atlas', 'g2c', 'm'] > > which makes sense, we haven't added the include directory that cblas.h > is in. > > So I added: > lapack_include_dirs += ['/usr/local/include/atlas/'] > > and now it seems to work. > > Perhaps an include line should be added to the stock cfc_packages.py Sounds good to me. I added this. Thanks, Todd From jmiller at stsci.edu Thu Jul 21 08:01:12 2005 From: jmiller at stsci.edu (Todd Miller) Date: Thu Jul 21 08:01:12 2005 Subject: [Numpy-discussion] Biulding Numarray with atlas. In-Reply-To: <42DEC894.5030304@noaa.gov> References: <42DEC894.5030304@noaa.gov> Message-ID: <1121954393.12577.0.camel@halloween.stsci.edu> On Wed, 2005-07-20 at 17:56, Chris Barker wrote: > Hi all, > > I'm working on building numarray with atlas on a Fedora core 4 system. I > installed atlas by downloading a binary from the atlas sourceforge site, > and copying stuff into /usr/local/... > > Then in cfc_packages.py, I added: > > USE_LAPACK = True > > at the top. That doesn't seem like where I should do it, but it works. > > then I got: > > Src/_dotblas.c:15:19: error: cblas.h: No such file or directory > > So I looked and saw: > > else: > lapack_dirs = ['/usr/local/lib/atlas'] > lapack_libs = ['lapack', 'cblas', 'f77blas', 'atlas', 'g2c', 'm'] > > which makes sense, we haven't added the include directory that cblas.h > is in. > > So I added: > lapack_include_dirs += ['/usr/local/include/atlas/'] > > and now it seems to work. > > Perhaps an include line should be added to the stock cfc_packages.py Sounds good to me. I added this. Thanks, Todd From jmiller at stsci.edu Thu Jul 21 08:30:23 2005 From: jmiller at stsci.edu (Todd Miller) Date: Thu Jul 21 08:30:23 2005 Subject: [Numpy-discussion] Biulding Numarray with atlas. In-Reply-To: <42DEC894.5030304@noaa.gov> References: <42DEC894.5030304@noaa.gov> Message-ID: <1121954393.12577.0.camel@halloween.stsci.edu> On Wed, 2005-07-20 at 17:56, Chris Barker wrote: > Hi all, > > I'm working on building numarray with atlas on a Fedora core 4 system. I > installed atlas by downloading a binary from the atlas sourceforge site, > and copying stuff into /usr/local/... > > Then in cfc_packages.py, I added: > > USE_LAPACK = True > > at the top. That doesn't seem like where I should do it, but it works. > > then I got: > > Src/_dotblas.c:15:19: error: cblas.h: No such file or directory > > So I looked and saw: > > else: > lapack_dirs = ['/usr/local/lib/atlas'] > lapack_libs = ['lapack', 'cblas', 'f77blas', 'atlas', 'g2c', 'm'] > > which makes sense, we haven't added the include directory that cblas.h > is in. > > So I added: > lapack_include_dirs += ['/usr/local/include/atlas/'] > > and now it seems to work. > > Perhaps an include line should be added to the stock cfc_packages.py Sounds good to me. I added this. Thanks, Todd From Chris.Barker at noaa.gov Thu Jul 21 11:24:13 2005 From: Chris.Barker at noaa.gov (Chris Barker) Date: Thu Jul 21 11:24:13 2005 Subject: [Numpy-discussion] Biulding Numarray with atlas. In-Reply-To: <07C6A61102C94148B8104D42DE95F7E86DEFA5@exchange2k.envision.co.il> References: <07C6A61102C94148B8104D42DE95F7E86DEFA5@exchange2k.envision.co.il> Message-ID: <42DFE82B.6030603@noaa.gov> Nadav Horesh wrote: > Whenever you need a patch to compile numarray with lapack/atlas, you > should check if the linear_algebra module works. Of course. And test I have, and no, it doesn't work: ImportError: /usr/lib/python2.4/site-packages/numarray/linear_algebra/lapack_lite2.so: undefined symbol: dgesdd_ I'm guessing this is a symbol mangling mismatch between what lapack_lite2 is expecting and what my atlas install used (i.e. too many or not enough underscores) For what it's worth, I get the same error with Numeric. which brings me back to my first question: Where should I get a good lapack/blas for Fedora core 4? There are lapack and blas rpms, but there is no indication of whether they are optimized or not. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From jmiller at stsci.edu Thu Jul 21 12:09:17 2005 From: jmiller at stsci.edu (Todd Miller) Date: Thu Jul 21 12:09:17 2005 Subject: [Numpy-discussion] Biulding Numarray with atlas. In-Reply-To: <42DFE82B.6030603@noaa.gov> References: <07C6A61102C94148B8104D42DE95F7E86DEFA5@exchange2k.envision.co.il> <42DFE82B.6030603@noaa.gov> Message-ID: <1121972619.4971.45.camel@halloween.stsci.edu> On Thu, 2005-07-21 at 14:23, Chris Barker wrote: > Nadav Horesh wrote: > > Whenever you need a patch to compile numarray with lapack/atlas, you > > should check if the linear_algebra module works. > > Of course. And test I have, and no, it doesn't work: > > ImportError: > /usr/lib/python2.4/site-packages/numarray/linear_algebra/lapack_lite2.so: > undefined symbol: dgesdd_ > > I'm guessing this is a symbol mangling mismatch between what > lapack_lite2 is expecting and what my atlas install used (i.e. too many > or not enough underscores) For what it's worth, I get the same error > with Numeric. I think this is a SciPy FAQ and following their advice is how I've worked around it in the past myself: http://www.scipy.org/documentation/mailman?fn=scipy-dev/2001-November/000105.html From Chris.Barker at noaa.gov Thu Jul 21 15:28:14 2005 From: Chris.Barker at noaa.gov (Chris Barker) Date: Thu Jul 21 15:28:14 2005 Subject: [Numpy-discussion] Biulding Numarray with atlas. In-Reply-To: <1121972619.4971.45.camel@halloween.stsci.edu> References: <07C6A61102C94148B8104D42DE95F7E86DEFA5@exchange2k.envision.co.il> <42DFE82B.6030603@noaa.gov> <1121972619.4971.45.camel@halloween.stsci.edu> Message-ID: <42E0212C.5060702@noaa.gov> Todd Miller wrote: > I think this is a SciPy FAQ and following their advice is how I've > worked around it in the past myself: > > http://www.scipy.org/documentation/mailman?fn=scipy-dev/2001-November/000105.html I'd seem that note before, and it wasn't very helpful, but then I noticed, in the Numeric customize.py: Example: Some Linux distributions have ATLAS, but not LAPACK (if you're liblapack file is smaller than a megabyte, this is probably you) Use our own f2c'd lapack libraries, but use ATLAS for BLAS. My linux distro (FC4) has a full lapack, but it's not optimized, in fact, it's slower than lapack-lite. However, I got atlas from the atlas site, and it doesn't come with a full lapack, which would explain the missing symbols. Using: # Using ATLAS blas in /usr/local with lapack-lite if 1: use_system_lapack = 0 use_system_blas = 1 lapack_library_dirs = ['/usr/local/lib/atlas'] lapack_libraries = ['cblas', 'f77blas', 'atlas', 'g2c'] It compiles, works, and is about twice as fast as using the the Numeric-supplied blas. I still would like to find a full, optimized lapack for Fedora Core 4. On my last machine, I got about a six times speed-up using the atlas that came with Gentoo. But for the moment, I'm OK. Also, I saw something in the atlas docs about how to merge the atlas lapack with another full lapack, so maybe I can get that to work. Now on to see if I can do something similar with numarray. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Chris.Barker at noaa.gov Thu Jul 21 17:07:24 2005 From: Chris.Barker at noaa.gov (Chris Barker) Date: Thu Jul 21 17:07:24 2005 Subject: [Numpy-discussion] Building Numeric+numarray with atals on Fedora Core 4 In-Reply-To: <42E0212C.5060702@noaa.gov> References: <07C6A61102C94148B8104D42DE95F7E86DEFA5@exchange2k.envision.co.il> <42DFE82B.6030603@noaa.gov> <1121972619.4971.45.camel@halloween.stsci.edu> <42E0212C.5060702@noaa.gov> Message-ID: <42E03313.7090208@noaa.gov> It looks like I'm managing to maintain this thread all by myself! I thought I'd post this final note, so that anyone searching the archives can see my solution. It's a bit ugly, but it seems to work. First the summary: I'm trying to build both Numeric and numarray on a new installation of Linux Fedora Core 4 on a Pentium M laptop. Fedora helpfully provides a full lapack and blas rpms, but they don't appear to be at all optimized. In fact, when I used them, it was slower than the built-in lapack-lite. I haven't been able to find a complete atlas+lapack distribution for Fedora core 4 (or anything like it), so this is what I did: Got the atlas binaries from the atlas site: https://sourceforge.net/projects/math-atlas/ I downloaded: atlas3.6.0_Linux_P4SSE2.tar.gz NOTE: these are pretty old...should I be using the 3.7.10 unstable version instead? I then merged these with the Fedora lapack (should be in /usr/lib/liblapack.a) by following the directions in the atlas README: """******** GETTING A FULL LAPACK LIB ************************** ATLAS does not provide a full lapack library. However, there is a simple way to get ATLAS to provide its faster LAPACK routines to a full LAPACK library. ATLAS's internal routines are distinct from LAPACK's, so it is safe to compile ATLAS's LAPACK routines directly into a netlib-style LAPACK library. First, obtain the LAPACK src from netlib and build the LAPACK library as normal. Then, in this directory (where you should have a liblapack.a), issue the following commands: mkdir tmp cd tmp ar x ../liblapack.a cp ../liblapack.a ar r ../liblapack.a *.o cd .. rm -rf tmp Just linking in ATLAS's liblapack.a first will not get you the best LAPACK performance, mainly because LAPACK's untuned ILAENV will be used instead of ATLAS's tuned one. """ I installed them in: /usr/local/lib/atlas and /usr/local/include/atlas Then I set up the numarray and Numeric build process to use them. For numarray, I edited cfg_packages.py, and put in: USE_LAPACK = True near the top, and edited on of the blocks to read: print "Adding paths for lapack/blas" lapack_dirs = ['/usr/local/lib/atlas'] lapack_libs = ['lapack', 'cblas', 'f77blas', 'atlas', 'g2c', 'gfortran'] lapack_include_dirs += ['/usr/local/include/atlas/'] NOTE: the gfortran lib is something I hadn't seen before, but I needed it to resolve missing symbols. I'm guess that the fedora lapack used gfortran. then a setup.py build, setup.py install, and it seems to work, and is about twice as fast as lapack-lite on my simple test case. In the past, I got about a 7 times speed-up with the atlas that I got from Gentoo. I'm a bit disappointed, but it's OK, and I'm not really doing anything where it matters much. For Numeric, I edited customize.py: # Using ATLAS blas in /usr/local if 1: use_system_lapack = 1 lapack_library_dirs = ['/usr/local/lib/atlas'] lapack_libraries = ['lapack', 'cblas', 'f77blas', 'atlas', 'g2c', 'gfortran'] And that did it. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From python-ml at nn7.de Sat Jul 23 09:04:16 2005 From: python-ml at nn7.de (Soeren Sonnenburg) Date: Sat Jul 23 09:04:16 2005 Subject: [Numpy-discussion] design issues - octave 'incompatibilities' Message-ID: <1122134630.26083.74.camel@localhost> Dear all, I am new to numarray and as I am trying to use it day-by-day I am now wondering about certain numeric/numarray design issues. Please forgive me / correct me as I probably misunderstood certain issues. -- Why did you choose row-major instead of column major format as practiced by R/octave/matlab... Doesn't that mean performance problems as fortran/blas is optimized if you work with the transposed version ? -- why do vectors have no 'orientation', i.e. there are only row but no column vectors (or why do you treat matrices/vectors differently, i.e. have vectors at all as a separate type) numarray: a=array([[1,2,3],[4,5,6],[7,8,9]]) >>> a[0,:] array([1, 2, 3]) >>> a[:,0] array([1, 4, 7]) vs. octave: a=[1,2,3;4,5,6;7,8,9]; >> a(1,:) ans = 1 2 3 >> a(:,1) ans = 1 4 7 -- How can one obtain submatrices from full matrices: numarray gives only elements (which is very, very rarely needed and should IMHO rather be some extra function): >>> i=[0,1] >>> j=[0,2] >>> a[i,j] array([1, 6]) vs octave: >> i=[1,2]; >> j=[1,3]; >> a(i,j) ans = 1 3 4 6 all one can do in numarray currently is as awkward as this: >>> a.take(i,0).take(j,1) array([[1, 3], [4, 6]]) also mixing slice and variable does not work a[:,i] - hmmhh seems that is a major issue for me... -- why don't you allow iterating over the whole matrix via a single index ? i.e. a[3] Traceback (most recent call last): File "", line 1, in ? IndexError: Index out of range vs octave >> a(4) ans = 2 Are there more elegant ways to do this ? Which issues are likely to be fixed in the future ? Best regards, Soeren From python-ml at nn7.de Sat Jul 23 09:13:16 2005 From: python-ml at nn7.de (Soeren Sonnenburg) Date: Sat Jul 23 09:13:16 2005 Subject: [Numpy-discussion] C API function to get the shape of an array ? Message-ID: <1122135157.26083.84.camel@localhost> Hi, I just want to use NA_get1_Int64 etc but I couldn't find a way to 'cleanly' find out about the size of an array, i.e. I always have to access raw fields in the PyArrayObject structure like: PyArrayObject* a; if (a->nd == 2) { if (a->dimensions[0] == m && b->dimensions[1] == n) ... Is this the intended way ? I hope not... Best, Soeren From rkern at ucsd.edu Sat Jul 23 12:07:11 2005 From: rkern at ucsd.edu (Robert Kern) Date: Sat Jul 23 12:07:11 2005 Subject: [Numpy-discussion] design issues - octave 'incompatibilities' In-Reply-To: <1122134630.26083.74.camel@localhost> References: <1122134630.26083.74.camel@localhost> Message-ID: <42E29537.7080702@ucsd.edu> Soeren Sonnenburg wrote: > Dear all, > > I am new to numarray and as I am trying to use it day-by-day > I am now wondering about certain numeric/numarray design issues. Please > forgive me / correct me as I probably misunderstood certain issues. > > -- Why did you choose row-major instead of column major format as > practiced by R/octave/matlab... Doesn't that mean performance problems > as fortran/blas is optimized if you work with the transposed version ? Not everyone is interfacing with optimized FORTRAN code. Those who are are usually using ATLAS as their BLAS, and ATLAS has row-major versions of the BLAS subroutines. Row-major is C's convention and numarray/Numeric largely follow that. There are of course some performance issues when interfacing with FORTRAN code that expects column-major, but there would be other performance problems if numarray/Numeric were column-major and interfacing with row-major code. > -- why do vectors have no 'orientation', i.e. there are only row but no > column vectors (or why do you treat matrices/vectors differently, i.e. > have vectors at all as a separate type) You can certainly make column vectors. In [1]: a = arange(5) In [2]: a Out[2]: NumPy array, format: long [0 1 2 3 4] In [3]: a.shape = (5,1) In [4]: a Out[4]: NumPy array, format: long [[0] [1] [2] [3] [4]] Often, though, a "row" vector can also be used in place of a "column" vector. Although sometimes not: > -- How can one obtain submatrices from full matrices: In [6]: import numarray In [7]: a = numarray.arange(1, 10) In [8]: a.shape = (3,3) In [9]: a Out[9]: array([[1, 2, 3], [4, 5, 6], [7, 8, 9]]) In [10]: i = [0,1] In [11]: j = [[0],[2]] In [12]: a[i,j] Out[12]: array([[1, 4], [3, 6]]) > -- why don't you allow iterating over the whole matrix via a single > index ? ravel() > Are there more elegant ways to do this ? Which issues are likely to be > fixed in the future ? None. They're not broken, just different from what you are used to. -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From python-ml at nn7.de Sun Jul 24 11:42:30 2005 From: python-ml at nn7.de (Soeren Sonnenburg) Date: Sun Jul 24 11:42:30 2005 Subject: [Numpy-discussion] design issues - octave 'incompatibilities' In-Reply-To: <42E29537.7080702@ucsd.edu> References: <1122134630.26083.74.camel@localhost> <42E29537.7080702@ucsd.edu> Message-ID: <1122230465.30698.72.camel@localhost> On Sat, 2005-07-23 at 12:06 -0700, Robert Kern wrote: > Soeren Sonnenburg wrote: > > Dear all, > > > > I am new to numarray and as I am trying to use it day-by-day > > I am now wondering about certain numeric/numarray design issues. Please > > forgive me / correct me as I probably misunderstood certain issues. > > > > -- Why did you choose row-major instead of column major format as > > practiced by R/octave/matlab... Doesn't that mean performance problems > > as fortran/blas is optimized if you work with the transposed version ? > > Not everyone is interfacing with optimized FORTRAN code. Those who are > are usually using ATLAS as their BLAS, and ATLAS has row-major versions > of the BLAS subroutines. Row-major is C's convention and > numarray/Numeric largely follow that. There are of course some > performance issues when interfacing with FORTRAN code that expects > column-major, but there would be other performance problems if > numarray/Numeric were column-major and interfacing with row-major code. Well but why did you change to the C version then ? Well maybe if it is about optimizing stuff seriously one could work with the transpose anyway... > Often, though, a "row" vector can also be used in place of a "column" > vector. Although sometimes not: You are right, thanks! > > Are there more elegant ways to do this ? Which issues are likely to be > > fixed in the future ? > > None. They're not broken, just different from what you are used to. Thanks a lot for your very helpful mail! Do you know whether mixing slices and arrays will be supported at some point a[:,[0,1]] ? Soeren. From rkern at ucsd.edu Sun Jul 24 11:49:25 2005 From: rkern at ucsd.edu (Robert Kern) Date: Sun Jul 24 11:49:25 2005 Subject: [Numpy-discussion] design issues - octave 'incompatibilities' In-Reply-To: <1122230465.30698.72.camel@localhost> References: <1122134630.26083.74.camel@localhost> <42E29537.7080702@ucsd.edu> <1122230465.30698.72.camel@localhost> Message-ID: <42E3E260.5050406@ucsd.edu> Soeren Sonnenburg wrote: > On Sat, 2005-07-23 at 12:06 -0700, Robert Kern wrote: > >>Soeren Sonnenburg wrote: >> >>>Dear all, >>> >>>I am new to numarray and as I am trying to use it day-by-day >>>I am now wondering about certain numeric/numarray design issues. Please >>>forgive me / correct me as I probably misunderstood certain issues. >>> >>>-- Why did you choose row-major instead of column major format as >>>practiced by R/octave/matlab... Doesn't that mean performance problems >>>as fortran/blas is optimized if you work with the transposed version ? >> >>Not everyone is interfacing with optimized FORTRAN code. Those who are >>are usually using ATLAS as their BLAS, and ATLAS has row-major versions >>of the BLAS subroutines. Row-major is C's convention and >>numarray/Numeric largely follow that. There are of course some >>performance issues when interfacing with FORTRAN code that expects >>column-major, but there would be other performance problems if >>numarray/Numeric were column-major and interfacing with row-major code. > > Well but why did you change to the C version then ? Well maybe if it is > about optimizing stuff seriously one could work with the transpose > anyway... Because numarray and Python are written in C. It's also consistent with lists-of-lists. In [8]: L = [[0, 1],[2,3]] In [9]: A = array(L) In [10]: L[1][0] Out[10]: 2 In [11]: A[1][0] Out[11]: 2 In [12]: A[1,0] Out[12]: 2 -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From perry at stsci.edu Mon Jul 25 07:11:17 2005 From: perry at stsci.edu (Perry Greenfield) Date: Mon Jul 25 07:11:17 2005 Subject: [Numpy-discussion] design issues - octave 'incompatibilities' In-Reply-To: <1122230465.30698.72.camel@localhost> References: <1122134630.26083.74.camel@localhost> <42E29537.7080702@ucsd.edu> <1122230465.30698.72.camel@localhost> Message-ID: <89be641d30d1e246ea18d3733b6c3357@stsci.edu> On Jul 24, 2005, at 2:41 PM, Soeren Sonnenburg wrote: > > Well but why did you change to the C version then ? Well maybe if it is > about optimizing stuff seriously one could work with the transpose > anyway... > To get a really solid answer to "why" you would have to ask those that wrote the original Numeric. (Jim Hugunin and Co.). My guess is was that it was just as much to preserve the following relation arr[i,j] = arr[i][j] (instead of arr[i,j] = arr[j][i]) But I could be wrong. Note that this is a confusing issue to many and often as not there are unstated assumptions that are repeatedly made by many that are *not* shared by everyone. To illustrate, there are at least two different approaches to handling this mismatch and it seems to me that many seem oblivious to the fact that there are two approaches. Approach 1: Keep the index convention the same. As a user of Numeric or numarray, you wish M[i,j] to mean the same as it does in Fortran/IDL/matlab... The consequence is that the data are ordered differently between Numeric/numarray and these other languages. This is seen as a data compatibility problem. Approach 2: Keep the data invariant and change the indexing convention. What was M[i,j] in Fortran is now M[j,i] in Numeric/numarray. This is not a data compatibility problem, but is now a brain compatibility problem ;-) Since we deal with big data sets we have adopted 2 (no doubt to the consternation of many). This ought to be in a FAQ, it comes up enough to be :-) > > Do you know whether mixing slices and arrays will be supported at some > point a[:,[0,1]] ? > > Soeren. No plans at the moment. We figured indexing was complicated enough as it was. I think Travis Oliphant did allow for this in Numeric3 (aka scipy.core); see the link below. But it may not mean what you think it should so you should check that out to see: http://cvs.sourceforge.net/viewcvs.py/numpy/Numeric3/PEP.txt?view=markup Perry Greenfield From Chris.Barker at noaa.gov Mon Jul 25 09:01:03 2005 From: Chris.Barker at noaa.gov (Chris Barker) Date: Mon Jul 25 09:01:03 2005 Subject: [Numpy-discussion] design issues - octave 'incompatibilities' In-Reply-To: <1122134630.26083.74.camel@localhost> References: <1122134630.26083.74.camel@localhost> Message-ID: <42E50C49.8080705@noaa.gov> Soeren Sonnenburg wrote: > -- why do vectors have no 'orientation', i.e. there are only row but no > column vectors (or why do you treat matrices/vectors differently, i.e. > have vectors at all as a separate type) I think the key to understanding this is that NumPy uses a fundamentally different data type that MATLAB ans it's derivatives. MATLAB was originally just what it is called: a "Matrix" laboratory. The basic data type of Matlab is a 2-d matrix. Even a scalar is really a 1X! matrix. Matlab has a few tricks that can make these look like row and column vectors, etc, but they are really always matrices. On the other hand, NumPy arrays are N-dimensional, where N is theoretically unlimited. In practice, I think the max N is defined and compiled in, but you could change it and re-compile if you wanted. In any case, many of us frequently use 3-d and higher arrays, and they can be very useful. When thought of this way, you can see why there is no such thing as a column vs. a row vector. A vector is a one-dimensional array: it has no orientation. However, NumPy does support NX1 and 1XN 2-d arrays which can be very handy: >>> import numarray as N >>> a = N.arange(5) >>> a.shape = (1,-1) >>> a array([[0, 1, 2, 3, 4]]) >>> b = N.arange(5) >>> b.shape = (-1,1) >>> a array([0, 1, 2, 3, 4]) >>> b array([[0], [1], [2], [3], [4]]) So a is a row vector and b is a column vector. If you multiply them, you get "array broadcasting": >>> a * b array([[ 0, 0, 0, 0, 0], [ 0, 1, 2, 3, 4], [ 0, 2, 4, 6, 8], [ 0, 3, 6, 9, 12], [ 0, 4, 8, 12, 16]]) This eliminates a LOT of extra duplicate arrays that you have to make in Matlab with meshgrid. When you index into an array, you reduce its rank (number of dimensions) by 1: >>> a = N.arange(27) >>> a.shape = (3,3,3) >>> a array([[[ 0, 1, 2], [ 3, 4, 5], [ 6, 7, 8]], [[ 9, 10, 11], [12, 13, 14], [15, 16, 17]], [[18, 19, 20], [21, 22, 23], [24, 25, 26]]]) >>> a.shape (3, 3, 3) >>> a[1].shape (3, 3) >>> a[1][1].shape (3,) When you slice, you keep the rank the same: >>> a[1:2].shape (1, 3, 3) This creates a way to make row and column "vectors" from your 2-d array (matrix) >>> a = N.arange(25) >>> a.shape = (5,5) >>> a array([[ 0, 1, 2, 3, 4], [ 5, 6, 7, 8, 9], [10, 11, 12, 13, 14], [15, 16, 17, 18, 19], [20, 21, 22, 23, 24]]) To make a "row vector" (really a 1XN matrix) >>> a[0:1,:] array([[0, 1, 2, 3, 4]]) To make a "column vector" (really a NX1 matrix) >>> a[:,0:1] array([[ 0], [ 5], [10], [15], [20]]) I hope that helps: -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From oliphant at ee.byu.edu Mon Jul 25 14:11:08 2005 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Jul 25 14:11:08 2005 Subject: [Numpy-discussion] design issues - octave 'incompatibilities' In-Reply-To: <1122134630.26083.74.camel@localhost> References: <1122134630.26083.74.camel@localhost> Message-ID: <42E5551D.1030203@ee.byu.edu> Soeren Sonnenburg wrote: >Dear all, > >I am new to numarray and as I am trying to use it day-by-day >I am now wondering about certain numeric/numarray design issues. Please >forgive me / correct me as I probably misunderstood certain issues. > >-- Why did you choose row-major instead of column major format as >practiced by R/octave/matlab... Doesn't that mean performance problems >as fortran/blas is optimized if you work with the transposed version ? > >-- why do vectors have no 'orientation', i.e. there are only row but no >column vectors (or why do you treat matrices/vectors differently, i.e. >have vectors at all as a separate type) > > I think others have responded to these questions well. >-- How can one obtain submatrices from full matrices: > >numarray gives only elements (which is very, very rarely needed and >should IMHO rather be some extra function): > > I thought about this quite a bit because I initially felt as you did before beginning work on scipy.base (Numeric3). I wondered if the right choice had been made. After some thought and discussion, I decided this approach was more general and flexible, and agreed with the numarray design. Therefore, this works in scipy.base the same. Also, scipy.base does allow mixing slices and lists like you want: import scipy.base a = scipy.base.array([[1,2,3],[4,5,6],[7,8,9]]) >>> a[:,[0,1]] array([[1, 2], [4, 5], [7, 8]], 'l') > >-- why don't you allow iterating over the whole matrix via a single >index ? > > You can with scipy.base with a.flat[index] but ravel(a)[index] always works too. The reason is that Numeric uses the concept of reduced-dimensionality arrays so that a[index] for a 2-d array returns a 1-d array not an element of the 2-d array. It may be different then you are used to but I think it is a better choice. Best regards, -Travis Oliphant From perry at stsci.edu Mon Jul 25 14:48:05 2005 From: perry at stsci.edu (Perry Greenfield) Date: Mon Jul 25 14:48:05 2005 Subject: [Numpy-discussion] design issues - octave 'incompatibilities' In-Reply-To: <1122134630.26083.74.camel@localhost> References: <1122134630.26083.74.camel@localhost> Message-ID: <20e127854b47f13f1210e2f3574584f9@stsci.edu> I missed this part and was reminded by Travis's message. On Jul 23, 2005, at 12:03 PM, Soeren Sonnenburg wrote: > -- How can one obtain submatrices from full matrices: > > numarray gives only elements (which is very, very rarely needed and > should IMHO rather be some extra function): > >>>> i=[0,1] >>>> j=[0,2] >>>> a[i,j] > array([1, 6]) > > vs octave: >>> i=[1,2]; >>> j=[1,3]; >>> a(i,j) > ans = > 1 3 > 4 6 > Maybe for you it is rarely needed, but not for us. The situation is reversed. The latter is rarely used in our case. This is largely a reflection of whether your orientation is matrices or multidimensional arrays. In our case it is quite handy to select out a list of point in an image by giving a list of their respective indices (e.g., stars). True, I do see that others may need the other view, but then the question is which should get the convenience. Since Numeric/numarray are primarily oriented towards multidimensional arrays (e.g., operations are element-by-element rather than matrix) it seemed to make sense to go this way for consistency, but I understand that there is another side to this. Perry From python-ml at nn7.de Mon Jul 25 21:36:23 2005 From: python-ml at nn7.de (Soeren Sonnenburg) Date: Mon Jul 25 21:36:23 2005 Subject: [Numpy-discussion] design issues - octave 'incompatibilities' In-Reply-To: <89be641d30d1e246ea18d3733b6c3357@stsci.edu> References: <1122134630.26083.74.camel@localhost> <42E29537.7080702@ucsd.edu> <1122230465.30698.72.camel@localhost> <89be641d30d1e246ea18d3733b6c3357@stsci.edu> Message-ID: <1122352506.30698.136.camel@localhost> On Mon, 2005-07-25 at 10:10 -0400, Perry Greenfield wrote: > On Jul 24, 2005, at 2:41 PM, Soeren Sonnenburg wrote: > > > > > Well but why did you change to the C version then ? Well maybe if it is > > about optimizing stuff seriously one could work with the transpose > > anyway... > > > > To get a really solid answer to "why" you would have to ask those that > wrote the original Numeric. (Jim Hugunin and Co.). My guess is was that > it was just as much to preserve the following relation > > arr[i,j] = arr[i][j] > > (instead of arr[i,j] = arr[j][i]) Ok, that sounds reasonable. > But I could be wrong. > > Note that this is a confusing issue to many and often as not there are > unstated assumptions that are repeatedly made by many that are *not* > shared by everyone. To illustrate, there are at least two different > approaches to handling this mismatch and it seems to me that many seem > oblivious to the fact that there are two approaches. > > Approach 1: Keep the index convention the same. As a user of Numeric or > numarray, you wish M[i,j] to mean the same as it does in > Fortran/IDL/matlab... The consequence is that the data are ordered > differently between Numeric/numarray and these other languages. This is > seen as a data compatibility problem. > > Approach 2: Keep the data invariant and change the indexing convention. > What was M[i,j] in Fortran is now M[j,i] in Numeric/numarray. This is > not a data compatibility problem, but is now a brain compatibility > problem ;-) well it might not be *that* bad in the end... only it is a tough decision to break with everything that is there (to put it to extreme: the world is wrong and but we did it right) and be compatible with C like array access... If one does so one needs to have very serious reasons to do so ... that is why I was asking. > Since we deal with big data sets we have adopted 2 (no doubt to the > consternation of many). hmmhh, there is no advantage with big datasets for any of the formats. > > Do you know whether mixing slices and arrays will be supported at some > > point a[:,[0,1]] ? > > > > Soeren. > > No plans at the moment. We figured indexing was complicated enough as > it was. I think Travis Oliphant did allow for this in Numeric3 (aka > scipy.core); see the link below. But it may not mean what you think it > should so you should check that out to see: > > http://cvs.sourceforge.net/viewcvs.py/numpy/Numeric3/PEP.txt?view=markup Hmmhh while we are at it, is there work on a consensus num* ? I've seen the PEP for inclusion of numarray, now I see numeric3 and scipy and also cvxopt - are people actually converging towards a single num* package ? Soeren. From python-ml at nn7.de Mon Jul 25 22:23:11 2005 From: python-ml at nn7.de (Soeren Sonnenburg) Date: Mon Jul 25 22:23:11 2005 Subject: [Numpy-discussion] design issues - octave 'incompatibilities' In-Reply-To: <42E50C49.8080705@noaa.gov> References: <1122134630.26083.74.camel@localhost> <42E50C49.8080705@noaa.gov> Message-ID: <1122355332.30698.160.camel@localhost> On Mon, 2005-07-25 at 08:59 -0700, Chris Barker wrote: > Soeren Sonnenburg wrote: > > -- why do vectors have no 'orientation', i.e. there are only row but no > > column vectors (or why do you treat matrices/vectors differently, i.e. > > have vectors at all as a separate type) > > I think the key to understanding this is that NumPy uses a > fundamentally different data type that MATLAB and it's derivatives. > MATLAB was originally just what it is called: a "Matrix" laboratory. The > basic data type of Matlab is a 2-d matrix. Even a scalar is really a 1X! > matrix. Matlab has a few tricks that can make these look like row and > column vectors, etc, but they are really always matrices. Ok, I am realizing that R also distinguishes between vectors and matrices. > On the other hand, NumPy arrays are N-dimensional, where N is > theoretically unlimited. In practice, I think the max N is defined and > compiled in, but you could change it and re-compile if you wanted. In > any case, many of us frequently use 3-d and higher arrays, and they can > be very useful. When thought of this way, you can see why there is no Well at least this is the same for octave/matlab. > such thing as a column vs. a row vector. A vector is a one-dimensional > array: it has no orientation. This makes life more difficult if one wants to convert from octave/matlab -> numarray and automated systems close to impossible. If vectors had the same properties/functions as matrices one would not have such problems, i.e. v^{transpose} * u == dot(v,u) and v*u -> error > However, NumPy does support NX1 and 1XN 2-d arrays which can be very handy: > >>> import numarray as N > >>> a = N.arange(5) > >>> a.shape = (1,-1) > >>> a > array([[0, 1, 2, 3, 4]]) > >>> b = N.arange(5) > >>> b.shape = (-1,1) > >>> a > array([0, 1, 2, 3, 4]) > >>> b > array([[0], > [1], > [2], > [3], > [4]]) > > So a is a row vector and b is a column vector. If you multiply them, you > get "array broadcasting": > >>> a * b > array([[ 0, 0, 0, 0, 0], > [ 0, 1, 2, 3, 4], > [ 0, 2, 4, 6, 8], > [ 0, 3, 6, 9, 12], > [ 0, 4, 8, 12, 16]]) > > This eliminates a LOT of extra duplicate arrays that you have to make in > Matlab with meshgrid. In my eyes 'array broadcasting' is confusing and should rather be in a function like meshgrid and instead a*b should return matrixmultiply(a,b) ... > When you index into an array, you reduce its rank (number of dimensions) > by 1: > >>> a = N.arange(27) > >>> a.shape = (3,3,3) > >>> a > array([[[ 0, 1, 2], > [ 3, 4, 5], > [ 6, 7, 8]], > > [[ 9, 10, 11], > [12, 13, 14], > [15, 16, 17]], > > [[18, 19, 20], > [21, 22, 23], > [24, 25, 26]]]) > >>> a.shape > (3, 3, 3) > >>> a[1].shape > (3, 3) > >>> a[1][1].shape > (3,) > > When you slice, you keep the rank the same: > > >>> a[1:2].shape > (1, 3, 3) > > This creates a way to make row and column "vectors" from your 2-d array > (matrix) > >>> a = N.arange(25) > >>> a.shape = (5,5) > >>> a > array([[ 0, 1, 2, 3, 4], > [ 5, 6, 7, 8, 9], > [10, 11, 12, 13, 14], > [15, 16, 17, 18, 19], > [20, 21, 22, 23, 24]]) > > To make a "row vector" (really a 1XN matrix) > >>> a[0:1,:] > array([[0, 1, 2, 3, 4]]) > > > To make a "column vector" (really a NX1 matrix) > >>> a[:,0:1] > array([[ 0], > [ 5], > [10], > [15], > [20]]) > > > I hope that helps: Indeed it does - Thanks!! Unfortunately I am not at all happy now that '*' != matrixmultiply (but outerproduct) for vectors/matrices... I realize that with lists it is ok to grow them via slicing. x=[] x[0]=1 IndexError: list assignment index out of range x[0:0]=[1] x [1] that seems not to work with numarray ... or ? y=array() y[0]=1 TypeError: object does not support item assignment y[0:0]=array([1]) TypeError: object does not support item assignment Soeren. From rkern at ucsd.edu Mon Jul 25 22:45:05 2005 From: rkern at ucsd.edu (Robert Kern) Date: Mon Jul 25 22:45:05 2005 Subject: [Numpy-discussion] design issues - octave 'incompatibilities' In-Reply-To: <1122355332.30698.160.camel@localhost> References: <1122134630.26083.74.camel@localhost> <42E50C49.8080705@noaa.gov> <1122355332.30698.160.camel@localhost> Message-ID: <42E5CCAF.4000201@ucsd.edu> Soeren Sonnenburg wrote: > On Mon, 2005-07-25 at 08:59 -0700, Chris Barker wrote: [snip] >>such thing as a column vs. a row vector. A vector is a one-dimensional >>array: it has no orientation. > > This makes life more difficult if one wants to convert > from octave/matlab -> numarray and automated systems close to > impossible. That wasn't a design goal. > If vectors had the same properties/functions as matrices one > would not have such problems, i.e. v^{transpose} * u == dot(v,u) and v*u > -> error That would be a big problem for those of us who don't use Numeric just for linear algebra. These are general arrays, not matrices and vectors. [snip] > In my eyes 'array broadcasting' is confusing and should rather be in a > function like meshgrid and instead a*b should return > matrixmultiply(a,b) ... Spend some time with it. It will probably grow on you. Numeric is not Matlab or Octave. It never will be, thank G-d. [snip] >>I hope that helps: > > Indeed it does - Thanks!! Unfortunately I am not at all happy now that > '*' != matrixmultiply (but outerproduct) for vectors/matrices... Again, Numeric is a package for arrays, not just linear algebra. Please spend some more time with Python and Numeric before deciding that they must be changed to match your preconceptions. > I realize that with lists it is ok to grow them via slicing. > > x=[] > x[0]=1 > IndexError: list assignment index out of range > x[0:0]=[1] > x > [1] > > that seems not to work with numarray ... or ? > > y=array() > y[0]=1 > TypeError: object does not support item assignment > y[0:0]=array([1]) > TypeError: object does not support item assignment Python lists are designed to grow dynamically. Their memory is preallocated so that growing them is on average pretty cheap. Numeric arrays are not, nor will they be. -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From python-ml at nn7.de Mon Jul 25 22:45:06 2005 From: python-ml at nn7.de (Soeren Sonnenburg) Date: Mon Jul 25 22:45:06 2005 Subject: [Numpy-discussion] design issues - octave 'incompatibilities' In-Reply-To: <20e127854b47f13f1210e2f3574584f9@stsci.edu> References: <1122134630.26083.74.camel@localhost> <20e127854b47f13f1210e2f3574584f9@stsci.edu> Message-ID: <1122356650.30698.176.camel@localhost> On Mon, 2005-07-25 at 17:47 -0400, Perry Greenfield wrote: > I missed this part and was reminded by Travis's message. > > On Jul 23, 2005, at 12:03 PM, Soeren Sonnenburg wrote: > > -- How can one obtain submatrices from full matrices: > > > > numarray gives only elements (which is very, very rarely needed and > > should IMHO rather be some extra function): > > > >>>> i=[0,1] > >>>> j=[0,2] > >>>> a[i,j] > > array([1, 6]) > > > > vs octave: > >>> i=[1,2]; > >>> j=[1,3]; > >>> a(i,j) > > ans = > > 1 3 > > 4 6 > > > Maybe for you it is rarely needed, but not for us. The situation is > reversed. The latter is rarely used in our case. This is largely a > reflection of whether your orientation is matrices or multidimensional > arrays. In our case it is quite handy to select out a list of point in > an image by giving a list of their respective indices (e.g., stars). Hmmhh. I see that this again breaks with R/octave/matlab. One should not do so if there is no serious reason. It just makes life harder for every potential convert from any of these languages. A funktion take() would have served this purposed much better... but this is ofcourse all IMHO and I can see your point: You design it according to your (or your communities) requirements and different communities different needs... I am realizing that this must have been why cvxopt switched away from numarray/numeric. There slicing/indexing and '*' work as I would have expected: In [71]: a=matrix([1,2,3,4,5,6,7,8,9],size=(3,3)) In [72]: a Out[72]: <3x3 matrix, tc='d'> In [73]: b=matrix([1,2,3,4,5,6,7,8,9],size=(3,3)) In [74]: a*b Out[74]: <3x3 matrix, tc='d'> In [75]: print a 1.0000e+00 4.0000e+00 7.0000e+00 2.0000e+00 5.0000e+00 8.0000e+00 3.0000e+00 6.0000e+00 9.0000e+00 In [76]: print b 1.0000e+00 4.0000e+00 7.0000e+00 2.0000e+00 5.0000e+00 8.0000e+00 3.0000e+00 6.0000e+00 9.0000e+00 In [77]: print a*b 3.0000e+01 6.6000e+01 1.0200e+02 3.6000e+01 8.1000e+01 1.2600e+02 4.2000e+01 9.6000e+01 1.5000e+02 In [78]: print a[:,0] 1.0000e+00 2.0000e+00 3.0000e+00 In [79]: print a[0,1] 4.0 In [80]: print a[0,:] 1.0000e+00 4.0000e+00 7.0000e+00 > True, I do see that others may need the other view, but then the > question is which should get the convenience. Since Numeric/numarray > are primarily oriented towards multidimensional arrays (e.g., > operations are element-by-element rather than matrix) it seemed to make > sense to go this way for consistency, but I understand that there is > another side to this. It now seems very difficult for me to end up with a single numeric/matrix package that makes it into core python - which is at the same time very sad. Soeren From aisaac at american.edu Tue Jul 26 06:32:25 2005 From: aisaac at american.edu (Alan G Isaac) Date: Tue Jul 26 06:32:25 2005 Subject: [Numpy-discussion] design issues - octave 'incompatibilities' In-Reply-To: <1122355332.30698.160.camel@localhost> References: <1122134630.26083.74.camel@localhost><42E50C49.8080705@noaa.gov><1122355332.30698.160.camel@localhost> Message-ID: On Tue, 26 Jul 2005, Soeren Sonnenburg apparently wrote: > In my eyes 'array broadcasting' is confusing and should rather be in a > function like meshgrid and instead a*b should return > matrixmultiply(a,b) ... It does, if you work with matrices. Try import scipy help(scipy.mat) Coming from GAUSS I had the same initial reaction. It did not last long. hth, Alan Isaac From perry at stsci.edu Tue Jul 26 06:39:09 2005 From: perry at stsci.edu (Perry Greenfield) Date: Tue Jul 26 06:39:09 2005 Subject: [Numpy-discussion] design issues - octave 'incompatibilities' In-Reply-To: <1122352506.30698.136.camel@localhost> References: <1122134630.26083.74.camel@localhost> <42E29537.7080702@ucsd.edu> <1122230465.30698.72.camel@localhost> <89be641d30d1e246ea18d3733b6c3357@stsci.edu> <1122352506.30698.136.camel@localhost> Message-ID: On Jul 26, 2005, at 12:35 AM, Soeren Sonnenburg wrote: >> Since we deal with big data sets we have adopted 2 (no doubt to the >> consternation of many). > > hmmhh, there is no advantage with big datasets for any of the formats. > It is if you have to reorder the data, or use non-optimum iteration through the array. The first is definitely slower. >>> Do you know whether mixing slices and arrays will be supported at >>> some >>> point a[:,[0,1]] ? >>> >>> Soeren. >> >> No plans at the moment. We figured indexing was complicated enough as >> it was. I think Travis Oliphant did allow for this in Numeric3 (aka >> scipy.core); see the link below. But it may not mean what you think it >> should so you should check that out to see: >> >> http://cvs.sourceforge.net/viewcvs.py/numpy/Numeric3/PEP.txt? >> view=markup > > Hmmhh while we are at it, is there work on a consensus num* ? I've seen > the PEP for inclusion of numarray, now I see numeric3 and scipy and > also > cvxopt - are people actually converging towards a single num* package ? > That's what the goal of Numeric3 is as far as different underlying array engines. But if you mean a merging of the array/matrix viewpoints, I think you've answered your own question. Perry From Chris.Barker at noaa.gov Tue Jul 26 09:32:07 2005 From: Chris.Barker at noaa.gov (Chris Barker) Date: Tue Jul 26 09:32:07 2005 Subject: [Numpy-discussion] design issues - octave 'incompatibilities' In-Reply-To: <1122356650.30698.176.camel@localhost> References: <1122134630.26083.74.camel@localhost> <20e127854b47f13f1210e2f3574584f9@stsci.edu> <1122356650.30698.176.camel@localhost> Message-ID: <42E6653F.7040403@noaa.gov> Soeren Sonnenburg wrote: > Hmmhh. I see that this again breaks with R/octave/matlab. One should not > do so if there is no serious reason. It just makes life harder for every > potential convert from any of these languages. If you're looking for a matlab clone, use octave or psilab, or.... Speaking as an ex-matlab user, I much prefer the NumPy approach. The reason is that I very rarely did linear algebra, and generally used matlab as a general computational environment. I got very tired of having to type that "." before all my operators. I also love array broadcasting, it seems so much cleaner and efficient. When I moved from Matlab to NumPy, I missed these things: Integrated plotting: many years later, this is still weak, but at least for 2-d matplotlib is wonderful. The large library of math functions: SciPy is moving to fill that gap. Automatic handling of IEEE special values: numarray now does that pretty well. That's what I missed. What I gained was a far more powerful and expressive language, AND a more powerful and flexible array type. I don't miss MATLAB at all. In fact, you'll find me on the matplotlib mailing list, often repeating the refrain: matplotlib is NOT MATLAB nor should it be! > It now seems very difficult for me to end up with a single > numeric/matrix package that makes it into core python - That is probably true. > which is at the same time very sad. But I'm not sure we should be sad about it. What we all want is the package best suited to our own needs to be in the standard library. However, I'd rather the current situation than having a package poorly suited to my needs in the standard library. As this thread proves, there is no one kind of array package that would fit even most people's needs. However, there is some promise for the future: 1) SciPy base may serve to unify Numeric/numarray 2) Travis has introduced the "array interface" http://numeric.scipy.org/array_interface.html this would provide an efficient way for the various array and matrix packages to exchange data. It does have a hope of making it into the standard library, though even if it doesn't, if a wide variety of add-on packages use it, then the same thing is accomplished. If fact, if packages that don't implement an array type, but might use one (like PIL, wxPython, etc) accept this interface, then any package that implements it can be used together with them. 3) What about PEP 225: Elementwise/Objectwise Operators? It's listed under: Deferred, Abandoned, Withdrawn, and Rejected PEPs Which of those applied here? I had high hope for it one time. By the way, I had not seen cvxopt before, thanks for the link. Perhaps it would serve as a better base for a full-featured linear algebra package than Numeric/numarray. Ideally, it could use the array interface, for easy exchange of data with other packages. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From haase at msg.ucsf.edu Tue Jul 26 09:42:28 2005 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue Jul 26 09:42:28 2005 Subject: [Numpy-discussion] is numerical python only for prototyping ? Message-ID: <200507260941.37821.haase@msg.ucsf.edu> Hi, This is not sopposed to be an evil question; instead I'm hoping for the answer: "No, generally we get >=95% the speed of a pure C/fortran implementation" ;-) But as I am the strongest Python/numarray advocate in our group I get often the answer that Matlab is (of course) also very convenient but generally memory handling and overall execution performance is so bad that for final implementation one would generally have to reimplement in C. We are a bio-physics group at UCSF developping new algorithms for deconvolution (often in 3D). Our data sets are regularly bigger than several 100MB. When deciding for numarray I was assuming that the "Hubble Crowd" had a similar situation and all the operations are therefore very much optimized for this type of data. Is 95% a reasonable number to hope for ? I did wrap my own version of FFTW (with "plan-caching"), which should give 100% of the C-speed. But concerns arise from expression like "a=b+c*a" (think "convenience"!): If a,b,c are each 3D-datastacks creation of temporary data-arrays for 'c*a' AND then also for 'b+...' would have to be very costly. (I think this is at least happening for Numeric - I don't know about Matlab and numarray) Hoping for comments, Thanks Sebastian Haase UCSF, Sedat Lab From verveer at embl-heidelberg.de Tue Jul 26 10:15:15 2005 From: verveer at embl-heidelberg.de (Peter Verveer) Date: Tue Jul 26 10:15:15 2005 Subject: [Numpy-discussion] is numerical python only for prototyping ? In-Reply-To: <200507260941.37821.haase@msg.ucsf.edu> References: <200507260941.37821.haase@msg.ucsf.edu> Message-ID: On 26 Jul 2005, at 18:41, Sebastian Haase wrote: > Hi, > This is not sopposed to be an evil question; instead I'm hoping for the > answer: "No, generally we get >=95% the speed of a pure C/fortran > implementation" ;-) you won't, generally. Question is: since you are certainly not going to gain an order of a magnitude doing it in C, do you really care? > But as I am the strongest Python/numarray advocate in our group I get > often > the answer that Matlab is (of course) also very convenient but > generally > memory handling and overall execution performance is so bad that for > final > implementation one would generally have to reimplement in C. Well its true that implementations in C will be faster. And memory handling in Numeric/numarray can be a pain since the tendency is to create and destroy a lot of arrays if you are not careful. > We are a bio-physics group at UCSF developping new algorithms for > deconvolution (often in 3D). Our data sets are regularly bigger than > several > 100MB. When deciding for numarray I was assuming that the "Hubble > Crowd" had > a similar situation and all the operations are therefore very much > optimized > for this type of data. Funny you mention that example. I did my PhD in exactly the same field (considering you are from Sedats lab I guess you are in exactly the same field as I was/am, i.e. fluorescence microscopy. What are you guys up to these days?) and I developed all my algorithms in C at the time. Now, about 7 years later, I returned to the field to re-implement and extend some of my old algorithms for use with microscopy data that can consist of multiple sets, each several 100MB at least. Now I use python with numarray, and I am actualy quite happy with. I am pushing it by using up to 2GB of memory, (per process, after splitting the problem up and distributing it on a cluster...), but it works. I am sure I could squeeze maybe a factor of two or three in terms of speed and memory usage by rewriting in C, but that is currently not worth my time. So I guess that counts as using numarray as a prototyping environment, but the result is also suitable for production use. > Is 95% a reasonable number to hope for ? I did wrap my own version of > FFTW > (with "plan-caching"), which should give 100% of the C-speed. That should help a lot, as the standard FFTs that come with Numarray/Numeric suck big time. I do use them, but have to go through all kind of tricks to get some decent memory usage in 32bit floating point. The FFT array module is in fact very badly written for use with large multi-dimensional data sets. > But concerns > arise from expression like "a=b+c*a" (think "convenience"!): If a,b,c > are > each 3D-datastacks creation of temporary data-arrays for 'c*a' AND > then also > for 'b+...' would have to be very costly. (I think this is at least > happening > for Numeric - I don't know about Matlab and numarray) That is indeed a problem, although I think in your case you maybe limited by your FFTs anyway, at least in terms of speed. One thing you should consider is replacing expressions such as ' c= a + b' with add(a, b, c). If you do that cleverly you can avoid quite some memory allocations and you 'should' get closer to C. That does not solve everything though: 1) Complex expressions still need to be broken up in sequences of operations which is likely slower then iterating once over you array and do the expression at each point. 2) Unfortunately not all numarray functions support an output array (maybe only the ufuncs?). This can be a big problem, as then temporary arrays must be allocated. (It sure was a problem for me.) You can of course always re-implement the parts that are critical in C and wrap them (as you did with FFTW). In fact, I think numarray now provides a relatively easy way to write ufuncs which would allow you to write a single python function in C for complex expressions. > Hoping for comments, Hope this gives some insights. I guess I have had similar experiences, there are definitely some limits to the use of numarray/Numeric that could be relieved, for instance by having a consistent implementation of output arrays. That would allow writing algorithms where you could very strictly control the allocation and de-allocation of arrays, which would be a big help for working with large arrays. Cheers, Peter PS. I would not mind to hear a bit about your experiences doing the big deconvolutions in numarray/Numeric, but that may not be a good topic for this list. > Thanks > Sebastian Haase > UCSF, Sedat Lab > -- Dr Peter J Verveer European Molecular Biology Laboratory Meyerhofstrasse 1 D-69117 Heidelberg Germany Tel. +49 6221 387 8245 Fax. +49 6221 397 8306 From perry at stsci.edu Tue Jul 26 10:21:26 2005 From: perry at stsci.edu (Perry Greenfield) Date: Tue Jul 26 10:21:26 2005 Subject: [Numpy-discussion] is numerical python only for prototyping ? In-Reply-To: <200507260941.37821.haase@msg.ucsf.edu> References: <200507260941.37821.haase@msg.ucsf.edu> Message-ID: <940f0530c22ba28fc2d655a59c9e5a44@stsci.edu> On Jul 26, 2005, at 12:41 PM, Sebastian Haase wrote: > Hi, > This is not sopposed to be an evil question; instead I'm hoping for the > answer: "No, generally we get >=95% the speed of a pure C/fortran > implementation" ;-) > But as I am the strongest Python/numarray advocate in our group I get > often > the answer that Matlab is (of course) also very convenient but > generally > memory handling and overall execution performance is so bad that for > final > implementation one would generally have to reimplement in C. > We are a bio-physics group at UCSF developping new algorithms for > deconvolution (often in 3D). Our data sets are regularly bigger than > several > 100MB. When deciding for numarray I was assuming that the "Hubble > Crowd" had > a similar situation and all the operations are therefore very much > optimized > for this type of data. > Is 95% a reasonable number to hope for ? I did wrap my own version of > FFTW > (with "plan-caching"), which should give 100% of the C-speed. But > concerns > arise from expression like "a=b+c*a" (think "convenience"!): If a,b,c > are > each 3D-datastacks creation of temporary data-arrays for 'c*a' AND > then also > for 'b+...' would have to be very costly. (I think this is at least > happening > for Numeric - I don't know about Matlab and numarray) > Is it speed or memory usage you are worried about? Where are you actually seeing unacceptable performance? Offhand, I would say the temporaries are not likely to be serious speed issues (unless you running out of memory). We did envision at some point (we haven't done it yet) of recognizing situations where the temporaries could be reused. As for 95% speed, it's not what required for our work (I think that what an acceptable speed ratio is depends on the problem). In many cases being within 50% is good enough except for heavily used things where it would need to be faster. But yes, we do plan on (and do indeed actually use it now) for large data sets where speed is important. We generally don't compare the numarray speed to C speed all the time (if we were writing C equivalents every time, that would defeat the purpose of using Python :-). Perhaps you could give a more specific example with some measurements? I don't think I would promise anyone that all one's code could be done in Python. Undoubtedly, there are going to be some cases where coding in C or similar is going to be needed. I'd just argue that Python let's you keep as much as possible in a higher level language and a little as necessary in a low level language such as C. Perry From rbastian at club-internet.fr Tue Jul 26 13:17:02 2005 From: rbastian at club-internet.fr (=?iso-8859-15?q?Ren=E9=20Bastian?=) Date: Tue Jul 26 13:17:02 2005 Subject: [Numpy-discussion] convolve Message-ID: <05072610581402.00755@rbastian> Hello, in the actual form convolve(a,b)==convolve(b,a)[::-1] but following the mathematics, one should obtain convolve(a,b)==convolve(b,a) can anyone explain me the reason ? thanks -- Ren? Bastian http://www.musiques-rb.org From sheltraw at unm.edu Tue Jul 26 16:10:07 2005 From: sheltraw at unm.edu (Daniel Sheltraw) Date: Tue Jul 26 16:10:07 2005 Subject: [Numpy-discussion] fft output and input storage Message-ID: Hello all Would someone please tell me how negative and positive frequencies are stored in the output of the Numerical Python fft? Cheers, Daniel From focke at slac.stanford.edu Tue Jul 26 17:11:20 2005 From: focke at slac.stanford.edu (Warren Focke) Date: Tue Jul 26 17:11:20 2005 Subject: [Numpy-discussion] fft output and input storage In-Reply-To: References: Message-ID: >>> print FFT.fft.__doc__ fft(a, n=None, axis=-1) Will return the n point discrete Fourier transform of a. n defaults to the length of a. If n is larger than a, then a will be zero-padded to make up the difference. If n is smaller than a, the first n items in a will be used. The packing of the result is "standard": If A = fft(a, n), then A[0] contains the zero-frequency term, A[1:n/2+1] contains the positive-frequency terms, and A[n/2+1:] contains the negative-frequency terms, in order of decreasingly negative frequency. So for an 8-point transform, the frequencies of the result are [ 0, 1, 2, 3, 4, -3, -2, -1]. This is most efficient for n a power of two. This also stores a cache of working memory for different sizes of fft's, so you could theoretically run into memory problems if you call this too many times with too many different n's. >>> Hmph. The style guide says I should've put a blank line on the end of that docstring. But I don't think there was a style guide then. w On Tue, 26 Jul 2005, Daniel Sheltraw wrote: > Hello all > > Would someone please tell me how negative and positive > frequencies are > stored in the output of the Numerical Python fft? > > Cheers, > Daniel > > > ------------------------------------------------------- > SF.Net email is sponsored by: Discover Easy Linux Migration Strategies > from IBM. Find simple to follow Roadmaps, straightforward articles, > informative Webcasts and more! Get everything you need to get up to > speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From rbastian at club-internet.fr Tue Jul 26 21:53:01 2005 From: rbastian at club-internet.fr (=?iso-8859-15?q?Ren=E9=20Bastian?=) Date: Tue Jul 26 21:53:01 2005 Subject: [Numpy-discussion] convolve In-Reply-To: <05072610581402.00755@rbastian> References: <05072610581402.00755@rbastian> Message-ID: <05072706514600.00755@rbastian> Le Mardi 26 Juillet 2005 10:58, Ren? Bastian a ?crit : > Hello, > > in the actual form > > convolve(a,b)==convolve(b,a)[::-1] > > but following the mathematics, > one should obtain > > convolve(a,b)==convolve(b,a) > > can anyone explain me the reason ? I think the reasons are the modes other than FULL and the re-use of "correlate" Are this good reasons ? > > thanks -- Ren? Bastian http://www.musiques-rb.org/Ecchym/voiture.html From oliphant at ee.byu.edu Wed Jul 27 01:03:09 2005 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed Jul 27 01:03:09 2005 Subject: [Numpy-discussion] design issues - octave 'incompatibilities' In-Reply-To: <1122356650.30698.176.camel@localhost> References: <1122134630.26083.74.camel@localhost> <20e127854b47f13f1210e2f3574584f9@stsci.edu> <1122356650.30698.176.camel@localhost> Message-ID: <42E73F90.402@ee.byu.edu> Soeren Sonnenburg wrote: >I am realizing that this must have been why cvxopt switched away from >numarray/numeric. There slicing/indexing and '*' work as I would have >expected: > > cvxopt uses it's own classes because they did not feel that a general purpose array was needed. They wanted to define a matrix class with sparse matrix and dense matrix sub-classes. In fact, cvxopt's matrix classes can not be used as ubiquitously as Numeric/numarray arrays. Everything is not a matrix. In fact, I would like to see more general linear algebra routines that allow people to more naturally deal with (for example) six-dimensional linear operators mapping from a three-dimensional space to a three-dimensional space. Currently, you are forced to perform an artificial row-scanning procedure just to interface with matrix libraries. Scipy can help with this kind of thing. I do not see cvxopt as a competing array implementation. At some point, hopefully cvxopt will be integrated with scipy. I am continually looking for feasible ways to make scipy more attractive to contributors. Everybody benefits when their is a standard infrastructure. For example, there are sparse matrices in SciPy. If cvxopt has better sparse matrix objects, I would love to use them. Hopefully, the array interface will assist on a more abstract scale so that memory re-use can occur for at least the dense cvxopt matrices. >It now seems very difficult for me to end up with a single >numeric/matrix package that makes it into core python - which is at the > > >same time very sad. > > Their are several issues here. But, yes a Matrix object will always be a separate object just as quaternions should be because they represent an interpretation to a memory block. In Numeric/numarray the focus is on generic multidimensional arrays. Therefore numeric operators must be element-by element. Note that Numeric does have a Matrix object that allows you to use '*' to represent matrix multiplication. It's only problem is that passing this object to a function usually returns an array again instead of a Matrix. -Travis From perry at stsci.edu Wed Jul 27 09:20:24 2005 From: perry at stsci.edu (Perry Greenfield) Date: Wed Jul 27 09:20:24 2005 Subject: [Numpy-discussion] "Using Python for Interactive Data Analysis" tutorial available Message-ID: <0b142e5bdd9a4f5996f579e0e6afb751@stsci.edu> A tutorial that illustrates how Python can be used to analyze and visualize astronomical data (much like one can do in IDL) has been written and now is available at: http://www.scipy.org/wikis/topical_software/Tutorial While much of the focus is on astronomical data, most of it probably could serve as a general tutorial as well if sections on more advanced pyfits usage are skipped (only a few pages). There are appendices that contrast Python/numarray with IDL, I'd be happy to include the equivalent for Matlab if anyone who is familiar with both would like to write such appendices. Suggestions for improvements are welcome. Perry From aisaac at american.edu Wed Jul 27 11:19:10 2005 From: aisaac at american.edu (Alan G Isaac) Date: Wed Jul 27 11:19:10 2005 Subject: [Numpy-discussion] convolve Message-ID: On Tue, 26 Jul 2005, Ren? Bastian apparently wrote: > convolve(a,b)==convolve(b,a)[::-1] I do not see this behavior. Can you post a self-contained example? Thanks, Alan Isaac From sheltraw at unm.edu Wed Jul 27 11:28:09 2005 From: sheltraw at unm.edu (Daniel Sheltraw) Date: Wed Jul 27 11:28:09 2005 Subject: [Numpy-discussion] loading Numeric module Message-ID: Hello all When I try doing a "from Numeric import *" at the Python command line I get the error message "ImportError: No module named Numeric" so obviously the module is not in the path known to command line Python. However when the same import statement is part of a Python script file the module is found correctly. How do I fix this? Is there a way to give the full path within the import statement (when I try I get a syntax error message)? Thanks, Daniel From rkern at ucsd.edu Wed Jul 27 11:33:06 2005 From: rkern at ucsd.edu (Robert Kern) Date: Wed Jul 27 11:33:06 2005 Subject: [Numpy-discussion] loading Numeric module In-Reply-To: References: Message-ID: <42E7D329.4070400@ucsd.edu> Daniel Sheltraw wrote: > Hello all > > When I try doing a "from Numeric import *" at the Python command line I get > the error message "ImportError: No module named Numeric" so obviously > the module is not in the path known to command line Python. However when > the same import statement is part of a Python script file the module is > found correctly. How do I fix this? Are you sure that the Python interpreter is the same in both instances? Print sys.path before importing in both instances to see what paths are being searched. > Is there a way to give the full path > within the import statement (when I try I get a syntax error message)? Not really, no. -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From sheltraw at unm.edu Wed Jul 27 12:04:15 2005 From: sheltraw at unm.edu (Daniel Sheltraw) Date: Wed Jul 27 12:04:15 2005 Subject: [Numpy-discussion] 2-D FFT input/ouput storage Message-ID: Hello All Now that I have Numeric loading properly at the Python command line I went in search of some docs on the 2-D FFT. How are the input and output to the forward and inverse 2-D FFTs stored? Is it simply a 2-D version of the 1-D storage? When I do "print FFT.inverse_fft2d.__doc__" I get no helpful info about storage. Thanks, Daniel From aisaac at american.edu Wed Jul 27 13:09:17 2005 From: aisaac at american.edu (Alan G Isaac) Date: Wed Jul 27 13:09:17 2005 Subject: [Numpy-discussion] convolve In-Reply-To: <05072710171601.00755@rbastian> References: <05072710171601.00755@rbastian> Message-ID: On Wed, 27 Jul 2005, Ren? Bastian apparently wrote: > Outcommenting > if len(data) < len(kernel): > data, kernel = kernel, data > in convolve.py gives the correct result. The author is not listed for this numarray module, so we cannot directly ask why this behavior was chosen. You might use Numeric.convolve instead of numarray.convolve.Convolve.convolve: Numeric gives the expected result here. fwiw, Alan Isaac From aisaac at american.edu Wed Jul 27 13:16:05 2005 From: aisaac at american.edu (Alan G Isaac) Date: Wed Jul 27 13:16:05 2005 Subject: [Numpy-discussion] loading Numeric module In-Reply-To: References: Message-ID: On Wed, 27 Jul 2005, Daniel Sheltraw apparently wrote: > Is there a way to give the full path within the import > statement (when I try I get a syntax error message)? import sys sys.path.append("/path/to/myfiles") import mystuff hth, Alan Isaac From rkern at ucsd.edu Wed Jul 27 13:44:20 2005 From: rkern at ucsd.edu (Robert Kern) Date: Wed Jul 27 13:44:20 2005 Subject: [Numpy-discussion] convolve In-Reply-To: References: <05072710171601.00755@rbastian> Message-ID: <42E7F1BE.2010800@ucsd.edu> Alan G Isaac wrote: > On Wed, 27 Jul 2005, Ren? Bastian apparently wrote: > >>Outcommenting >>if len(data) < len(kernel): >> data, kernel = kernel, data >>in convolve.py gives the correct result. > > The author is not listed for this numarray module, > so we cannot directly ask why this behavior was chosen. This behavior was chosen for performance reasons. Since convolution is commutative, it doesn't matter what order the arguments come in. I do not see the behavior Ren? is claiming. Perhaps Ren? can tell us what version of numarray he is using? or provide an example? I am using CVS as of a week ago, I think. In [10]: import numarray In [11]: import numarray.convolve In [12]: a = numarray.arange(10) + 1. In [13]: b = numarray.arange(5) + 2. In [14]: c = a - 1. In [15]: numarray.convolve.convolve(a,b) Out[15]: array([ 2., 7., 16., 30., 50., 70., 90., 110., 130., 150., 148., 133., 104., 60.]) In [16]: numarray.convolve.convolve(b,a) Out[16]: array([ 2., 7., 16., 30., 50., 70., 90., 110., 130., 150., 148., 133., 104., 60.]) In [17]: numarray.convolve.convolve(a,c) Out[17]: array([ 0., 1., 4., 10., 20., 35., 56., 84., 120., 165., 210., 244., 266., 275., 270., 250., 214., 161., 90.]) In [18]: numarray.convolve.convolve(c,a) Out[18]: array([ 0., 1., 4., 10., 20., 35., 56., 84., 120., 165., 210., 244., 266., 275., 270., 250., 214., 161., 90.]) -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From aisaac at american.edu Wed Jul 27 13:58:11 2005 From: aisaac at american.edu (Alan G Isaac) Date: Wed Jul 27 13:58:11 2005 Subject: [Numpy-discussion] convolve In-Reply-To: <42E7F1BE.2010800@ucsd.edu> References: <05072710171601.00755@rbastian> <42E7F1BE.2010800@ucsd.edu> Message-ID: On Wed, 27 Jul 2005, Robert Kern apparently wrote: > I do not see the behavior Ren? is claiming. Perhaps Ren? > can tell us what version of numarray he is using? I see it in version 1.1, for the example he provided. Cheers, Alan Isaac From rkern at ucsd.edu Wed Jul 27 14:46:12 2005 From: rkern at ucsd.edu (Robert Kern) Date: Wed Jul 27 14:46:12 2005 Subject: [Numpy-discussion] convolve In-Reply-To: References: <05072710171601.00755@rbastian> <42E7F1BE.2010800@ucsd.edu> Message-ID: <42E80089.3090901@ucsd.edu> Alan G Isaac wrote: > On Wed, 27 Jul 2005, Robert Kern apparently wrote: > >>I do not see the behavior Ren? is claiming. Perhaps Ren? >>can tell us what version of numarray he is using? > > I see it in version 1.1, > for the example he provided. What example? Either his message has not yet propogated to me or the mailing list archives, or he sent it to you privately. The problem does not seem to exist any longer in CVS. -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From Chris.Barker at noaa.gov Wed Jul 27 14:47:24 2005 From: Chris.Barker at noaa.gov (Chris Barker) Date: Wed Jul 27 14:47:24 2005 Subject: [Numpy-discussion] Suggested Addition to the new array PEP Message-ID: <42E800B0.7050803@noaa.gov> Hi all, but particularly Travis, I just took the time to read the new array PEP: http://numeric.scipy.org/PEP.txt It looks great. I do have one suggestion for an addition method: change_type(NewType) # or some other name: Change the type that the bytes in the array represent, without changing the data at. Dimensions will change if the new type has a different itemsize than the old. This would be the equivalent of: A = Numeric.fromstring(A.tostring(), NewType) but without the extra copies of the data. When would anyone want this? I've written a function that does exactly this with Numeric arrays in order to efficiently slice, dice and re-arange a bunch of binary data files. The files consist of a bunch of C structs written sequentially to a file. In this case the structs are: (long, long, char) I have a file which is N of these structs written sequentially to a file, I can do: NumBytes = 9 # number of bytes in a record # Read the file data = fromstring(file.read(NumBytes*NumTimesteps*NumLEs),UnsignedInt8) file.close() data.shape = (NumTimesteps,NumLEs,NumBytes) # extract LE data: LEs = data[:,:,:8].copy() # extract the flags flags = data[:,:,8].copy() del data # here I can create the Int32 array without making a copy: # changetype returns a rank-1 array changetype(LEs,Int32) # It now gets reshaped to fit what it represents. LEs.shape = (NumTimesteps,NumLEs,2) I think it would only work in the general case with a contiguous array. All this really requires is to re-set the attributes of the array. with Numeric, the code is (for contiguous arrays): NewDescription = PyArray_DescrFromType(typecode); NewElementSize = NewDescription->elsize; if (TotalBytes % NewElementSize > 0){ PyErr_SetString (PyExc_ValueError, "The input array is the wrong size for the requested type"); return NULL; } Array->descr = NewDescription; // does the old descr need to be freed?? how?? Array->nd = 1; Array->strides[0] = NewElementSize; Array->dimensions[0] = TotalBytes / NewElementSize; The core By the way, I'm also not seeing tostring() or tobuffer() listed in the PEP -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From python-ml at nn7.de Wed Jul 27 15:02:11 2005 From: python-ml at nn7.de (Soeren Sonnenburg) Date: Wed Jul 27 15:02:11 2005 Subject: [Numpy-discussion] design issues - octave 'incompatibilities' In-Reply-To: <42E6653F.7040403@noaa.gov> References: <1122134630.26083.74.camel@localhost> <20e127854b47f13f1210e2f3574584f9@stsci.edu> <1122356650.30698.176.camel@localhost> <42E6653F.7040403@noaa.gov> Message-ID: <1122483294.28910.13.camel@localhost> On Tue, 2005-07-26 at 09:30 -0700, Chris Barker wrote: > Soeren Sonnenburg wrote: > > Hmmhh. I see that this again breaks with R/octave/matlab. One should not > > do so if there is no serious reason. It just makes life harder for every > > potential convert from any of these languages. > > If you're looking for a matlab clone, use octave or psilab, or.... No I am not, I am sold to python. > Speaking as an ex-matlab user, I much prefer the NumPy approach. The > reason is that I very rarely did linear algebra, and generally used > matlab as a general computational environment. I got very tired of > having to type that "." before all my operators. I also love array > broadcasting, it seems so much cleaner and efficient. Well coming from machine learning I am used to having just matrices. > When I moved from Matlab to NumPy, I missed these things: > > Integrated plotting: many years later, this is still weak, but at least > for 2-d matplotlib is wonderful. I agree here. [no single numeric package in python] > But I'm not sure we should be sad about it. What we all want is the > package best suited to our own needs to be in the standard library. > However, I'd rather the current situation than having a package poorly > suited to my needs in the standard library. As this thread proves, there > is no one kind of array package that would fit even most people's needs. > > However, there is some promise for the future: > > 1) SciPy base may serve to unify Numeric/numarray > > 2) Travis has introduced the "array interface" > > http://numeric.scipy.org/array_interface.html > > this would provide an efficient way for the various array and matrix > packages to exchange data. It does have a hope of making it into the > standard library, though even if it doesn't, if a wide variety of add-on > packages use it, then the same thing is accomplished. If fact, if > packages that don't implement an array type, but might use one (like > PIL, wxPython, etc) accept this interface, then any package that > implements it can be used together with them. > > 3) What about PEP 225: Elementwise/Objectwise Operators? > It's listed under: > > Deferred, Abandoned, Withdrawn, and Rejected PEPs > > Which of those applied here? I had high hope for it one time. > > By the way, I had not seen cvxopt before, thanks for the link. Perhaps > it would serve as a better base for a full-featured linear algebra > package than Numeric/numarray. Ideally, it could use the array > interface, for easy exchange of data with other packages. Actually that would be nice. Having had a closer look at cvxopt that might suit *my* needs more, but having an interface to get cvxopt matrices -> numarray/numeric without trouble to get certain potentially missing functionality would be nice. Thanks, Soeren From python-ml at nn7.de Wed Jul 27 15:03:02 2005 From: python-ml at nn7.de (Soeren Sonnenburg) Date: Wed Jul 27 15:03:02 2005 Subject: [Numpy-discussion] design issues - octave 'incompatibilities' In-Reply-To: <42E73F90.402@ee.byu.edu> References: <1122134630.26083.74.camel@localhost> <20e127854b47f13f1210e2f3574584f9@stsci.edu> <1122356650.30698.176.camel@localhost> <42E73F90.402@ee.byu.edu> Message-ID: <1122497705.28910.41.camel@localhost> On Wed, 2005-07-27 at 02:02 -0600, Travis Oliphant wrote: > Soeren Sonnenburg wrote: > > >I am realizing that this must have been why cvxopt switched away from > >numarray/numeric. There slicing/indexing and '*' work as I would have > >expected: > > > > > cvxopt uses it's own classes because they did not feel that a general > purpose array was needed. They wanted to define a matrix class with > sparse matrix and dense matrix sub-classes. In fact, cvxopt's matrix > classes can not be used as ubiquitously as Numeric/numarray arrays. > Everything is not a matrix. In fact, I would like to see more general > linear algebra routines that allow people to more naturally deal with > (for example) six-dimensional linear operators mapping from a > three-dimensional space to a three-dimensional space. Currently, you > are forced to perform an artificial row-scanning procedure just to > interface with matrix libraries. Scipy can help with this kind of thing. Hmmhh, > I do not see cvxopt as a competing array implementation. At some > point, hopefully cvxopt will be integrated with scipy. I am continually > looking for feasible ways to make scipy more attractive to > contributors. Everybody benefits when their is a standard > infrastructure. For example, there are sparse matrices in SciPy. If > cvxopt has better sparse matrix objects, I would love to use them. > Hopefully, the array interface will assist on a more abstract scale so > that memory re-use can occur for at least the dense cvxopt matrices. I guess we now observe the different communities different expectations problem :/ In any case I agree that a standard infrastructure is very desirable. However it might come at a cost one might not want to pay, but still at least conversion functions from say cvxopt <-> numarray are worth spending time on. > >It now seems very difficult for me to end up with a single > >numeric/matrix package that makes it into core python - which is at the > > > > > >same time very sad. > > > > > > Their are several issues here. But, yes a Matrix object will always be > a separate object just as quaternions should be because they represent > an interpretation to a memory block. In Numeric/numarray the focus is > on generic multidimensional arrays. Therefore numeric operators must be > element-by element. OK. > Note that Numeric does have a Matrix object that allows you to use '*' > to represent matrix multiplication. It's only problem is that passing > this object to a function usually returns an array again instead of a > Matrix. So the cvxopt approach is pretty valid, doing everything for matrices as they do, but allowing other types as 'int' etc.. Soeren From python-ml at nn7.de Wed Jul 27 15:03:06 2005 From: python-ml at nn7.de (Soeren Sonnenburg) Date: Wed Jul 27 15:03:06 2005 Subject: [Numpy-discussion] design issues - octave 'incompatibilities' In-Reply-To: References: <1122134630.26083.74.camel@localhost> <42E29537.7080702@ucsd.edu> <1122230465.30698.72.camel@localhost> <89be641d30d1e246ea18d3733b6c3357@stsci.edu> <1122352506.30698.136.camel@localhost> Message-ID: <1122500138.28910.59.camel@localhost> On Tue, 2005-07-26 at 09:37 -0400, Perry Greenfield wrote: > On Jul 26, 2005, at 12:35 AM, Soeren Sonnenburg wrote: [...] > > Hmmhh while we are at it, is there work on a consensus num* ? I've seen > > the PEP for inclusion of numarray, now I see numeric3 and scipy and > > also > > cvxopt - are people actually converging towards a single num* package ? > > > That's what the goal of Numeric3 is as far as different underlying > array engines. But if you mean a merging of the array/matrix > viewpoints, I think you've answered your own question. As Alan and Travis suggested Matrices scipy.mat, one could have matrices more or less independently of num* if one could convert them back and forth. Doing so would even allow for doing things differently, like getting submatrices, allowing for real matrix-matrix multiplication etc. However that of course kills consistency. Soeren From rkern at ucsd.edu Wed Jul 27 15:03:08 2005 From: rkern at ucsd.edu (Robert Kern) Date: Wed Jul 27 15:03:08 2005 Subject: [Numpy-discussion] 2-D FFT input/ouput storage In-Reply-To: References: Message-ID: <42E8045A.3090609@ucsd.edu> Daniel Sheltraw wrote: > Hello All > > Now that I have Numeric loading properly at the Python command line I > went in search of some docs on the 2-D FFT. How are the input and output > to the forward and inverse 2-D FFTs stored? Is it simply a 2-D version > of the 1-D storage? When I do "print FFT.inverse_fft2d.__doc__" I get no > helpful info about storage. The 2-D layout follows directly as a consequence of the 1-D layout and the definition of a 2-D FFT. In [14]: import FFT In [15]: def my2dfft(A): ....: A = array(A, Complex) ....: for i in range(A.shape[0]): ....: A[i] = FFT.fft(A[i]) ....: for i in range(A.shape[-1]): ....: A[:,i] = FFT.fft(A[:,i]) ....: return A ....: In [16]: A = reshape(arange(16.), (4,4)) In [17]: my2dfft(A) Out[17]: array([[ 120. +0.j, -8. +8.j, -8. +0.j, -8. -8.j], [ -32.+32.j, 0. +0.j, 0. +0.j, 0. +0.j], [ -32. +0.j, 0. +0.j, 0. +0.j, 0. +0.j], [ -32.-32.j, 0. +0.j, 0. +0.j, 0. +0.j]]) In [18]: FFT.fft2d(A) Out[18]: array([[ 120. +0.j, -8. +8.j, -8. +0.j, -8. -8.j], [ -32.+32.j, 0. +0.j, 0. +0.j, 0. +0.j], [ -32. +0.j, 0. +0.j, 0. +0.j, 0. +0.j], [ -32.-32.j, 0. +0.j, 0. +0.j, 0. +0.j]]) -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From python-ml at nn7.de Wed Jul 27 15:03:10 2005 From: python-ml at nn7.de (Soeren Sonnenburg) Date: Wed Jul 27 15:03:10 2005 Subject: [Numpy-discussion] design issues - octave 'incompatibilities' In-Reply-To: <42E5CCAF.4000201@ucsd.edu> References: <1122134630.26083.74.camel@localhost> <42E50C49.8080705@noaa.gov> <1122355332.30698.160.camel@localhost> <42E5CCAF.4000201@ucsd.edu> Message-ID: <1122499368.28910.52.camel@localhost> On Mon, 2005-07-25 at 22:39 -0700, Robert Kern wrote: > Soeren Sonnenburg wrote: > > [snip] > > > In my eyes 'array broadcasting' is confusing and should rather be in a > > function like meshgrid and instead a*b should return > > matrixmultiply(a,b) ... > > Spend some time with it. It will probably grow on you. Numeric is not [...] > Again, Numeric is a package for arrays, not just linear algebra. Please > spend some more time with Python and Numeric before deciding that they > must be changed to match your preconceptions. I realize that I just need plain matrices and operations on them so I am probably better off with cvxopt at that point. > > I realize that with lists it is ok to grow them via slicing. > > > > x=[] > > x[0]=1 > > IndexError: list assignment index out of range > > x[0:0]=[1] > > x > > [1] > > > > that seems not to work with numarray ... or ? > > > > y=array() > > y[0]=1 > > TypeError: object does not support item assignment > > y[0:0]=array([1]) > > TypeError: object does not support item assignment > > Python lists are designed to grow dynamically. Their memory is > preallocated so that growing them is on average pretty cheap. Numeric > arrays are not, nor will they be. Well I don't claim that this is/must be efficient. It is just what I would have expected to work if I use standard python arrays and now numarrays. Soeren. From aisaac at american.edu Wed Jul 27 18:19:04 2005 From: aisaac at american.edu (Alan G Isaac) Date: Wed Jul 27 18:19:04 2005 Subject: [Numpy-discussion] convolve In-Reply-To: <42E80089.3090901@ucsd.edu> References: <05072710171601.00755@rbastian> <42E7F1BE.2010800@ucsd.edu> <42E80089.3090901@ucsd.edu> Message-ID: >> On Wed, 27 Jul 2005, Robert Kern apparently wrote: >>> I do not see the behavior Ren? is claiming. Perhaps Ren? >>> can tell us what version of numarray he is using? > Alan G Isaac wrote: >> I see it in version 1.1, >> for the example he provided. On Wed, 27 Jul 2005, Robert Kern apparently wrote: > What example? Either his message has not yet propogated to me or the > mailing list archives, or he sent it to you privately. The problem does > not seem to exist any longer in CVS. My bad. It was indeed off list. It is (essentially) replicated below. Alan ================================================== #-*-coding:latin-1-*- #from Numeric import arrayrange,asarray #from Numeric import convolve from numarray import arrayrange,asarray from numarray.convolve.Convolve import convolve def explo0(): a=arrayrange(10.) b=asarray([1., 0., 2., -1., 3., -2.]) c=convolve(a, b) print c d=convolve(b, a) print d if __name__=="__main__": explo0() From rkern at ucsd.edu Wed Jul 27 18:24:11 2005 From: rkern at ucsd.edu (Robert Kern) Date: Wed Jul 27 18:24:11 2005 Subject: [Numpy-discussion] convolve In-Reply-To: References: <05072710171601.00755@rbastian> <42E7F1BE.2010800@ucsd.edu> <42E80089.3090901@ucsd.edu> Message-ID: <42E83383.3010308@ucsd.edu> Alan G Isaac wrote: >>>On Wed, 27 Jul 2005, Robert Kern apparently wrote: >>> >>>>I do not see the behavior Ren? is claiming. Perhaps Ren? >>>>can tell us what version of numarray he is using? > >>Alan G Isaac wrote: >> >>>I see it in version 1.1, >>>for the example he provided. > > On Wed, 27 Jul 2005, Robert Kern apparently wrote: > >>What example? Either his message has not yet propogated to me or the >>mailing list archives, or he sent it to you privately. The problem does >>not seem to exist any longer in CVS. > > My bad. > It was indeed off list. > It is (essentially) replicated below. Whatever problem that once existed is now fixed in CVS. The example you give works fine. -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From aisaac at american.edu Wed Jul 27 18:25:04 2005 From: aisaac at american.edu (Alan G Isaac) Date: Wed Jul 27 18:25:04 2005 Subject: [Numpy-discussion] convolve In-Reply-To: <42E83383.3010308@ucsd.edu> References: <05072710171601.00755@rbastian> <42E7F1BE.2010800@ucsd.edu> <42E80089.3090901@ucsd.edu> <42E83383.3010308@ucsd.edu> Message-ID: On Wed, 27 Jul 2005, Robert Kern apparently wrote: > Whatever problem that once existed is now fixed in CVS. Great! Thanks, Alan From jochen at fhi-berlin.mpg.de Wed Jul 27 23:33:07 2005 From: jochen at fhi-berlin.mpg.de (=?iso-8859-1?Q?Jochen_K=FCpper?=) Date: Wed Jul 27 23:33:07 2005 Subject: [Numpy-discussion] LAPACK/BLAS for Fedora core 4 i386 ? In-Reply-To: <42DE9BEC.9070902@noaa.gov> (Chris Barker's message of "Wed, 20 Jul 2005 11:46:04 -0700") References: <42DE9BEC.9070902@noaa.gov> Message-ID: "Chris Barker" writes: > Does anyone have a suggestion for how to add the addional lapack > stuff I need, while still using atlas stuff where possible? See the ATLAS installation manual. Greetings, Jochen -- Einigkeit und Recht und Freiheit http://www.Jochen-Kuepper.de Libert?, ?galit?, Fraternit? GnuPG key: CC1B0B4D (Part 3 you find in my messages before fall 2003.) From rbastian at club-internet.fr Thu Jul 28 00:25:17 2005 From: rbastian at club-internet.fr (=?iso-8859-15?q?Ren=E9=20Bastian?=) Date: Thu Jul 28 00:25:17 2005 Subject: [Numpy-discussion] convolve In-Reply-To: References: <42E83383.3010308@ucsd.edu> Message-ID: <05072722060300.00753@rbastian> Le Jeudi 28 Juillet 2005 03:26, Alan G Isaac a ?crit : > On Wed, 27 Jul 2005, Robert Kern apparently wrote: > > Whatever problem that once existed is now fixed in CVS. Yes, it works ! Thanks, Ren? > > Great! > > Thanks, > Alan > > From Chris.Barker at noaa.gov Thu Jul 28 08:31:27 2005 From: Chris.Barker at noaa.gov (Chris Barker) Date: Thu Jul 28 08:31:27 2005 Subject: [Numpy-discussion] LAPACK/BLAS for Fedora core 4 i386 ? In-Reply-To: References: <42DE9BEC.9070902@noaa.gov> Message-ID: <42E8FA0B.3050107@noaa.gov> Jochen K?pper wrote: >>Does anyone have a suggestion for how to add the addional lapack >>stuff I need, while still using atlas stuff where possible? > > See the ATLAS installation manual. Thanks for the tip. Do you mean this? : """ ******** GETTING A FULL LAPACK LIB ************************** ATLAS does not provide a full lapack library. However, there is a simple way to get ATLAS to provide its faster LAPACK routines to a full LAPACK library. ATLAS's internal routines are distinct from LAPACK's, so it is safe to compile ATLAS's LAPACK routines directly into a netlib-style LAPACK library. First, obtain the LAPACK src from netlib and build the LAPACK library as normal. Then, in this directory (where you should have a liblapack.a), issue the following commands: mkdir tmp cd tmp ar x ../liblapack.a cp ../liblapack.a ar r ../liblapack.a *.o cd .. rm -rf tmp Just linking in ATLAS's liblapack.a first will not get you the best LAPACK performance, mainly because LAPACK's untuned ILAENV will be used instead of ATLAS's tuned one. """ If so, I did that (see a later post to this list). However, I found that I got about the same performance improvement doing this as I had gotten compiling against just the atlas BLAS and lapack_lite. It makes me wonder if I'm really getting the benefit of atlas lapack. I don't' have hard-core performance needs at the moment, so I'm done, but I'm still surprised I couldn't find a ready-to-go atlas rpm for Fedora Core 4. It actually makes me miss Gentoo. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From joachim at ee.ucla.edu Thu Jul 28 10:55:12 2005 From: joachim at ee.ucla.edu (Joachim Dahl) Date: Thu Jul 28 10:55:12 2005 Subject: [Numpy-discussion] Interfacing to other array classes Message-ID: <42E91B85.50701@ee.ucla.edu> I am one the authors of 'cvxopt', a Python package that includes simple matrix classes aimed at numerical linear algebra for convex solvers, and I saw the discussion on the list regarding interfacing to such classes. I think the idea of an advanced interface or buffer protocol to exchange information between simple array types without using (or knowing) the API of the different classes is great. What is the status of the new array interface? When it is ready for testing, I will definitely try it out. - Joachim From pieter at djinnit.com Thu Jul 28 14:38:45 2005 From: pieter at djinnit.com (Pieter van Beek) Date: Thu Jul 28 14:38:45 2005 Subject: [Numpy-discussion] New e-mail address Message-ID: *English follows below* Geachte relatie, Ooit (misschien recent, misschien al weer jaren geleden) heeft u e-mail verstuurd naar mijn voormalige e-mail adres: _vanbeek at q-factor.nl_. Bij deze wil ik u mededelen dat mijn e-mail adres is gewijzigd in pieter at djinnit.com. Gelieve deze wijziging door te voeren in uw adresbestand. Met vriendelijke groet, Pieter van Beek ------------------------------------------------------------------------ Dear relation, Once (maybe recently, maybe years ago) you have been sending mail to my former e-mail address: _vanbeek at q-factor.nl_. Hereby I'd like to inform you that my e-mail address has changed into pieter at djinnit.com. Please change my e-mail address in your address book. Kind regards, Pieter van beek -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Fri Jul 29 09:24:05 2005 From: Chris.Barker at noaa.gov (Chris Barker) Date: Fri Jul 29 09:24:05 2005 Subject: [Numpy-discussion] LAPACK/BLAS for Fedora core 4 i386 ? In-Reply-To: References: <42DE9BEC.9070902@noaa.gov> <42E8FA0B.3050107@noaa.gov> Message-ID: <42EA57D1.7010709@noaa.gov> Jochen K?pper wrote: > "Chris Barker" writes: >>Do you mean this? : > yes OK, so I've done what I can. > To find out you might have to look at the LAPACK functions you are > using and compare them against the ones provided by ATLAS. Well, I'm testing using LinearAlgebra.solve_linear_equations() from Numeric (and numarray). The reason I'm wondering is that on my last machine, a 1ghz PIII laptop running Gentoo, using the Gentoo provided atlas/lapack, I got a 6-7 times speed up over lapack_lite. On my new machine, a 2GHz Pentium M laptop running FC4, I got about a 2 times speed-up, which is nice, but not nearly as impressive. Another possible issue is that I used pre-complied binaries form the atlas site, which are a bit old, maybe I should compile myself. Any thoughts? -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From dd55 at cornell.edu Fri Jul 29 09:29:19 2005 From: dd55 at cornell.edu (Darren Dale) Date: Fri Jul 29 09:29:19 2005 Subject: [Numpy-discussion] LAPACK/BLAS for Fedora core 4 i386 ? In-Reply-To: <42EA57D1.7010709@noaa.gov> References: <42DE9BEC.9070902@noaa.gov> <42EA57D1.7010709@noaa.gov> Message-ID: <200507291228.38305.dd55@cornell.edu> On Friday 29 July 2005 12:22 pm, Chris Barker wrote: > > To find out you might have to look at the LAPACK functions you are > > using and compare them against the ones provided by ATLAS. > > Well, I'm testing using LinearAlgebra.solve_linear_equations() from > Numeric (and numarray). The reason I'm wondering is that on my last > machine, a 1ghz PIII laptop running Gentoo, using the Gentoo provided > atlas/lapack, I got a 6-7 times speed up over lapack_lite. > > On my new machine, a 2GHz Pentium M laptop running FC4, I got about a 2 > times speed-up, which is nice, but not nearly as impressive. > > Another possible issue is that I used pre-complied binaries form the > atlas site, which are a bit old, maybe I should compile myself. Any > thoughts? Maybe I don't understanding something, but isn't the point of ATLAS that the libraries are tuned at compile time for your specific setup? -- Darren From Chris.Barker at noaa.gov Fri Jul 29 11:48:14 2005 From: Chris.Barker at noaa.gov (Chris Barker) Date: Fri Jul 29 11:48:14 2005 Subject: [Numpy-discussion] LAPACK/BLAS for Fedora core 4 i386 ? In-Reply-To: <200507291228.38305.dd55@cornell.edu> References: <42DE9BEC.9070902@noaa.gov> <42EA57D1.7010709@noaa.gov> <200507291228.38305.dd55@cornell.edu> Message-ID: <42EA7964.5050806@noaa.gov> Darren Dale wrote: > Maybe I don't understanding something, but isn't the point of ATLAS that the > libraries are tuned at compile time for your specific setup? yes, but the binaries I downloaded are for the P4 processor, so I'm not sure how much more specific I can get. I'm going to give it a try. This does bring up a point, however. Why is lapack usually linked statically? I usually link statically so that I can move applications to a different system without worrying about what libraries are installed there. However, with lapack/blas, you want to use a library compiled for that particular machine, so it seems to make more sense to use dynamic ones. Besides, it would be easier to compare performance if all I had to do was drop a new *.so in place. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From matthew.brett at gmail.com Fri Jul 29 12:14:21 2005 From: matthew.brett at gmail.com (Matthew Brett) Date: Fri Jul 29 12:14:21 2005 Subject: [Numpy-discussion] LAPACK/BLAS for Fedora core 4 i386 ? In-Reply-To: <42EA7964.5050806@noaa.gov> References: <42DE9BEC.9070902@noaa.gov> <42EA57D1.7010709@noaa.gov> <200507291228.38305.dd55@cornell.edu> <42EA7964.5050806@noaa.gov> Message-ID: <1e2af89e0507291212235360f7@mail.gmail.com> Hi, > yes, but the binaries I downloaded are for the P4 processor, so I'm not > sure how much more specific I can get. I'm going to give it a try. There is some tuning that is specific to the cache, which might change things a bit. Is it possible your test code uses NaNs at some point? That will result in a huge slowdown on the P4 with the default ATLAS binaries: http://www.mrc-cbu.cam.ac.uk/Imaging/Common/spm_intel_tune.shtml#nan_problem Matthew From luszczek at cs.utk.edu Fri Jul 29 12:18:48 2005 From: luszczek at cs.utk.edu (Piotr Luszczek) Date: Fri Jul 29 12:18:48 2005 Subject: [Numpy-discussion] LAPACK/BLAS for Fedora core 4 i386 ? In-Reply-To: <42EA7964.5050806@noaa.gov> References: <42DE9BEC.9070902@noaa.gov> <42EA57D1.7010709@noaa.gov> <200507291228.38305.dd55@cornell.edu> <42EA7964.5050806@noaa.gov> Message-ID: <42EA80C3.2060601@cs.utk.edu> Chris Barker wrote: > Darren Dale wrote: > >> Maybe I don't understanding something, but isn't the point of ATLAS >> that the libraries are tuned at compile time for your specific setup? > > > yes, but the binaries I downloaded are for the P4 processor, so I'm not > sure how much more specific I can get. I'm going to give it a try. Unfortunately, you have to get very specific to get the most out of Atlas. The two things that come to mind are: - cache size which differs across various Pentium 4s/Xeons - clock speed The former means that you have to block operations on your matrices in the right way so you keep data in each level of memory hierarchy as long as possible. Clock speed matters because it introduces varying memory latencies so, for example, you might not be prefetching data at the right rate. That's why I usually compile Atlas without any builtin presets even though it takes very long. However, I also try the prebuilt binaries to make sure that Clint (Atlas' author) didn't turn some secret knob that I don't know about. Piotr From edcjones at comcast.net Sun Jul 31 20:57:33 2005 From: edcjones at comcast.net (Edward C. Jones) Date: Sun Jul 31 20:57:33 2005 Subject: [Numpy-discussion] numarray: PyArrayObject supports number protocols? Message-ID: <42ED9D73.1010400@comcast.net> I just saw in the docs, Section 14.2.4: It should be noted that unlike earlier versions of numarray, the present PyArrayObject structure is a first class python object, with full support for the number protocols in C. Does this mean that I can add two numarrays in C using "PyNumber_Add"? From joostvanevert at gmail.com Fri Jul 1 04:44:58 2005 From: joostvanevert at gmail.com (Joost van Evert) Date: Fri Jul 1 04:44:58 2005 Subject: [Numpy-discussion] numarray problem creating large memmap In-Reply-To: <42B721E1.2040503@cs.utk.edu> References: <42B437AD.3040805@sympatico.ca> <42B71450.6090102@cs.utk.edu> <42B721E1.2040503@cs.utk.edu> Message-ID: <1120217998.20139.10.camel@inpc93.et.tudelft.nl> Hello, does anyone know why it is not possible to create a memmap beyond a certain length. For example: m = MM.open('/tmp/test','write',len=500000000) works, while the same command using a length of 600000000 gives the following error message: /shannon/joost/lib/python/numarray/memmap.py in __init__(self, filename, mode, len) 299 else: 300 acc = mmap.ACCESS_WRITE --> 301 self._mmap = mmap.mmap(file.fileno(), len, access=acc) 302 else: 303 self._mmap = None EnvironmentError: [Errno 22] Invalid argument I am using: - numarray version 1.3.1 - python 2.4.1 - linux Greetings, Joost From jmiller at stsci.edu Fri Jul 1 09:52:26 2005 From: jmiller at stsci.edu (Todd Miller) Date: Fri Jul 1 09:52:26 2005 Subject: [Numpy-discussion] numarray problem creating large memmap In-Reply-To: <1120217998.20139.10.camel@inpc93.et.tudelft.nl> References: <42B437AD.3040805@sympatico.ca> <42B71450.6090102@cs.utk.edu> <42B721E1.2040503@cs.utk.edu> <1120217998.20139.10.camel@inpc93.et.tudelft.nl> Message-ID: <1120236614.30306.126.camel@halloween.stsci.edu> On Fri, 2005-07-01 at 07:39, Joost van Evert wrote: > Hello, > > does anyone know why it is not possible to create a memmap beyond a > certain length. For example: m = > MM.open('/tmp/test','write',len=500000000) works, while the same command > using a length of 600000000 gives the following error message: > > /shannon/joost/lib/python/numarray/memmap.py in __init__(self, filename, > mode, len) > 299 else: > 300 acc = mmap.ACCESS_WRITE > --> 301 self._mmap = mmap.mmap(file.fileno(), len, > access=acc) > 302 else: > 303 self._mmap = None > > EnvironmentError: [Errno 22] Invalid argument > > I am using: > - numarray version 1.3.1 > - python 2.4.1 > - linux I'm not sure it matters, but how much RAM and swap do you have? How much free space do you have on /tmp? Regards, Todd From rays at blue-cove.com Fri Jul 1 14:01:38 2005 From: rays at blue-cove.com (Ray Schumacher) Date: Fri Jul 1 14:01:38 2005 Subject: [Numpy-discussion] a Numeric.where re-coded for weave.inline: very fast... Message-ID: <5.2.0.4.2.20050701134209.09aecff8@blue-cove.com> I tried weave for executing a C routine over a Numeric array. The Numeric data array is c_int16 with 14 significant bits (bit 13 is the sign) and I wanted it as "normal" Int16. So: t1=clock() m = where(greater(dataArray[:1000], 8191), dataArray[:1000]-16383, dataArray[:1000]) t2 = clock() print 'where', round((t2-t1)*1000000, 1), 'us' Usually gives: where 634.3 us PyInline has about the same calling overhead as ctypes (I had tried a DLL and ctypes: http://sourceforge.net/mailarchive/forum.php?thread_id=7630224&forum_id=24606 ), apparently, about 240us for a simple C statement. It also requires some extra hoops to digest the pointer to short I need ( void b2int(short *f, int N) ), so I didn't fully try it. weave.inline testing was very good, with different issues. I found I had to patch msvccompiler.py for MS C 7.1 .NET ( http://www.vrplumber.com/programming/mstoolkit/ ). After the free but massive download/install... import weave inline = weave.inline # for speed N = 1000 code="int i; for(i = 0; i < N; i++){if (dataArray[i]>8191){dataArray[i]-=16383;}}" inline(code, ['dataArray', 'N']) # just pass Python object's names This created sc_019a1cf36209cb2dfc688820080541ef0.pyd in C:\Documents and Settings\rays\Local Settings\Temp\rays\python24_compiled\ Using the above code is slow; ~32000 us, as the compiler checks or runs each time. However, after much fiddling and dir()s, I copied the long-name.pyd to C:\Python24\DLLs and just did import sc_019a1cf36209cb2dfc688820080541ef0 b2iw = sc_019a1cf36209cb2dfc688820080541ef0.compiled_func # note the exposed function! N = 1 t1=clock() b2iw({'dataArray':dataArray}, {'N':N}) # note the dicts! t2 = clock() print 'weave', round((t2-t1)*1000000, 1), 'us' 25us! (~300us for N=10000) I think that's as close to C speed as I can expect, although I'm looking at the compiler options in msvccompiler.py for P4 optimization... I still get "Missing compiler_cxx fix for MSVCCompiler" on the initial compile, but apparently to no harm. As a final note, I also found that psyco _slows down_ both ctypes and weave calls. I did psyco.full() at the app's start. Without pysco: b2i 210.8 us weave 53.7 us with: b2i 250.0 us weave 234.7 us Comment/criticism ? Ray From rowen at cesmail.net Fri Jul 1 15:05:15 2005 From: rowen at cesmail.net (Russell E. Owen) Date: Fri Jul 1 15:05:15 2005 Subject: [Numpy-discussion] Trouble installing numarray on Solaris Message-ID: A user of my application reports that installing numarray 1.3.2 on Solaris using Python 2.3.1 has run into trouble: prompt> python Python 2.3.1 (#1, Sep 30 2003, 20:29:58) [C] on sunos5 Type "help", "copyright", "credits" or "license" for more information. >>> import numarray Fatal Python error: Call to API function without first calling import_libnumarray() in Src/_convmodule.c Abort (core dumped) I've seen a one or two recent reports of this error when running extension modules, but I've never heard of it happening when simply trying to import numarray. Any suggestions? -- Russell P.S. I do realize that python 2.3.1 is old and should be updated, and I have already suggested it. But it seems a bit unlikely to fix this particular problem. From nicopernetty at yahoo.fr Sat Jul 2 03:35:23 2005 From: nicopernetty at yahoo.fr (Nicolas Pernetty) Date: Sat Jul 2 03:35:23 2005 Subject: [Numpy-discussion] Trouble installing numarray on Solaris In-Reply-To: References: Message-ID: <20050702123323.05f93476@linuxcestcomplique.fr> On Fri, 01 Jul 2005 15:02:44 -0700, "Russell E. Owen" wrote : > A user of my application reports that installing numarray 1.3.2 on > Solaris using Python 2.3.1 has run into trouble: > > prompt> python > Python 2.3.1 (#1, Sep 30 2003, 20:29:58) [C] on sunos5 > Type "help", "copyright", "credits" or "license" for more > information. > >>> import numarray > Fatal Python error: Call to API function without first calling > import_libnumarray() in Src/_convmodule.c > Abort (core dumped) > > I've seen a one or two recent reports of this error when running > extension modules, but I've never heard of it happening when simply > trying to import numarray. > > Any suggestions? > > -- Russell > > P.S. I do realize that python 2.3.1 is old and should be updated, and > I have already suggested it. But it seems a bit unlikely to fix this > particular problem. > Hello, I'm happily using Python and numarray on Solaris 5.8 without any problems. But it's Python 2.4.1 and numarray 1.3.2. Maybe it's some sort of incompatibility ? Then you should try to install older version of numarray... From jmiller at stsci.edu Sat Jul 2 03:44:31 2005 From: jmiller at stsci.edu (Todd Miller) Date: Sat Jul 2 03:44:31 2005 Subject: [Numpy-discussion] Trouble installing numarray on Solaris In-Reply-To: References: Message-ID: <1120300999.5263.40.camel@localhost.localdomain> On Fri, 2005-07-01 at 15:02 -0700, Russell E. Owen wrote: > A user of my application reports that installing numarray 1.3.2 on > Solaris using Python 2.3.1 has run into trouble: > > prompt> python > Python 2.3.1 (#1, Sep 30 2003, 20:29:58) [C] on sunos5 > Type "help", "copyright", "credits" or "license" for more information. > >>> import numarray > Fatal Python error: Call to API function without first calling > import_libnumarray() in Src/_convmodule.c > Abort (core dumped) It looks vaguely like an binary API mismatch of some kind... or one of you numarray modules is stale or broken. To restate the obvious, it's likely that numarray won't import (in C!) because import_libnumarray() is very defintely in _convmodule.c. Since it's there and the API pointer was not initialized, the libnumarray import failed. I just tried: Python-2.3.4 (non-debug) numarray-1.3.2 (should be irrelevant that it's not 1.3.1) % uname -a SunOS 5.8 Generic_117350-18 sun4u sparc SUNW,Ultra-5_10 It checked out fine. I recommend finding your old numarray install and deleting it and re-installing 1.3.2. Please let me know if that doesn't solve the problem... I realize it's an unsatisfying answer. Regards, Todd > I've seen a one or two recent reports of this error when running > extension modules, but I've never heard of it happening when simply > trying to import numarray. > > Any suggestions? > > -- Russell > > P.S. I do realize that python 2.3.1 is old and should be updated, and I > have already suggested it. But it seems a bit unlikely to fix this > particular problem. > > > > ------------------------------------------------------- > SF.Net email is sponsored by: Discover Easy Linux Migration Strategies > from IBM. Find simple to follow Roadmaps, straightforward articles, > informative Webcasts and more! Get everything you need to get up to > speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From accounts at eBay.com Sat Jul 2 07:28:18 2005 From: accounts at eBay.com (eBay) Date: Sat Jul 2 07:28:18 2005 Subject: [Numpy-discussion] You have successfully added a new email address Message-ID: <1120314405.124551.qmail@paypal.com> An HTML attachment was scrubbed... URL: From schaffer at optonline.net Sat Jul 2 19:40:22 2005 From: schaffer at optonline.net (Les Schaffer) Date: Sat Jul 2 19:40:22 2005 Subject: [Numpy-discussion] subclassing RecordArrays Message-ID: <42C74FDD.9030908@optonline.net> Is there a simple way to subclass RecordArrays? it seems not. suppose i want to have a class RecArrayD(RecArray) that returns a list of its fields, much like a Dict does: recarrayD.keys() the problem is that the fromXXX functions in records.py know only about the one RecArray. and RecArrayD._copyFrom() seems to be of little help, since the new style RecArray has to be framed out first, so to speak. which leaves me to write a new fromRecArray() function that uses the new class. all to get a list of the fields. is this by design, or am i missing something? yes, i can get the keys like so: recarray._names, but i assume that the '_' in front of '_names' is there for a reason, like maybe don't count on it??? or would the designers be willing to add a keys()-like method in for the next release? les schaffer From rowen at u.washington.edu Sat Jul 2 20:31:48 2005 From: rowen at u.washington.edu (Russell E Owen) Date: Sat Jul 2 20:31:48 2005 Subject: [Numpy-discussion] Trouble installing numarray on Solaris In-Reply-To: <1120300999.5263.40.camel@localhost.localdomain> References: <1120300999.5263.40.camel@localhost.localdomain> Message-ID: At 6:43 AM -0400 7/2/05, Todd Miller wrote: >On Fri, 2005-07-01 at 15:02 -0700, Russell E. Owen wrote: >> A user of my application reports that installing numarray 1.3.2 on >> Solaris using Python 2.3.1 has run into trouble: >> >> prompt> python >> Python 2.3.1 (#1, Sep 30 2003, 20:29:58) [C] on sunos5 >> Type "help", "copyright", "credits" or "license" for more information. >> >>> import numarray >> Fatal Python error: Call to API function without first calling >> import_libnumarray() in Src/_convmodule.c >> Abort (core dumped) > >It looks vaguely like an binary API mismatch of some kind... or one of >you numarray modules is stale or broken. To restate the obvious, it's >likely that numarray won't import (in C!) because import_libnumarray() >is very defintely in _convmodule.c. Since it's there and the API >pointer was not initialized, the libnumarray import failed. Here's what the person had to say about it. I found the solution rather puzzling, but perhaps it makes sense to somebody who understands more about such things: Got it working; it was indeed that the shared object modules weren't being linked correctly at build time, so when python tried to load them, they were missing symbols (eg, that are defined in the C math library, for example). Probably the result of compiling numarray w/Solaris's cc rather than gcc. I manually linked the modules for now, and numarry can now be imported... It even passed all its tests. -- Russell From theller at python.net Mon Jul 4 04:43:22 2005 From: theller at python.net (Thomas Heller) Date: Mon Jul 4 04:43:22 2005 Subject: [Numpy-discussion] Re: Trouble installing numarray on Solaris References: <1120300999.5263.40.camel@localhost.localdomain> Message-ID: Todd Miller writes: > On Fri, 2005-07-01 at 15:02 -0700, Russell E. Owen wrote: >> A user of my application reports that installing numarray 1.3.2 on >> Solaris using Python 2.3.1 has run into trouble: >> >> prompt> python >> Python 2.3.1 (#1, Sep 30 2003, 20:29:58) [C] on sunos5 >> Type "help", "copyright", "credits" or "license" for more information. >> >>> import numarray >> Fatal Python error: Call to API function without first calling >> import_libnumarray() in Src/_convmodule.c >> Abort (core dumped) > > It looks vaguely like an binary API mismatch of some kind... or one of > you numarray modules is stale or broken. To restate the obvious, it's > likely that numarray won't import (in C!) because import_libnumarray() > is very defintely in _convmodule.c. Since it's there and the API > pointer was not initialized, the libnumarray import failed. To the numarray authors/maintainers: Isn't it a bit harsh to call Py_FatalError() in the init function? Thomas From jmiller at stsci.edu Mon Jul 4 05:58:32 2005 From: jmiller at stsci.edu (Todd Miller) Date: Mon Jul 4 05:58:32 2005 Subject: [Numpy-discussion] Re: Trouble installing numarray on Solaris In-Reply-To: References: <1120300999.5263.40.camel@localhost.localdomain> Message-ID: <1120481830.5222.23.camel@localhost.localdomain> On Mon, 2005-07-04 at 13:40 +0200, Thomas Heller wrote: > Todd Miller writes: > > > On Fri, 2005-07-01 at 15:02 -0700, Russell E. Owen wrote: > >> A user of my application reports that installing numarray 1.3.2 on > >> Solaris using Python 2.3.1 has run into trouble: > >> > >> prompt> python > >> Python 2.3.1 (#1, Sep 30 2003, 20:29:58) [C] on sunos5 > >> Type "help", "copyright", "credits" or "license" for more information. > >> >>> import numarray > >> Fatal Python error: Call to API function without first calling > >> import_libnumarray() in Src/_convmodule.c > >> Abort (core dumped) > > > > It looks vaguely like an binary API mismatch of some kind... or one of > > you numarray modules is stale or broken. To restate the obvious, it's > > likely that numarray won't import (in C!) because import_libnumarray() > > is very defintely in _convmodule.c. Since it's there and the API > > pointer was not initialized, the libnumarray import failed. > > To the numarray authors/maintainers: Isn't it a bit harsh to call > Py_FatalError() in the init function? That's an interesting observation... the old recourse was to dump core by calling through a NULL function pointer. :-) Since this error can occur in the context of any API function, possibly called from any sub-function within an extension function, I'm not sure there is a soft & general way to handle it: getting it implies the function must fail but that can't always be reported. I'm open to suggestions... it's just not obvious to me how to handle it better. I've concluded that the message is overly specific and may shorten it to just the facts: "call through NULL numarray C-API pointer in ." Regards, Todd From theller at python.net Mon Jul 4 10:01:17 2005 From: theller at python.net (Thomas Heller) Date: Mon Jul 4 10:01:17 2005 Subject: [Numpy-discussion] Re: Trouble installing numarray on Solaris References: <1120300999.5263.40.camel@localhost.localdomain> <1120481830.5222.23.camel@localhost.localdomain> Message-ID: Todd Miller writes: > On Mon, 2005-07-04 at 13:40 +0200, Thomas Heller wrote: >> Todd Miller writes: >> >> > On Fri, 2005-07-01 at 15:02 -0700, Russell E. Owen wrote: >> >> A user of my application reports that installing numarray 1.3.2 on >> >> Solaris using Python 2.3.1 has run into trouble: >> >> >> >> prompt> python >> >> Python 2.3.1 (#1, Sep 30 2003, 20:29:58) [C] on sunos5 >> >> Type "help", "copyright", "credits" or "license" for more information. >> >> >>> import numarray >> >> Fatal Python error: Call to API function without first calling >> >> import_libnumarray() in Src/_convmodule.c >> >> Abort (core dumped) >> > >> > It looks vaguely like an binary API mismatch of some kind... or one of >> > you numarray modules is stale or broken. To restate the obvious, it's >> > likely that numarray won't import (in C!) because import_libnumarray() >> > is very defintely in _convmodule.c. Since it's there and the API >> > pointer was not initialized, the libnumarray import failed. >> >> To the numarray authors/maintainers: Isn't it a bit harsh to call >> Py_FatalError() in the init function? > > That's an interesting observation... the old recourse was to dump core > by calling through a NULL function pointer. :-) > > Since this error can occur in the context of any API function, possibly > called from any sub-function within an extension function, I'm not sure > there is a soft & general way to handle it: getting it implies the > function must fail but that can't always be reported. I'm open to > suggestions... it's just not obvious to me how to handle it better. Maybe I was confusing numarray and numeric. In numeric's _numpymodule.c, function init_numpy(), there is this [...] if (PyErr_Occurred()) goto err; return; err: Py_FatalError("can't initialize module _numpy"); } Sorry for the confusion. > I've concluded that the message is overly specific and may shorten it to > just the facts: "call through NULL numarray C-API pointer in ." Wouldn't it be possible to raise an exception - that would make it easier to find the bug? Thomas From jmiller at stsci.edu Mon Jul 4 17:27:54 2005 From: jmiller at stsci.edu (Todd Miller) Date: Mon Jul 4 17:27:54 2005 Subject: [Numpy-discussion] Re: Trouble installing numarray on Solaris In-Reply-To: References: <1120300999.5263.40.camel@localhost.localdomain> <1120481830.5222.23.camel@localhost.localdomain> Message-ID: <1120523113.4725.17.camel@localhost.localdomain> On Mon, 2005-07-04 at 18:59 +0200, Thomas Heller wrote: > Todd Miller writes: > > > On Mon, 2005-07-04 at 13:40 +0200, Thomas Heller wrote: > >> Todd Miller writes: > >> > >> > On Fri, 2005-07-01 at 15:02 -0700, Russell E. Owen wrote: > >> >> A user of my application reports that installing numarray 1.3.2 on > >> >> Solaris using Python 2.3.1 has run into trouble: > >> >> > >> >> prompt> python > >> >> Python 2.3.1 (#1, Sep 30 2003, 20:29:58) [C] on sunos5 > >> >> Type "help", "copyright", "credits" or "license" for more information. > >> >> >>> import numarray > >> >> Fatal Python error: Call to API function without first calling > >> >> import_libnumarray() in Src/_convmodule.c > >> >> Abort (core dumped) > >> > > >> > It looks vaguely like an binary API mismatch of some kind... or one of > >> > you numarray modules is stale or broken. To restate the obvious, it's > >> > likely that numarray won't import (in C!) because import_libnumarray() > >> > is very defintely in _convmodule.c. Since it's there and the API > >> > pointer was not initialized, the libnumarray import failed. > >> > >> To the numarray authors/maintainers: Isn't it a bit harsh to call > >> Py_FatalError() in the init function? > > > > That's an interesting observation... the old recourse was to dump core > > by calling through a NULL function pointer. :-) > > > > Since this error can occur in the context of any API function, possibly > > called from any sub-function within an extension function, I'm not sure > > there is a soft & general way to handle it: getting it implies the > > function must fail but that can't always be reported. I'm open to > > suggestions... it's just not obvious to me how to handle it better. > > Maybe I was confusing numarray and numeric. In numeric's > _numpymodule.c, function init_numpy(), there is this > > [...] > if (PyErr_Occurred()) goto err; > return; > err: > Py_FatalError("can't initialize module _numpy"); > } > > Sorry for the confusion. > > > I've concluded that the message is overly specific and may shorten it to > > just the facts: "call through NULL numarray C-API pointer in ." > > Wouldn't it be possible to raise an exception - that would make it > easier to find the bug? The two cases where this error have been triggered are: 1. A bug in a C extension module which an author will catch before release. An exception is not worth much there. 2. A corrupted numarray installation which will no longer import. I agree it would be nice there. But... it's hard to raise an exception because the failure macro would have to handle a variety of different return values, even no return value. I don't see how to do it without too much extra complexity for a relatively rare problem. Regards, Todd From faltet at carabos.com Tue Jul 5 09:58:23 2005 From: faltet at carabos.com (Francesc Altet) Date: Tue Jul 5 09:58:23 2005 Subject: [Numpy-discussion] subclassing RecordArrays In-Reply-To: <42C74FDD.9030908@optonline.net> References: <42C74FDD.9030908@optonline.net> Message-ID: <200507051856.28581.faltet@carabos.com> A Sunday 03 July 2005 04:39, Les Schaffer va escriure: > suppose i want to have a class RecArrayD(RecArray) that returns a list > of its fields, much like a Dict does: recarrayD.keys() > > the problem is that the fromXXX functions in records.py know only about > the one RecArray. and RecArrayD._copyFrom() seems to be of little help, > since the new style RecArray has to be framed out first, so to speak. I don't know exactly what do you mean here. If RecArrayD inherits from RecArray, then RecArrayD should be a RecArray object (and also a RecArrayD, of course!). So I would say that the fromXXX fucntions should work just fine for these objects. May you put some example on what are you trying to do?. > yes, i can get the keys like so: recarray._names, but i assume that the > '_' in front of '_names' is there for a reason, like maybe don't count > on it??? Maybe. However I guess that "_names" would not change for some time so it might be safe using it for a while :-P. Nevertheless I agree that "_names" attribute should be promoted to "names" because it's a first-class attribute for RecArray object IMO. Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From schaffer at optonline.net Tue Jul 5 10:36:49 2005 From: schaffer at optonline.net (Les Schaffer) Date: Tue Jul 5 10:36:49 2005 Subject: [Numpy-discussion] subclassing RecordArrays In-Reply-To: <200507051856.28581.faltet@carabos.com> References: <42C74FDD.9030908@optonline.net> <200507051856.28581.faltet@carabos.com> Message-ID: <42CAC4EB.1020304@optonline.net> Francesc Altet wrote: >I don't know exactly what do you mean here. If RecArrayD inherits from >RecArray, then RecArrayD should be a RecArray object (and also a >RecArrayD, of course!). So I would say that the fromXXX fucntions >should work just fine for these objects. May you put some example on >what are you trying to do?. > > consider the calling sequence: <<<<<<<<<<<<<<<<< def array(buffer=None, formats=None, shape=0, names=None, byteorder=sys.byteorder, aligned=0): ... [code to decide which fromXXXX to use] def fromrecords(recList, formats=None, names=None, shape=0, byteorder=sys.byteorder, aligned=0): ... _array = fromarrays(arrlist, formats=formats, shape=_shape, names=names, byteorder=byteorder, aligned=aligned) ... def fromarrays(arrayList, formats=None, names=None, shape=0, byteorder=sys.byteorder, aligned=0): ... _array = RecArray(None, formats=formats, shape=_shape, names=names, byteorder=byteorder, aligned=aligned) >>>>>>>>>>>>>>>>>>>>> there is no way to tell fromXXXX to use my new class. one would need something like: >>>>>>>>>>>>>>>>>>>>>>>>>>>>> def array(buffer=None, formats=None, shape=0, names=None, byteorder=sys.byteorder, aligned=0,rec_class = RecArray ): def fromrecords(recList, formats=None, names=None, shape=0, byteorder=sys.byteorder, aligned=0, rec_class): _array = fromarrays(arrlist, formats=formats, shape=_shape, names=names, byteorder=byteorder, aligned=aligned, rec_class) def fromarrays(arrayList, formats=None, names=None, shape=0, byteorder=sys.byteorder, aligned=0, rec_class): ... _array = rec_class(None, formats=formats, shape=_shape, names=names, byteorder=byteorder, aligned=aligned) <<<<<<<<<<<<<<<<<<<<<<<<<<<<<< THEN i could use my MyRecArrayWithDict class to my hearts content. all that though to have a RecArray.keys() or .names() or .RemindMeWhatWasTheNamesOfTheFields() ???? >Maybe. However I guess that "_names" would not change for some time so >it might be safe using it for a while :-P. Nevertheless I agree that >"_names" attribute should be promoted to "names" because it's a >first-class attribute for RecArray object IMO. > agreed. seems the safest design tho would be to GRAB the "names" entered when instantiating the RecArray (names='XXX, YYY, ...') plus whatever col_N's are added for un-named columns and keep it in an attribute accessible by keys() (or whatever name you like). les schaffer From faltet at carabos.com Tue Jul 5 11:31:03 2005 From: faltet at carabos.com (Francesc Altet) Date: Tue Jul 5 11:31:03 2005 Subject: [Numpy-discussion] subclassing RecordArrays In-Reply-To: <42CAC4EB.1020304@optonline.net> References: <42C74FDD.9030908@optonline.net> <200507051856.28581.faltet@carabos.com> <42CAC4EB.1020304@optonline.net> Message-ID: <200507052028.57782.faltet@carabos.com> A Tuesday 05 July 2005 19:35, Les Schaffer va escriure: > there is no way to tell fromXXXX to use my new class. one would need > something like: I see. Yep, you are right. [snip...] > THEN i could use my MyRecArrayWithDict class to my hearts content. Yes. We encountered similar problems to this that you reported in a subclass of RecArray to support nested columns. We finally ended implementing specific versions of array() and fromarray() factories. Sorry, as I was not the main author of the module, I did not remember that issue. Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From edcjones at comcast.net Tue Jul 5 11:47:29 2005 From: edcjones at comcast.net (Edward C. Jones) Date: Tue Jul 5 11:47:29 2005 Subject: [Numpy-discussion] numarray: Bug printing an object array after unpickling it Message-ID: <42CAD56A.4070707@comcast.net> #! /usr/bin/env python # I have a PC with up-to-date Debian unstable Linux. I have installed # Python 2.4.1 and numarray 1.3.2 from source (not from a Debian # package). The following program fails for both pickle and cPickle: import pickle import numarray, numarray.objects obj = numarray.objects.array([0,0,0,0], (2,2)) print obj s = pickle.dumps(obj, pickle.HIGHEST_PROTOCOL) open('mess2', 'w').write(s) s = open('mess2', 'r').read() obj = pickle.loads(s) print obj # Fails here From faltet at carabos.com Tue Jul 5 11:49:17 2005 From: faltet at carabos.com (Francesc Altet) Date: Tue Jul 5 11:49:17 2005 Subject: [Numpy-discussion] nestedrecords.py module Message-ID: <200507052047.17356.faltet@carabos.com> Hi, In our company we had the need for an extension of RecArray objects so that they would be able to keep nested fields (that is, fields that can have other subfields). The outcome was a module that is intended to do that. We have tested it quite extensively and we think it is quite free of bugs. The conversation with Les has reminded me about it, and thought that it would interesting for the numarray community. I'm attaching a tarball with the next files: - nestedrecords.py -- Where the NestedRecArray lives - nriterators.py -- A suite of iterator utilities for nestedrecords.py - nestedrecords.txt -- Documentation in restructured text The nestedrecords module is mainly a job of Vicent Mas (vmas at carabos.com) and is offered with a BSD license. Of course, we would be glad if this stuff ends in numarray sometime :-) -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" -------------- next part -------------- A non-text attachment was scrubbed... Name: nestedrecords.tar.gz Type: application/x-tgz Size: 12901 bytes Desc: not available URL: From faltet at carabos.com Tue Jul 5 12:01:07 2005 From: faltet at carabos.com (Francesc Altet) Date: Tue Jul 5 12:01:07 2005 Subject: [Numpy-discussion] nestedrecords.py module In-Reply-To: <200507052047.17356.faltet@carabos.com> References: <200507052047.17356.faltet@carabos.com> Message-ID: <200507052058.57780.faltet@carabos.com> Ops, forgot to include the test units in the tarball. Hopefully they are there now. -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" -------------- next part -------------- A non-text attachment was scrubbed... Name: nestedrecords.tar.gz Type: application/x-tgz Size: 16801 bytes Desc: not available URL: From haase at msg.ucsf.edu Tue Jul 5 16:06:27 2005 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue Jul 5 16:06:27 2005 Subject: [Numpy-discussion] patch numarray.fromfuntion with optional type argument In-Reply-To: <200506161112.49144.haase@msg.ucsf.edu> References: <200506161112.49144.haase@msg.ucsf.edu> Message-ID: <200507051604.34642.haase@msg.ucsf.edu> Hi, (this is regarding my post from 16 June 2005) Todd, maybe you can confirm that this patch is a good idea and that you will put it into upcoming numarray versions !? I would like to streamline my code - but only of course if you OK this. Sebastian On Thursday 16 June 2005 11:12, Sebastian Haase wrote: > Hi, > Please tell if this patch is a good idea. (Use: For large image data we > always use Float32 and I thought it would be extra overhead if I first > create everything in Float64 and then have to convert to Float32 > afterwards, myself) > > haase at gorilla:~: diff -p ~/myCVS/numarray/Lib/generic.py > ~/myCVS/numarray/Lib/generic.py.~1.74.~ > *** /home/haase/myCVS/numarray/Lib/generic.py Thu Jun 16 11:04:10 2005 > --- /home/haase/myCVS/numarray/Lib/generic.py.~1.74.~ Fri Apr 22 13:35:26 > 2005 > *************** def indices(shape, type=None): > *** 1167,1179 **** > a = a.astype(type) > return a > > ! def fromfunction(function, dimensions, type=None): # from Numeric > """fromfunction() returns an array constructed by calling function > on a tuple of number grids. The function should accept as many > arguments as there are dimensions which is a list of numbers > indicating the length of the desired output for each axis. > """ > ! return apply(function, tuple(indices(dimensions,type))) > > def _broadcast_or_resize(a, b): > try: > --- 1167,1179 ---- > a = a.astype(type) > return a > > ! def fromfunction(function, dimensions): # from Numeric > """fromfunction() returns an array constructed by calling function > on a tuple of number grids. The function should accept as many > arguments as there are dimensions which is a list of numbers > indicating the length of the desired output for each axis. > """ > ! return apply(function, tuple(indices(dimensions))) > > def _broadcast_or_resize(a, b): > try: > > Thanks, > - Sebastian Haase > > > ------------------------------------------------------- > SF.Net email is sponsored by: Discover Easy Linux Migration Strategies > from IBM. Find simple to follow Roadmaps, straightforward articles, > informative Webcasts and more! Get everything you need to get up to > speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From haase at msg.ucsf.edu Tue Jul 5 17:37:01 2005 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue Jul 5 17:37:01 2005 Subject: [Numpy-discussion] unexpected behavior of na.where (no short-circuiting) Message-ID: <200507051734.59020.haase@msg.ucsf.edu> Hi, I was very surprised when I got this warning: >>> a = na.arange(4)-2 >>> na.where(a != 0,1./a, 999) Warning: Encountered divide by zero(s) in divide [ -0.5 -1. 999. 1. ] Then I realized that this is generally referred to as (not) "short circuiting" (e.g. in the case of the '?:'-C-operator, the middle part never gets evaluated at all if the first part evals to 0 ) Especially annoying was this because (for debugging) I had set this error-mode: >>> na.Error.setMode(dividebyzero="error") My questions are: a) Did other people encounter this problem ? b) What is the general feeling about this actually being a "problem" ? c) Could this (at all possible) be implemented differently ? Thanks, Sebastian Haase From jmiller at stsci.edu Wed Jul 6 05:58:42 2005 From: jmiller at stsci.edu (Todd Miller) Date: Wed Jul 6 05:58:42 2005 Subject: [Numpy-discussion] patch numarray.fromfuntion with optional type argument In-Reply-To: <200507051604.34642.haase@msg.ucsf.edu> References: <200506161112.49144.haase@msg.ucsf.edu> <200507051604.34642.haase@msg.ucsf.edu> Message-ID: <1120654644.5565.15.camel@localhost.localdomain> On Tue, 2005-07-05 at 16:04 -0700, Sebastian Haase wrote: > Hi, (this is regarding my post from 16 June 2005) > Todd, maybe you can confirm that this patch is a good idea and that you will > put it into upcoming numarray versions !? > I would like to streamline my code - but only of course if you OK this. > > Sebastian I added a 'type' parameter to fromfunction() and committed it to CVS this morning. It'll be in 1.3.3. Regards, Todd From jmiller at stsci.edu Wed Jul 6 06:10:44 2005 From: jmiller at stsci.edu (Todd Miller) Date: Wed Jul 6 06:10:44 2005 Subject: [Numpy-discussion] Trouble installing numarray on Solaris In-Reply-To: References: <1120300999.5263.40.camel@localhost.localdomain> Message-ID: <1120655414.5565.26.camel@localhost.localdomain> On Sat, 2005-07-02 at 20:30 -0700, Russell E Owen wrote: > At 6:43 AM -0400 7/2/05, Todd Miller wrote: > >On Fri, 2005-07-01 at 15:02 -0700, Russell E. Owen wrote: > >> A user of my application reports that installing numarray 1.3.2 on > >> Solaris using Python 2.3.1 has run into trouble: > >> > >> prompt> python > >> Python 2.3.1 (#1, Sep 30 2003, 20:29:58) [C] on sunos5 > >> Type "help", "copyright", "credits" or "license" for more information. > >> >>> import numarray > >> Fatal Python error: Call to API function without first calling > >> import_libnumarray() in Src/_convmodule.c > >> Abort (core dumped) > > > >It looks vaguely like an binary API mismatch of some kind... or one of > >you numarray modules is stale or broken. To restate the obvious, it's > >likely that numarray won't import (in C!) because import_libnumarray() > >is very defintely in _convmodule.c. Since it's there and the API > >pointer was not initialized, the libnumarray import failed. > > Here's what the person had to say about it. I found the solution > rather puzzling, but perhaps it makes sense to somebody who > understands more about such things: > > Got it working; it was indeed that the shared object modules weren't > being linked correctly at build time, so when python tried to load > them, they were missing symbols (eg, that are defined in the C math > library, for example). Probably the result of compiling numarray > w/Solaris's cc rather than gcc. I manually linked the modules for > now, and numarry can now be imported... It even passed all its tests. One thing I should have mentioned earlier is that numarray has been built at STScI using Solaris's cc from the beginning. Although our primary development platform is now x86 Linux, we still ultimately deploy to Solaris so I think Solaris cc per-say is probably not the problem here. Here are the tools I use personally w/o problems: basil> uname -a SunOS anysun.stsci.edu 5.8 Generic_117350-18 sun4u sparc SUNW,Ultra-5_10 basil> cc -V cc: Sun WorkShop 6 2000/06/19 C 5.1 Patch 109491-02 Regards, Todd From cjw at sympatico.ca Wed Jul 6 08:58:38 2005 From: cjw at sympatico.ca (Colin J. Williams) Date: Wed Jul 6 08:58:38 2005 Subject: [Numpy-discussion] _fastCopyAndTranpose Message-ID: <42CBFF51.4030606@sympatico.ca> numarray.LinearAlgebra2.py has # _fastCopyAndTranpose is an optimized version of _castCopyAndTranspose. # It assumes the input is 2D (as all the calls in here are). However the following code is commented out. Is this considered a satisactory replacement for _castCopyAndTranspose? Colin W From rowen at u.washington.edu Wed Jul 6 16:24:36 2005 From: rowen at u.washington.edu (Russell E Owen) Date: Wed Jul 6 16:24:36 2005 Subject: [Numpy-discussion] Trouble installing numarray on Solaris In-Reply-To: <1120655414.5565.26.camel@localhost.localdomain> References: <1120300999.5263.40.camel@localhost.localdomain> <1120655414.5565.26.camel@localhost.localdomain> Message-ID: At 9:10 AM -0400 2005-07-06, Todd Miller wrote: >One thing I should have mentioned earlier is that numarray has been >built at STScI using Solaris's cc from the beginning. Although our >primary development platform is now x86 Linux, we still ultimately >deploy to Solaris so I think Solaris cc per-say is probably not the >problem here. > >Here are the tools I use personally w/o problems: > >basil> uname -a >SunOS anysun.stsci.edu 5.8 Generic_117350-18 sun4u sparc SUNW,Ultra-5_10 >basil> cc -V >cc: Sun WorkShop 6 2000/06/19 C 5.1 Patch 109491-02 I asked John for more information and got the following: --- begin quote --- First: The numarray modules (shared library/object files) weren't being "linked" against any libraries when built, and as a result, any symbols (functions) that they referenced which are in shared libraries not already being loaded by python itself would go unresolved whenever python tried to load the modules. In particular, my dynamically linked copy of python (which I compiled from source, btw) isn't linked against the C math library, so any math functions being used by numarray were unresolved when the modules were loaded. The fix was to add a "-lm" to the command being used to link the module, eg, change: cc -G build/temp.solaris-2.7-sun4u-2.3/Src/_ufuncFloat32module.o -o \ build/lib.solaris-2.7-sun4u-2.3/numarray/_ufuncFloat32.so to cc -G build/temp.solaris-2.7-sun4u-2.3/Src/_ufuncFloat32module.o -o \ build/lib.solaris-2.7-sun4u-2.3/numarray/_ufuncFloat32.so -lm With that, now whenever this module gets loaded, the dynamic loader routines will automatically pull in any required routines from the math library (libm.so). Given the time constraints and my lack of python experience, I couldn't figure out how to feed the standard "python setup.py build" command a list of libraries to link against, so instead I just let setup.py build it as is, logged the commands used to create the shared objects, and then turned the log into a script to relink the objects after manually added a "-lm" to the end of each link command. Looking through distutils now, I think I can feed setup.py a "--libraries" option with the name of the math library, and with that, it will then build/link the modules correctly. > Before I...pass it along, though, I just thought of something: you > did specify the --gencode argument, didn't you? The need for that > argument is an unwelcome numarray oddity. Yes I did, although I wasn't sure why that was needed. BTW, PIL built OK, and likewise, its modules link against libraries which python doesn't automatically pull in itself (eg, libjpeg, zlib, etc), so something in its setup.py is automatically including these libraries when it builds/links the shared objects. Another datapoint: I *think* (or at least it used to be the case) that gcc always includes the math library when linking, whereas with Sun's (and may other vendors') compilers, -lm needs to be given explicitly. In general, I usually try to compile things with an OS's/vendor's native compilers instead of gcc, just to keep developers honest (ie, in the interest of weeding out compiler dependencies/bugs that inhibit source portability). :) I'm guessing that if I had compiled w/ gcc, I wouldn't have had this problem. However, I had compiled python and the other python modules I have installed with Sun's cc (from Sun's Studio 8 compiler collection, btw), so for consistency, I wouldn't have wanted to use gcc anyway. --- end quote --- Unless you have any questions, that's the last I'll say on the subject. I hope something in here may help track down a bug. Regards, -- Russell From jmiller at stsci.edu Thu Jul 7 04:34:52 2005 From: jmiller at stsci.edu (Todd Miller) Date: Thu Jul 7 04:34:52 2005 Subject: [Numpy-discussion] Trouble installing numarray on Solaris In-Reply-To: References: <1120300999.5263.40.camel@localhost.localdomain> <1120655414.5565.26.camel@localhost.localdomain> Message-ID: <1120736002.4694.25.camel@localhost.localdomain> On Wed, 2005-07-06 at 16:22 -0700, Russell E Owen wrote: > Unless you have any questions, that's the last I'll say on the > subject. I hope something in here may help track down a bug. > > Regards, > > -- Russell I added "-lm" globally to the extra link arguments... hopefully that should take care of it. Regards, Todd From jmiller at stsci.edu Sat Jul 9 10:26:15 2005 From: jmiller at stsci.edu (Todd Miller) Date: Sat Jul 9 10:26:15 2005 Subject: [Numpy-discussion] ANN: numarray-1.3.3 Message-ID: <1120929968.4884.2.camel@localhost.localdomain> Numarray is an array processing package designed to efficiently manipulate large multi-dimensional arrays. Numarray is modelled after Numeric and features c-code generated from python template scripts, the capacity to operate directly on arrays in files, arrays of heterogeneous records, string arrays, and in-place operation on memory mapped files. ========================================================================= Release Notes for numarray-1.3.3 I. ENHANCEMENTS None. 1.3.3 fixes a fatal problem with embedding numarray and loading it more than once. It also fixes some numarray memory leaks associated with restarting the Python interpreter. II. BUGS FIXED / CLOSED 1233431 Add 'type' to fromfunction() 1230490 Numarray: integer overflow in sum() is not detected 1216688 ufunc's ignoring complex operand 1213675 Problems embedding after 1.1.1 1209946 Mac OS-X --use_lapack 1212975 numarray.fromfunction(...) misbehavior 1209929 config.h and Python-2.2 1235278 Add -lm to numarray link lists See http://sourceforge.net/tracker/?atid=450446&group_id=1369&func=browse for more details. ========================================================================= Release Notes for numarray-1.3.2 I. ENHANCEMENTS None. 1.3.2 fixes the problem with gcc4 on Mac OS-X 10.4 Tiger II. BUGS FIXED / CLOSED 1193125 Build failure on Mac OS 10.4 (Tiger) 1195370 numarray1.3.1 setup.py broken See http://sourceforge.net/tracker/?atid=450446&group_id=1369&func=browse for more details. ========================================================================= Release Notes for numarray-1.3.1 I. ENHANCEMENTS None. 1.3.1 fixes the problem with gcc-3.4.3 II. BUGS FIXED / CLOSED 1152323 /usr/include/fenv.h:96: error: conflicting types for 'fegete 1185024 numarray-1.2.3 fails to compile with gcc-3.4.3 1187162 Numarray 1.3.0 installation failure See http://sourceforge.net/tracker/?atid=450446&group_id=1369&func=browse for more details. ========================================================================= Release Notes for numarray-1.3.0 Numarray is an array processing package designed to efficiently manipulate large multi-dimensional arrays. Numarray is modelled after Numeric and features c-code generated from python template scripts, the capacity to operate directly on arrays in files, arrays of heterogeneous records, string arrays, and in-place operation on memory mapped files. I. ENHANCEMENTS 1. Migration of NumArray.__del__ to C (tp_dealloc). Overall performance. 2. Removal of dictionary update from array view creation improves performance of view/slice/subarray creation. This should e.g. improve the performance of wxPython sequence protocol access to Nx2 arrays. Subclasses now need to do a.flags |= numarray.generic._UPDATEDICT to ensure that dictionary based attributes are inherited by views. NumArrays no longer do this by default. 2. Modifications to support scipy.special. 3. Removal of an unnecessary getattr() from ufunc calling sequence. Ufunc performance. II. BUGS FIXED / CLOSED 1179355 average() broken in numarray 1.2.3 1167184 Floating point exception in numarray's dot() 1151892 Bug in matrixmultiply with zero size arrays 1160184 RecArray reversal 1156172 Incorect error message for shape incompatability 1155538 Incorrect error message when multiplying arrays See http://sourceforge.net/tracker/?atid=450446&group_id=1369&func=browse for more details. III. CAUTIONS This release should be backward binary compatible with numarray-1.1.1 and 1.2.3. WHERE ----------- Numarray-1.3.0 windows executable installers, source code, and manual is here: http://sourceforge.net/project/showfiles.php?group_id=1369 Numarray is hosted by Source Forge in the same project which hosts Numeric: http://sourceforge.net/projects/numpy/ The web page for Numarray information is at: http://stsdas.stsci.edu/numarray/index.html Trackers for Numarray Bugs, Feature Requests, Support, and Patches are at the Source Forge project for NumPy at: http://sourceforge.net/tracker/?group_id=1369 REQUIREMENTS ------------------------------ numarray-1.3.0 requires Python 2.2.2 or greater. AUTHORS, LICENSE ------------------------------ Numarray was written by Perry Greenfield, Rick White, Todd Miller, JC Hsu, Paul Barrett, Phil Hodge at the Space Telescope Science Institute. We'd like to acknowledge the assitance of Francesc Alted, Paul Dubois, Sebastian Haase, Chuck Harris, Tim Hochberg, Nadav Horesh, Edward C. Jones, Eric Jones, Jochen Kuepper, Travis Oliphant, Pearu Peterson, Peter Verveer, Colin Williams, Rory Yorke, and everyone else who has contributed with comments and feedback. Numarray is made available under a BSD-style License. See LICENSE.txt in the source distribution for details. -- Todd Miller jmiller at stsci.edu From jmiller at stsci.edu Sun Jul 10 18:21:58 2005 From: jmiller at stsci.edu (Todd Miller) Date: Sun Jul 10 18:21:58 2005 Subject: [Numpy-discussion] ANN: numarray-1.3.3 In-Reply-To: <42D14226.4050701@optonline.net> References: <1120929968.4884.2.camel@localhost.localdomain> <42D00C20.6010402@optonline.net> <1120998167.4921.52.camel@localhost.localdomain> <42D14226.4050701@optonline.net> Message-ID: <1121044171.4694.31.camel@localhost.localdomain> On Sun, 2005-07-10 at 11:43 -0400, Les Schaffer wrote: > Todd Miller wrote: > > >numarray-1.3.3 is a "maintenance release" which solves a couple esoteric > >(embedded Python + numarray) but serious (core dumping) problems. > > > > > > > cool > > >I think there's a good chance of adding something for the next release > >but I don't think it will be called keys() because that's a dictionary > >cue and RecArrays aren't dictionaries. > > > > > > tho as column spaces, they essentially act as dicts. > > >I'm in favor of adding .names to make _names public. > > > > > that would work, at least it tells outsiders .names is public and stable > attribute. > > and even better is help us subclass easily by making the constructor > functions fromarray, fromrecord, and array take an optional argument > which is the RecArray class to use. as it stands RecArray cannot be > subclasses without re-doing all those constructor functions. > > i would be happy to contribute code if i knew it would be used. What you want to do sounds reasonable so chances are good a patch would be used. I would: 1. Make .names another reference to ._names 2. Either convert the functions you want to be subclassable into "class functions" (and add backward compatible globals) or else add a "kind" parameter for passing in the subclass (see strings.py) to the current functions. I'm not sure which approach is better. Regards, Todd From gr at grrrr.org Mon Jul 11 01:51:01 2005 From: gr at grrrr.org (Thomas Grill) Date: Mon Jul 11 01:51:01 2005 Subject: [Numpy-discussion] optimizations In-Reply-To: <1120929968.4884.2.camel@localhost.localdomain> References: <1120929968.4884.2.camel@localhost.localdomain> Message-ID: <78581171a9e444def1705b4a1eb5067c@grrrr.org> Hi all, i'm using numarrays with audio and video processing with the real-time framework pure data (http://www.puredata.org). I realized that the ufuncs in numarrays are not as fast as possible because they are coded very straightforward. Is there any particular reason why this is the case, like cleanness of code, confidence in the compiler or because the code was automatically generated? I would like to contribute assembly SIMD codelets (SSE and Altivec), i have been successfully using in my projects for quite some time. Is it feasible to submit patches, so that these go into the main numarray distribution, or should i rather implement this as separate ufuncs? best greetings, Thomas From onlinebanking at chevychasebank.com Mon Jul 11 06:17:11 2005 From: onlinebanking at chevychasebank.com (Chevy Chase Bank) Date: Mon Jul 11 06:17:11 2005 Subject: [Numpy-discussion] Online Banking - Verify Your Account Information Message-ID: An HTML attachment was scrubbed... URL: From curzio.basso at gmail.com Mon Jul 11 10:40:18 2005 From: curzio.basso at gmail.com (Curzio Basso) Date: Mon Jul 11 10:40:18 2005 Subject: [Numpy-discussion] bug or feature? Message-ID: Hi all, I noticed the following (unexpected for me) behaviour: >>> a = NA.arange(12) >>> a.shape = (3,4) >>> a[[0,2]][:,:2] = a[[0,2]][:,2:] >>> print a array([[ 0, 1, 2, 3], [ 4, 5, 6, 7], [ 8, 9, 10, 11]]) rather than array([[ 2, 3, 2, 3], [ 4, 5, 6, 7], [ 10, 11, 10, 11]]) as I was expecting. This happens only if I try to combine indices lists with slices. Is it by design or I ran into a bug? curzio From perry at stsci.edu Mon Jul 11 10:53:35 2005 From: perry at stsci.edu (Perry Greenfield) Date: Mon Jul 11 10:53:35 2005 Subject: [Numpy-discussion] bug or feature? In-Reply-To: References: Message-ID: <5deb92a4eb50a0ed681dbf3e8d8e3a50@stsci.edu> On Jul 11, 2005, at 1:38 PM, Curzio Basso wrote: > Hi all, > > I noticed the following (unexpected for me) behaviour: > >>>> a = NA.arange(12) >>>> a.shape = (3,4) >>>> a[[0,2]][:,:2] = a[[0,2]][:,2:] >>>> print a > array([[ 0, 1, 2, 3], > [ 4, 5, 6, 7], > [ 8, 9, 10, 11]]) > > rather than > > array([[ 2, 3, 2, 3], > [ 4, 5, 6, 7], > [ 10, 11, 10, 11]]) > > as I was expecting. This happens only if I try to combine indices > lists with slices. > Is it by design or I ran into a bug? > > curzio > An array used as an index produces a new copy (if you think about it, given the possibilities, there isn't much else that can be done). So the entity on the left of your expression is modified, but since nothing refers to it, it is discarded immediately thereafter. The array 'a' itself was never changed. I'd say that it isn't either a bug or feature. Perry Greenfield From perry at stsci.edu Mon Jul 11 12:24:24 2005 From: perry at stsci.edu (Perry Greenfield) Date: Mon Jul 11 12:24:24 2005 Subject: [Numpy-discussion] bug or feature? In-Reply-To: References: <5deb92a4eb50a0ed681dbf3e8d8e3a50@stsci.edu> Message-ID: <23974d4bfbd55596f8212bbfd6ff6723@stsci.edu> On Jul 11, 2005, at 2:38 PM, Curzio Basso wrote: >> An array used as an index produces a new copy (if you think about it, >> given the possibilities, there isn't much else that can be done). So >> the entity on the left of your expression is modified, but since >> nothing refers to it, it is discarded immediately thereafter. The >> array >> 'a' itself was never changed. I'd say that it isn't either a bug or >> feature. > > ok. on the other hand: > >>>> a[[0,2],0] = 1 >>>> print a > array([[ 1, 1, 2, 3], > [ 4, 5, 6, 7], > [ 1, 9, 10, 11]]) > > I have some problem understanding the difference at the implementation > level between the two cases. I would have assumed that at end they > would operate on a similar way... > > curzio I was sloppy in my description. For array indexing, assignment does modify the original array (as your case above illustrates), but once you sliced the array-indexed array, it changed the context of the indexing. I.e., (to take a simpler, 1-d array x) x[[0,2]] = 1 # modifies x This form results in __setitem__ being called on x x[[0,2]][:4] = 1 # doesn't modify x This form results in __getitem__ being called on x (and thus producing a new copy of the array) for the array indexing and __setitem__ being called for the slice. At least that's what I think is happening. Todd or someone more recently familiar with how Python handles this can correct me if I'm wrong. Perry From fccoelho at gmail.com Tue Jul 12 13:34:21 2005 From: fccoelho at gmail.com (Flavio Coelho) Date: Tue Jul 12 13:34:21 2005 Subject: [Numpy-discussion] problem with poisson generators Message-ID: Hi, I am having problems with the poisson random number generators of both Numarray and Numeric. I can't replicate it when calling the function from the python cosonle, but it has consistently malfunctioned when used within one of my scripts. What happens is that it frequently return a value greater than zero when called with zero mean: poisson(0.0) Unfortunately My program is too big to send attached but I have confirmed the malfunction by printing both the mean and the result whenever it spits out a wrong result. This is very weird indeed, I have run poisson millions of times by itsel on the python console, without any problems... I hope it is some stupid mistake, but when I replace the poisson function call within my program by the R equivalent command (rpois) via the rpy wrapper, everything works just fine... any Ideas? -- Fl?vio Code?o Coelho registered Linux user # 386432 --------------------------- Great spirits have always found violent opposition from mediocrities. The latter cannot understand it when a man does not thoughtlessly submit to hereditary prejudices but honestly and courageously uses his intelligence. Albert Einstein -------------- next part -------------- An HTML attachment was scrubbed... URL: From jdhunter at ace.bsd.uchicago.edu Tue Jul 12 13:54:47 2005 From: jdhunter at ace.bsd.uchicago.edu (John Hunter) Date: Tue Jul 12 13:54:47 2005 Subject: [Numpy-discussion] problem with poisson generators In-Reply-To: (Flavio Coelho's message of "Tue, 12 Jul 2005 17:32:25 -0300") References: Message-ID: <87eka39281.fsf@peds-pc311.bsd.uchicago.edu> >>>>> "Flavio" == Flavio Coelho writes: Flavio> This is very weird indeed, I have run poisson millions of Flavio> times by itsel on the python console, without any Flavio> problems... Does your code include any custom or non-standard extension code? If so, you could have a pointer error somewhere which is fouling up the ranlib memory layout and causing the error. If you can't reproduce the error with a pure python + Numeric script, this may be the explanation. JDH From haase at msg.ucsf.edu Tue Jul 12 14:51:55 2005 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue Jul 12 14:51:55 2005 Subject: [Numpy-discussion] problem with poisson generators In-Reply-To: References: Message-ID: <200507121447.01356.haase@msg.ucsf.edu> Hi Flavio! I had reported this long time ago and this list (about numarray). Somehow this got more or less dismissed. If I recall correctly the argument was that nobody could reproduce it (I ran this on Debian Woody ,py2.2, (with CVS numarray at the time). I ended up writting my own wrapper(s): def poissonArr(shape=defshape, mean=1): from numarray import random_array as ra if mean == 0: return zeroArrF(shape) elif mean < 0: raise "poisson not defined for mean < 0" else: return ra.poisson(mean, shape).astype(na.UInt16) def poissonize(arr): from numarray import random_array as ra return na.where(arr<=0, 0, ra.poisson(arr)).astype(na.UInt16) (I use the astype(na.UInt16) because of some OpenGL code) Just last week had this problem on a windows98 computer (python2.4). This should get sorted out ... Thanks for reporting this problem. Sebastian Haase On Tuesday 12 July 2005 13:32, Flavio Coelho wrote: > Hi, > > I am having problems with the poisson random number generators of both > Numarray and Numeric. > I can't replicate it when calling the function from the python cosonle, but > it has consistently malfunctioned when used within one of my scripts. > > What happens is that it frequently return a value greater than zero when > called with zero mean: poisson(0.0) > > Unfortunately My program is too big to send attached but I have confirmed > the malfunction by printing both the mean and the result whenever it spits > out a wrong result. > > This is very weird indeed, I have run poisson millions of times by itsel on > the python console, without any problems... > > I hope it is some stupid mistake, but when I replace the poisson function > call within my program by the R equivalent command (rpois) via the rpy > wrapper, everything works just fine... > > any Ideas? From fccoelho at gmail.com Wed Jul 13 04:06:16 2005 From: fccoelho at gmail.com (Flavio Coelho) Date: Wed Jul 13 04:06:16 2005 Subject: [Numpy-discussion] problem with poisson generators In-Reply-To: <87eka39281.fsf@peds-pc311.bsd.uchicago.edu> References: <87eka39281.fsf@peds-pc311.bsd.uchicago.edu> Message-ID: No, its a pure python code... 2005/7/12, John Hunter : > > >>>>> "Flavio" == Flavio Coelho writes: > > Flavio> This is very weird indeed, I have run poisson millions of > Flavio> times by itsel on the python console, without any > Flavio> problems... > > Does your code include any custom or non-standard extension code? If > so, you could have a pointer error somewhere which is fouling up the > ranlib memory layout and causing the error. If you can't reproduce > the error with a pure python + Numeric script, this may be the > explanation. > > > > JDH > -- Fl?vio Code?o Coelho registered Linux user # 386432 --------------------------- Great spirits have always found violent opposition from mediocrities. The latter cannot understand it when a man does not thoughtlessly submit to hereditary prejudices but honestly and courageously uses his intelligence. Albert Einstein -------------- next part -------------- An HTML attachment was scrubbed... URL: From fccoelho at gmail.com Wed Jul 13 04:22:05 2005 From: fccoelho at gmail.com (Flavio Coelho) Date: Wed Jul 13 04:22:05 2005 Subject: [Numpy-discussion] problem with poisson generators In-Reply-To: <200507121447.01356.haase@msg.ucsf.edu> References: <200507121447.01356.haase@msg.ucsf.edu> Message-ID: Well, I am definetly glad that someone has also stumbled onto the same problem. But it is nevertheless intriguing, that you can run poisson a million times with mean zero or negative(it assumes zero mean inthis case) without any problems by itself. But when I call it within my code, the rate of error is very high (I would say it returns a wrong result every time, but I haven't checked every answer). Meanwhile, my solution will be: import rpy n = rpy.r.rpois(n,mean) I don't feel I can trust poisson while this "funny" behavior is still there... If someone has any Idea of how I could trace this bug please tell me, and I'll hunt it down. At least I can reproduce it in a very specific context. thanks, Fl?vio 2005/7/12, Sebastian Haase : > > Hi Flavio! > I had reported this long time ago and this list (about numarray). > Somehow this got more or less dismissed. If I recall correctly the > argument > was that nobody could reproduce it (I ran this on Debian Woody ,py2.2, > (with > CVS numarray at the time). > > I ended up writting my own wrapper(s): > def poissonArr(shape=defshape, mean=1): > from numarray import random_array as ra > if mean == 0: > return zeroArrF(shape) > elif mean < 0: > raise "poisson not defined for mean < 0" > else: > return ra.poisson(mean, shape).astype(na.UInt16) > > def poissonize(arr): > from numarray import random_array as ra > return na.where(arr<=0, 0, ra.poisson(arr)).astype(na.UInt16) > > (I use the astype(na.UInt16) because of some OpenGL code) > > Just last week had this problem on a windows98 computer (python2.4). > > This should get sorted out ... > > Thanks for reporting this problem. > Sebastian Haase > > > > > On Tuesday 12 July 2005 13:32, Flavio Coelho wrote: > > Hi, > > > > I am having problems with the poisson random number generators of both > > Numarray and Numeric. > > I can't replicate it when calling the function from the python cosonle, > but > > it has consistently malfunctioned when used within one of my scripts. > > > > What happens is that it frequently return a value greater than zero when > > called with zero mean: poisson(0.0) > > > > Unfortunately My program is too big to send attached but I have > confirmed > > the malfunction by printing both the mean and the result whenever it > spits > > out a wrong result. > > > > This is very weird indeed, I have run poisson millions of times by itsel > on > > the python console, without any problems... > > > > I hope it is some stupid mistake, but when I replace the poisson > function > > call within my program by the R equivalent command (rpois) via the rpy > > wrapper, everything works just fine... > > > > any Ideas? > -- Fl?vio Code?o Coelho registered Linux user # 386432 --------------------------- Great spirits have always found violent opposition from mediocrities. The latter cannot understand it when a man does not thoughtlessly submit to hereditary prejudices but honestly and courageously uses his intelligence. Albert Einstein -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsouthey at gmail.com Wed Jul 13 06:20:42 2005 From: bsouthey at gmail.com (Bruce Southey) Date: Wed Jul 13 06:20:42 2005 Subject: [Numpy-discussion] problem with poisson generators In-Reply-To: References: <200507121447.01356.haase@msg.ucsf.edu> Message-ID: Hi, What is the seed used? It is really important that you set the seed? Did you build Python and numarray from source? Can you reduce your code to a few lines that have the problem? Ranlib uses static floats so it may relate to numerical precision (as well as the random uniform random number generator). Really the only way is for you to debug the ranlib.c file Note that R provides a standalone math library in C that includes the random number generators so you could you those instead. SWIG handles it rather well. Regards Bruce On 7/13/05, Flavio Coelho wrote: > Well, > I am definetly glad that someone has also stumbled onto the same problem. > > But it is nevertheless intriguing, that you can run poisson a million times > with mean zero or negative(it assumes zero mean inthis case) without any > problems by itself. But when I call it within my code, the rate of error is > very high (I would say it returns a wrong result every time, but I haven't > checked every answer). > > Meanwhile, my solution will be: > > import rpy > > n = rpy.r.rpois(n,mean) > > I don't feel I can trust poisson while this "funny" behavior is still > there... > If someone has any Idea of how I could trace this bug please tell me, and > I'll hunt it down. At least I can reproduce it in a very specific context. > > thanks, > > Fl?vio > > 2005/7/12, Sebastian Haase : > > Hi Flavio! > > I had reported this long time ago and this list (about numarray). > > Somehow this got more or less dismissed. If I recall correctly the > argument > > was that nobody could reproduce it (I ran this on Debian Woody , py2.2, > (with > > CVS numarray at the time). > > > > I ended up writting my own wrapper(s): > > def poissonArr(shape=defshape, mean=1): > > from numarray import random_array as ra > > if mean == 0: > > return zeroArrF(shape) > > elif mean < 0: > > raise "poisson not defined for mean < 0" > > else: > > return ra.poisson(mean, shape).astype(na.UInt16) > > > > def poissonize(arr): > > from numarray import random_array as ra > > return na.where(arr<=0, 0, ra.poisson(arr)).astype(na.UInt16) > > > > (I use the astype( na.UInt16) because of some OpenGL code) > > > > Just last week had this problem on a windows98 computer (python2.4). > > > > This should get sorted out ... > > > > Thanks for reporting this problem. > > Sebastian Haase > > > > > > > > > > On Tuesday 12 July 2005 13:32, Flavio Coelho wrote: > > > Hi, > > > > > > I am having problems with the poisson random number generators of both > > > Numarray and Numeric. > > > I can't replicate it when calling the function from the python cosonle, > but > > > it has consistently malfunctioned when used within one of my scripts. > > > > > > What happens is that it frequently return a value greater than zero when > > > called with zero mean: poisson(0.0) > > > > > > Unfortunately My program is too big to send attached but I have > confirmed > > > the malfunction by printing both the mean and the result whenever it > spits > > > out a wrong result. > > > > > > This is very weird indeed, I have run poisson millions of times by itsel > on > > > the python console, without any problems... > > > > > > I hope it is some stupid mistake, but when I replace the poisson > function > > > call within my program by the R equivalent command (rpois) via the rpy > > > wrapper, everything works just fine... > > > > > > any Ideas? > > > > > > -- > > Fl?vio Code?o Coelho > registered Linux user # 386432 > --------------------------- > Great spirits have always found violent opposition from mediocrities. The > latter cannot understand it when a man does not thoughtlessly submit to > hereditary prejudices but honestly and courageously uses his intelligence. > Albert Einstein > From fccoelho at gmail.com Wed Jul 13 08:17:21 2005 From: fccoelho at gmail.com (Flavio Coelho) Date: Wed Jul 13 08:17:21 2005 Subject: [Numpy-discussion] problem with poisson generators In-Reply-To: References: <200507121447.01356.haase@msg.ucsf.edu> Message-ID: 2005/7/13, Bruce Southey : > > Hi, > What is the seed used? I am not setting the seed. It is really important that you set the seed? No. Did you build Python and numarray from source? Yes, I use Gentoo. I build everything from source. Can you reduce your code to a few lines that have the problem? Not really, poisson and binomial are the only two functions from Numeric that I use but they are called inside a rather complex object oriented code environment (objects within objetcs, being called recursively...) Bellow is the function within which poisson is called: def stepSEIR_s(self,inits=(0,0,0),par=(0.001,1,1,0.5,1,0,0,0),theta=0, npass=0,dist='poisson'): """ Defines an stochastic model SEIR: - inits = (E,I,S) - par = (Beta, alpha, E,r,delta,B,w,p) see docs. - theta = infectious individuals from neighbor sites """ E,I,S = inits N = self.parentSite.totpop beta,alpha,e,r,delta,B,w,p = par Lpos_esp = beta*S*((I+theta)/(N+npass))**alpha #Number of new cases if dist == 'poisson': Lpos = poisson(Lpos_esp) ## if theta == 0 and Lpos_esp == 0 and Lpos > 0: ## print Lpos,Lpos_esp,S,I,theta,N,self.parentSite.sitename elif dist == 'negbin': prob = I/(I+Lpos_esp) #convertin between parameterizations Lpos = negative_binomial(I,prob) self.parentSite.totalcases += Lpos #update number of cases Epos = (1-e)*E + Lpos Ipos = e*E + (1-r)*I Spos = S + B - Lpos Rpos = N-(Spos+Epos+Ipos) #self.parentSite.totpop = Spos+Epos+Ipos+Rpos self.parentSite.incidence.append(Lpos) if not self.parentSite.infected: if Lpos > 0: self.parentSite.infected = self.parentSite.parentGraph.simstep self.parentSite.parentGraph.epipath.append(( self.parentSite.parentGraph.simstep,self.parentSite,self.parentSite.infector )) return [Epos,Ipos,Spos] I wonder if called by itself it would trigger the problem... The commented Lines Is where I catch the errors: when poisson returns a greater than zero number, when called with mean 0. Ranlib uses static floats so it may relate to numerical precision (as > well as the random uniform random number generator). Really the only > way is for you to debug the ranlib.c file I'll see what I can do. Note that R provides a standalone math library in C that includes the > random number generators so you could you those instead. SWIG handles > it rather well. I think thats what Rpy already does...is it not? thanks Bruce, Fl?vio Regards > Bruce > > On 7/13/05, Flavio Coelho wrote: > > Well, > > I am definetly glad that someone has also stumbled onto the same > problem. > > > > But it is nevertheless intriguing, that you can run poisson a million > times > > with mean zero or negative(it assumes zero mean inthis case) without any > > problems by itself. But when I call it within my code, the rate of error > is > > very high (I would say it returns a wrong result every time, but I > haven't > > checked every answer). > > > > Meanwhile, my solution will be: > > > > import rpy > > > > n = rpy.r.rpois(n,mean) > > > > I don't feel I can trust poisson while this "funny" behavior is still > > there... > > If someone has any Idea of how I could trace this bug please tell me, > and > > I'll hunt it down. At least I can reproduce it in a very specific > context. > > > > thanks, > > > > Fl?vio > > > > 2005/7/12, Sebastian Haase : > > > Hi Flavio! > > > I had reported this long time ago and this list (about numarray). > > > Somehow this got more or less dismissed. If I recall correctly the > > argument > > > was that nobody could reproduce it (I ran this on Debian Woody , py2.2 > , > > (with > > > CVS numarray at the time). > > > > > > I ended up writting my own wrapper(s): > > > def poissonArr(shape=defshape, mean=1): > > > from numarray import random_array as ra > > > if mean == 0: > > > return zeroArrF(shape) > > > elif mean < 0: > > > raise "poisson not defined for mean < 0" > > > else: > > > return ra.poisson(mean, shape).astype(na.UInt16) > > > > > > def poissonize(arr): > > > from numarray import random_array as ra > > > return na.where(arr<=0, 0, ra.poisson(arr)).astype(na.UInt16) > > > > > > (I use the astype( na.UInt16) because of some OpenGL code) > > > > > > Just last week had this problem on a windows98 computer (python2.4). > > > > > > This should get sorted out ... > > > > > > Thanks for reporting this problem. > > > Sebastian Haase > > > > > > > > > > > > > > > On Tuesday 12 July 2005 13:32, Flavio Coelho wrote: > > > > Hi, > > > > > > > > I am having problems with the poisson random number generators of > both > > > > Numarray and Numeric. > > > > I can't replicate it when calling the function from the python > cosonle, > > but > > > > it has consistently malfunctioned when used within one of my > scripts. > > > > > > > > What happens is that it frequently return a value greater than zero > when > > > > called with zero mean: poisson(0.0) > > > > > > > > Unfortunately My program is too big to send attached but I have > > confirmed > > > > the malfunction by printing both the mean and the result whenever it > > spits > > > > out a wrong result. > > > > > > > > This is very weird indeed, I have run poisson millions of times by > itsel > > on > > > > the python console, without any problems... > > > > > > > > I hope it is some stupid mistake, but when I replace the poisson > > function > > > > call within my program by the R equivalent command (rpois) via the > rpy > > > > wrapper, everything works just fine... > > > > > > > > any Ideas? > > > > > > > > > > > -- > > > > Fl?vio Code?o Coelho > > registered Linux user # 386432 > > --------------------------- > > Great spirits have always found violent opposition from mediocrities. > The > > latter cannot understand it when a man does not thoughtlessly submit to > > hereditary prejudices but honestly and courageously uses his > intelligence. > > Albert Einstein > > > -- Fl?vio Code?o Coelho registered Linux user # 386432 --------------------------- Great spirits have always found violent opposition from mediocrities. The latter cannot understand it when a man does not thoughtlessly submit to hereditary prejudices but honestly and courageously uses his intelligence. Albert Einstein -------------- next part -------------- An HTML attachment was scrubbed... URL: From cookedm at physics.mcmaster.ca Wed Jul 13 08:28:41 2005 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Wed Jul 13 08:28:41 2005 Subject: [Numpy-discussion] problem with poisson generators In-Reply-To: References: Message-ID: <20050713152444.GA31450@arbutus.physics.mcmaster.ca> On Tue, Jul 12, 2005 at 05:32:25PM -0300, Flavio Coelho wrote: > Hi, > > I am having problems with the poisson random number generators of both > Numarray and Numeric. > I can't replicate it when calling the function from the python cosonle, but > it has consistently malfunctioned when used within one of my scripts. > > What happens is that it frequently return a value greater than zero when > called with zero mean: poisson(0.0) > > Unfortunately My program is too big to send attached but I have confirmed > the malfunction by printing both the mean and the result whenever it spits > out a wrong result. > > This is very weird indeed, I have run poisson millions of times by itsel on > the python console, without any problems... > > I hope it is some stupid mistake, but when I replace the poisson function > call within my program by the R equivalent command (rpois) via the rpy > wrapper, everything works just fine... > > any Ideas? This looks like bug #1123145 in Numeric: http://sourceforge.net/tracker/index.php?func=detail&aid=1123145&group_id=1369&atid=101369 which was fixed a few months ago. numarray, I believe, originally took ranlib.c from Numeric, so it doesn't have this bug fix. Try replacing numarray's ranlib.c with the version from Numeric 24.0b2 (or CVS). -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From rowen at cesmail.net Wed Jul 13 10:15:40 2005 From: rowen at cesmail.net (Russell E. Owen) Date: Wed Jul 13 10:15:40 2005 Subject: [Numpy-discussion] numarray 1.3.3 build question Message-ID: I'm on MacOS X 10.3.9 and am getting mysterious warnings when building numarray 1.3.3 which I've never seen before. Building for the unix/x11 version yields: [dhcp-254:~/Archives\342\200\242/PythonPackages/numarray-1.3.3] rowen% python setup.py config build --gencode Running sitecustomize.pyc Using EXTRA_COMPILE_ARGS = ['-Ddarwin'] Running sitecustomize.pyc Running sitecustomize.pyc generating new API module 'libnumarray' .c & .h generating new API module 'libteacup' .c & .h generating new API module 'libnumeric' .c & .h Using external BLAS and LAPACK running config Wrote config.h ... Packages/LinearAlgebra2/Src/lapack_litemodule.c:851: warning: function declaration isn't a prototype Packages/LinearAlgebra2/Src/lapack_litemodule.c:848: warning: `lapack_liteError' defined but not used gcc: -framework: linker input file unused because linking not done gcc: vecLib: linker input file unused because linking not done Should I worry? I haven't figured out how to run the self-tests w/out installing and I'm reluctant to install w/out knowing if the warnings are a problem. also: - why do we need --gencode? Is there some way to make that the default so we can leave it off? - why do we need "config"? That's new to 1.3.3 isn't it? -- Russell P.S. I have two completely different versions of python installed: - the normal built-in Python 2.3.0, which I run using 'pythonw' - a pure unix/x11 python that I installed from source using an x11 version of tcl/tk. This is run using "python" (which runs /usr/local/python). However, the errors occur while building with either version (much to my surprised -- I expected the framework complain to go away when building with pythonw, the framework python). From perry at stsci.edu Wed Jul 13 10:53:11 2005 From: perry at stsci.edu (Perry Greenfield) Date: Wed Jul 13 10:53:11 2005 Subject: [Numpy-discussion] problem with poisson generators In-Reply-To: <20050713152444.GA31450@arbutus.physics.mcmaster.ca> References: <20050713152444.GA31450@arbutus.physics.mcmaster.ca> Message-ID: <0a6666184dc9fa774a8a6dcecb6a771c@stsci.edu> On Jul 13, 2005, at 11:24 AM, David M. Cooke wrote: > > This looks like bug #1123145 in Numeric: > > http://sourceforge.net/tracker/index.php? > func=detail&aid=1123145&group_id=1369&atid=101369 > > which was fixed a few months ago. numarray, I believe, originally took > ranlib.c from Numeric, so it doesn't have this bug fix. Try replacing > numarray's ranlib.c with the version from Numeric 24.0b2 (or CVS). > Thanks for noticing this. By the way, Todd is away this week so numarray issues won't likely be responded to this week by him. Perry From fccoelho at gmail.com Thu Jul 14 00:19:26 2005 From: fccoelho at gmail.com (Flavio Coelho) Date: Thu Jul 14 00:19:26 2005 Subject: [Numpy-discussion] problem with poisson generators In-Reply-To: <20050713152444.GA31450@arbutus.physics.mcmaster.ca> References: <20050713152444.GA31450@arbutus.physics.mcmaster.ca> Message-ID: 2005/7/13, David M. Cooke : > > On Tue, Jul 12, 2005 at 05:32:25PM -0300, Flavio Coelho wrote: > > Hi, > > > > I am having problems with the poisson random number generators of both > > Numarray and Numeric. > > I can't replicate it when calling the function from the python cosonle, > but > > it has consistently malfunctioned when used within one of my scripts. > > > > What happens is that it frequently return a value greater than zero when > > called with zero mean: poisson(0.0) > > > > Unfortunately My program is too big to send attached but I have > confirmed > > the malfunction by printing both the mean and the result whenever it > spits > > out a wrong result. > > > > This is very weird indeed, I have run poisson millions of times by itsel > on > > the python console, without any problems... > > > > I hope it is some stupid mistake, but when I replace the poisson > function > > call within my program by the R equivalent command (rpois) via the rpy > > wrapper, everything works just fine... > > > > any Ideas? > > This looks like bug #1123145 in Numeric: > > > http://sourceforge.net/tracker/index.php?func=detail&aid=1123145&group_id=1369&atid=101369 > > which was fixed a few months ago. numarray, I believe, originally took > ranlib.c from Numeric, so it doesn't have this bug fix. Try replacing > numarray's ranlib.c with the version from Numeric 24.0b2 (or CVS). I have both numeric 23.7 and numarray 1.3.1 installed and on neither of them I could reproduce the bug when I called them directly from the python interpreter. However they fail on mean 0.0 every time when called within my code. So it appears to me that the Bug you mentioned is not what is causing my problem. It seems to stem from interaction with the way its being called, maybe some carryover from previous calls... this is the test I ran on the interpreter: [(poisson(i),i) for i in uniform(-20,20,1000) if i<=0] I also ran: sum(poisson(0,100000)) they both worked flawlessly. In the first test I wanted to see if there was some carry over from previous runs when called with various means (which is closer to the way it is used within my code), but it returned zero every time. (I don't use negative means in my code, but I wanted to try it here just in case) -- > |>|\/|< > > /--------------------------------------------------------------------------\ > |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ > |cookedm at physics.mcmaster.ca > -- Fl?vio Code?o Coelho registered Linux user # 386432 --------------------------- Great spirits have always found violent opposition from mediocrities. The latter cannot understand it when a man does not thoughtlessly submit to hereditary prejudices but honestly and courageously uses his intelligence. Albert Einstein -------------- next part -------------- An HTML attachment was scrubbed... URL: From rkern at ucsd.edu Thu Jul 14 00:28:06 2005 From: rkern at ucsd.edu (Robert Kern) Date: Thu Jul 14 00:28:06 2005 Subject: [Numpy-discussion] problem with poisson generators In-Reply-To: References: <20050713152444.GA31450@arbutus.physics.mcmaster.ca> Message-ID: <42D61349.5060700@ucsd.edu> Flavio Coelho wrote: > I have both numeric 23.7 and numarray 1.3.1 installed and on neither > of them I could reproduce the bug when I called them directly from the > python interpreter. However they fail on mean 0.0 every time when called > within my code. So it appears to me that the Bug you mentioned is not > what is causing my problem. It seems to stem from interaction with the > way its being called, maybe some carryover from previous calls... > this is the test I ran on the interpreter: > [(poisson(i),i) for i in uniform(-20,20,1000) if i<=0] > > I also ran: > > sum(poisson(0,100000)) > > they both worked flawlessly. In the first test I wanted to see if there > was some carry over from previous runs when called with various means > (which is closer to the way it is used within my code), but it returned > zero every time. (I don't use negative means in my code, but I wanted to > try it here just in case) Are you sure that you don't have other versions of Numeric or numarray installed that might be getting picked up by your code? -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From jmiller at stsci.edu Thu Jul 14 02:44:59 2005 From: jmiller at stsci.edu (Todd Miller) Date: Thu Jul 14 02:44:59 2005 Subject: [Numpy-discussion] bug or feature? In-Reply-To: <23974d4bfbd55596f8212bbfd6ff6723@stsci.edu> References: <5deb92a4eb50a0ed681dbf3e8d8e3a50@stsci.edu> <23974d4bfbd55596f8212bbfd6ff6723@stsci.edu> Message-ID: <1121334126.4697.5.camel@localhost.localdomain> On Mon, 2005-07-11 at 15:22 -0400, Perry Greenfield wrote: > For array indexing, assignment does > modify the original array (as your case above illustrates), but once > you sliced the array-indexed array, it changed the context of the > indexing. I.e., (to take a simpler, 1-d array x) > > x[[0,2]] = 1 # modifies x > > This form results in __setitem__ being called on x > > x[[0,2]][:4] = 1 # doesn't modify x > > This form results in __getitem__ being called on x (and thus producing > a new copy of the array) for the array indexing and __setitem__ being > called for the slice. At least that's what I think is happening. Todd > or someone more recently familiar with how Python handles this can > correct me if I'm wrong. > > Perry > This is indeed what happens. A temporary is created by the first sub- expression x[[0,2]], modified by the slice subexpression [:4] = 1, and then discarded. I think the array indexing "grammar" could be changed to support x[[0,2],:4] as a single expression which would create the temporary and then copy-back at the end... but numarray's array indexing is not currently that fancy. Todd From jmiller at stsci.edu Thu Jul 14 03:13:10 2005 From: jmiller at stsci.edu (Todd Miller) Date: Thu Jul 14 03:13:10 2005 Subject: [Numpy-discussion] numarray 1.3.3 build question In-Reply-To: References: Message-ID: <1121335822.4697.25.camel@localhost.localdomain> On Wed, 2005-07-13 at 10:09 -0700, Russell E. Owen wrote: > I'm on MacOS X 10.3.9 and am getting mysterious warnings when building > numarray 1.3.3 which I've never seen before. > > Building for the unix/x11 version yields: > [dhcp-254:~/Archives\342\200\242/PythonPackages/numarray-1.3.3] rowen% > python setup.py config build --gencode > Running sitecustomize.pyc > Using EXTRA_COMPILE_ARGS = ['-Ddarwin'] > Running sitecustomize.pyc > Running sitecustomize.pyc > generating new API module 'libnumarray' .c & .h > generating new API module 'libteacup' .c & .h > generating new API module 'libnumeric' .c & .h > Using external BLAS and LAPACK > running config > Wrote config.h > ... > Packages/LinearAlgebra2/Src/lapack_litemodule.c:851: warning: function > declaration isn't a prototype > Packages/LinearAlgebra2/Src/lapack_litemodule.c:848: warning: > `lapack_liteError' defined but not used > gcc: -framework: linker input file unused because linking not done > gcc: vecLib: linker input file unused because linking not done > > Should I worry? > I haven't figured out how to run the self-tests w/out > installing and I'm reluctant to install w/out knowing if the warnings > are a problem. > > also: > - why do we need --gencode? You don't need it, I do. In general, it's not necessary to use but it's there because the code generation needs to remain "live" in order to be an asset but is also time consuming for people doing repetitive installs to fix problems. Prior to Python-2.4 and VC.NET, --gen_code had meaning on win32 because VC6.0 didn't fully support UInt64. Everywhere else, the CVS'ed C source just works. > Is there some way to make that the default > so we can leave it off? Sure. I would say just don't worry about it unless you're building on win32 from source using VC6.0. > - why do we need "config"? That's new to 1.3.3 isn't it? No, I think that's been there since 1.2.1 or so. It's an artifact of user submitted fixes to the linear algebra package which I accepted as- is because they were submitted by someone who both knew enough technically and had enough idealism to make the effort. Regards, Todd From fccoelho at gmail.com Thu Jul 14 07:12:35 2005 From: fccoelho at gmail.com (Flavio Coelho) Date: Thu Jul 14 07:12:35 2005 Subject: [Numpy-discussion] problem with poisson generators In-Reply-To: <42D61349.5060700@ucsd.edu> References: <20050713152444.GA31450@arbutus.physics.mcmaster.ca> <42D61349.5060700@ucsd.edu> Message-ID: 2005/7/14, Robert Kern : > > Flavio Coelho wrote: > > > I have both numeric 23.7 and numarray 1.3.1 installed and on neither > > of them I could reproduce the bug when I called them directly from the > > python interpreter. However they fail on mean 0.0 every time when called > > within my code. So it appears to me that the Bug you mentioned is not > > what is causing my problem. It seems to stem from interaction with the > > way its being called, maybe some carryover from previous calls... > > this is the test I ran on the interpreter: > > [(poisson(i),i) for i in uniform(-20,20,1000) if i<=0] > > > > I also ran: > > > > sum(poisson(0,100000)) > > > > they both worked flawlessly. In the first test I wanted to see if there > > was some carry over from previous runs when called with various means > > (which is closer to the way it is used within my code), but it returned > > zero every time. (I don't use negative means in my code, but I wanted to > > try it here just in case) > > Are you sure that you don't have other versions of Numeric or numarray > installed that might be getting picked up by your code? I have just double checked. And the answer is no, there is a single version of each of the modules. -- > Robert Kern > rkern at ucsd.edu > > "In the fields of hell where the grass grows high > Are the graves of dreams allowed to die." > -- Richard Harter > > > ------------------------------------------------------- > This SF.Net email is sponsored by the 'Do More With Dual!' > webinar happening > July 14 at 8am PDT/11am EDT. We invite you to explore the latest in dual > core and dual graphics technology at this free one hour event hosted by > HP, > AMD, and NVIDIA. To register visit http://www.hp.com/go/dualwebinar > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -- Fl?vio Code?o Coelho registered Linux user # 386432 --------------------------- Great spirits have always found violent opposition from mediocrities. The latter cannot understand it when a man does not thoughtlessly submit to hereditary prejudices but honestly and courageously uses his intelligence. Albert Einstein -------------- next part -------------- An HTML attachment was scrubbed... URL: From jmiller at stsci.edu Fri Jul 15 03:57:02 2005 From: jmiller at stsci.edu (Todd Miller) Date: Fri Jul 15 03:57:02 2005 Subject: [Numpy-discussion] problem with poisson generators In-Reply-To: References: <200507121447.01356.haase@msg.ucsf.edu> Message-ID: <1121424865.4707.16.camel@localhost.localdomain> On Wed, 2005-07-13 at 12:13 -0300, Flavio Coelho wrote: > > > 2005/7/13, Bruce Southey : > Hi, > What is the seed used? > > I am not setting the seed. > > > It is really important that you set the seed? > > No. > > > Did you build Python and numarray from source? > > > Yes, I use Gentoo. I build everything from source. > > > Can you reduce your code to a few lines that have the problem? > > > Not really, poisson and binomial are the only two functions from > Numeric that I use but they are called inside a rather complex object > oriented code environment (objects within objetcs, being called > recursively...) Bellow is the function within which poisson is called: > > def stepSEIR_s(self,inits=(0,0,0),par=(0.001,1,1,0.5,1,0,0,0),theta=0, > npass=0,dist='poisson'): > """ > Defines an stochastic model SEIR: > - inits = (E,I,S) > - par = (Beta, alpha, E,r,delta,B,w,p) see docs. > - theta = infectious individuals from neighbor sites > """ > E,I,S = inits > N = self.parentSite.totpop > beta,alpha,e,r,delta,B,w,p = par > Lpos_esp = beta*S*((I+theta)/(N+npass))**alpha #Number of new > cases > > if dist == 'poisson': > Lpos = poisson(Lpos_esp) > ## if theta == 0 and Lpos_esp == 0 and Lpos > 0: > ## print > Lpos,Lpos_esp,S,I,theta,N,self.parentSite.sitename > elif dist == 'negbin': > prob = I/(I+Lpos_esp) #convertin between parameterizations > Lpos = negative_binomial(I,prob) > self.parentSite.totalcases += Lpos #update number of cases > Epos = (1-e)*E + Lpos > Ipos = e*E + (1-r)*I > Spos = S + B - Lpos > Rpos = N-(Spos+Epos+Ipos) > #self.parentSite.totpop = Spos+Epos+Ipos+Rpos > self.parentSite.incidence.append(Lpos) > if not self.parentSite.infected: > if Lpos > 0: > self.parentSite.infected = > self.parentSite.parentGraph.simstep > self.parentSite.parentGraph.epipath.append > ((self.parentSite.parentGraph.simstep,self.parentSite,self.parentSite.infector)) > > > return [Epos,Ipos,Spos] > Having this code is a good start to solving the problem. I think the next step is to simplify your example to make it runnable and provide known inputs for all the parameters which lead to a failure for you. Being really literal (spelling out every single damn thing) cuts down on speculation and inefficiency on our end. It would also be good to express the condition (or it's inverse) you think is an error as an assertion, so something like this might be what we need: from numarray.random_array import poisson E, I, S = (0,0,0) beta,alpha,e,r,delta,B,w,p = (0.001,1,1,0.5,1,0,0,0) theta = 0 npass = 0 N = 100 # total guess here for me Lpos_esp = beta*S*((I+theta)/(N+npass))**alpha #Number of new cases Lpos = poisson(Lpos_esp) assert not (theta == 0 and Lpos_esp == 0 and Lpos > 0) The problem is, the above works for me. Make it sane and get it to expose the error for you and I'll look into this some more. Regards, Todd > > I wonder if called by itself it would trigger the problem... The > commented Lines Is where I catch the errors: when poisson returns a > greater than zero number, when called with mean 0. > > > > Ranlib uses static floats so it may relate to numerical > precision (as > well as the random uniform random number generator). Really > the only > way is for you to debug the ranlib.c file > > I'll see what I can do. > > > Note that R provides a standalone math library in C that > includes the > random number generators so you could you those instead. SWIG > handles > it rather well. > > > I think thats what Rpy already does...is it not? > > thanks Bruce, > > Fl?vio > > > Regards > Bruce > > On 7/13/05, Flavio Coelho wrote: > > Well, > > I am definetly glad that someone has also stumbled onto > the same problem. > > > > But it is nevertheless intriguing, that you can run poisson > a million times > > with mean zero or negative(it assumes zero mean inthis > case) without any > > problems by itself. But when I call it within my code, the > rate of error is > > very high (I would say it returns a wrong result every time, > but I haven't > > checked every answer). > > > > Meanwhile, my solution will be: > > > > import rpy > > > > n = rpy.r.rpois(n,mean) > > > > I don't feel I can trust poisson while this "funny" > behavior is still > > there... > > If someone has any Idea of how I could trace this bug > please tell me, and > > I'll hunt it down. At least I can reproduce it in a very > specific context. > > > > thanks, > > > > Fl?vio > > > > 2005/7/12, Sebastian Haase : > > > Hi Flavio! > > > I had reported this long time ago and this list (about > numarray). > > > Somehow this got more or less dismissed. If I recall > correctly the > > argument > > > was that nobody could reproduce it (I ran this on Debian > Woody , py2.2, > > (with > > > CVS numarray at the time). > > > > > > I ended up writting my own wrapper(s): > > > def poissonArr(shape=defshape, mean=1): > > > from numarray import random_array as ra > > > if mean == 0: > > > return zeroArrF(shape) > > > elif mean < 0: > > > raise "poisson not defined for mean < 0" > > > else: > > > return ra.poisson(mean, shape).astype > (na.UInt16) > > > > > > def poissonize(arr): > > > from numarray import random_array as ra > > > return na.where(arr<=0, 0, ra.poisson(arr)).astype > ( na.UInt16) > > > > > > (I use the astype( na.UInt16) because of some OpenGL code) > > > > > > Just last week had this problem on a windows98 computer > (python2.4). > > > > > > This should get sorted out ... > > > > > > Thanks for reporting this problem. > > > Sebastian Haase > > > > > > > > > > > > > > > On Tuesday 12 July 2005 13:32, Flavio Coelho wrote: > > > > Hi, > > > > > > > > I am having problems with the poisson random number > generators of both > > > > Numarray and Numeric. > > > > I can't replicate it when calling the function from the > python cosonle, > > but > > > > it has consistently malfunctioned when used within one > of my scripts. > > > > > > > > What happens is that it frequently return a value > greater than zero when > > > > called with zero mean: poisson( 0.0) > > > > > > > > Unfortunately My program is too big to send attached but > I have > > confirmed > > > > the malfunction by printing both the mean and the result > whenever it > > spits > > > > out a wrong result. > > > > > > > > This is very weird indeed, I have run poisson millions > of times by itsel > > on > > > > the python console, without any problems... > > > > > > > > I hope it is some stupid mistake, but when I replace the > poisson > > function > > > > call within my program by the R equivalent command > (rpois) via the rpy > > > > wrapper, everything works just fine... > > > > > > > > any Ideas? > > > > > > > > > > > -- > > > > Fl?vio Code?o Coelho > > registered Linux user # 386432 > > --------------------------- > > Great spirits have always found violent opposition from > mediocrities. The > > latter cannot understand it when a man does not > thoughtlessly submit to > > hereditary prejudices but honestly and courageously uses his > intelligence. > > Albert Einstein > > > > > > -- > Fl?vio Code?o Coelho > registered Linux user # 386432 > --------------------------- > Great spirits have always found violent opposition from mediocrities. > The > latter cannot understand it when a man does not thoughtlessly submit > to > hereditary prejudices but honestly and courageously uses his > intelligence. > Albert Einstein From faltet at carabos.com Fri Jul 15 04:01:00 2005 From: faltet at carabos.com (Francesc Altet) Date: Fri Jul 15 04:01:00 2005 Subject: [Numpy-discussion] ANN: PyTables 1.1 released Message-ID: <200507151258.28250.faltet@carabos.com> ========================= Announcing PyTables 1.1 ========================= The PyTables development team is happy to announce the availability of a new version of PyTables package. On this version you will find support for a nice set of new features, like nested datatypes, enumerated datatypes, nested iterators (for reading only), support for native HDF5 multidimensional attributes, a new object for dealing with compressed, non-enlargeable arrays (CArray), bzip2 compression support and more. Many bugs has been addressed as well. Go to the PyTables web site for downloading the beast: http://pytables.sourceforge.net/ Or keep reading for more info about the new features and bugs fixed. Changes more in depth ===================== Improvements: - Support for nested datatypes is in place. You can now made columns of tables that host another columns for an unlimited depth (well, theoretically, in practice until the python recursive limit would be reached). Convenient NestedRecArray objects has been implemented as data containers. Cols and Description accessors has been improved so you can navigate on the type hierarchy very easily (natural naming is has been implemented for the task). - ``Table``, ``EArray`` and ``VLArray`` objects now support enumerated types. ``Array`` objects support opening existing HDF5 enumerated arrays. Enumerated types are restricted sets of ``(name, value)`` pairs. Use the ``Enum`` class to easily define new enumerations that will be saved along with your data. - Now, the HDF5 library is responsible to do data conversions when the datasets are written in a machine with different byte-ordering than the machine that reads the dataset. With this, all the data is converted on-the-fly and you always get native datatypes in memory. I think this approach to be more convenient in terms of CPU consumption when using these datasets. Right now, this only works for tables, though. - Added support for native HDF5 multidimensional attributes. Now, you can load native HDF5 files that contains fully multidimensional attributes; these attributes will be mapped to NumArray objects. Also, when you save NumArray objects as attributes, they get saved as native HDF5 attributes (before, NumArray attributes where pickled). - A brand-new class, called CArray, has been introduced. It's mainly like an Array class (i.e. non-enlargeable), but with compression capabilities enabled. The existence of CArray also allows PyTables to read native HDF5 chunked, non-enlargeable datasets. - Bzip2 compressor is supported. Such a support was already in PyTables 1.0, but forgot to announce it. - New LZO2 (http://www.oberhumer.com/opensource/lzo/lzonews.php) compressor is supported. The installer now recognizes whether LZO1 or LZO2 is installed, and adapts automatically to it. If both are installed in your system, then LZO2 is chosen. LZO2 claims to be fully compatible (both backward and forward) with LZO1, so you should not experience any problem during this transition. - The old limit of 256 columns in a table has been released. Now, you can have tables with any number of columns, although if you try to use a too high number (i.e. > 1024), you will start to consume a lot of system resources. You have been warned!. - The limit in the length of column names has been released also. - Nested iterators for reading in tables are supported now. - A new section in tutorial about how to modify values in tables and arrays has been added to the User's Manual. Backward-incompatible changes: - None. Bug fixes: - VLArray now correctly updates the number of rows internal counter when opening an existing VLArray object. Now you can add new rows to existing VLA's without problems. - Tuple flavor for VLArrays now works as intended, i.e. reading VLArray objects will always return tuples even in the case of multidimensional Atoms. Before, this operations returned a mix of tuples and lists. - If a column was not able to be indexed because it has too few entries, then _whereInRange is called instead of _whereIndexed. Fixes #1203202. - You can call now Row.append() in the middle of Table iterators without resetting loop counters. Fixes #1205588. - PyTables used to give a segmentation fault when removing the last row out of a table with the table.removeRows() method. This is due to a limitation in the HDF5 library. Until this get fixed in HDF5, a NotImplemented error is raised when trying to do that. Address #1201023. - You can safely break a loop over an iterator returned by Table.where(). Fixes #1234637. - When removing a Group with hidden child groups, those are effectively closed now. - Now, there is a distinction between shapes 1 and (1,) in tables. The former represents a scalar, and the later a 1-D array with just one element. That follows the numarray convention for records, and makes more sense as well. Before 1.1, shapes 1 and (1,) were represented by an scalar on disk. Known bugs: - Classes inheriting from IsDescription subclasses do not inherit columns defined in the super-class. See SF bug #1207732 for more info. - Time datatypes are non-portable between big-endian and little-endian architectures. This is ultimately a consequence of a HDF5 limitation. See SF bug #1234709 for more info. Known issues: - UCL compressor seems to work badly on MacOSX platforms. Until the problem would be isolated and eventually solved, UCL will not be compiled by default on MacOSX platforms, even if the installer finds it in the system. However, if you still want to get UCL support on MacOSX, you can use the --force-ucl flag in setup.py. Important note for Python 2.4 and Windows users =============================================== If you are willing to use PyTables with Python 2.4 in Windows platforms, you will need to get the HDF5 library compiled for MSVC 7.1, aka .NET 2003. It can be found at: ftp://ftp.ncsa.uiuc.edu/HDF/HDF5/current/bin/windows/5-164-win-net.ZIP Users of Python 2.3 on Windows will have to download the version of HDF5 compiled with MSVC 6.0 available in: ftp://ftp.ncsa.uiuc.edu/HDF/HDF5/current/bin/windows/5-164-win.ZIP What it is ========== **PyTables** is a package for managing hierarchical datasets and designed to efficiently cope with extremely large amounts of data (with support for full 64-bit file addressing). It features an object-oriented interface that, combined with C extensions for the performance-critical parts of the code, makes it a very easy-to-use tool for high performance data storage and retrieval. Perhaps its more interesting feature is that it optimizes memory and disk resources so that data take much less space (between a factor 3 to 5, and more if the data is compressible) than other solutions, like for example, relational or object oriented databases. Besides, PyTables I/O for table objects is buffered, implemented in C and carefully tuned so that you can reach much better performance with PyTables than with your own home-grown wrappings to the HDF5 library. PyTables sports indexing capabilities as well, allowing doing selections in tables exceeding one billion of rows in just seconds. Where can PyTables be applied? ============================== PyTables is not designed to work as a relational database competitor, but rather as a teammate. If you want to work with large datasets of multidimensional data (for example, for multidimensional analysis), or just provide a categorized structure for some portions of your cluttered RDBS, then give PyTables a try. It works well for storing data from data acquisition systems (DAS), simulation software, network data monitoring systems (for example, traffic measurements of IP packets on routers), very large XML files, or for creating a centralized repository for system logs, to name only a few possible uses. What is a table? ================ A table is defined as a collection of records whose values are stored in fixed-length fields. All records have the same structure and all values in each field have the same data type. The terms "fixed-length" and "strict data types" seem to be quite a strange requirement for a language like Python that supports dynamic data types, but they serve a useful function if the goal is to save very large quantities of data (such as is generated by many scientific applications, for example) in an efficient manner that reduces demand on CPU time and I/O resources. What is HDF5? ============= For those people who know nothing about HDF5, it is a general purpose library and file format for storing scientific data made at NCSA. HDF5 can store two primary objects: datasets and groups. A dataset is essentially a multidimensional array of data elements, and a group is a structure for organizing objects in an HDF5 file. Using these two basic constructs, one can create and store almost any kind of scientific data structure, such as images, arrays of vectors, and structured and unstructured grids. You can also mix and match them in HDF5 files according to your needs. Platforms ========= We are using Linux on top of Intel32 as the main development platform, but PyTables should be easy to compile/install on other UNIX machines. This package has also been successfully compiled and tested on a FreeBSD 5.4 with Opteron64 processors, a UltraSparc platform with Solaris 7 and Solaris 8, a SGI Origin3000 with Itanium processors running IRIX 6.5 (using the gcc compiler), Microsoft Windows and MacOSX (10.2 although 10.3 should work fine as well). In particular, it has been thoroughly tested on 64-bit platforms, like Linux-64 on top of an Intel Itanium, AMD Opteron (in 64-bit mode) or PowerPC G5 (in 64-bit mode) where all the tests pass successfully. Regarding Windows platforms, PyTables has been tested with Windows 2000 and Windows XP (using the Microsoft Visual C compiler), but it should also work with other flavors as well. Web site ======== Go to the PyTables web site for more details: http://pytables.sourceforge.net/ To know more about the company behind the PyTables development, see: http://www.carabos.com/ Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. ---- **Enjoy data!** -- The PyTables Team From Sebastien.deMentendeHorne at electrabel.com Fri Jul 15 04:13:04 2005 From: Sebastien.deMentendeHorne at electrabel.com (Sebastien.deMentendeHorne at electrabel.com) Date: Fri Jul 15 04:13:04 2005 Subject: [Numpy-discussion] problem with poisson generators Message-ID: <6E48F3D185CF644788F55917A0D50A93017CF4A3@seebex02.eib.electrabel.be> Hi Flavio, Could you give us every call to poisson or negative_binomial (ie all functions related to random numbers) preceeded with the seed ? Adding before your declaration of stepSEIR_s code like: randoutput = file("randoutput.py", "w") old_poisson = poisson def poisson(m): print >> randoutput, "print get_seed(), poisson(%s)"%m result = old_poisson(m) print >> randoutput, "# result = %s"%result return result old_negative_binomial = negative_binomial def negative_binomial(i,p): print >> randoutput, "print get_seed(), negative_binomial(%s,%s)"%(i,p) result = old_negative_binomial(i,p) print >> randoutput, "# result = %s"%result return result will ouput every call. We can then easily try on our machines what it gives. Best, Sebastien > -----Original Message----- > From: numpy-discussion-admin at lists.sourceforge.net > [mailto:numpy-discussion-admin at lists.sourceforge.net]On Behalf Of Todd > Miller > Sent: vendredi 15 juillet 2005 12:54 > To: Flavio Coelho > Cc: Bruce Southey; Sebastian Haase; numpy-discussion > Subject: Re: [Numpy-discussion] problem with poisson generators > > > On Wed, 2005-07-13 at 12:13 -0300, Flavio Coelho wrote: > > > > > > 2005/7/13, Bruce Southey : > > Hi, > > What is the seed used? > > > > I am not setting the seed. > > > > > > It is really important that you set the seed? > > > > No. > > > > > > Did you build Python and numarray from source? > > > > > > Yes, I use Gentoo. I build everything from source. > > > > > > Can you reduce your code to a few lines that have > the problem? > > > > > > Not really, poisson and binomial are the only two functions from > > Numeric that I use but they are called inside a rather > complex object > > oriented code environment (objects within objetcs, being called > > recursively...) Bellow is the function within which poisson > is called: > > > > def > stepSEIR_s(self,inits=(0,0,0),par=(0.001,1,1,0.5,1,0,0,0),theta=0, > > npass=0,dist='poisson'): > > """ > > Defines an stochastic model SEIR: > > - inits = (E,I,S) > > - par = (Beta, alpha, E,r,delta,B,w,p) see docs. > > - theta = infectious individuals from neighbor sites > > """ > > E,I,S = inits > > N = self.parentSite.totpop > > beta,alpha,e,r,delta,B,w,p = par > > Lpos_esp = beta*S*((I+theta)/(N+npass))**alpha > #Number of new > > cases > > > > if dist == 'poisson': > > Lpos = poisson(Lpos_esp) > > ## if theta == 0 and Lpos_esp == 0 and Lpos > 0: > > ## print > > Lpos,Lpos_esp,S,I,theta,N,self.parentSite.sitename > > elif dist == 'negbin': > > prob = I/(I+Lpos_esp) #convertin between > parameterizations > > Lpos = negative_binomial(I,prob) > > self.parentSite.totalcases += Lpos #update number > of cases > > Epos = (1-e)*E + Lpos > > Ipos = e*E + (1-r)*I > > Spos = S + B - Lpos > > Rpos = N-(Spos+Epos+Ipos) > > #self.parentSite.totpop = Spos+Epos+Ipos+Rpos > > self.parentSite.incidence.append(Lpos) > > if not self.parentSite.infected: > > if Lpos > 0: > > self.parentSite.infected = > > self.parentSite.parentGraph.simstep > > self.parentSite.parentGraph.epipath.append > > > ((self.parentSite.parentGraph.simstep,self.parentSite,self.par > entSite.infector)) > > > > > > return [Epos,Ipos,Spos] > > > > Having this code is a good start to solving the problem. I think the > next step is to simplify your example to make it runnable and provide > known inputs for all the parameters which lead to a failure for you. > > Being really literal (spelling out every single damn thing) > cuts down on > speculation and inefficiency on our end. > > It would also be good to express the condition (or it's inverse) you > think is an error as an assertion, so something like this > might be what > we need: > > from numarray.random_array import poisson > > E, I, S = (0,0,0) > beta,alpha,e,r,delta,B,w,p = (0.001,1,1,0.5,1,0,0,0) > theta = 0 > npass = 0 > N = 100 # total guess here for me > Lpos_esp = beta*S*((I+theta)/(N+npass))**alpha #Number of new cases > Lpos = poisson(Lpos_esp) > assert not (theta == 0 and Lpos_esp == 0 and Lpos > 0) > > The problem is, the above works for me. Make it sane and get it to > expose the error for you and I'll look into this some more. > > Regards, > Todd > > > > > I wonder if called by itself it would trigger the problem... The > > commented Lines Is where I catch the errors: when poisson returns a > > greater than zero number, when called with mean 0. > > > > > > > > Ranlib uses static floats so it may relate to numerical > > precision (as > > well as the random uniform random number generator). Really > > the only > > way is for you to debug the ranlib.c file > > > > I'll see what I can do. > > > > > > Note that R provides a standalone math library in C that > > includes the > > random number generators so you could you those > instead. SWIG > > handles > > it rather well. > > > > > > I think thats what Rpy already does...is it not? > > > > thanks Bruce, > > > > Fl?vio > > > > > > Regards > > Bruce > > > > On 7/13/05, Flavio Coelho wrote: > > > Well, > > > I am definetly glad that someone has also stumbled onto > > the same problem. > > > > > > But it is nevertheless intriguing, that you can > run poisson > > a million times > > > with mean zero or negative(it assumes zero mean inthis > > case) without any > > > problems by itself. But when I call it within my code, the > > rate of error is > > > very high (I would say it returns a wrong result > every time, > > but I haven't > > > checked every answer). > > > > > > Meanwhile, my solution will be: > > > > > > import rpy > > > > > > n = rpy.r.rpois(n,mean) > > > > > > I don't feel I can trust poisson while this "funny" > > behavior is still > > > there... > > > If someone has any Idea of how I could trace this bug > > please tell me, and > > > I'll hunt it down. At least I can reproduce it in a very > > specific context. > > > > > > thanks, > > > > > > Fl?vio > > > > > > 2005/7/12, Sebastian Haase : > > > > Hi Flavio! > > > > I had reported this long time ago and this list (about > > numarray). > > > > Somehow this got more or less dismissed. If I recall > > correctly the > > > argument > > > > was that nobody could reproduce it (I ran this on Debian > > Woody , py2.2, > > > (with > > > > CVS numarray at the time). > > > > > > > > I ended up writting my own wrapper(s): > > > > def poissonArr(shape=defshape, mean=1): > > > > from numarray import random_array as ra > > > > if mean == 0: > > > > return zeroArrF(shape) > > > > elif mean < 0: > > > > raise "poisson not defined for mean < 0" > > > > else: > > > > return ra.poisson(mean, shape).astype > > (na.UInt16) > > > > > > > > def poissonize(arr): > > > > from numarray import random_array as ra > > > > return na.where(arr<=0, 0, > ra.poisson(arr)).astype > > ( na.UInt16) > > > > > > > > (I use the astype( na.UInt16) because of some > OpenGL code) > > > > > > > > Just last week had this problem on a windows98 computer > > (python2.4). > > > > > > > > This should get sorted out ... > > > > > > > > Thanks for reporting this problem. > > > > Sebastian Haase > > > > > > > > > > > > > > > > > > > > On Tuesday 12 July 2005 13:32, Flavio Coelho wrote: > > > > > Hi, > > > > > > > > > > I am having problems with the poisson random number > > generators of both > > > > > Numarray and Numeric. > > > > > I can't replicate it when calling the > function from the > > python cosonle, > > > but > > > > > it has consistently malfunctioned when used within one > > of my scripts. > > > > > > > > > > What happens is that it frequently return a value > > greater than zero when > > > > > called with zero mean: poisson( 0.0) > > > > > > > > > > Unfortunately My program is too big to send > attached but > > I have > > > confirmed > > > > > the malfunction by printing both the mean and > the result > > whenever it > > > spits > > > > > out a wrong result. > > > > > > > > > > This is very weird indeed, I have run poisson millions > > of times by itsel > > > on > > > > > the python console, without any problems... > > > > > > > > > > I hope it is some stupid mistake, but when I > replace the > > poisson > > > function > > > > > call within my program by the R equivalent command > > (rpois) via the rpy > > > > > wrapper, everything works just fine... > > > > > > > > > > any Ideas? > > > > > > > > > > > > > > > > -- > > > > > > Fl?vio Code?o Coelho > > > registered Linux user # 386432 > > > --------------------------- > > > Great spirits have always found violent opposition from > > mediocrities. The > > > latter cannot understand it when a man does not > > thoughtlessly submit to > > > hereditary prejudices but honestly and > courageously uses his > > intelligence. > > > Albert Einstein > > > > > > > > > > > -- > > Fl?vio Code?o Coelho > > registered Linux user # 386432 > > --------------------------- > > Great spirits have always found violent opposition from > mediocrities. > > The > > latter cannot understand it when a man does not thoughtlessly submit > > to > > hereditary prejudices but honestly and courageously uses his > > intelligence. > > Albert Einstein > > > > ------------------------------------------------------- > SF.Net email is sponsored by: Discover Easy Linux Migration Strategies > from IBM. Find simple to follow Roadmaps, straightforward articles, > informative Webcasts and more! Get everything you need to get up to > speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > ======================================================= This message is confidential. It may also be privileged or otherwise protected by work product immunity or other legal rules. If you have received it by mistake please let us know by reply and then delete it from your system; you should not copy it or disclose its contents to anyone. All messages sent to and from Electrabel may be monitored to ensure compliance with internal policies and to protect our business. Emails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, lost or destroyed, or contain viruses. Anyone who communicates with us by email is taken to accept these risks. http://www.electrabel.be/homepage/general/disclaimer_EN.asp ======================================================= From fccoelho at gmail.com Fri Jul 15 17:41:29 2005 From: fccoelho at gmail.com (Flavio Coelho) Date: Fri Jul 15 17:41:29 2005 Subject: [Numpy-discussion] problem with poisson generators In-Reply-To: <1121424865.4707.16.camel@localhost.localdomain> References: <200507121447.01356.haase@msg.ucsf.edu> <1121424865.4707.16.camel@localhost.localdomain> Message-ID: 2005/7/15, Todd Miller : Todd, the function within which the problem happens runs fine by itself, as well as the simplified version you sent me. I am running the code on a AMD64 (though all my software and OS is compiled to 686 - Gentoo Linux). Is there any issue I should be aware of, regarding this specific cpu? I can try to run the software on another machine... Having this code is a good start to solving the problem. I think the > next step is to simplify your example to make it runnable and provide > known inputs for all the parameters which lead to a failure for you. > > Being really literal (spelling out every single damn thing) cuts down on > speculation and inefficiency on our end. > > It would also be good to express the condition (or it's inverse) you > think is an error as an assertion, so something like this might be what > we need: > > from numarray.random_array import poisson > > E, I, S = (0,0,0) > beta,alpha,e,r,delta,B,w,p = (0.001,1,1,0.5,1,0,0,0) > theta = 0 > npass = 0 > N = 100 # total guess here for me > Lpos_esp = beta*S*((I+theta)/(N+npass))**alpha #Number of new cases > Lpos = poisson(Lpos_esp) > assert not (theta == 0 and Lpos_esp == 0 and Lpos > 0) > > The problem is, the above works for me. Make it sane and get it to > expose the error for you and I'll look into this some more. > > Regards, > Todd > > > > > I wonder if called by itself it would trigger the problem... The > > commented Lines Is where I catch the errors: when poisson returns a > > greater than zero number, when called with mean 0. > > > > > > > > Ranlib uses static floats so it may relate to numerical > > precision (as > > well as the random uniform random number generator). Really > > the only > > way is for you to debug the ranlib.c file > > > > I'll see what I can do. > > > > > > Note that R provides a standalone math library in C that > > includes the > > random number generators so you could you those instead. SWIG > > handles > > it rather well. > > > > > > I think thats what Rpy already does...is it not? > > > > thanks Bruce, > > > > Fl?vio > > > > > > Regards > > Bruce > > > > On 7/13/05, Flavio Coelho wrote: > > > Well, > > > I am definetly glad that someone has also stumbled onto > > the same problem. > > > > > > But it is nevertheless intriguing, that you can run poisson > > a million times > > > with mean zero or negative(it assumes zero mean inthis > > case) without any > > > problems by itself. But when I call it within my code, the > > rate of error is > > > very high (I would say it returns a wrong result every time, > > but I haven't > > > checked every answer). > > > > > > Meanwhile, my solution will be: > > > > > > import rpy > > > > > > n = rpy.r.rpois(n,mean) > > > > > > I don't feel I can trust poisson while this "funny" > > behavior is still > > > there... > > > If someone has any Idea of how I could trace this bug > > please tell me, and > > > I'll hunt it down. At least I can reproduce it in a very > > specific context. > > > > > > thanks, > > > > > > Fl?vio > > > > > > 2005/7/12, Sebastian Haase : > > > > Hi Flavio! > > > > I had reported this long time ago and this list (about > > numarray). > > > > Somehow this got more or less dismissed. If I recall > > correctly the > > > argument > > > > was that nobody could reproduce it (I ran this on Debian > > Woody , py2.2, > > > (with > > > > CVS numarray at the time). > > > > > > > > I ended up writting my own wrapper(s): > > > > def poissonArr(shape=defshape, mean=1): > > > > from numarray import random_array as ra > > > > if mean == 0: > > > > return zeroArrF(shape) > > > > elif mean < 0: > > > > raise "poisson not defined for mean < 0" > > > > else: > > > > return ra.poisson(mean, shape).astype > > (na.UInt16) > > > > > > > > def poissonize(arr): > > > > from numarray import random_array as ra > > > > return na.where(arr<=0, 0, ra.poisson(arr)).astype > > ( na.UInt16) > > > > > > > > (I use the astype( na.UInt16) because of some OpenGL code) > > > > > > > > Just last week had this problem on a windows98 computer > > (python2.4). > > > > > > > > This should get sorted out ... > > > > > > > > Thanks for reporting this problem. > > > > Sebastian Haase > > > > > > > > > > > > > > > > > > > > On Tuesday 12 July 2005 13:32, Flavio Coelho wrote: > > > > > Hi, > > > > > > > > > > I am having problems with the poisson random number > > generators of both > > > > > Numarray and Numeric. > > > > > I can't replicate it when calling the function from the > > python cosonle, > > > but > > > > > it has consistently malfunctioned when used within one > > of my scripts. > > > > > > > > > > What happens is that it frequently return a value > > greater than zero when > > > > > called with zero mean: poisson( 0.0) > > > > > > > > > > Unfortunately My program is too big to send attached but > > I have > > > confirmed > > > > > the malfunction by printing both the mean and the result > > whenever it > > > spits > > > > > out a wrong result. > > > > > > > > > > This is very weird indeed, I have run poisson millions > > of times by itsel > > > on > > > > > the python console, without any problems... > > > > > > > > > > I hope it is some stupid mistake, but when I replace the > > poisson > > > function > > > > > call within my program by the R equivalent command > > (rpois) via the rpy > > > > > wrapper, everything works just fine... > > > > > > > > > > any Ideas? > > > > > > > > > > > > > > > > -- > > > > > > Fl?vio Code?o Coelho > > > registered Linux user # 386432 > > > --------------------------- > > > Great spirits have always found violent opposition from > > mediocrities. The > > > latter cannot understand it when a man does not > > thoughtlessly submit to > > > hereditary prejudices but honestly and courageously uses his > > intelligence. > > > Albert Einstein > > > > > > > > > > > -- > > Fl?vio Code?o Coelho > > registered Linux user # 386432 > > --------------------------- > > Great spirits have always found violent opposition from mediocrities. > > The > > latter cannot understand it when a man does not thoughtlessly submit > > to > > hereditary prejudices but honestly and courageously uses his > > intelligence. > > Albert Einstein > > -- Fl?vio Code?o Coelho registered Linux user # 386432 --------------------------- Great spirits have always found violent opposition from mediocrities. The latter cannot understand it when a man does not thoughtlessly submit to hereditary prejudices but honestly and courageously uses his intelligence. Albert Einstein -------------- next part -------------- An HTML attachment was scrubbed... URL: From jmiller at stsci.edu Fri Jul 15 18:52:31 2005 From: jmiller at stsci.edu (Todd Miller) Date: Fri Jul 15 18:52:31 2005 Subject: [Numpy-discussion] problem with poisson generators In-Reply-To: References: <200507121447.01356.haase@msg.ucsf.edu> <1121424865.4707.16.camel@localhost.localdomain> Message-ID: <1121478711.4695.14.camel@localhost.localdomain> On Fri, 2005-07-15 at 21:30 -0300, Flavio Coelho wrote: > > > 2005/7/15, Todd Miller : > > Todd, > > the function within which the problem happens runs fine by itself, > as well as the simplified version you sent me. Maybe you could stick in an assert rather than the print and then use pdb.pm() to find the actual parameters leading to the failure. (The parameters may or may not matter, i.e. the problem may be hidden state somewhere your program accumulates in some complicated way, but maybe we'll get lucky...) > I am running the code on a AMD64 (though all my software and OS is > compiled to 686 - Gentoo Linux). Is there any issue I should be aware > of, regarding this specific cpu? Other than the fact that 64-bit addressing functionality isn't where we want it to be, there are no remaining issues I'm aware of for AMD64. But the addressing is currently hamstrung by 32-bit limits in Python's sequence protocol so you should be aware that even with your AMD64 you may run out of head room. > I can try to run the software on another machine... I wouldn't bother: our goal should be to replicate the problem simply and that suggests to me maintaining identical initial conditions. BTW, I'm using AMD64 too so that won't be another variable. Regards, Todd > > Having this code is a good start to solving the problem. I > think the > next step is to simplify your example to make it runnable and > provide > known inputs for all the parameters which lead to a failure > for you. > > Being really literal (spelling out every single damn thing) > cuts down on > speculation and inefficiency on our end. > > It would also be good to express the condition (or it's > inverse) you > think is an error as an assertion, so something like this > might be what > we need: > > from numarray.random_array import poisson > > E, I, S = (0,0,0) > beta,alpha,e,r,delta,B,w,p = (0.001,1,1,0.5,1,0,0,0) > theta = 0 > npass = 0 > N = 100 # total guess here for me > Lpos_esp = beta*S*((I+theta)/(N+npass))**alpha #Number of new > cases > Lpos = poisson(Lpos_esp) > assert not (theta == 0 and Lpos_esp == 0 and Lpos > 0) > > The problem is, the above works for me. Make it sane and get > it to > expose the error for you and I'll look into this some more. > > Regards, > Todd > > > > > I wonder if called by itself it would trigger the > problem... The > > commented Lines Is where I catch the errors: when poisson > returns a > > greater than zero number, when called with mean 0. > > > > > > > > Ranlib uses static floats so it may relate to > numerical > > precision (as > > well as the random uniform random number generator). > Really > > the only > > way is for you to debug the ranlib.c file > > > > I'll see what I can do. > > > > > > Note that R provides a standalone math library in C > that > > includes the > > random number generators so you could you those > instead. SWIG > > handles > > it rather well. > > > > > > I think thats what Rpy already does...is it not? > > > > thanks Bruce, > > > > Fl?vio > > > > > > Regards > > Bruce > > > > On 7/13/05, Flavio Coelho > wrote: > > > Well, > > > I am definetly glad that someone has also > stumbled onto > > the same problem. > > > > > > But it is nevertheless intriguing, that you can > run poisson > > a million times > > > with mean zero or negative(it assumes zero mean > inthis > > case) without any > > > problems by itself. But when I call it within my > code, the > > rate of error is > > > very high (I would say it returns a wrong result > every time, > > but I haven't > > > checked every answer). > > > > > > Meanwhile, my solution will be: > > > > > > import rpy > > > > > > n = rpy.r.rpois(n,mean) > > > > > > I don't feel I can trust poisson while this > "funny" > > behavior is still > > > there... > > > If someone has any Idea of how I could trace this > bug > > please tell me, and > > > I'll hunt it down. At least I can reproduce it in > a very > > specific context. > > > > > > thanks, > > > > > > Fl?vio > > > > > > 2005/7/12, Sebastian Haase : > > > > Hi Flavio! > > > > I had reported this long time ago and this list > (about > > numarray). > > > > Somehow this got more or less dismissed. If I > recall > > correctly the > > > argument > > > > was that nobody could reproduce it (I ran this > on Debian > > Woody , py2.2, > > > (with > > > > CVS numarray at the time). > > > > > > > > I ended up writting my own wrapper(s): > > > > def poissonArr(shape=defshape, mean=1): > > > > from numarray import random_array as ra > > > > if mean == 0: > > > > return zeroArrF(shape) > > > > elif mean < 0: > > > > raise "poisson not defined for > mean < 0" > > > > else: > > > > return ra.poisson(mean, > shape).astype > > (na.UInt16) > > > > > > > > def poissonize(arr): > > > > from numarray import random_array as ra > > > > return na.where(arr<=0, 0, ra.poisson > (arr)).astype > > ( na.UInt16) > > > > > > > > (I use the astype( na.UInt16) because of some > OpenGL code) > > > > > > > > Just last week had this problem on a windows98 > computer > > (python2.4). > > > > > > > > This should get sorted out ... > > > > > > > > Thanks for reporting this problem. > > > > Sebastian Haase > > > > > > > > > > > > > > > > > > > > On Tuesday 12 July 2005 13:32, Flavio Coelho > wrote: > > > > > Hi, > > > > > > > > > > I am having problems with the poisson random > number > > generators of both > > > > > Numarray and Numeric. > > > > > I can't replicate it when calling the function > from the > > python cosonle, > > > but > > > > > it has consistently malfunctioned when used > within one > > of my scripts. > > > > > > > > > > What happens is that it frequently return a > value > > greater than zero when > > > > > called with zero mean: poisson( 0.0) > > > > > > > > > > Unfortunately My program is too big to send > attached but > > I have > > > confirmed > > > > > the malfunction by printing both the mean and > the result > > whenever it > > > spits > > > > > out a wrong result. > > > > > > > > > > This is very weird indeed, I have run poisson > millions > > of times by itsel > > > on > > > > > the python console, without any problems... > > > > > > > > > > I hope it is some stupid mistake, but when I > replace the > > poisson > > > function > > > > > call within my program by the R equivalent > command > > (rpois) via the rpy > > > > > wrapper, everything works just fine... > > > > > > > > > > any Ideas? > > > > > > > > > > > > > > > > -- > > > > > > Fl?vio Code?o Coelho > > > registered Linux user # 386432 > > > --------------------------- > > > Great spirits have always found violent opposition > from > > mediocrities. The > > > latter cannot understand it when a man does not > > thoughtlessly submit to > > > hereditary prejudices but honestly and > courageously uses his > > intelligence. > > > Albert Einstein > > > > > > > > > > > -- > > Fl?vio Code?o Coelho > > registered Linux user # 386432 > > --------------------------- > > Great spirits have always found violent opposition from > mediocrities. > > The > > latter cannot understand it when a man does not > thoughtlessly submit > > to > > hereditary prejudices but honestly and courageously uses his > > intelligence. > > Albert Einstein > > > > > -- > Fl?vio Code?o Coelho > registered Linux user # 386432 > --------------------------- > Great spirits have always found violent opposition from mediocrities. > The > latter cannot understand it when a man does not thoughtlessly submit > to > hereditary prejudices but honestly and courageously uses his > intelligence. > Albert Einstein From bsouthey at gmail.com Sat Jul 16 20:53:32 2005 From: bsouthey at gmail.com (Bruce Southey) Date: Sat Jul 16 20:53:32 2005 Subject: [Numpy-discussion] problem with poisson generators In-Reply-To: References: <200507121447.01356.haase@msg.ucsf.edu> <1121424865.4707.16.camel@localhost.localdomain> Message-ID: Hi, This seems more and more like some coding problem. The fact that everything you have tried seems to work especially replacing the ranlib generator with R's (using the standalone library just avoids the calls to R). I am not an expert in this but it seems that your code may not be correctly calling things correctly. Perhaps you should find the types of the variables like Lpos_esp as this really should be an C int. Also, if it remains a problem I wouldn't mind seeing the values and types of the variables used to compute Lpos_esp. Regards Bruce On 7/15/05, Flavio Coelho wrote: > > > 2005/7/15, Todd Miller : > > Todd, > > the function within which the problem happens runs fine by itself, as > well as the simplified version you sent me. > > I am running the code on a AMD64 (though all my software and OS is compiled > to 686 - Gentoo Linux). Is there any issue I should be aware of, regarding > this specific cpu? I can try to run the software on another machine... > > > > Having this code is a good start to solving the problem. I think the > > next step is to simplify your example to make it runnable and provide > > known inputs for all the parameters which lead to a failure for you. > > > > Being really literal (spelling out every single damn thing) cuts down on > > speculation and inefficiency on our end. > > > > It would also be good to express the condition (or it's inverse) you > > think is an error as an assertion, so something like this might be what > > we need: > > > > from numarray.random_array import poisson > > > > E, I, S = (0,0,0) > > beta,alpha,e,r,delta,B,w,p = (0.001,1,1,0.5,1,0,0,0) > > theta = 0 > > npass = 0 > > N = 100 # total guess here for me > > Lpos_esp = beta*S*((I+theta)/(N+npass))**alpha #Number of > new cases > > Lpos = poisson(Lpos_esp) > > assert not (theta == 0 and Lpos_esp == 0 and Lpos > 0) > > > > The problem is, the above works for me. Make it sane and get it to > > expose the error for you and I'll look into this some more. > > > > Regards, > > Todd > > > > > > > > I wonder if called by itself it would trigger the problem... The > > > commented Lines Is where I catch the errors: when poisson returns a > > > greater than zero number, when called with mean 0. > > > > > > > > > > > > Ranlib uses static floats so it may relate to numerical > > > precision (as > > > well as the random uniform random number generator). Really > > > the only > > > way is for you to debug the ranlib.c file > > > > > > I'll see what I can do. > > > > > > > > > Note that R provides a standalone math library in C that > > > includes the > > > random number generators so you could you those instead. SWIG > > > handles > > > it rather well. > > > > > > > > > I think thats what Rpy already does...is it not? > > > > > > thanks Bruce, > > > > > > Fl?vio > > > > > > > > > Regards > > > Bruce > > > > > > On 7/13/05, Flavio Coelho wrote: > > > > Well, > > > > I am definetly glad that someone has also stumbled onto > > > the same problem. > > > > > > > > But it is nevertheless intriguing, that you can run poisson > > > a million times > > > > with mean zero or negative(it assumes zero mean inthis > > > case) without any > > > > problems by itself. But when I call it within my code, the > > > rate of error is > > > > very high (I would say it returns a wrong result every time, > > > but I haven't > > > > checked every answer). > > > > > > > > Meanwhile, my solution will be: > > > > > > > > import rpy > > > > > > > > n = rpy.r.rpois(n,mean) > > > > > > > > I don't feel I can trust poisson while this "funny" > > > behavior is still > > > > there... > > > > If someone has any Idea of how I could trace this bug > > > please tell me, and > > > > I'll hunt it down. At least I can reproduce it in a very > > > specific context. > > > > > > > > thanks, > > > > > > > > Fl?vio > > > > > > > > 2005/7/12, Sebastian Haase < haase at msg.ucsf.edu>: > > > > > Hi Flavio! > > > > > I had reported this long time ago and this list (about > > > numarray). > > > > > Somehow this got more or less dismissed. If I recall > > > correctly the > > > > argument > > > > > was that nobody could reproduce it (I ran this on Debian > > > Woody , py2.2, > > > > (with > > > > > CVS numarray at the time). > > > > > > > > > > I ended up writting my own wrapper(s): > > > > > def poissonArr(shape=defshape, mean=1): > > > > > from numarray import random_array as ra > > > > > if mean == 0: > > > > > return zeroArrF(shape) > > > > > elif mean < 0: > > > > > raise "poisson not defined for mean < 0" > > > > > else: > > > > > return ra.poisson(mean, shape).astype > > > (na.UInt16) > > > > > > > > > > def poissonize(arr): > > > > > from numarray import random_array as ra > > > > > return na.where(arr<=0, 0, ra.poisson(arr)).astype > > > ( na.UInt16) > > > > > > > > > > (I use the astype( na.UInt16) because of some OpenGL code) > > > > > > > > > > Just last week had this problem on a windows98 computer > > > (python2.4). > > > > > > > > > > This should get sorted out ... > > > > > > > > > > Thanks for reporting this problem. > > > > > Sebastian Haase > > > > > > > > > > > > > > > > > > > > > > > > > On Tuesday 12 July 2005 13:32, Flavio Coelho wrote: > > > > > > Hi, > > > > > > > > > > > > I am having problems with the poisson random number > > > generators of both > > > > > > Numarray and Numeric. > > > > > > I can't replicate it when calling the function from the > > > python cosonle, > > > > but > > > > > > it has consistently malfunctioned when used within one > > > of my scripts. > > > > > > > > > > > > What happens is that it frequently return a value > > > greater than zero when > > > > > > called with zero mean: poisson( 0.0) > > > > > > > > > > > > Unfortunately My program is too big to send attached but > > > I have > > > > confirmed > > > > > > the malfunction by printing both the mean and the result > > > whenever it > > > > spits > > > > > > out a wrong result. > > > > > > > > > > > > This is very weird indeed, I have run poisson millions > > > of times by itsel > > > > on > > > > > > the python console, without any problems... > > > > > > > > > > > > I hope it is some stupid mistake, but when I replace the > > > poisson > > > > function > > > > > > call within my program by the R equivalent command > > > (rpois) via the rpy > > > > > > wrapper, everything works just fine... > > > > > > > > > > > > any Ideas? > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Fl?vio Code?o Coelho > > > > registered Linux user # 386432 > > > > --------------------------- > > > > Great spirits have always found violent opposition from > > > mediocrities. The > > > > latter cannot understand it when a man does not > > > thoughtlessly submit to > > > > hereditary prejudices but honestly and courageously uses his > > > intelligence. > > > > Albert Einstein > > > > > > > > > > > > > > > > -- > > > Fl?vio Code?o Coelho > > > registered Linux user # 386432 > > > --------------------------- > > > Great spirits have always found violent opposition from mediocrities. > > > The > > > latter cannot understand it when a man does not thoughtlessly submit > > > to > > > hereditary prejudices but honestly and courageously uses his > > > intelligence. > > > Albert Einstein > > > > > > > > -- > > Fl?vio Code?o Coelho > registered Linux user # 386432 > --------------------------- > Great spirits have always found violent opposition from mediocrities. The > latter cannot understand it when a man does not thoughtlessly submit to > hereditary prejudices but honestly and courageously uses his intelligence. > Albert Einstein > From gr at grrrr.org Mon Jul 18 01:46:08 2005 From: gr at grrrr.org (Thomas Grill) Date: Mon Jul 18 01:46:08 2005 Subject: [Numpy-discussion] optimizations In-Reply-To: <78581171a9e444def1705b4a1eb5067c@grrrr.org> References: <1120929968.4884.2.camel@localhost.localdomain> <78581171a9e444def1705b4a1eb5067c@grrrr.org> Message-ID: <2df447904c0dc255e826fcffc2c3703d@grrrr.org> Ok, i'm answering myself partly since no-one else did > I realized that the ufuncs in numarrays are not as fast as possible > because they are coded very straightforward. Is there any particular > reason why this is the case, like cleanness of code, confidence in the > compiler or because the code was automatically generated? > sorry for disregarding the manual in this respect... clearly the codelets are generated > I would like to contribute assembly SIMD codelets (SSE and Altivec), i > have been successfully using in my projects for quite some time. Is it > feasible to submit patches, so that these go into the main numarray > distribution, or should i rather implement this as separate ufuncs? > well then, is it an option to provide hand-written ufunc implementations, circumventing the code generation? Is the function nomenclature stable enough that this would work for a longer time? thanks, Thomas From webb.sprague at gmail.com Mon Jul 18 11:14:35 2005 From: webb.sprague at gmail.com (Egg) Date: Mon Jul 18 11:14:35 2005 Subject: [Numpy-discussion] Reversing elements of array Message-ID: Hi all, I would like to reverse the elements of an array (or vector). If it were a regular Python list, I could say L.reverse(). If it were in Matlab (ick), I could say U(4:-1:1). Can anyone give me a hint on doing this in Numeric (not Numarray)? Thx From faltet at carabos.com Mon Jul 18 11:23:07 2005 From: faltet at carabos.com (Francesc Altet) Date: Mon Jul 18 11:23:07 2005 Subject: [Numpy-discussion] Reversing elements of array In-Reply-To: References: Message-ID: <200507182021.18324.faltet@carabos.com> A Monday 18 July 2005 20:00, Egg va escriure: > I would like to reverse the elements of an array (or vector). If it were a > regular Python list, I could say L.reverse(). If it were in Matlab (ick), > I could say U(4:-1:1). Can anyone give me a hint on doing this in Numeric > (not Numarray)? >>> import Numeric >>> a=Numeric.arange(10) >>> a[::-1] array([9, 8, 7, 6, 5, 4, 3, 2, 1, 0]) (the same works for python lists) Cheers, -- >0,0< Francesc Altet ? ? http://www.carabos.com/ V V C?rabos Coop. V. ??Enjoy Data "-" From rkern at ucsd.edu Mon Jul 18 11:50:26 2005 From: rkern at ucsd.edu (Robert Kern) Date: Mon Jul 18 11:50:26 2005 Subject: [Numpy-discussion] Reversing elements of array In-Reply-To: References: Message-ID: <42DBF1FE.6090700@ucsd.edu> Egg wrote: > Hi all, > > I would like to reverse the elements of an array (or vector). If it were a > regular Python list, I could say L.reverse(). If it were in Matlab (ick), I > could say U(4:-1:1). Can anyone give me a hint on doing this in Numeric (not > Numarray)? In [2]: U = arange(10) In [3]: U[::-1] Out[3]: array([9, 8, 7, 6, 5, 4, 3, 2, 1, 0]) -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From haase at msg.ucsf.edu Tue Jul 19 21:12:03 2005 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue Jul 19 21:12:03 2005 Subject: [Numpy-discussion] unexpected behavior of na.where (no short-circuiting) In-Reply-To: <200507051734.59020.haase@msg.ucsf.edu> References: <200507051734.59020.haase@msg.ucsf.edu> Message-ID: <200507191740.43694.haase@msg.ucsf.edu> Hi, any thoughts on this ? I really think it's counter intuitive ? Thanks, Sebastian Haase On Tuesday 05 July 2005 17:34, Sebastian Haase wrote: > Hi, > > I was very surprised when I got this warning: > >>> a = na.arange(4)-2 > >>> na.where(a != 0,1./a, 999) > > Warning: Encountered divide by zero(s) in divide > [ -0.5 -1. 999. 1. ] > > Then I realized that this is generally referred to as (not) "short > circuiting" (e.g. in the case of the '?:'-C-operator, the middle part never > gets evaluated at all if the first part evals to 0 ) > > Especially annoying was this because (for debugging) I had set this > > error-mode: > >>> na.Error.setMode(dividebyzero="error") > > My questions are: > a) Did other people encounter this problem ? > b) What is the general feeling about this actually being a "problem" ? > c) Could this (at all possible) be implemented differently ? > > Thanks, > Sebastian Haase > > > > > > > > ------------------------------------------------------- > SF.Net email is sponsored by: Discover Easy Linux Migration Strategies > from IBM. Find simple to follow Roadmaps, straightforward articles, > informative Webcasts and more! Get everything you need to get up to > speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From focke at slac.stanford.edu Wed Jul 20 10:37:17 2005 From: focke at slac.stanford.edu (Warren Focke) Date: Wed Jul 20 10:37:17 2005 Subject: [Numpy-discussion] unexpected behavior of na.where (no short-circuiting) In-Reply-To: <200507191740.43694.haase@msg.ucsf.edu> References: <200507051734.59020.haase@msg.ucsf.edu> <200507191740.43694.haase@msg.ucsf.edu> Message-ID: Well, that's the way Python works - function arguments are evaluated before the function is called. Even if there were some way to pass the arguments in as expressions, and evaluate them on the fly, you'd need some sort of lazy evaluation, and some pretty deep parsing of the expression - in this case, you could calc 1./a[x,y,z,whatever] for the appropriate values of [x,y,z,whatever], but what if, instead of 1./a, you have divide.outer(b,c)? Or videoCamera.getNextFrame()? I don't think the general problem of figuring out how to iterate over the indices of subexpressions in the argument to get the appropriate index into the evaluated argument is soluble, even when there are indexable subexpressions. In this case, you can pre-mask the array: >>> a = na.arange(4)-2 a[a==0] = 1./999 >>> na.where(a != 0,1./a, 999) Warren Focke On Tue, 19 Jul 2005, Sebastian Haase wrote: > Hi, > any thoughts on this ? I really think it's counter intuitive ? > > Thanks, > Sebastian Haase > > On Tuesday 05 July 2005 17:34, Sebastian Haase wrote: > > Hi, > > > > I was very surprised when I got this warning: > > >>> a = na.arange(4)-2 > > >>> na.where(a != 0,1./a, 999) > > > > Warning: Encountered divide by zero(s) in divide > > [ -0.5 -1. 999. 1. ] > > > > Then I realized that this is generally referred to as (not) "short > > circuiting" (e.g. in the case of the '?:'-C-operator, the middle part never > > gets evaluated at all if the first part evals to 0 ) > > > > Especially annoying was this because (for debugging) I had set this > > > > error-mode: > > >>> na.Error.setMode(dividebyzero="error") > > > > My questions are: > > a) Did other people encounter this problem ? > > b) What is the general feeling about this actually being a "problem" ? > > c) Could this (at all possible) be implemented differently ? > > > > Thanks, > > Sebastian Haase > > > > > > > > > > > > > > > > ------------------------------------------------------- > > SF.Net email is sponsored by: Discover Easy Linux Migration Strategies > > from IBM. Find simple to follow Roadmaps, straightforward articles, > > informative Webcasts and more! Get everything you need to get up to > > speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > ------------------------------------------------------- > SF.Net email is sponsored by: Discover Easy Linux Migration Strategies > from IBM. Find simple to follow Roadmaps, straightforward articles, > informative Webcasts and more! Get everything you need to get up to > speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From Chris.Barker at noaa.gov Wed Jul 20 11:48:14 2005 From: Chris.Barker at noaa.gov (Chris Barker) Date: Wed Jul 20 11:48:14 2005 Subject: [Numpy-discussion] LAPACK/BLAS for Fedora core 4 i386 ? Message-ID: <42DE9BEC.9070902@noaa.gov> Hi all, Can anyone recommend the best way to get a native LAPACK installed on Fedora core 4? I'm quite surprised that I can't find an atlas rpm with yum. While I'm at it, when I was using Gentoo, it came with a nice atlas blas/lapack. Unfortunately, the atlas lapack does not include all of lapack. It has everything Numeric/numarray needs, but not some stuff i need for another project (Banded Matrix solvers). Does anyone have a suggestion for how to add the addional lapack stuff I need, while still using atlas stuff where possible? Frankly, I don't understand why atlas doesn't just include all of lapack, using generic versions for anything they haven't optimized, it would be a lot easier to get one stop shopping. -Chris From dd55 at cornell.edu Wed Jul 20 11:59:20 2005 From: dd55 at cornell.edu (Darren Dale) Date: Wed Jul 20 11:59:20 2005 Subject: [Numpy-discussion] LAPACK/BLAS for Fedora core 4 i386 ? In-Reply-To: <42DE9BEC.9070902@noaa.gov> References: <42DE9BEC.9070902@noaa.gov> Message-ID: <200507201458.39116.dd55@cornell.edu> Hi Chris, On Wednesday 20 July 2005 02:46 pm, Chris Barker wrote: > Hi all, > > Can anyone recommend the best way to get a native LAPACK installed on > Fedora core 4? > > I'm quite surprised that I can't find an atlas rpm with yum. > > While I'm at it, when I was using Gentoo, it came with a nice atlas > blas/lapack. Unfortunately, the atlas lapack does not include all of > lapack. It has everything Numeric/numarray needs, but not some stuff i > need for another project (Banded Matrix solvers). Does anyone have a > suggestion for how to add the addional lapack stuff I need, while still > using atlas stuff where possible? I can't comment on the Fedora question. With Gentoo, you should not use the "atlas" package, because it comes with an incomplete LAPACK, as you pointed out. Instead, Gentoo-ers should use the "blas-atlas" and "lapack-atlas" packages. -- Darren From Chris.Barker at noaa.gov Wed Jul 20 12:11:13 2005 From: Chris.Barker at noaa.gov (Chris Barker) Date: Wed Jul 20 12:11:13 2005 Subject: [Numpy-discussion] LAPACK/BLAS for Fedora core 4 i386 ? In-Reply-To: <200507201458.39116.dd55@cornell.edu> References: <42DE9BEC.9070902@noaa.gov> <200507201458.39116.dd55@cornell.edu> Message-ID: <42DEA1AB.7070200@noaa.gov> Darren Dale wrote: > I can't comment on the Fedora question. With Gentoo, you should not use the > "atlas" package, because it comes with an incomplete LAPACK, as you pointed > out. > > Instead, Gentoo-ers should use the "blas-atlas" and "lapack-atlas" packages. Well, I've moved away from Gentoo, but I just may go back. However, I'm pretty sure I was using the lapack-atlas package, and while it was described as "complete", it did not, in fact, have all of lapack. Maybe I was doing something wrong, however. -thanks, Chris From Chris.Barker at noaa.gov Wed Jul 20 14:57:15 2005 From: Chris.Barker at noaa.gov (Chris Barker) Date: Wed Jul 20 14:57:15 2005 Subject: [Numpy-discussion] Biulding Numarray with atlas. Message-ID: <42DEC894.5030304@noaa.gov> Hi all, I'm working on building numarray with atlas on a Fedora core 4 system. I installed atlas by downloading a binary from the atlas sourceforge site, and copying stuff into /usr/local/... Then in cfc_packages.py, I added: USE_LAPACK = True at the top. That doesn't seem like where I should do it, but it works. then I got: Src/_dotblas.c:15:19: error: cblas.h: No such file or directory So I looked and saw: else: lapack_dirs = ['/usr/local/lib/atlas'] lapack_libs = ['lapack', 'cblas', 'f77blas', 'atlas', 'g2c', 'm'] which makes sense, we haven't added the include directory that cblas.h is in. So I added: lapack_include_dirs += ['/usr/local/include/atlas/'] and now it seems to work. Perhaps an include line should be added to the stock cfc_packages.py -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From jmiller at stsci.edu Thu Jul 21 07:36:27 2005 From: jmiller at stsci.edu (Todd Miller) Date: Thu Jul 21 07:36:27 2005 Subject: [Numpy-discussion] Biulding Numarray with atlas. In-Reply-To: <42DEC894.5030304@noaa.gov> References: <42DEC894.5030304@noaa.gov> Message-ID: <1121954393.12577.0.camel@halloween.stsci.edu> On Wed, 2005-07-20 at 17:56, Chris Barker wrote: > Hi all, > > I'm working on building numarray with atlas on a Fedora core 4 system. I > installed atlas by downloading a binary from the atlas sourceforge site, > and copying stuff into /usr/local/... > > Then in cfc_packages.py, I added: > > USE_LAPACK = True > > at the top. That doesn't seem like where I should do it, but it works. > > then I got: > > Src/_dotblas.c:15:19: error: cblas.h: No such file or directory > > So I looked and saw: > > else: > lapack_dirs = ['/usr/local/lib/atlas'] > lapack_libs = ['lapack', 'cblas', 'f77blas', 'atlas', 'g2c', 'm'] > > which makes sense, we haven't added the include directory that cblas.h > is in. > > So I added: > lapack_include_dirs += ['/usr/local/include/atlas/'] > > and now it seems to work. > > Perhaps an include line should be added to the stock cfc_packages.py Sounds good to me. I added this. Thanks, Todd From jmiller at stsci.edu Thu Jul 21 08:01:12 2005 From: jmiller at stsci.edu (Todd Miller) Date: Thu Jul 21 08:01:12 2005 Subject: [Numpy-discussion] Biulding Numarray with atlas. In-Reply-To: <42DEC894.5030304@noaa.gov> References: <42DEC894.5030304@noaa.gov> Message-ID: <1121954393.12577.0.camel@halloween.stsci.edu> On Wed, 2005-07-20 at 17:56, Chris Barker wrote: > Hi all, > > I'm working on building numarray with atlas on a Fedora core 4 system. I > installed atlas by downloading a binary from the atlas sourceforge site, > and copying stuff into /usr/local/... > > Then in cfc_packages.py, I added: > > USE_LAPACK = True > > at the top. That doesn't seem like where I should do it, but it works. > > then I got: > > Src/_dotblas.c:15:19: error: cblas.h: No such file or directory > > So I looked and saw: > > else: > lapack_dirs = ['/usr/local/lib/atlas'] > lapack_libs = ['lapack', 'cblas', 'f77blas', 'atlas', 'g2c', 'm'] > > which makes sense, we haven't added the include directory that cblas.h > is in. > > So I added: > lapack_include_dirs += ['/usr/local/include/atlas/'] > > and now it seems to work. > > Perhaps an include line should be added to the stock cfc_packages.py Sounds good to me. I added this. Thanks, Todd From jmiller at stsci.edu Thu Jul 21 08:30:23 2005 From: jmiller at stsci.edu (Todd Miller) Date: Thu Jul 21 08:30:23 2005 Subject: [Numpy-discussion] Biulding Numarray with atlas. In-Reply-To: <42DEC894.5030304@noaa.gov> References: <42DEC894.5030304@noaa.gov> Message-ID: <1121954393.12577.0.camel@halloween.stsci.edu> On Wed, 2005-07-20 at 17:56, Chris Barker wrote: > Hi all, > > I'm working on building numarray with atlas on a Fedora core 4 system. I > installed atlas by downloading a binary from the atlas sourceforge site, > and copying stuff into /usr/local/... > > Then in cfc_packages.py, I added: > > USE_LAPACK = True > > at the top. That doesn't seem like where I should do it, but it works. > > then I got: > > Src/_dotblas.c:15:19: error: cblas.h: No such file or directory > > So I looked and saw: > > else: > lapack_dirs = ['/usr/local/lib/atlas'] > lapack_libs = ['lapack', 'cblas', 'f77blas', 'atlas', 'g2c', 'm'] > > which makes sense, we haven't added the include directory that cblas.h > is in. > > So I added: > lapack_include_dirs += ['/usr/local/include/atlas/'] > > and now it seems to work. > > Perhaps an include line should be added to the stock cfc_packages.py Sounds good to me. I added this. Thanks, Todd From Chris.Barker at noaa.gov Thu Jul 21 11:24:13 2005 From: Chris.Barker at noaa.gov (Chris Barker) Date: Thu Jul 21 11:24:13 2005 Subject: [Numpy-discussion] Biulding Numarray with atlas. In-Reply-To: <07C6A61102C94148B8104D42DE95F7E86DEFA5@exchange2k.envision.co.il> References: <07C6A61102C94148B8104D42DE95F7E86DEFA5@exchange2k.envision.co.il> Message-ID: <42DFE82B.6030603@noaa.gov> Nadav Horesh wrote: > Whenever you need a patch to compile numarray with lapack/atlas, you > should check if the linear_algebra module works. Of course. And test I have, and no, it doesn't work: ImportError: /usr/lib/python2.4/site-packages/numarray/linear_algebra/lapack_lite2.so: undefined symbol: dgesdd_ I'm guessing this is a symbol mangling mismatch between what lapack_lite2 is expecting and what my atlas install used (i.e. too many or not enough underscores) For what it's worth, I get the same error with Numeric. which brings me back to my first question: Where should I get a good lapack/blas for Fedora core 4? There are lapack and blas rpms, but there is no indication of whether they are optimized or not. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From jmiller at stsci.edu Thu Jul 21 12:09:17 2005 From: jmiller at stsci.edu (Todd Miller) Date: Thu Jul 21 12:09:17 2005 Subject: [Numpy-discussion] Biulding Numarray with atlas. In-Reply-To: <42DFE82B.6030603@noaa.gov> References: <07C6A61102C94148B8104D42DE95F7E86DEFA5@exchange2k.envision.co.il> <42DFE82B.6030603@noaa.gov> Message-ID: <1121972619.4971.45.camel@halloween.stsci.edu> On Thu, 2005-07-21 at 14:23, Chris Barker wrote: > Nadav Horesh wrote: > > Whenever you need a patch to compile numarray with lapack/atlas, you > > should check if the linear_algebra module works. > > Of course. And test I have, and no, it doesn't work: > > ImportError: > /usr/lib/python2.4/site-packages/numarray/linear_algebra/lapack_lite2.so: > undefined symbol: dgesdd_ > > I'm guessing this is a symbol mangling mismatch between what > lapack_lite2 is expecting and what my atlas install used (i.e. too many > or not enough underscores) For what it's worth, I get the same error > with Numeric. I think this is a SciPy FAQ and following their advice is how I've worked around it in the past myself: http://www.scipy.org/documentation/mailman?fn=scipy-dev/2001-November/000105.html From Chris.Barker at noaa.gov Thu Jul 21 15:28:14 2005 From: Chris.Barker at noaa.gov (Chris Barker) Date: Thu Jul 21 15:28:14 2005 Subject: [Numpy-discussion] Biulding Numarray with atlas. In-Reply-To: <1121972619.4971.45.camel@halloween.stsci.edu> References: <07C6A61102C94148B8104D42DE95F7E86DEFA5@exchange2k.envision.co.il> <42DFE82B.6030603@noaa.gov> <1121972619.4971.45.camel@halloween.stsci.edu> Message-ID: <42E0212C.5060702@noaa.gov> Todd Miller wrote: > I think this is a SciPy FAQ and following their advice is how I've > worked around it in the past myself: > > http://www.scipy.org/documentation/mailman?fn=scipy-dev/2001-November/000105.html I'd seem that note before, and it wasn't very helpful, but then I noticed, in the Numeric customize.py: Example: Some Linux distributions have ATLAS, but not LAPACK (if you're liblapack file is smaller than a megabyte, this is probably you) Use our own f2c'd lapack libraries, but use ATLAS for BLAS. My linux distro (FC4) has a full lapack, but it's not optimized, in fact, it's slower than lapack-lite. However, I got atlas from the atlas site, and it doesn't come with a full lapack, which would explain the missing symbols. Using: # Using ATLAS blas in /usr/local with lapack-lite if 1: use_system_lapack = 0 use_system_blas = 1 lapack_library_dirs = ['/usr/local/lib/atlas'] lapack_libraries = ['cblas', 'f77blas', 'atlas', 'g2c'] It compiles, works, and is about twice as fast as using the the Numeric-supplied blas. I still would like to find a full, optimized lapack for Fedora Core 4. On my last machine, I got about a six times speed-up using the atlas that came with Gentoo. But for the moment, I'm OK. Also, I saw something in the atlas docs about how to merge the atlas lapack with another full lapack, so maybe I can get that to work. Now on to see if I can do something similar with numarray. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Chris.Barker at noaa.gov Thu Jul 21 17:07:24 2005 From: Chris.Barker at noaa.gov (Chris Barker) Date: Thu Jul 21 17:07:24 2005 Subject: [Numpy-discussion] Building Numeric+numarray with atals on Fedora Core 4 In-Reply-To: <42E0212C.5060702@noaa.gov> References: <07C6A61102C94148B8104D42DE95F7E86DEFA5@exchange2k.envision.co.il> <42DFE82B.6030603@noaa.gov> <1121972619.4971.45.camel@halloween.stsci.edu> <42E0212C.5060702@noaa.gov> Message-ID: <42E03313.7090208@noaa.gov> It looks like I'm managing to maintain this thread all by myself! I thought I'd post this final note, so that anyone searching the archives can see my solution. It's a bit ugly, but it seems to work. First the summary: I'm trying to build both Numeric and numarray on a new installation of Linux Fedora Core 4 on a Pentium M laptop. Fedora helpfully provides a full lapack and blas rpms, but they don't appear to be at all optimized. In fact, when I used them, it was slower than the built-in lapack-lite. I haven't been able to find a complete atlas+lapack distribution for Fedora core 4 (or anything like it), so this is what I did: Got the atlas binaries from the atlas site: https://sourceforge.net/projects/math-atlas/ I downloaded: atlas3.6.0_Linux_P4SSE2.tar.gz NOTE: these are pretty old...should I be using the 3.7.10 unstable version instead? I then merged these with the Fedora lapack (should be in /usr/lib/liblapack.a) by following the directions in the atlas README: """******** GETTING A FULL LAPACK LIB ************************** ATLAS does not provide a full lapack library. However, there is a simple way to get ATLAS to provide its faster LAPACK routines to a full LAPACK library. ATLAS's internal routines are distinct from LAPACK's, so it is safe to compile ATLAS's LAPACK routines directly into a netlib-style LAPACK library. First, obtain the LAPACK src from netlib and build the LAPACK library as normal. Then, in this directory (where you should have a liblapack.a), issue the following commands: mkdir tmp cd tmp ar x ../liblapack.a cp ../liblapack.a ar r ../liblapack.a *.o cd .. rm -rf tmp Just linking in ATLAS's liblapack.a first will not get you the best LAPACK performance, mainly because LAPACK's untuned ILAENV will be used instead of ATLAS's tuned one. """ I installed them in: /usr/local/lib/atlas and /usr/local/include/atlas Then I set up the numarray and Numeric build process to use them. For numarray, I edited cfg_packages.py, and put in: USE_LAPACK = True near the top, and edited on of the blocks to read: print "Adding paths for lapack/blas" lapack_dirs = ['/usr/local/lib/atlas'] lapack_libs = ['lapack', 'cblas', 'f77blas', 'atlas', 'g2c', 'gfortran'] lapack_include_dirs += ['/usr/local/include/atlas/'] NOTE: the gfortran lib is something I hadn't seen before, but I needed it to resolve missing symbols. I'm guess that the fedora lapack used gfortran. then a setup.py build, setup.py install, and it seems to work, and is about twice as fast as lapack-lite on my simple test case. In the past, I got about a 7 times speed-up with the atlas that I got from Gentoo. I'm a bit disappointed, but it's OK, and I'm not really doing anything where it matters much. For Numeric, I edited customize.py: # Using ATLAS blas in /usr/local if 1: use_system_lapack = 1 lapack_library_dirs = ['/usr/local/lib/atlas'] lapack_libraries = ['lapack', 'cblas', 'f77blas', 'atlas', 'g2c', 'gfortran'] And that did it. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From python-ml at nn7.de Sat Jul 23 09:04:16 2005 From: python-ml at nn7.de (Soeren Sonnenburg) Date: Sat Jul 23 09:04:16 2005 Subject: [Numpy-discussion] design issues - octave 'incompatibilities' Message-ID: <1122134630.26083.74.camel@localhost> Dear all, I am new to numarray and as I am trying to use it day-by-day I am now wondering about certain numeric/numarray design issues. Please forgive me / correct me as I probably misunderstood certain issues. -- Why did you choose row-major instead of column major format as practiced by R/octave/matlab... Doesn't that mean performance problems as fortran/blas is optimized if you work with the transposed version ? -- why do vectors have no 'orientation', i.e. there are only row but no column vectors (or why do you treat matrices/vectors differently, i.e. have vectors at all as a separate type) numarray: a=array([[1,2,3],[4,5,6],[7,8,9]]) >>> a[0,:] array([1, 2, 3]) >>> a[:,0] array([1, 4, 7]) vs. octave: a=[1,2,3;4,5,6;7,8,9]; >> a(1,:) ans = 1 2 3 >> a(:,1) ans = 1 4 7 -- How can one obtain submatrices from full matrices: numarray gives only elements (which is very, very rarely needed and should IMHO rather be some extra function): >>> i=[0,1] >>> j=[0,2] >>> a[i,j] array([1, 6]) vs octave: >> i=[1,2]; >> j=[1,3]; >> a(i,j) ans = 1 3 4 6 all one can do in numarray currently is as awkward as this: >>> a.take(i,0).take(j,1) array([[1, 3], [4, 6]]) also mixing slice and variable does not work a[:,i] - hmmhh seems that is a major issue for me... -- why don't you allow iterating over the whole matrix via a single index ? i.e. a[3] Traceback (most recent call last): File "", line 1, in ? IndexError: Index out of range vs octave >> a(4) ans = 2 Are there more elegant ways to do this ? Which issues are likely to be fixed in the future ? Best regards, Soeren From python-ml at nn7.de Sat Jul 23 09:13:16 2005 From: python-ml at nn7.de (Soeren Sonnenburg) Date: Sat Jul 23 09:13:16 2005 Subject: [Numpy-discussion] C API function to get the shape of an array ? Message-ID: <1122135157.26083.84.camel@localhost> Hi, I just want to use NA_get1_Int64 etc but I couldn't find a way to 'cleanly' find out about the size of an array, i.e. I always have to access raw fields in the PyArrayObject structure like: PyArrayObject* a; if (a->nd == 2) { if (a->dimensions[0] == m && b->dimensions[1] == n) ... Is this the intended way ? I hope not... Best, Soeren From rkern at ucsd.edu Sat Jul 23 12:07:11 2005 From: rkern at ucsd.edu (Robert Kern) Date: Sat Jul 23 12:07:11 2005 Subject: [Numpy-discussion] design issues - octave 'incompatibilities' In-Reply-To: <1122134630.26083.74.camel@localhost> References: <1122134630.26083.74.camel@localhost> Message-ID: <42E29537.7080702@ucsd.edu> Soeren Sonnenburg wrote: > Dear all, > > I am new to numarray and as I am trying to use it day-by-day > I am now wondering about certain numeric/numarray design issues. Please > forgive me / correct me as I probably misunderstood certain issues. > > -- Why did you choose row-major instead of column major format as > practiced by R/octave/matlab... Doesn't that mean performance problems > as fortran/blas is optimized if you work with the transposed version ? Not everyone is interfacing with optimized FORTRAN code. Those who are are usually using ATLAS as their BLAS, and ATLAS has row-major versions of the BLAS subroutines. Row-major is C's convention and numarray/Numeric largely follow that. There are of course some performance issues when interfacing with FORTRAN code that expects column-major, but there would be other performance problems if numarray/Numeric were column-major and interfacing with row-major code. > -- why do vectors have no 'orientation', i.e. there are only row but no > column vectors (or why do you treat matrices/vectors differently, i.e. > have vectors at all as a separate type) You can certainly make column vectors. In [1]: a = arange(5) In [2]: a Out[2]: NumPy array, format: long [0 1 2 3 4] In [3]: a.shape = (5,1) In [4]: a Out[4]: NumPy array, format: long [[0] [1] [2] [3] [4]] Often, though, a "row" vector can also be used in place of a "column" vector. Although sometimes not: > -- How can one obtain submatrices from full matrices: In [6]: import numarray In [7]: a = numarray.arange(1, 10) In [8]: a.shape = (3,3) In [9]: a Out[9]: array([[1, 2, 3], [4, 5, 6], [7, 8, 9]]) In [10]: i = [0,1] In [11]: j = [[0],[2]] In [12]: a[i,j] Out[12]: array([[1, 4], [3, 6]]) > -- why don't you allow iterating over the whole matrix via a single > index ? ravel() > Are there more elegant ways to do this ? Which issues are likely to be > fixed in the future ? None. They're not broken, just different from what you are used to. -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From python-ml at nn7.de Sun Jul 24 11:42:30 2005 From: python-ml at nn7.de (Soeren Sonnenburg) Date: Sun Jul 24 11:42:30 2005 Subject: [Numpy-discussion] design issues - octave 'incompatibilities' In-Reply-To: <42E29537.7080702@ucsd.edu> References: <1122134630.26083.74.camel@localhost> <42E29537.7080702@ucsd.edu> Message-ID: <1122230465.30698.72.camel@localhost> On Sat, 2005-07-23 at 12:06 -0700, Robert Kern wrote: > Soeren Sonnenburg wrote: > > Dear all, > > > > I am new to numarray and as I am trying to use it day-by-day > > I am now wondering about certain numeric/numarray design issues. Please > > forgive me / correct me as I probably misunderstood certain issues. > > > > -- Why did you choose row-major instead of column major format as > > practiced by R/octave/matlab... Doesn't that mean performance problems > > as fortran/blas is optimized if you work with the transposed version ? > > Not everyone is interfacing with optimized FORTRAN code. Those who are > are usually using ATLAS as their BLAS, and ATLAS has row-major versions > of the BLAS subroutines. Row-major is C's convention and > numarray/Numeric largely follow that. There are of course some > performance issues when interfacing with FORTRAN code that expects > column-major, but there would be other performance problems if > numarray/Numeric were column-major and interfacing with row-major code. Well but why did you change to the C version then ? Well maybe if it is about optimizing stuff seriously one could work with the transpose anyway... > Often, though, a "row" vector can also be used in place of a "column" > vector. Although sometimes not: You are right, thanks! > > Are there more elegant ways to do this ? Which issues are likely to be > > fixed in the future ? > > None. They're not broken, just different from what you are used to. Thanks a lot for your very helpful mail! Do you know whether mixing slices and arrays will be supported at some point a[:,[0,1]] ? Soeren. From rkern at ucsd.edu Sun Jul 24 11:49:25 2005 From: rkern at ucsd.edu (Robert Kern) Date: Sun Jul 24 11:49:25 2005 Subject: [Numpy-discussion] design issues - octave 'incompatibilities' In-Reply-To: <1122230465.30698.72.camel@localhost> References: <1122134630.26083.74.camel@localhost> <42E29537.7080702@ucsd.edu> <1122230465.30698.72.camel@localhost> Message-ID: <42E3E260.5050406@ucsd.edu> Soeren Sonnenburg wrote: > On Sat, 2005-07-23 at 12:06 -0700, Robert Kern wrote: > >>Soeren Sonnenburg wrote: >> >>>Dear all, >>> >>>I am new to numarray and as I am trying to use it day-by-day >>>I am now wondering about certain numeric/numarray design issues. Please >>>forgive me / correct me as I probably misunderstood certain issues. >>> >>>-- Why did you choose row-major instead of column major format as >>>practiced by R/octave/matlab... Doesn't that mean performance problems >>>as fortran/blas is optimized if you work with the transposed version ? >> >>Not everyone is interfacing with optimized FORTRAN code. Those who are >>are usually using ATLAS as their BLAS, and ATLAS has row-major versions >>of the BLAS subroutines. Row-major is C's convention and >>numarray/Numeric largely follow that. There are of course some >>performance issues when interfacing with FORTRAN code that expects >>column-major, but there would be other performance problems if >>numarray/Numeric were column-major and interfacing with row-major code. > > Well but why did you change to the C version then ? Well maybe if it is > about optimizing stuff seriously one could work with the transpose > anyway... Because numarray and Python are written in C. It's also consistent with lists-of-lists. In [8]: L = [[0, 1],[2,3]] In [9]: A = array(L) In [10]: L[1][0] Out[10]: 2 In [11]: A[1][0] Out[11]: 2 In [12]: A[1,0] Out[12]: 2 -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From perry at stsci.edu Mon Jul 25 07:11:17 2005 From: perry at stsci.edu (Perry Greenfield) Date: Mon Jul 25 07:11:17 2005 Subject: [Numpy-discussion] design issues - octave 'incompatibilities' In-Reply-To: <1122230465.30698.72.camel@localhost> References: <1122134630.26083.74.camel@localhost> <42E29537.7080702@ucsd.edu> <1122230465.30698.72.camel@localhost> Message-ID: <89be641d30d1e246ea18d3733b6c3357@stsci.edu> On Jul 24, 2005, at 2:41 PM, Soeren Sonnenburg wrote: > > Well but why did you change to the C version then ? Well maybe if it is > about optimizing stuff seriously one could work with the transpose > anyway... > To get a really solid answer to "why" you would have to ask those that wrote the original Numeric. (Jim Hugunin and Co.). My guess is was that it was just as much to preserve the following relation arr[i,j] = arr[i][j] (instead of arr[i,j] = arr[j][i]) But I could be wrong. Note that this is a confusing issue to many and often as not there are unstated assumptions that are repeatedly made by many that are *not* shared by everyone. To illustrate, there are at least two different approaches to handling this mismatch and it seems to me that many seem oblivious to the fact that there are two approaches. Approach 1: Keep the index convention the same. As a user of Numeric or numarray, you wish M[i,j] to mean the same as it does in Fortran/IDL/matlab... The consequence is that the data are ordered differently between Numeric/numarray and these other languages. This is seen as a data compatibility problem. Approach 2: Keep the data invariant and change the indexing convention. What was M[i,j] in Fortran is now M[j,i] in Numeric/numarray. This is not a data compatibility problem, but is now a brain compatibility problem ;-) Since we deal with big data sets we have adopted 2 (no doubt to the consternation of many). This ought to be in a FAQ, it comes up enough to be :-) > > Do you know whether mixing slices and arrays will be supported at some > point a[:,[0,1]] ? > > Soeren. No plans at the moment. We figured indexing was complicated enough as it was. I think Travis Oliphant did allow for this in Numeric3 (aka scipy.core); see the link below. But it may not mean what you think it should so you should check that out to see: http://cvs.sourceforge.net/viewcvs.py/numpy/Numeric3/PEP.txt?view=markup Perry Greenfield From Chris.Barker at noaa.gov Mon Jul 25 09:01:03 2005 From: Chris.Barker at noaa.gov (Chris Barker) Date: Mon Jul 25 09:01:03 2005 Subject: [Numpy-discussion] design issues - octave 'incompatibilities' In-Reply-To: <1122134630.26083.74.camel@localhost> References: <1122134630.26083.74.camel@localhost> Message-ID: <42E50C49.8080705@noaa.gov> Soeren Sonnenburg wrote: > -- why do vectors have no 'orientation', i.e. there are only row but no > column vectors (or why do you treat matrices/vectors differently, i.e. > have vectors at all as a separate type) I think the key to understanding this is that NumPy uses a fundamentally different data type that MATLAB ans it's derivatives. MATLAB was originally just what it is called: a "Matrix" laboratory. The basic data type of Matlab is a 2-d matrix. Even a scalar is really a 1X! matrix. Matlab has a few tricks that can make these look like row and column vectors, etc, but they are really always matrices. On the other hand, NumPy arrays are N-dimensional, where N is theoretically unlimited. In practice, I think the max N is defined and compiled in, but you could change it and re-compile if you wanted. In any case, many of us frequently use 3-d and higher arrays, and they can be very useful. When thought of this way, you can see why there is no such thing as a column vs. a row vector. A vector is a one-dimensional array: it has no orientation. However, NumPy does support NX1 and 1XN 2-d arrays which can be very handy: >>> import numarray as N >>> a = N.arange(5) >>> a.shape = (1,-1) >>> a array([[0, 1, 2, 3, 4]]) >>> b = N.arange(5) >>> b.shape = (-1,1) >>> a array([0, 1, 2, 3, 4]) >>> b array([[0], [1], [2], [3], [4]]) So a is a row vector and b is a column vector. If you multiply them, you get "array broadcasting": >>> a * b array([[ 0, 0, 0, 0, 0], [ 0, 1, 2, 3, 4], [ 0, 2, 4, 6, 8], [ 0, 3, 6, 9, 12], [ 0, 4, 8, 12, 16]]) This eliminates a LOT of extra duplicate arrays that you have to make in Matlab with meshgrid. When you index into an array, you reduce its rank (number of dimensions) by 1: >>> a = N.arange(27) >>> a.shape = (3,3,3) >>> a array([[[ 0, 1, 2], [ 3, 4, 5], [ 6, 7, 8]], [[ 9, 10, 11], [12, 13, 14], [15, 16, 17]], [[18, 19, 20], [21, 22, 23], [24, 25, 26]]]) >>> a.shape (3, 3, 3) >>> a[1].shape (3, 3) >>> a[1][1].shape (3,) When you slice, you keep the rank the same: >>> a[1:2].shape (1, 3, 3) This creates a way to make row and column "vectors" from your 2-d array (matrix) >>> a = N.arange(25) >>> a.shape = (5,5) >>> a array([[ 0, 1, 2, 3, 4], [ 5, 6, 7, 8, 9], [10, 11, 12, 13, 14], [15, 16, 17, 18, 19], [20, 21, 22, 23, 24]]) To make a "row vector" (really a 1XN matrix) >>> a[0:1,:] array([[0, 1, 2, 3, 4]]) To make a "column vector" (really a NX1 matrix) >>> a[:,0:1] array([[ 0], [ 5], [10], [15], [20]]) I hope that helps: -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From oliphant at ee.byu.edu Mon Jul 25 14:11:08 2005 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon Jul 25 14:11:08 2005 Subject: [Numpy-discussion] design issues - octave 'incompatibilities' In-Reply-To: <1122134630.26083.74.camel@localhost> References: <1122134630.26083.74.camel@localhost> Message-ID: <42E5551D.1030203@ee.byu.edu> Soeren Sonnenburg wrote: >Dear all, > >I am new to numarray and as I am trying to use it day-by-day >I am now wondering about certain numeric/numarray design issues. Please >forgive me / correct me as I probably misunderstood certain issues. > >-- Why did you choose row-major instead of column major format as >practiced by R/octave/matlab... Doesn't that mean performance problems >as fortran/blas is optimized if you work with the transposed version ? > >-- why do vectors have no 'orientation', i.e. there are only row but no >column vectors (or why do you treat matrices/vectors differently, i.e. >have vectors at all as a separate type) > > I think others have responded to these questions well. >-- How can one obtain submatrices from full matrices: > >numarray gives only elements (which is very, very rarely needed and >should IMHO rather be some extra function): > > I thought about this quite a bit because I initially felt as you did before beginning work on scipy.base (Numeric3). I wondered if the right choice had been made. After some thought and discussion, I decided this approach was more general and flexible, and agreed with the numarray design. Therefore, this works in scipy.base the same. Also, scipy.base does allow mixing slices and lists like you want: import scipy.base a = scipy.base.array([[1,2,3],[4,5,6],[7,8,9]]) >>> a[:,[0,1]] array([[1, 2], [4, 5], [7, 8]], 'l') > >-- why don't you allow iterating over the whole matrix via a single >index ? > > You can with scipy.base with a.flat[index] but ravel(a)[index] always works too. The reason is that Numeric uses the concept of reduced-dimensionality arrays so that a[index] for a 2-d array returns a 1-d array not an element of the 2-d array. It may be different then you are used to but I think it is a better choice. Best regards, -Travis Oliphant From perry at stsci.edu Mon Jul 25 14:48:05 2005 From: perry at stsci.edu (Perry Greenfield) Date: Mon Jul 25 14:48:05 2005 Subject: [Numpy-discussion] design issues - octave 'incompatibilities' In-Reply-To: <1122134630.26083.74.camel@localhost> References: <1122134630.26083.74.camel@localhost> Message-ID: <20e127854b47f13f1210e2f3574584f9@stsci.edu> I missed this part and was reminded by Travis's message. On Jul 23, 2005, at 12:03 PM, Soeren Sonnenburg wrote: > -- How can one obtain submatrices from full matrices: > > numarray gives only elements (which is very, very rarely needed and > should IMHO rather be some extra function): > >>>> i=[0,1] >>>> j=[0,2] >>>> a[i,j] > array([1, 6]) > > vs octave: >>> i=[1,2]; >>> j=[1,3]; >>> a(i,j) > ans = > 1 3 > 4 6 > Maybe for you it is rarely needed, but not for us. The situation is reversed. The latter is rarely used in our case. This is largely a reflection of whether your orientation is matrices or multidimensional arrays. In our case it is quite handy to select out a list of point in an image by giving a list of their respective indices (e.g., stars). True, I do see that others may need the other view, but then the question is which should get the convenience. Since Numeric/numarray are primarily oriented towards multidimensional arrays (e.g., operations are element-by-element rather than matrix) it seemed to make sense to go this way for consistency, but I understand that there is another side to this. Perry From python-ml at nn7.de Mon Jul 25 21:36:23 2005 From: python-ml at nn7.de (Soeren Sonnenburg) Date: Mon Jul 25 21:36:23 2005 Subject: [Numpy-discussion] design issues - octave 'incompatibilities' In-Reply-To: <89be641d30d1e246ea18d3733b6c3357@stsci.edu> References: <1122134630.26083.74.camel@localhost> <42E29537.7080702@ucsd.edu> <1122230465.30698.72.camel@localhost> <89be641d30d1e246ea18d3733b6c3357@stsci.edu> Message-ID: <1122352506.30698.136.camel@localhost> On Mon, 2005-07-25 at 10:10 -0400, Perry Greenfield wrote: > On Jul 24, 2005, at 2:41 PM, Soeren Sonnenburg wrote: > > > > > Well but why did you change to the C version then ? Well maybe if it is > > about optimizing stuff seriously one could work with the transpose > > anyway... > > > > To get a really solid answer to "why" you would have to ask those that > wrote the original Numeric. (Jim Hugunin and Co.). My guess is was that > it was just as much to preserve the following relation > > arr[i,j] = arr[i][j] > > (instead of arr[i,j] = arr[j][i]) Ok, that sounds reasonable. > But I could be wrong. > > Note that this is a confusing issue to many and often as not there are > unstated assumptions that are repeatedly made by many that are *not* > shared by everyone. To illustrate, there are at least two different > approaches to handling this mismatch and it seems to me that many seem > oblivious to the fact that there are two approaches. > > Approach 1: Keep the index convention the same. As a user of Numeric or > numarray, you wish M[i,j] to mean the same as it does in > Fortran/IDL/matlab... The consequence is that the data are ordered > differently between Numeric/numarray and these other languages. This is > seen as a data compatibility problem. > > Approach 2: Keep the data invariant and change the indexing convention. > What was M[i,j] in Fortran is now M[j,i] in Numeric/numarray. This is > not a data compatibility problem, but is now a brain compatibility > problem ;-) well it might not be *that* bad in the end... only it is a tough decision to break with everything that is there (to put it to extreme: the world is wrong and but we did it right) and be compatible with C like array access... If one does so one needs to have very serious reasons to do so ... that is why I was asking. > Since we deal with big data sets we have adopted 2 (no doubt to the > consternation of many). hmmhh, there is no advantage with big datasets for any of the formats. > > Do you know whether mixing slices and arrays will be supported at some > > point a[:,[0,1]] ? > > > > Soeren. > > No plans at the moment. We figured indexing was complicated enough as > it was. I think Travis Oliphant did allow for this in Numeric3 (aka > scipy.core); see the link below. But it may not mean what you think it > should so you should check that out to see: > > http://cvs.sourceforge.net/viewcvs.py/numpy/Numeric3/PEP.txt?view=markup Hmmhh while we are at it, is there work on a consensus num* ? I've seen the PEP for inclusion of numarray, now I see numeric3 and scipy and also cvxopt - are people actually converging towards a single num* package ? Soeren. From python-ml at nn7.de Mon Jul 25 22:23:11 2005 From: python-ml at nn7.de (Soeren Sonnenburg) Date: Mon Jul 25 22:23:11 2005 Subject: [Numpy-discussion] design issues - octave 'incompatibilities' In-Reply-To: <42E50C49.8080705@noaa.gov> References: <1122134630.26083.74.camel@localhost> <42E50C49.8080705@noaa.gov> Message-ID: <1122355332.30698.160.camel@localhost> On Mon, 2005-07-25 at 08:59 -0700, Chris Barker wrote: > Soeren Sonnenburg wrote: > > -- why do vectors have no 'orientation', i.e. there are only row but no > > column vectors (or why do you treat matrices/vectors differently, i.e. > > have vectors at all as a separate type) > > I think the key to understanding this is that NumPy uses a > fundamentally different data type that MATLAB and it's derivatives. > MATLAB was originally just what it is called: a "Matrix" laboratory. The > basic data type of Matlab is a 2-d matrix. Even a scalar is really a 1X! > matrix. Matlab has a few tricks that can make these look like row and > column vectors, etc, but they are really always matrices. Ok, I am realizing that R also distinguishes between vectors and matrices. > On the other hand, NumPy arrays are N-dimensional, where N is > theoretically unlimited. In practice, I think the max N is defined and > compiled in, but you could change it and re-compile if you wanted. In > any case, many of us frequently use 3-d and higher arrays, and they can > be very useful. When thought of this way, you can see why there is no Well at least this is the same for octave/matlab. > such thing as a column vs. a row vector. A vector is a one-dimensional > array: it has no orientation. This makes life more difficult if one wants to convert from octave/matlab -> numarray and automated systems close to impossible. If vectors had the same properties/functions as matrices one would not have such problems, i.e. v^{transpose} * u == dot(v,u) and v*u -> error > However, NumPy does support NX1 and 1XN 2-d arrays which can be very handy: > >>> import numarray as N > >>> a = N.arange(5) > >>> a.shape = (1,-1) > >>> a > array([[0, 1, 2, 3, 4]]) > >>> b = N.arange(5) > >>> b.shape = (-1,1) > >>> a > array([0, 1, 2, 3, 4]) > >>> b > array([[0], > [1], > [2], > [3], > [4]]) > > So a is a row vector and b is a column vector. If you multiply them, you > get "array broadcasting": > >>> a * b > array([[ 0, 0, 0, 0, 0], > [ 0, 1, 2, 3, 4], > [ 0, 2, 4, 6, 8], > [ 0, 3, 6, 9, 12], > [ 0, 4, 8, 12, 16]]) > > This eliminates a LOT of extra duplicate arrays that you have to make in > Matlab with meshgrid. In my eyes 'array broadcasting' is confusing and should rather be in a function like meshgrid and instead a*b should return matrixmultiply(a,b) ... > When you index into an array, you reduce its rank (number of dimensions) > by 1: > >>> a = N.arange(27) > >>> a.shape = (3,3,3) > >>> a > array([[[ 0, 1, 2], > [ 3, 4, 5], > [ 6, 7, 8]], > > [[ 9, 10, 11], > [12, 13, 14], > [15, 16, 17]], > > [[18, 19, 20], > [21, 22, 23], > [24, 25, 26]]]) > >>> a.shape > (3, 3, 3) > >>> a[1].shape > (3, 3) > >>> a[1][1].shape > (3,) > > When you slice, you keep the rank the same: > > >>> a[1:2].shape > (1, 3, 3) > > This creates a way to make row and column "vectors" from your 2-d array > (matrix) > >>> a = N.arange(25) > >>> a.shape = (5,5) > >>> a > array([[ 0, 1, 2, 3, 4], > [ 5, 6, 7, 8, 9], > [10, 11, 12, 13, 14], > [15, 16, 17, 18, 19], > [20, 21, 22, 23, 24]]) > > To make a "row vector" (really a 1XN matrix) > >>> a[0:1,:] > array([[0, 1, 2, 3, 4]]) > > > To make a "column vector" (really a NX1 matrix) > >>> a[:,0:1] > array([[ 0], > [ 5], > [10], > [15], > [20]]) > > > I hope that helps: Indeed it does - Thanks!! Unfortunately I am not at all happy now that '*' != matrixmultiply (but outerproduct) for vectors/matrices... I realize that with lists it is ok to grow them via slicing. x=[] x[0]=1 IndexError: list assignment index out of range x[0:0]=[1] x [1] that seems not to work with numarray ... or ? y=array() y[0]=1 TypeError: object does not support item assignment y[0:0]=array([1]) TypeError: object does not support item assignment Soeren. From rkern at ucsd.edu Mon Jul 25 22:45:05 2005 From: rkern at ucsd.edu (Robert Kern) Date: Mon Jul 25 22:45:05 2005 Subject: [Numpy-discussion] design issues - octave 'incompatibilities' In-Reply-To: <1122355332.30698.160.camel@localhost> References: <1122134630.26083.74.camel@localhost> <42E50C49.8080705@noaa.gov> <1122355332.30698.160.camel@localhost> Message-ID: <42E5CCAF.4000201@ucsd.edu> Soeren Sonnenburg wrote: > On Mon, 2005-07-25 at 08:59 -0700, Chris Barker wrote: [snip] >>such thing as a column vs. a row vector. A vector is a one-dimensional >>array: it has no orientation. > > This makes life more difficult if one wants to convert > from octave/matlab -> numarray and automated systems close to > impossible. That wasn't a design goal. > If vectors had the same properties/functions as matrices one > would not have such problems, i.e. v^{transpose} * u == dot(v,u) and v*u > -> error That would be a big problem for those of us who don't use Numeric just for linear algebra. These are general arrays, not matrices and vectors. [snip] > In my eyes 'array broadcasting' is confusing and should rather be in a > function like meshgrid and instead a*b should return > matrixmultiply(a,b) ... Spend some time with it. It will probably grow on you. Numeric is not Matlab or Octave. It never will be, thank G-d. [snip] >>I hope that helps: > > Indeed it does - Thanks!! Unfortunately I am not at all happy now that > '*' != matrixmultiply (but outerproduct) for vectors/matrices... Again, Numeric is a package for arrays, not just linear algebra. Please spend some more time with Python and Numeric before deciding that they must be changed to match your preconceptions. > I realize that with lists it is ok to grow them via slicing. > > x=[] > x[0]=1 > IndexError: list assignment index out of range > x[0:0]=[1] > x > [1] > > that seems not to work with numarray ... or ? > > y=array() > y[0]=1 > TypeError: object does not support item assignment > y[0:0]=array([1]) > TypeError: object does not support item assignment Python lists are designed to grow dynamically. Their memory is preallocated so that growing them is on average pretty cheap. Numeric arrays are not, nor will they be. -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From python-ml at nn7.de Mon Jul 25 22:45:06 2005 From: python-ml at nn7.de (Soeren Sonnenburg) Date: Mon Jul 25 22:45:06 2005 Subject: [Numpy-discussion] design issues - octave 'incompatibilities' In-Reply-To: <20e127854b47f13f1210e2f3574584f9@stsci.edu> References: <1122134630.26083.74.camel@localhost> <20e127854b47f13f1210e2f3574584f9@stsci.edu> Message-ID: <1122356650.30698.176.camel@localhost> On Mon, 2005-07-25 at 17:47 -0400, Perry Greenfield wrote: > I missed this part and was reminded by Travis's message. > > On Jul 23, 2005, at 12:03 PM, Soeren Sonnenburg wrote: > > -- How can one obtain submatrices from full matrices: > > > > numarray gives only elements (which is very, very rarely needed and > > should IMHO rather be some extra function): > > > >>>> i=[0,1] > >>>> j=[0,2] > >>>> a[i,j] > > array([1, 6]) > > > > vs octave: > >>> i=[1,2]; > >>> j=[1,3]; > >>> a(i,j) > > ans = > > 1 3 > > 4 6 > > > Maybe for you it is rarely needed, but not for us. The situation is > reversed. The latter is rarely used in our case. This is largely a > reflection of whether your orientation is matrices or multidimensional > arrays. In our case it is quite handy to select out a list of point in > an image by giving a list of their respective indices (e.g., stars). Hmmhh. I see that this again breaks with R/octave/matlab. One should not do so if there is no serious reason. It just makes life harder for every potential convert from any of these languages. A funktion take() would have served this purposed much better... but this is ofcourse all IMHO and I can see your point: You design it according to your (or your communities) requirements and different communities different needs... I am realizing that this must have been why cvxopt switched away from numarray/numeric. There slicing/indexing and '*' work as I would have expected: In [71]: a=matrix([1,2,3,4,5,6,7,8,9],size=(3,3)) In [72]: a Out[72]: <3x3 matrix, tc='d'> In [73]: b=matrix([1,2,3,4,5,6,7,8,9],size=(3,3)) In [74]: a*b Out[74]: <3x3 matrix, tc='d'> In [75]: print a 1.0000e+00 4.0000e+00 7.0000e+00 2.0000e+00 5.0000e+00 8.0000e+00 3.0000e+00 6.0000e+00 9.0000e+00 In [76]: print b 1.0000e+00 4.0000e+00 7.0000e+00 2.0000e+00 5.0000e+00 8.0000e+00 3.0000e+00 6.0000e+00 9.0000e+00 In [77]: print a*b 3.0000e+01 6.6000e+01 1.0200e+02 3.6000e+01 8.1000e+01 1.2600e+02 4.2000e+01 9.6000e+01 1.5000e+02 In [78]: print a[:,0] 1.0000e+00 2.0000e+00 3.0000e+00 In [79]: print a[0,1] 4.0 In [80]: print a[0,:] 1.0000e+00 4.0000e+00 7.0000e+00 > True, I do see that others may need the other view, but then the > question is which should get the convenience. Since Numeric/numarray > are primarily oriented towards multidimensional arrays (e.g., > operations are element-by-element rather than matrix) it seemed to make > sense to go this way for consistency, but I understand that there is > another side to this. It now seems very difficult for me to end up with a single numeric/matrix package that makes it into core python - which is at the same time very sad. Soeren From aisaac at american.edu Tue Jul 26 06:32:25 2005 From: aisaac at american.edu (Alan G Isaac) Date: Tue Jul 26 06:32:25 2005 Subject: [Numpy-discussion] design issues - octave 'incompatibilities' In-Reply-To: <1122355332.30698.160.camel@localhost> References: <1122134630.26083.74.camel@localhost><42E50C49.8080705@noaa.gov><1122355332.30698.160.camel@localhost> Message-ID: On Tue, 26 Jul 2005, Soeren Sonnenburg apparently wrote: > In my eyes 'array broadcasting' is confusing and should rather be in a > function like meshgrid and instead a*b should return > matrixmultiply(a,b) ... It does, if you work with matrices. Try import scipy help(scipy.mat) Coming from GAUSS I had the same initial reaction. It did not last long. hth, Alan Isaac From perry at stsci.edu Tue Jul 26 06:39:09 2005 From: perry at stsci.edu (Perry Greenfield) Date: Tue Jul 26 06:39:09 2005 Subject: [Numpy-discussion] design issues - octave 'incompatibilities' In-Reply-To: <1122352506.30698.136.camel@localhost> References: <1122134630.26083.74.camel@localhost> <42E29537.7080702@ucsd.edu> <1122230465.30698.72.camel@localhost> <89be641d30d1e246ea18d3733b6c3357@stsci.edu> <1122352506.30698.136.camel@localhost> Message-ID: On Jul 26, 2005, at 12:35 AM, Soeren Sonnenburg wrote: >> Since we deal with big data sets we have adopted 2 (no doubt to the >> consternation of many). > > hmmhh, there is no advantage with big datasets for any of the formats. > It is if you have to reorder the data, or use non-optimum iteration through the array. The first is definitely slower. >>> Do you know whether mixing slices and arrays will be supported at >>> some >>> point a[:,[0,1]] ? >>> >>> Soeren. >> >> No plans at the moment. We figured indexing was complicated enough as >> it was. I think Travis Oliphant did allow for this in Numeric3 (aka >> scipy.core); see the link below. But it may not mean what you think it >> should so you should check that out to see: >> >> http://cvs.sourceforge.net/viewcvs.py/numpy/Numeric3/PEP.txt? >> view=markup > > Hmmhh while we are at it, is there work on a consensus num* ? I've seen > the PEP for inclusion of numarray, now I see numeric3 and scipy and > also > cvxopt - are people actually converging towards a single num* package ? > That's what the goal of Numeric3 is as far as different underlying array engines. But if you mean a merging of the array/matrix viewpoints, I think you've answered your own question. Perry From Chris.Barker at noaa.gov Tue Jul 26 09:32:07 2005 From: Chris.Barker at noaa.gov (Chris Barker) Date: Tue Jul 26 09:32:07 2005 Subject: [Numpy-discussion] design issues - octave 'incompatibilities' In-Reply-To: <1122356650.30698.176.camel@localhost> References: <1122134630.26083.74.camel@localhost> <20e127854b47f13f1210e2f3574584f9@stsci.edu> <1122356650.30698.176.camel@localhost> Message-ID: <42E6653F.7040403@noaa.gov> Soeren Sonnenburg wrote: > Hmmhh. I see that this again breaks with R/octave/matlab. One should not > do so if there is no serious reason. It just makes life harder for every > potential convert from any of these languages. If you're looking for a matlab clone, use octave or psilab, or.... Speaking as an ex-matlab user, I much prefer the NumPy approach. The reason is that I very rarely did linear algebra, and generally used matlab as a general computational environment. I got very tired of having to type that "." before all my operators. I also love array broadcasting, it seems so much cleaner and efficient. When I moved from Matlab to NumPy, I missed these things: Integrated plotting: many years later, this is still weak, but at least for 2-d matplotlib is wonderful. The large library of math functions: SciPy is moving to fill that gap. Automatic handling of IEEE special values: numarray now does that pretty well. That's what I missed. What I gained was a far more powerful and expressive language, AND a more powerful and flexible array type. I don't miss MATLAB at all. In fact, you'll find me on the matplotlib mailing list, often repeating the refrain: matplotlib is NOT MATLAB nor should it be! > It now seems very difficult for me to end up with a single > numeric/matrix package that makes it into core python - That is probably true. > which is at the same time very sad. But I'm not sure we should be sad about it. What we all want is the package best suited to our own needs to be in the standard library. However, I'd rather the current situation than having a package poorly suited to my needs in the standard library. As this thread proves, there is no one kind of array package that would fit even most people's needs. However, there is some promise for the future: 1) SciPy base may serve to unify Numeric/numarray 2) Travis has introduced the "array interface" http://numeric.scipy.org/array_interface.html this would provide an efficient way for the various array and matrix packages to exchange data. It does have a hope of making it into the standard library, though even if it doesn't, if a wide variety of add-on packages use it, then the same thing is accomplished. If fact, if packages that don't implement an array type, but might use one (like PIL, wxPython, etc) accept this interface, then any package that implements it can be used together with them. 3) What about PEP 225: Elementwise/Objectwise Operators? It's listed under: Deferred, Abandoned, Withdrawn, and Rejected PEPs Which of those applied here? I had high hope for it one time. By the way, I had not seen cvxopt before, thanks for the link. Perhaps it would serve as a better base for a full-featured linear algebra package than Numeric/numarray. Ideally, it could use the array interface, for easy exchange of data with other packages. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From haase at msg.ucsf.edu Tue Jul 26 09:42:28 2005 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue Jul 26 09:42:28 2005 Subject: [Numpy-discussion] is numerical python only for prototyping ? Message-ID: <200507260941.37821.haase@msg.ucsf.edu> Hi, This is not sopposed to be an evil question; instead I'm hoping for the answer: "No, generally we get >=95% the speed of a pure C/fortran implementation" ;-) But as I am the strongest Python/numarray advocate in our group I get often the answer that Matlab is (of course) also very convenient but generally memory handling and overall execution performance is so bad that for final implementation one would generally have to reimplement in C. We are a bio-physics group at UCSF developping new algorithms for deconvolution (often in 3D). Our data sets are regularly bigger than several 100MB. When deciding for numarray I was assuming that the "Hubble Crowd" had a similar situation and all the operations are therefore very much optimized for this type of data. Is 95% a reasonable number to hope for ? I did wrap my own version of FFTW (with "plan-caching"), which should give 100% of the C-speed. But concerns arise from expression like "a=b+c*a" (think "convenience"!): If a,b,c are each 3D-datastacks creation of temporary data-arrays for 'c*a' AND then also for 'b+...' would have to be very costly. (I think this is at least happening for Numeric - I don't know about Matlab and numarray) Hoping for comments, Thanks Sebastian Haase UCSF, Sedat Lab From verveer at embl-heidelberg.de Tue Jul 26 10:15:15 2005 From: verveer at embl-heidelberg.de (Peter Verveer) Date: Tue Jul 26 10:15:15 2005 Subject: [Numpy-discussion] is numerical python only for prototyping ? In-Reply-To: <200507260941.37821.haase@msg.ucsf.edu> References: <200507260941.37821.haase@msg.ucsf.edu> Message-ID: On 26 Jul 2005, at 18:41, Sebastian Haase wrote: > Hi, > This is not sopposed to be an evil question; instead I'm hoping for the > answer: "No, generally we get >=95% the speed of a pure C/fortran > implementation" ;-) you won't, generally. Question is: since you are certainly not going to gain an order of a magnitude doing it in C, do you really care? > But as I am the strongest Python/numarray advocate in our group I get > often > the answer that Matlab is (of course) also very convenient but > generally > memory handling and overall execution performance is so bad that for > final > implementation one would generally have to reimplement in C. Well its true that implementations in C will be faster. And memory handling in Numeric/numarray can be a pain since the tendency is to create and destroy a lot of arrays if you are not careful. > We are a bio-physics group at UCSF developping new algorithms for > deconvolution (often in 3D). Our data sets are regularly bigger than > several > 100MB. When deciding for numarray I was assuming that the "Hubble > Crowd" had > a similar situation and all the operations are therefore very much > optimized > for this type of data. Funny you mention that example. I did my PhD in exactly the same field (considering you are from Sedats lab I guess you are in exactly the same field as I was/am, i.e. fluorescence microscopy. What are you guys up to these days?) and I developed all my algorithms in C at the time. Now, about 7 years later, I returned to the field to re-implement and extend some of my old algorithms for use with microscopy data that can consist of multiple sets, each several 100MB at least. Now I use python with numarray, and I am actualy quite happy with. I am pushing it by using up to 2GB of memory, (per process, after splitting the problem up and distributing it on a cluster...), but it works. I am sure I could squeeze maybe a factor of two or three in terms of speed and memory usage by rewriting in C, but that is currently not worth my time. So I guess that counts as using numarray as a prototyping environment, but the result is also suitable for production use. > Is 95% a reasonable number to hope for ? I did wrap my own version of > FFTW > (with "plan-caching"), which should give 100% of the C-speed. That should help a lot, as the standard FFTs that come with Numarray/Numeric suck big time. I do use them, but have to go through all kind of tricks to get some decent memory usage in 32bit floating point. The FFT array module is in fact very badly written for use with large multi-dimensional data sets. > But concerns > arise from expression like "a=b+c*a" (think "convenience"!): If a,b,c > are > each 3D-datastacks creation of temporary data-arrays for 'c*a' AND > then also > for 'b+...' would have to be very costly. (I think this is at least > happening > for Numeric - I don't know about Matlab and numarray) That is indeed a problem, although I think in your case you maybe limited by your FFTs anyway, at least in terms of speed. One thing you should consider is replacing expressions such as ' c= a + b' with add(a, b, c). If you do that cleverly you can avoid quite some memory allocations and you 'should' get closer to C. That does not solve everything though: 1) Complex expressions still need to be broken up in sequences of operations which is likely slower then iterating once over you array and do the expression at each point. 2) Unfortunately not all numarray functions support an output array (maybe only the ufuncs?). This can be a big problem, as then temporary arrays must be allocated. (It sure was a problem for me.) You can of course always re-implement the parts that are critical in C and wrap them (as you did with FFTW). In fact, I think numarray now provides a relatively easy way to write ufuncs which would allow you to write a single python function in C for complex expressions. > Hoping for comments, Hope this gives some insights. I guess I have had similar experiences, there are definitely some limits to the use of numarray/Numeric that could be relieved, for instance by having a consistent implementation of output arrays. That would allow writing algorithms where you could very strictly control the allocation and de-allocation of arrays, which would be a big help for working with large arrays. Cheers, Peter PS. I would not mind to hear a bit about your experiences doing the big deconvolutions in numarray/Numeric, but that may not be a good topic for this list. > Thanks > Sebastian Haase > UCSF, Sedat Lab > -- Dr Peter J Verveer European Molecular Biology Laboratory Meyerhofstrasse 1 D-69117 Heidelberg Germany Tel. +49 6221 387 8245 Fax. +49 6221 397 8306 From perry at stsci.edu Tue Jul 26 10:21:26 2005 From: perry at stsci.edu (Perry Greenfield) Date: Tue Jul 26 10:21:26 2005 Subject: [Numpy-discussion] is numerical python only for prototyping ? In-Reply-To: <200507260941.37821.haase@msg.ucsf.edu> References: <200507260941.37821.haase@msg.ucsf.edu> Message-ID: <940f0530c22ba28fc2d655a59c9e5a44@stsci.edu> On Jul 26, 2005, at 12:41 PM, Sebastian Haase wrote: > Hi, > This is not sopposed to be an evil question; instead I'm hoping for the > answer: "No, generally we get >=95% the speed of a pure C/fortran > implementation" ;-) > But as I am the strongest Python/numarray advocate in our group I get > often > the answer that Matlab is (of course) also very convenient but > generally > memory handling and overall execution performance is so bad that for > final > implementation one would generally have to reimplement in C. > We are a bio-physics group at UCSF developping new algorithms for > deconvolution (often in 3D). Our data sets are regularly bigger than > several > 100MB. When deciding for numarray I was assuming that the "Hubble > Crowd" had > a similar situation and all the operations are therefore very much > optimized > for this type of data. > Is 95% a reasonable number to hope for ? I did wrap my own version of > FFTW > (with "plan-caching"), which should give 100% of the C-speed. But > concerns > arise from expression like "a=b+c*a" (think "convenience"!): If a,b,c > are > each 3D-datastacks creation of temporary data-arrays for 'c*a' AND > then also > for 'b+...' would have to be very costly. (I think this is at least > happening > for Numeric - I don't know about Matlab and numarray) > Is it speed or memory usage you are worried about? Where are you actually seeing unacceptable performance? Offhand, I would say the temporaries are not likely to be serious speed issues (unless you running out of memory). We did envision at some point (we haven't done it yet) of recognizing situations where the temporaries could be reused. As for 95% speed, it's not what required for our work (I think that what an acceptable speed ratio is depends on the problem). In many cases being within 50% is good enough except for heavily used things where it would need to be faster. But yes, we do plan on (and do indeed actually use it now) for large data sets where speed is important. We generally don't compare the numarray speed to C speed all the time (if we were writing C equivalents every time, that would defeat the purpose of using Python :-). Perhaps you could give a more specific example with some measurements? I don't think I would promise anyone that all one's code could be done in Python. Undoubtedly, there are going to be some cases where coding in C or similar is going to be needed. I'd just argue that Python let's you keep as much as possible in a higher level language and a little as necessary in a low level language such as C. Perry From rbastian at club-internet.fr Tue Jul 26 13:17:02 2005 From: rbastian at club-internet.fr (=?iso-8859-15?q?Ren=E9=20Bastian?=) Date: Tue Jul 26 13:17:02 2005 Subject: [Numpy-discussion] convolve Message-ID: <05072610581402.00755@rbastian> Hello, in the actual form convolve(a,b)==convolve(b,a)[::-1] but following the mathematics, one should obtain convolve(a,b)==convolve(b,a) can anyone explain me the reason ? thanks -- Ren? Bastian http://www.musiques-rb.org From sheltraw at unm.edu Tue Jul 26 16:10:07 2005 From: sheltraw at unm.edu (Daniel Sheltraw) Date: Tue Jul 26 16:10:07 2005 Subject: [Numpy-discussion] fft output and input storage Message-ID: Hello all Would someone please tell me how negative and positive frequencies are stored in the output of the Numerical Python fft? Cheers, Daniel From focke at slac.stanford.edu Tue Jul 26 17:11:20 2005 From: focke at slac.stanford.edu (Warren Focke) Date: Tue Jul 26 17:11:20 2005 Subject: [Numpy-discussion] fft output and input storage In-Reply-To: References: Message-ID: >>> print FFT.fft.__doc__ fft(a, n=None, axis=-1) Will return the n point discrete Fourier transform of a. n defaults to the length of a. If n is larger than a, then a will be zero-padded to make up the difference. If n is smaller than a, the first n items in a will be used. The packing of the result is "standard": If A = fft(a, n), then A[0] contains the zero-frequency term, A[1:n/2+1] contains the positive-frequency terms, and A[n/2+1:] contains the negative-frequency terms, in order of decreasingly negative frequency. So for an 8-point transform, the frequencies of the result are [ 0, 1, 2, 3, 4, -3, -2, -1]. This is most efficient for n a power of two. This also stores a cache of working memory for different sizes of fft's, so you could theoretically run into memory problems if you call this too many times with too many different n's. >>> Hmph. The style guide says I should've put a blank line on the end of that docstring. But I don't think there was a style guide then. w On Tue, 26 Jul 2005, Daniel Sheltraw wrote: > Hello all > > Would someone please tell me how negative and positive > frequencies are > stored in the output of the Numerical Python fft? > > Cheers, > Daniel > > > ------------------------------------------------------- > SF.Net email is sponsored by: Discover Easy Linux Migration Strategies > from IBM. Find simple to follow Roadmaps, straightforward articles, > informative Webcasts and more! Get everything you need to get up to > speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From rbastian at club-internet.fr Tue Jul 26 21:53:01 2005 From: rbastian at club-internet.fr (=?iso-8859-15?q?Ren=E9=20Bastian?=) Date: Tue Jul 26 21:53:01 2005 Subject: [Numpy-discussion] convolve In-Reply-To: <05072610581402.00755@rbastian> References: <05072610581402.00755@rbastian> Message-ID: <05072706514600.00755@rbastian> Le Mardi 26 Juillet 2005 10:58, Ren? Bastian a ?crit : > Hello, > > in the actual form > > convolve(a,b)==convolve(b,a)[::-1] > > but following the mathematics, > one should obtain > > convolve(a,b)==convolve(b,a) > > can anyone explain me the reason ? I think the reasons are the modes other than FULL and the re-use of "correlate" Are this good reasons ? > > thanks -- Ren? Bastian http://www.musiques-rb.org/Ecchym/voiture.html From oliphant at ee.byu.edu Wed Jul 27 01:03:09 2005 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed Jul 27 01:03:09 2005 Subject: [Numpy-discussion] design issues - octave 'incompatibilities' In-Reply-To: <1122356650.30698.176.camel@localhost> References: <1122134630.26083.74.camel@localhost> <20e127854b47f13f1210e2f3574584f9@stsci.edu> <1122356650.30698.176.camel@localhost> Message-ID: <42E73F90.402@ee.byu.edu> Soeren Sonnenburg wrote: >I am realizing that this must have been why cvxopt switched away from >numarray/numeric. There slicing/indexing and '*' work as I would have >expected: > > cvxopt uses it's own classes because they did not feel that a general purpose array was needed. They wanted to define a matrix class with sparse matrix and dense matrix sub-classes. In fact, cvxopt's matrix classes can not be used as ubiquitously as Numeric/numarray arrays. Everything is not a matrix. In fact, I would like to see more general linear algebra routines that allow people to more naturally deal with (for example) six-dimensional linear operators mapping from a three-dimensional space to a three-dimensional space. Currently, you are forced to perform an artificial row-scanning procedure just to interface with matrix libraries. Scipy can help with this kind of thing. I do not see cvxopt as a competing array implementation. At some point, hopefully cvxopt will be integrated with scipy. I am continually looking for feasible ways to make scipy more attractive to contributors. Everybody benefits when their is a standard infrastructure. For example, there are sparse matrices in SciPy. If cvxopt has better sparse matrix objects, I would love to use them. Hopefully, the array interface will assist on a more abstract scale so that memory re-use can occur for at least the dense cvxopt matrices. >It now seems very difficult for me to end up with a single >numeric/matrix package that makes it into core python - which is at the > > >same time very sad. > > Their are several issues here. But, yes a Matrix object will always be a separate object just as quaternions should be because they represent an interpretation to a memory block. In Numeric/numarray the focus is on generic multidimensional arrays. Therefore numeric operators must be element-by element. Note that Numeric does have a Matrix object that allows you to use '*' to represent matrix multiplication. It's only problem is that passing this object to a function usually returns an array again instead of a Matrix. -Travis From perry at stsci.edu Wed Jul 27 09:20:24 2005 From: perry at stsci.edu (Perry Greenfield) Date: Wed Jul 27 09:20:24 2005 Subject: [Numpy-discussion] "Using Python for Interactive Data Analysis" tutorial available Message-ID: <0b142e5bdd9a4f5996f579e0e6afb751@stsci.edu> A tutorial that illustrates how Python can be used to analyze and visualize astronomical data (much like one can do in IDL) has been written and now is available at: http://www.scipy.org/wikis/topical_software/Tutorial While much of the focus is on astronomical data, most of it probably could serve as a general tutorial as well if sections on more advanced pyfits usage are skipped (only a few pages). There are appendices that contrast Python/numarray with IDL, I'd be happy to include the equivalent for Matlab if anyone who is familiar with both would like to write such appendices. Suggestions for improvements are welcome. Perry From aisaac at american.edu Wed Jul 27 11:19:10 2005 From: aisaac at american.edu (Alan G Isaac) Date: Wed Jul 27 11:19:10 2005 Subject: [Numpy-discussion] convolve Message-ID: On Tue, 26 Jul 2005, Ren? Bastian apparently wrote: > convolve(a,b)==convolve(b,a)[::-1] I do not see this behavior. Can you post a self-contained example? Thanks, Alan Isaac From sheltraw at unm.edu Wed Jul 27 11:28:09 2005 From: sheltraw at unm.edu (Daniel Sheltraw) Date: Wed Jul 27 11:28:09 2005 Subject: [Numpy-discussion] loading Numeric module Message-ID: Hello all When I try doing a "from Numeric import *" at the Python command line I get the error message "ImportError: No module named Numeric" so obviously the module is not in the path known to command line Python. However when the same import statement is part of a Python script file the module is found correctly. How do I fix this? Is there a way to give the full path within the import statement (when I try I get a syntax error message)? Thanks, Daniel From rkern at ucsd.edu Wed Jul 27 11:33:06 2005 From: rkern at ucsd.edu (Robert Kern) Date: Wed Jul 27 11:33:06 2005 Subject: [Numpy-discussion] loading Numeric module In-Reply-To: References: Message-ID: <42E7D329.4070400@ucsd.edu> Daniel Sheltraw wrote: > Hello all > > When I try doing a "from Numeric import *" at the Python command line I get > the error message "ImportError: No module named Numeric" so obviously > the module is not in the path known to command line Python. However when > the same import statement is part of a Python script file the module is > found correctly. How do I fix this? Are you sure that the Python interpreter is the same in both instances? Print sys.path before importing in both instances to see what paths are being searched. > Is there a way to give the full path > within the import statement (when I try I get a syntax error message)? Not really, no. -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From sheltraw at unm.edu Wed Jul 27 12:04:15 2005 From: sheltraw at unm.edu (Daniel Sheltraw) Date: Wed Jul 27 12:04:15 2005 Subject: [Numpy-discussion] 2-D FFT input/ouput storage Message-ID: Hello All Now that I have Numeric loading properly at the Python command line I went in search of some docs on the 2-D FFT. How are the input and output to the forward and inverse 2-D FFTs stored? Is it simply a 2-D version of the 1-D storage? When I do "print FFT.inverse_fft2d.__doc__" I get no helpful info about storage. Thanks, Daniel From aisaac at american.edu Wed Jul 27 13:09:17 2005 From: aisaac at american.edu (Alan G Isaac) Date: Wed Jul 27 13:09:17 2005 Subject: [Numpy-discussion] convolve In-Reply-To: <05072710171601.00755@rbastian> References: <05072710171601.00755@rbastian> Message-ID: On Wed, 27 Jul 2005, Ren? Bastian apparently wrote: > Outcommenting > if len(data) < len(kernel): > data, kernel = kernel, data > in convolve.py gives the correct result. The author is not listed for this numarray module, so we cannot directly ask why this behavior was chosen. You might use Numeric.convolve instead of numarray.convolve.Convolve.convolve: Numeric gives the expected result here. fwiw, Alan Isaac From aisaac at american.edu Wed Jul 27 13:16:05 2005 From: aisaac at american.edu (Alan G Isaac) Date: Wed Jul 27 13:16:05 2005 Subject: [Numpy-discussion] loading Numeric module In-Reply-To: References: Message-ID: On Wed, 27 Jul 2005, Daniel Sheltraw apparently wrote: > Is there a way to give the full path within the import > statement (when I try I get a syntax error message)? import sys sys.path.append("/path/to/myfiles") import mystuff hth, Alan Isaac From rkern at ucsd.edu Wed Jul 27 13:44:20 2005 From: rkern at ucsd.edu (Robert Kern) Date: Wed Jul 27 13:44:20 2005 Subject: [Numpy-discussion] convolve In-Reply-To: References: <05072710171601.00755@rbastian> Message-ID: <42E7F1BE.2010800@ucsd.edu> Alan G Isaac wrote: > On Wed, 27 Jul 2005, Ren? Bastian apparently wrote: > >>Outcommenting >>if len(data) < len(kernel): >> data, kernel = kernel, data >>in convolve.py gives the correct result. > > The author is not listed for this numarray module, > so we cannot directly ask why this behavior was chosen. This behavior was chosen for performance reasons. Since convolution is commutative, it doesn't matter what order the arguments come in. I do not see the behavior Ren? is claiming. Perhaps Ren? can tell us what version of numarray he is using? or provide an example? I am using CVS as of a week ago, I think. In [10]: import numarray In [11]: import numarray.convolve In [12]: a = numarray.arange(10) + 1. In [13]: b = numarray.arange(5) + 2. In [14]: c = a - 1. In [15]: numarray.convolve.convolve(a,b) Out[15]: array([ 2., 7., 16., 30., 50., 70., 90., 110., 130., 150., 148., 133., 104., 60.]) In [16]: numarray.convolve.convolve(b,a) Out[16]: array([ 2., 7., 16., 30., 50., 70., 90., 110., 130., 150., 148., 133., 104., 60.]) In [17]: numarray.convolve.convolve(a,c) Out[17]: array([ 0., 1., 4., 10., 20., 35., 56., 84., 120., 165., 210., 244., 266., 275., 270., 250., 214., 161., 90.]) In [18]: numarray.convolve.convolve(c,a) Out[18]: array([ 0., 1., 4., 10., 20., 35., 56., 84., 120., 165., 210., 244., 266., 275., 270., 250., 214., 161., 90.]) -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From aisaac at american.edu Wed Jul 27 13:58:11 2005 From: aisaac at american.edu (Alan G Isaac) Date: Wed Jul 27 13:58:11 2005 Subject: [Numpy-discussion] convolve In-Reply-To: <42E7F1BE.2010800@ucsd.edu> References: <05072710171601.00755@rbastian> <42E7F1BE.2010800@ucsd.edu> Message-ID: On Wed, 27 Jul 2005, Robert Kern apparently wrote: > I do not see the behavior Ren? is claiming. Perhaps Ren? > can tell us what version of numarray he is using? I see it in version 1.1, for the example he provided. Cheers, Alan Isaac From rkern at ucsd.edu Wed Jul 27 14:46:12 2005 From: rkern at ucsd.edu (Robert Kern) Date: Wed Jul 27 14:46:12 2005 Subject: [Numpy-discussion] convolve In-Reply-To: References: <05072710171601.00755@rbastian> <42E7F1BE.2010800@ucsd.edu> Message-ID: <42E80089.3090901@ucsd.edu> Alan G Isaac wrote: > On Wed, 27 Jul 2005, Robert Kern apparently wrote: > >>I do not see the behavior Ren? is claiming. Perhaps Ren? >>can tell us what version of numarray he is using? > > I see it in version 1.1, > for the example he provided. What example? Either his message has not yet propogated to me or the mailing list archives, or he sent it to you privately. The problem does not seem to exist any longer in CVS. -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From Chris.Barker at noaa.gov Wed Jul 27 14:47:24 2005 From: Chris.Barker at noaa.gov (Chris Barker) Date: Wed Jul 27 14:47:24 2005 Subject: [Numpy-discussion] Suggested Addition to the new array PEP Message-ID: <42E800B0.7050803@noaa.gov> Hi all, but particularly Travis, I just took the time to read the new array PEP: http://numeric.scipy.org/PEP.txt It looks great. I do have one suggestion for an addition method: change_type(NewType) # or some other name: Change the type that the bytes in the array represent, without changing the data at. Dimensions will change if the new type has a different itemsize than the old. This would be the equivalent of: A = Numeric.fromstring(A.tostring(), NewType) but without the extra copies of the data. When would anyone want this? I've written a function that does exactly this with Numeric arrays in order to efficiently slice, dice and re-arange a bunch of binary data files. The files consist of a bunch of C structs written sequentially to a file. In this case the structs are: (long, long, char) I have a file which is N of these structs written sequentially to a file, I can do: NumBytes = 9 # number of bytes in a record # Read the file data = fromstring(file.read(NumBytes*NumTimesteps*NumLEs),UnsignedInt8) file.close() data.shape = (NumTimesteps,NumLEs,NumBytes) # extract LE data: LEs = data[:,:,:8].copy() # extract the flags flags = data[:,:,8].copy() del data # here I can create the Int32 array without making a copy: # changetype returns a rank-1 array changetype(LEs,Int32) # It now gets reshaped to fit what it represents. LEs.shape = (NumTimesteps,NumLEs,2) I think it would only work in the general case with a contiguous array. All this really requires is to re-set the attributes of the array. with Numeric, the code is (for contiguous arrays): NewDescription = PyArray_DescrFromType(typecode); NewElementSize = NewDescription->elsize; if (TotalBytes % NewElementSize > 0){ PyErr_SetString (PyExc_ValueError, "The input array is the wrong size for the requested type"); return NULL; } Array->descr = NewDescription; // does the old descr need to be freed?? how?? Array->nd = 1; Array->strides[0] = NewElementSize; Array->dimensions[0] = TotalBytes / NewElementSize; The core By the way, I'm also not seeing tostring() or tobuffer() listed in the PEP -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From python-ml at nn7.de Wed Jul 27 15:02:11 2005 From: python-ml at nn7.de (Soeren Sonnenburg) Date: Wed Jul 27 15:02:11 2005 Subject: [Numpy-discussion] design issues - octave 'incompatibilities' In-Reply-To: <42E6653F.7040403@noaa.gov> References: <1122134630.26083.74.camel@localhost> <20e127854b47f13f1210e2f3574584f9@stsci.edu> <1122356650.30698.176.camel@localhost> <42E6653F.7040403@noaa.gov> Message-ID: <1122483294.28910.13.camel@localhost> On Tue, 2005-07-26 at 09:30 -0700, Chris Barker wrote: > Soeren Sonnenburg wrote: > > Hmmhh. I see that this again breaks with R/octave/matlab. One should not > > do so if there is no serious reason. It just makes life harder for every > > potential convert from any of these languages. > > If you're looking for a matlab clone, use octave or psilab, or.... No I am not, I am sold to python. > Speaking as an ex-matlab user, I much prefer the NumPy approach. The > reason is that I very rarely did linear algebra, and generally used > matlab as a general computational environment. I got very tired of > having to type that "." before all my operators. I also love array > broadcasting, it seems so much cleaner and efficient. Well coming from machine learning I am used to having just matrices. > When I moved from Matlab to NumPy, I missed these things: > > Integrated plotting: many years later, this is still weak, but at least > for 2-d matplotlib is wonderful. I agree here. [no single numeric package in python] > But I'm not sure we should be sad about it. What we all want is the > package best suited to our own needs to be in the standard library. > However, I'd rather the current situation than having a package poorly > suited to my needs in the standard library. As this thread proves, there > is no one kind of array package that would fit even most people's needs. > > However, there is some promise for the future: > > 1) SciPy base may serve to unify Numeric/numarray > > 2) Travis has introduced the "array interface" > > http://numeric.scipy.org/array_interface.html > > this would provide an efficient way for the various array and matrix > packages to exchange data. It does have a hope of making it into the > standard library, though even if it doesn't, if a wide variety of add-on > packages use it, then the same thing is accomplished. If fact, if > packages that don't implement an array type, but might use one (like > PIL, wxPython, etc) accept this interface, then any package that > implements it can be used together with them. > > 3) What about PEP 225: Elementwise/Objectwise Operators? > It's listed under: > > Deferred, Abandoned, Withdrawn, and Rejected PEPs > > Which of those applied here? I had high hope for it one time. > > By the way, I had not seen cvxopt before, thanks for the link. Perhaps > it would serve as a better base for a full-featured linear algebra > package than Numeric/numarray. Ideally, it could use the array > interface, for easy exchange of data with other packages. Actually that would be nice. Having had a closer look at cvxopt that might suit *my* needs more, but having an interface to get cvxopt matrices -> numarray/numeric without trouble to get certain potentially missing functionality would be nice. Thanks, Soeren From python-ml at nn7.de Wed Jul 27 15:03:02 2005 From: python-ml at nn7.de (Soeren Sonnenburg) Date: Wed Jul 27 15:03:02 2005 Subject: [Numpy-discussion] design issues - octave 'incompatibilities' In-Reply-To: <42E73F90.402@ee.byu.edu> References: <1122134630.26083.74.camel@localhost> <20e127854b47f13f1210e2f3574584f9@stsci.edu> <1122356650.30698.176.camel@localhost> <42E73F90.402@ee.byu.edu> Message-ID: <1122497705.28910.41.camel@localhost> On Wed, 2005-07-27 at 02:02 -0600, Travis Oliphant wrote: > Soeren Sonnenburg wrote: > > >I am realizing that this must have been why cvxopt switched away from > >numarray/numeric. There slicing/indexing and '*' work as I would have > >expected: > > > > > cvxopt uses it's own classes because they did not feel that a general > purpose array was needed. They wanted to define a matrix class with > sparse matrix and dense matrix sub-classes. In fact, cvxopt's matrix > classes can not be used as ubiquitously as Numeric/numarray arrays. > Everything is not a matrix. In fact, I would like to see more general > linear algebra routines that allow people to more naturally deal with > (for example) six-dimensional linear operators mapping from a > three-dimensional space to a three-dimensional space. Currently, you > are forced to perform an artificial row-scanning procedure just to > interface with matrix libraries. Scipy can help with this kind of thing. Hmmhh, > I do not see cvxopt as a competing array implementation. At some > point, hopefully cvxopt will be integrated with scipy. I am continually > looking for feasible ways to make scipy more attractive to > contributors. Everybody benefits when their is a standard > infrastructure. For example, there are sparse matrices in SciPy. If > cvxopt has better sparse matrix objects, I would love to use them. > Hopefully, the array interface will assist on a more abstract scale so > that memory re-use can occur for at least the dense cvxopt matrices. I guess we now observe the different communities different expectations problem :/ In any case I agree that a standard infrastructure is very desirable. However it might come at a cost one might not want to pay, but still at least conversion functions from say cvxopt <-> numarray are worth spending time on. > >It now seems very difficult for me to end up with a single > >numeric/matrix package that makes it into core python - which is at the > > > > > >same time very sad. > > > > > > Their are several issues here. But, yes a Matrix object will always be > a separate object just as quaternions should be because they represent > an interpretation to a memory block. In Numeric/numarray the focus is > on generic multidimensional arrays. Therefore numeric operators must be > element-by element. OK. > Note that Numeric does have a Matrix object that allows you to use '*' > to represent matrix multiplication. It's only problem is that passing > this object to a function usually returns an array again instead of a > Matrix. So the cvxopt approach is pretty valid, doing everything for matrices as they do, but allowing other types as 'int' etc.. Soeren From python-ml at nn7.de Wed Jul 27 15:03:06 2005 From: python-ml at nn7.de (Soeren Sonnenburg) Date: Wed Jul 27 15:03:06 2005 Subject: [Numpy-discussion] design issues - octave 'incompatibilities' In-Reply-To: References: <1122134630.26083.74.camel@localhost> <42E29537.7080702@ucsd.edu> <1122230465.30698.72.camel@localhost> <89be641d30d1e246ea18d3733b6c3357@stsci.edu> <1122352506.30698.136.camel@localhost> Message-ID: <1122500138.28910.59.camel@localhost> On Tue, 2005-07-26 at 09:37 -0400, Perry Greenfield wrote: > On Jul 26, 2005, at 12:35 AM, Soeren Sonnenburg wrote: [...] > > Hmmhh while we are at it, is there work on a consensus num* ? I've seen > > the PEP for inclusion of numarray, now I see numeric3 and scipy and > > also > > cvxopt - are people actually converging towards a single num* package ? > > > That's what the goal of Numeric3 is as far as different underlying > array engines. But if you mean a merging of the array/matrix > viewpoints, I think you've answered your own question. As Alan and Travis suggested Matrices scipy.mat, one could have matrices more or less independently of num* if one could convert them back and forth. Doing so would even allow for doing things differently, like getting submatrices, allowing for real matrix-matrix multiplication etc. However that of course kills consistency. Soeren From rkern at ucsd.edu Wed Jul 27 15:03:08 2005 From: rkern at ucsd.edu (Robert Kern) Date: Wed Jul 27 15:03:08 2005 Subject: [Numpy-discussion] 2-D FFT input/ouput storage In-Reply-To: References: Message-ID: <42E8045A.3090609@ucsd.edu> Daniel Sheltraw wrote: > Hello All > > Now that I have Numeric loading properly at the Python command line I > went in search of some docs on the 2-D FFT. How are the input and output > to the forward and inverse 2-D FFTs stored? Is it simply a 2-D version > of the 1-D storage? When I do "print FFT.inverse_fft2d.__doc__" I get no > helpful info about storage. The 2-D layout follows directly as a consequence of the 1-D layout and the definition of a 2-D FFT. In [14]: import FFT In [15]: def my2dfft(A): ....: A = array(A, Complex) ....: for i in range(A.shape[0]): ....: A[i] = FFT.fft(A[i]) ....: for i in range(A.shape[-1]): ....: A[:,i] = FFT.fft(A[:,i]) ....: return A ....: In [16]: A = reshape(arange(16.), (4,4)) In [17]: my2dfft(A) Out[17]: array([[ 120. +0.j, -8. +8.j, -8. +0.j, -8. -8.j], [ -32.+32.j, 0. +0.j, 0. +0.j, 0. +0.j], [ -32. +0.j, 0. +0.j, 0. +0.j, 0. +0.j], [ -32.-32.j, 0. +0.j, 0. +0.j, 0. +0.j]]) In [18]: FFT.fft2d(A) Out[18]: array([[ 120. +0.j, -8. +8.j, -8. +0.j, -8. -8.j], [ -32.+32.j, 0. +0.j, 0. +0.j, 0. +0.j], [ -32. +0.j, 0. +0.j, 0. +0.j, 0. +0.j], [ -32.-32.j, 0. +0.j, 0. +0.j, 0. +0.j]]) -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From python-ml at nn7.de Wed Jul 27 15:03:10 2005 From: python-ml at nn7.de (Soeren Sonnenburg) Date: Wed Jul 27 15:03:10 2005 Subject: [Numpy-discussion] design issues - octave 'incompatibilities' In-Reply-To: <42E5CCAF.4000201@ucsd.edu> References: <1122134630.26083.74.camel@localhost> <42E50C49.8080705@noaa.gov> <1122355332.30698.160.camel@localhost> <42E5CCAF.4000201@ucsd.edu> Message-ID: <1122499368.28910.52.camel@localhost> On Mon, 2005-07-25 at 22:39 -0700, Robert Kern wrote: > Soeren Sonnenburg wrote: > > [snip] > > > In my eyes 'array broadcasting' is confusing and should rather be in a > > function like meshgrid and instead a*b should return > > matrixmultiply(a,b) ... > > Spend some time with it. It will probably grow on you. Numeric is not [...] > Again, Numeric is a package for arrays, not just linear algebra. Please > spend some more time with Python and Numeric before deciding that they > must be changed to match your preconceptions. I realize that I just need plain matrices and operations on them so I am probably better off with cvxopt at that point. > > I realize that with lists it is ok to grow them via slicing. > > > > x=[] > > x[0]=1 > > IndexError: list assignment index out of range > > x[0:0]=[1] > > x > > [1] > > > > that seems not to work with numarray ... or ? > > > > y=array() > > y[0]=1 > > TypeError: object does not support item assignment > > y[0:0]=array([1]) > > TypeError: object does not support item assignment > > Python lists are designed to grow dynamically. Their memory is > preallocated so that growing them is on average pretty cheap. Numeric > arrays are not, nor will they be. Well I don't claim that this is/must be efficient. It is just what I would have expected to work if I use standard python arrays and now numarrays. Soeren. From aisaac at american.edu Wed Jul 27 18:19:04 2005 From: aisaac at american.edu (Alan G Isaac) Date: Wed Jul 27 18:19:04 2005 Subject: [Numpy-discussion] convolve In-Reply-To: <42E80089.3090901@ucsd.edu> References: <05072710171601.00755@rbastian> <42E7F1BE.2010800@ucsd.edu> <42E80089.3090901@ucsd.edu> Message-ID: >> On Wed, 27 Jul 2005, Robert Kern apparently wrote: >>> I do not see the behavior Ren? is claiming. Perhaps Ren? >>> can tell us what version of numarray he is using? > Alan G Isaac wrote: >> I see it in version 1.1, >> for the example he provided. On Wed, 27 Jul 2005, Robert Kern apparently wrote: > What example? Either his message has not yet propogated to me or the > mailing list archives, or he sent it to you privately. The problem does > not seem to exist any longer in CVS. My bad. It was indeed off list. It is (essentially) replicated below. Alan ================================================== #-*-coding:latin-1-*- #from Numeric import arrayrange,asarray #from Numeric import convolve from numarray import arrayrange,asarray from numarray.convolve.Convolve import convolve def explo0(): a=arrayrange(10.) b=asarray([1., 0., 2., -1., 3., -2.]) c=convolve(a, b) print c d=convolve(b, a) print d if __name__=="__main__": explo0() From rkern at ucsd.edu Wed Jul 27 18:24:11 2005 From: rkern at ucsd.edu (Robert Kern) Date: Wed Jul 27 18:24:11 2005 Subject: [Numpy-discussion] convolve In-Reply-To: References: <05072710171601.00755@rbastian> <42E7F1BE.2010800@ucsd.edu> <42E80089.3090901@ucsd.edu> Message-ID: <42E83383.3010308@ucsd.edu> Alan G Isaac wrote: >>>On Wed, 27 Jul 2005, Robert Kern apparently wrote: >>> >>>>I do not see the behavior Ren? is claiming. Perhaps Ren? >>>>can tell us what version of numarray he is using? > >>Alan G Isaac wrote: >> >>>I see it in version 1.1, >>>for the example he provided. > > On Wed, 27 Jul 2005, Robert Kern apparently wrote: > >>What example? Either his message has not yet propogated to me or the >>mailing list archives, or he sent it to you privately. The problem does >>not seem to exist any longer in CVS. > > My bad. > It was indeed off list. > It is (essentially) replicated below. Whatever problem that once existed is now fixed in CVS. The example you give works fine. -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From aisaac at american.edu Wed Jul 27 18:25:04 2005 From: aisaac at american.edu (Alan G Isaac) Date: Wed Jul 27 18:25:04 2005 Subject: [Numpy-discussion] convolve In-Reply-To: <42E83383.3010308@ucsd.edu> References: <05072710171601.00755@rbastian> <42E7F1BE.2010800@ucsd.edu> <42E80089.3090901@ucsd.edu> <42E83383.3010308@ucsd.edu> Message-ID: On Wed, 27 Jul 2005, Robert Kern apparently wrote: > Whatever problem that once existed is now fixed in CVS. Great! Thanks, Alan From jochen at fhi-berlin.mpg.de Wed Jul 27 23:33:07 2005 From: jochen at fhi-berlin.mpg.de (=?iso-8859-1?Q?Jochen_K=FCpper?=) Date: Wed Jul 27 23:33:07 2005 Subject: [Numpy-discussion] LAPACK/BLAS for Fedora core 4 i386 ? In-Reply-To: <42DE9BEC.9070902@noaa.gov> (Chris Barker's message of "Wed, 20 Jul 2005 11:46:04 -0700") References: <42DE9BEC.9070902@noaa.gov> Message-ID: "Chris Barker" writes: > Does anyone have a suggestion for how to add the addional lapack > stuff I need, while still using atlas stuff where possible? See the ATLAS installation manual. Greetings, Jochen -- Einigkeit und Recht und Freiheit http://www.Jochen-Kuepper.de Libert?, ?galit?, Fraternit? GnuPG key: CC1B0B4D (Part 3 you find in my messages before fall 2003.) From rbastian at club-internet.fr Thu Jul 28 00:25:17 2005 From: rbastian at club-internet.fr (=?iso-8859-15?q?Ren=E9=20Bastian?=) Date: Thu Jul 28 00:25:17 2005 Subject: [Numpy-discussion] convolve In-Reply-To: References: <42E83383.3010308@ucsd.edu> Message-ID: <05072722060300.00753@rbastian> Le Jeudi 28 Juillet 2005 03:26, Alan G Isaac a ?crit : > On Wed, 27 Jul 2005, Robert Kern apparently wrote: > > Whatever problem that once existed is now fixed in CVS. Yes, it works ! Thanks, Ren? > > Great! > > Thanks, > Alan > > From Chris.Barker at noaa.gov Thu Jul 28 08:31:27 2005 From: Chris.Barker at noaa.gov (Chris Barker) Date: Thu Jul 28 08:31:27 2005 Subject: [Numpy-discussion] LAPACK/BLAS for Fedora core 4 i386 ? In-Reply-To: References: <42DE9BEC.9070902@noaa.gov> Message-ID: <42E8FA0B.3050107@noaa.gov> Jochen K?pper wrote: >>Does anyone have a suggestion for how to add the addional lapack >>stuff I need, while still using atlas stuff where possible? > > See the ATLAS installation manual. Thanks for the tip. Do you mean this? : """ ******** GETTING A FULL LAPACK LIB ************************** ATLAS does not provide a full lapack library. However, there is a simple way to get ATLAS to provide its faster LAPACK routines to a full LAPACK library. ATLAS's internal routines are distinct from LAPACK's, so it is safe to compile ATLAS's LAPACK routines directly into a netlib-style LAPACK library. First, obtain the LAPACK src from netlib and build the LAPACK library as normal. Then, in this directory (where you should have a liblapack.a), issue the following commands: mkdir tmp cd tmp ar x ../liblapack.a cp ../liblapack.a ar r ../liblapack.a *.o cd .. rm -rf tmp Just linking in ATLAS's liblapack.a first will not get you the best LAPACK performance, mainly because LAPACK's untuned ILAENV will be used instead of ATLAS's tuned one. """ If so, I did that (see a later post to this list). However, I found that I got about the same performance improvement doing this as I had gotten compiling against just the atlas BLAS and lapack_lite. It makes me wonder if I'm really getting the benefit of atlas lapack. I don't' have hard-core performance needs at the moment, so I'm done, but I'm still surprised I couldn't find a ready-to-go atlas rpm for Fedora Core 4. It actually makes me miss Gentoo. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From joachim at ee.ucla.edu Thu Jul 28 10:55:12 2005 From: joachim at ee.ucla.edu (Joachim Dahl) Date: Thu Jul 28 10:55:12 2005 Subject: [Numpy-discussion] Interfacing to other array classes Message-ID: <42E91B85.50701@ee.ucla.edu> I am one the authors of 'cvxopt', a Python package that includes simple matrix classes aimed at numerical linear algebra for convex solvers, and I saw the discussion on the list regarding interfacing to such classes. I think the idea of an advanced interface or buffer protocol to exchange information between simple array types without using (or knowing) the API of the different classes is great. What is the status of the new array interface? When it is ready for testing, I will definitely try it out. - Joachim From pieter at djinnit.com Thu Jul 28 14:38:45 2005 From: pieter at djinnit.com (Pieter van Beek) Date: Thu Jul 28 14:38:45 2005 Subject: [Numpy-discussion] New e-mail address Message-ID: *English follows below* Geachte relatie, Ooit (misschien recent, misschien al weer jaren geleden) heeft u e-mail verstuurd naar mijn voormalige e-mail adres: _vanbeek at q-factor.nl_. Bij deze wil ik u mededelen dat mijn e-mail adres is gewijzigd in pieter at djinnit.com. Gelieve deze wijziging door te voeren in uw adresbestand. Met vriendelijke groet, Pieter van Beek ------------------------------------------------------------------------ Dear relation, Once (maybe recently, maybe years ago) you have been sending mail to my former e-mail address: _vanbeek at q-factor.nl_. Hereby I'd like to inform you that my e-mail address has changed into pieter at djinnit.com. Please change my e-mail address in your address book. Kind regards, Pieter van beek -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Fri Jul 29 09:24:05 2005 From: Chris.Barker at noaa.gov (Chris Barker) Date: Fri Jul 29 09:24:05 2005 Subject: [Numpy-discussion] LAPACK/BLAS for Fedora core 4 i386 ? In-Reply-To: References: <42DE9BEC.9070902@noaa.gov> <42E8FA0B.3050107@noaa.gov> Message-ID: <42EA57D1.7010709@noaa.gov> Jochen K?pper wrote: > "Chris Barker" writes: >>Do you mean this? : > yes OK, so I've done what I can. > To find out you might have to look at the LAPACK functions you are > using and compare them against the ones provided by ATLAS. Well, I'm testing using LinearAlgebra.solve_linear_equations() from Numeric (and numarray). The reason I'm wondering is that on my last machine, a 1ghz PIII laptop running Gentoo, using the Gentoo provided atlas/lapack, I got a 6-7 times speed up over lapack_lite. On my new machine, a 2GHz Pentium M laptop running FC4, I got about a 2 times speed-up, which is nice, but not nearly as impressive. Another possible issue is that I used pre-complied binaries form the atlas site, which are a bit old, maybe I should compile myself. Any thoughts? -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From dd55 at cornell.edu Fri Jul 29 09:29:19 2005 From: dd55 at cornell.edu (Darren Dale) Date: Fri Jul 29 09:29:19 2005 Subject: [Numpy-discussion] LAPACK/BLAS for Fedora core 4 i386 ? In-Reply-To: <42EA57D1.7010709@noaa.gov> References: <42DE9BEC.9070902@noaa.gov> <42EA57D1.7010709@noaa.gov> Message-ID: <200507291228.38305.dd55@cornell.edu> On Friday 29 July 2005 12:22 pm, Chris Barker wrote: > > To find out you might have to look at the LAPACK functions you are > > using and compare them against the ones provided by ATLAS. > > Well, I'm testing using LinearAlgebra.solve_linear_equations() from > Numeric (and numarray). The reason I'm wondering is that on my last > machine, a 1ghz PIII laptop running Gentoo, using the Gentoo provided > atlas/lapack, I got a 6-7 times speed up over lapack_lite. > > On my new machine, a 2GHz Pentium M laptop running FC4, I got about a 2 > times speed-up, which is nice, but not nearly as impressive. > > Another possible issue is that I used pre-complied binaries form the > atlas site, which are a bit old, maybe I should compile myself. Any > thoughts? Maybe I don't understanding something, but isn't the point of ATLAS that the libraries are tuned at compile time for your specific setup? -- Darren From Chris.Barker at noaa.gov Fri Jul 29 11:48:14 2005 From: Chris.Barker at noaa.gov (Chris Barker) Date: Fri Jul 29 11:48:14 2005 Subject: [Numpy-discussion] LAPACK/BLAS for Fedora core 4 i386 ? In-Reply-To: <200507291228.38305.dd55@cornell.edu> References: <42DE9BEC.9070902@noaa.gov> <42EA57D1.7010709@noaa.gov> <200507291228.38305.dd55@cornell.edu> Message-ID: <42EA7964.5050806@noaa.gov> Darren Dale wrote: > Maybe I don't understanding something, but isn't the point of ATLAS that the > libraries are tuned at compile time for your specific setup? yes, but the binaries I downloaded are for the P4 processor, so I'm not sure how much more specific I can get. I'm going to give it a try. This does bring up a point, however. Why is lapack usually linked statically? I usually link statically so that I can move applications to a different system without worrying about what libraries are installed there. However, with lapack/blas, you want to use a library compiled for that particular machine, so it seems to make more sense to use dynamic ones. Besides, it would be easier to compare performance if all I had to do was drop a new *.so in place. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From matthew.brett at gmail.com Fri Jul 29 12:14:21 2005 From: matthew.brett at gmail.com (Matthew Brett) Date: Fri Jul 29 12:14:21 2005 Subject: [Numpy-discussion] LAPACK/BLAS for Fedora core 4 i386 ? In-Reply-To: <42EA7964.5050806@noaa.gov> References: <42DE9BEC.9070902@noaa.gov> <42EA57D1.7010709@noaa.gov> <200507291228.38305.dd55@cornell.edu> <42EA7964.5050806@noaa.gov> Message-ID: <1e2af89e0507291212235360f7@mail.gmail.com> Hi, > yes, but the binaries I downloaded are for the P4 processor, so I'm not > sure how much more specific I can get. I'm going to give it a try. There is some tuning that is specific to the cache, which might change things a bit. Is it possible your test code uses NaNs at some point? That will result in a huge slowdown on the P4 with the default ATLAS binaries: http://www.mrc-cbu.cam.ac.uk/Imaging/Common/spm_intel_tune.shtml#nan_problem Matthew From luszczek at cs.utk.edu Fri Jul 29 12:18:48 2005 From: luszczek at cs.utk.edu (Piotr Luszczek) Date: Fri Jul 29 12:18:48 2005 Subject: [Numpy-discussion] LAPACK/BLAS for Fedora core 4 i386 ? In-Reply-To: <42EA7964.5050806@noaa.gov> References: <42DE9BEC.9070902@noaa.gov> <42EA57D1.7010709@noaa.gov> <200507291228.38305.dd55@cornell.edu> <42EA7964.5050806@noaa.gov> Message-ID: <42EA80C3.2060601@cs.utk.edu> Chris Barker wrote: > Darren Dale wrote: > >> Maybe I don't understanding something, but isn't the point of ATLAS >> that the libraries are tuned at compile time for your specific setup? > > > yes, but the binaries I downloaded are for the P4 processor, so I'm not > sure how much more specific I can get. I'm going to give it a try. Unfortunately, you have to get very specific to get the most out of Atlas. The two things that come to mind are: - cache size which differs across various Pentium 4s/Xeons - clock speed The former means that you have to block operations on your matrices in the right way so you keep data in each level of memory hierarchy as long as possible. Clock speed matters because it introduces varying memory latencies so, for example, you might not be prefetching data at the right rate. That's why I usually compile Atlas without any builtin presets even though it takes very long. However, I also try the prebuilt binaries to make sure that Clint (Atlas' author) didn't turn some secret knob that I don't know about. Piotr From edcjones at comcast.net Sun Jul 31 20:57:33 2005 From: edcjones at comcast.net (Edward C. Jones) Date: Sun Jul 31 20:57:33 2005 Subject: [Numpy-discussion] numarray: PyArrayObject supports number protocols? Message-ID: <42ED9D73.1010400@comcast.net> I just saw in the docs, Section 14.2.4: It should be noted that unlike earlier versions of numarray, the present PyArrayObject structure is a first class python object, with full support for the number protocols in C. Does this mean that I can add two numarrays in C using "PyNumber_Add"?