From jmiller at stsci.edu Mon May 2 12:53:06 2005 From: jmiller at stsci.edu (Todd Miller) Date: Mon May 2 12:53:06 2005 Subject: [Numpy-discussion] numarray-1.3 manual Message-ID: <1115063516.30278.404.camel@halloween.stsci.edu> There's a new release of the numarray manual (1.3) on Source Forge here: http://sourceforge.net/project/showfiles.php?group_id=1369 The manual has new documentation on the Ufunc code generation API (used for porting scipy.special) as well as a number of corrections from Steven Walton, Rory Yorke, Russell Owen, and Olivier Bornet, and others. Thank you all for your efforts. Windows users may also notice numarray-1.3.1 binaries are now available. The new binaries are equivalent to numnarray-1.3.0 so there's no reason to upgrade if you already have 1.3.0. Regards, Todd From meesters at uni-mainz.de Tue May 3 02:01:11 2005 From: meesters at uni-mainz.de (Christian Meesters) Date: Tue May 3 02:01:11 2005 Subject: [Numpy-discussion] which module to use Message-ID: <7ac98c6f297c152a105e4c62c730d2b2@uni-mainz.de> Hi Numarray and Numeric are great modules. I was using them a lot in my scripts. But now the time has come that I would like to write a somewhat bigger application and perhaps even need to write extensions for one of the numerical modules or Scipy. Now, I'm wondering which module to choose: Numarray (my preferred choice so far), Numpy, or "Numeric3"? How far is the latter grown with respect to the other two? How soon will it replace Numpy in SciPy releases? How long will all three co-exist, when will the merger be finished? How long will Numarray and Numpy be supported once Numeric3 "is ready"? Squeezed into one question: What is the state of the art and how can people like me (at the borderline between users and developers) plan ahead? I know that some of these questions are hard if not impossible to answer, but this was really a burning issue to me, since after all the discussions here on the list I am a little clueless. TIA Cheers Christian From konrad.hinsen at laposte.net Tue May 3 03:25:03 2005 From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net) Date: Tue May 3 03:25:03 2005 Subject: [Numpy-discussion] which module to use In-Reply-To: <7ac98c6f297c152a105e4c62c730d2b2@uni-mainz.de> References: <7ac98c6f297c152a105e4c62c730d2b2@uni-mainz.de> Message-ID: On May 3, 2005, at 11:00, Christian Meesters wrote: > Numarray and Numeric are great modules. I was using them a lot in my > scripts. But now the time has come that I would like to write a > somewhat bigger application and perhaps even need to write extensions > for one of the numerical modules or Scipy. > Now, I'm wondering which module to choose: Numarray (my preferred > choice so far), Numpy, or "Numeric3"? How far is the latter grown with > respect to the other My advice would be to stick to the common interface of Numarray and NumPy if you can. If you can't - well, then you are facing a difficult choice. Konrad. From NadavH at VisionSense.com Tue May 3 04:39:12 2005 From: NadavH at VisionSense.com (Nadav Horesh) Date: Tue May 3 04:39:12 2005 Subject: [Numpy-discussion] A numarray bug? can not get help on array attributes Message-ID: <42776407.1070705@VisionSense.com> >>> import numarray as N >>> a = N.array((2.3, 4.3)) >>> N.__version__ '1.3.1' >>> help(a.stdev) Traceback (most recent call last): File "", line 1, in -toplevel- help(a.stdev) AttributeError: 'NumArray' object has no attribute 'stdev' >>> help(a.stddev) Traceback (most recent call last): File "", line 1, in -toplevel- help(a.stddev) File "/usr/local/lib/python2.4/site.py", line 328, in __call__ return pydoc.help(*args, **kwds) File "/usr/local/lib/python2.4/pydoc.py", line 1647, in __call__ self.help(request) File "/usr/local/lib/python2.4/pydoc.py", line 1691, in help else: doc(request, 'Help on %s:') File "/usr/local/lib/python2.4/pydoc.py", line 1475, in doc pager(title % desc + '\n\n' + text.document(object, name)) File "/usr/local/lib/python2.4/pydoc.py", line 297, in document if inspect.isroutine(object): return self.docroutine(*args) File "/usr/local/lib/python2.4/pydoc.py", line 1226, in docroutine if object.im_self: File "/usr/local/lib/python2.4/site-packages/numarray/generic.py", line 537, in __nonzero__ raise RuntimeError("An array doesn't make sense as a truth value. Use any(a) or all(a).") RuntimeError: An array doesn't make sense as a truth value. Use any(a) or all(a). >>> help(a.sum) Traceback (most recent call last): File "", line 1, in -toplevel- help(a.sum) File "/usr/local/lib/python2.4/site.py", line 328, in __call__ return pydoc.help(*args, **kwds) File "/usr/local/lib/python2.4/pydoc.py", line 1647, in __call__ self.help(request) File "/usr/local/lib/python2.4/pydoc.py", line 1691, in help else: doc(request, 'Help on %s:') File "/usr/local/lib/python2.4/pydoc.py", line 1475, in doc pager(title % desc + '\n\n' + text.document(object, name)) File "/usr/local/lib/python2.4/pydoc.py", line 297, in document if inspect.isroutine(object): return self.docroutine(*args) File "/usr/local/lib/python2.4/pydoc.py", line 1226, in docroutine if object.im_self: File "/usr/local/lib/python2.4/site-packages/numarray/generic.py", line 537, in __nonzero__ raise RuntimeError("An array doesn't make sense as a truth value. Use any(a) or all(a).") RuntimeError: An array doesn't make sense as a truth value. Use any(a) or all(a). >>> Nadav. From jmiller at stsci.edu Tue May 3 06:20:14 2005 From: jmiller at stsci.edu (Todd Miller) Date: Tue May 3 06:20:14 2005 Subject: [Numpy-discussion] A numarray bug? can not get help on array attributes In-Reply-To: <42776407.1070705@VisionSense.com> References: <42776407.1070705@VisionSense.com> Message-ID: <1115126354.6484.38.camel@halloween.stsci.edu> On Tue, 2005-05-03 at 07:44, Nadav Horesh wrote: > >>> import numarray as N > >>> a = N.array((2.3, 4.3)) > >>> N.__version__ > '1.3.1' > >>> help(a.stdev) > > Traceback (most recent call last): > File "", line 1, in -toplevel- > help(a.stdev) > AttributeError: 'NumArray' object has no attribute 'stdev' > >>> help(a.stddev) > > Traceback (most recent call last): > File "", line 1, in -toplevel- > help(a.stddev) > File "/usr/local/lib/python2.4/site.py", line 328, in __call__ > return pydoc.help(*args, **kwds) > File "/usr/local/lib/python2.4/pydoc.py", line 1647, in __call__ > self.help(request) > File "/usr/local/lib/python2.4/pydoc.py", line 1691, in help > else: doc(request, 'Help on %s:') > File "/usr/local/lib/python2.4/pydoc.py", line 1475, in doc > pager(title % desc + '\n\n' + text.document(object, name)) > File "/usr/local/lib/python2.4/pydoc.py", line 297, in document > if inspect.isroutine(object): return self.docroutine(*args) > File "/usr/local/lib/python2.4/pydoc.py", line 1226, in docroutine > if object.im_self: > File "/usr/local/lib/python2.4/site-packages/numarray/generic.py", > line 537, in __nonzero__ > raise RuntimeError("An array doesn't make sense as a truth value. > Use any(a) or all(a).") > RuntimeError: An array doesn't make sense as a truth value. Use any(a) > or all(a). > >>> help(a.sum) > In my opinion, this is a limitation (bug is maybe too strong) of pydoc, in that pydoc's docroutine method requires that an object be evaluable as a truth value and that the result be True. I think that's a common, but wrong, idiom. I believe it's widely recognized that array's don't make much sense as truth values because it leads to code like this: >>> a = array(5); b = array(8) >>> a & b array(0) >>> if a & b: ... print "foo!" foo! Numeric expressions like the above implicitly mean any(a & b) and appear to work but really contain subtle bugs. Hence, in numarray and Numeric3 (now or eventually), the exception: >>> if a & b: ... print "foo!" ... Traceback (most recent call last): ... RuntimeError: An array doesn't make sense as a truth value. Use any(a) or all(a). I think the pydoc docroutine() code should be changed to read: if object.im_self is not None: and that particular limitation will be removed. I submitted a patch. Regards, Todd From oliphant at ee.byu.edu Tue May 3 12:25:06 2005 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue May 3 12:25:06 2005 Subject: [Numpy-discussion] which module to use In-Reply-To: <7ac98c6f297c152a105e4c62c730d2b2@uni-mainz.de> References: <7ac98c6f297c152a105e4c62c730d2b2@uni-mainz.de> Message-ID: <4277CFB2.501@ee.byu.edu> Christian Meesters wrote: > Hi > > Numarray and Numeric are great modules. I was using them a lot in my > scripts. But now the time has come that I would like to write a > somewhat bigger application and perhaps even need to write extensions > for one of the numerical modules or Scipy. > Now, I'm wondering which module to choose: Numarray (my preferred > choice so far), Numpy, or "Numeric3"? How far is the latter grown with > respect to the other two? How soon will it replace Numpy in SciPy > releases? How long will all three co-exist, when will the merger be > finished? How long will Numarray and Numpy be supported once Numeric3 > "is ready"? > As I've reiterated I won't support NumPy once Numeric3 is ready, but others may, so it's hard to give definitive answers like that. > Squeezed into one question: What is the state of the art and how can > people like me (at the borderline between users and developers) plan > ahead? This unfortunately has been a problem for a long time. The "state of the art" has been a mixed bag for a while. I have been trying to push scipy as a package which makes the choice of numarray and/or Numeric not a big deal for some time. So, of course, I would suggest using scipy which currently uses Numeric (though parts of it can also use numarray). Recent versions of Numarray and Numeric do not copy data between the arrays so you can really use Numeric for scipy internals and numarray arrays on the outside. Within a few months, scipy will be moving towards what used to be called Numeric3 (the package is now called scipy.base). So, if you stick with scipy you will be using the latest (notice that current scipy adds a few of the features of Numarray already (like 1-d integer indexing and non-scalar coercion). So, on the python side I would use scipy calls. On the C-side, I would use the old NumPy C-API whenever possible. Good luck, -Travis O. From strawman at astraw.com Tue May 3 14:33:19 2005 From: strawman at astraw.com (Andrew Straw) Date: Tue May 3 14:33:19 2005 Subject: [Numpy-discussion] patch for Debian libc bug "feclearexcept() error on CPUs with SSE" Message-ID: <4277EDF0.6010105@astraw.com> Hi all, Not of interest to most, and you'll probably know it you need it: Debian users users of P4 machines using the atlas3-sse2 package may be interested in a set of .debs which fix the "feclearexcept() error on CPUs with SSE" bug [1] The .debs generated are available here: http://www.its.caltech.edu/~astraw/coding.html 1. http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=279294 From meesters at uni-mainz.de Wed May 4 05:00:10 2005 From: meesters at uni-mainz.de (Christian Meesters) Date: Wed May 4 05:00:10 2005 Subject: [Numpy-discussion] which module to use In-Reply-To: <4277CFB2.501@ee.byu.edu> References: <7ac98c6f297c152a105e4c62c730d2b2@uni-mainz.de> <4277CFB2.501@ee.byu.edu> Message-ID: <7fe0c19185ca96790c7ae61faede0237@uni-mainz.de> Hi Thanks Travis and Konrad for your answers. On 3 May 2005, at 21:23, Travis Oliphant wrote: > I have been trying to push scipy as a package which makes the choice > of numarray and/or Numeric not a big deal for some time. Oh, yes. I noticed that. Thank you for this. > > Within a few months, scipy will be moving towards what used to be > called Numeric3 (the package is now called scipy.base). So, if you > stick with scipy you will be using the latest (notice that current > scipy adds a few of the features of Numarray already (like 1-d integer > indexing and non-scalar coercion). > So, on the python side I would use scipy calls. Although some things have been said before, this makes it really clear. I think I'm not going for Matlab then ... ;-) Thanks, Christian From postmaster at interface.ru Thu May 5 05:36:21 2005 From: postmaster at interface.ru (postmaster at interface.ru) Date: Thu May 5 05:36:21 2005 Subject: [Numpy-discussion] InterScan NT Alert Message-ID: Sender, InterScan has detected virus(es) in your e-mail attachment. Date: Thu, 05 May 2005 16:36:18 +0400 Method: Mail From: To: oldman at silicontaiga.ru File: noname.htm Action: clean failed - deleted Virus: HTML_Netsky.P From postmaster at interface.ru Thu May 5 05:36:25 2005 From: postmaster at interface.ru (postmaster at interface.ru) Date: Thu May 5 05:36:25 2005 Subject: [Numpy-discussion] InterScan NT Alert Message-ID: Sender, InterScan has detected virus(es) in your e-mail attachment. Date: Thu, 05 May 2005 16:36:18 +0400 Method: Mail From: To: oldman at silicontaiga.ru File: message.scr Action: clean failed - deleted Virus: WORM_NETSKY.P From confirm-s2-IhAOe02AQnsgWDuiE_hAZuBBgho-numpy-discussion=lists.sourceforge.net at yahoogroups.com Thu May 5 07:53:15 2005 From: confirm-s2-IhAOe02AQnsgWDuiE_hAZuBBgho-numpy-discussion=lists.sourceforge.net at yahoogroups.com (Yahoo! Groups) Date: Thu May 5 07:53:15 2005 Subject: [Numpy-discussion] Please confirm your request to join IErussian Message-ID: <1115304546.23.91860.w123@yahoogroups.com> Hello numpy-discussion at lists.sourceforge.net, We have received your request to join the IErussian group hosted by Yahoo! Groups, a free, easy-to-use community service. This request will expire in 7 days. TO BECOME A MEMBER OF THE GROUP: 1) Go to the Yahoo! Groups site by clicking on this link: http://groups.yahoo.com/i?i=IhAOe02AQnsgWDuiE_hAZuBBgho&e=numpy-discussion%40lists%2Esourceforge%2Enet (If clicking doesn't work, "Cut" and "Paste" the line above into your Web browser's address bar.) -OR- 2) REPLY to this email by clicking "Reply" and then "Send" in your email program If you did not request, or do not want, a membership in the IErussian group, please accept our apologies and ignore this message. Regards, Yahoo! Groups Customer Care Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/ From haase at msg.ucsf.edu Mon May 9 18:00:39 2005 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Mon May 9 18:00:39 2005 Subject: [Numpy-discussion] Re: ANN: numarray-1.2.3 -- segfault in in my C program In-Reply-To: <200503041146.50434.haase@msg.ucsf.edu> References: <1109869619.19608.16.camel@halloween.stsci.edu> <1109948612.19608.210.camel@halloween.stsci.edu> <200503041146.50434.haase@msg.ucsf.edu> Message-ID: <200505091758.11466.haase@msg.ucsf.edu> Hi, Now I'm back to this problem. In the meantime I upgraded again: >>> na.__version__ '1.4.0' All my python2.2 + numarray + my-extensions work. (Debian Woody) Only my C++ program segfaults. It looked to me like a problem with NA_NewAllFromBuffer -- was there any recent change ? (I saw that I was using a bytestride of 0 - but it used to work and giving a 1 for that argument still segfaults -- is 0 OK ??) Now I noticed in the gdb traceback that I chokes is the function deferred_libnumarray_init (( ((#0 0x406d68d5 in PyObject_GetAttrString () from ((/usr/lib/libpython2.2.so.0.0 #1 0x410f905e in ((deferred_libnumarray_init () at Src/libnumarraymodule.c:149 Is this function the standard function that always get called to init numarray (in embedded Python AND using "normal" python) ? Have you tested this with Python2.2 ? (See for more info below == I did follow Todd's suggestions 1 & 2, but hope not having to recompile python from source.) Thanks Sebastian Haase On Friday 04 March 2005 11:46, Sebastian Haase wrote: > On Friday 04 March 2005 07:03, Todd Miller wrote: > From what you're showing me, it looks like libnumarray initialization > > > is failing which makes me suspect a corrupted numarray installation. > > It understood it saying it fails in MyApp::OnInit omx_app.cpp:519 while > doing: NA_NewAllFromBuffer > (ndim=3, shape=0xbffff2e4, type=tFloat32, bufferObject=0x8a03988, > byteoffset=0, > bytestride=0, byteorder=0, aligned=1, writeable=1) > > the "initialize libnumarray"-stuff is in the 20 lines above that. > Do you use > NA_NewAllFromBuffer > anywhere ? > > Thanks, > Sebastian Haase > > > Here are some things to try: > > > > 1. Completely delete your existing site-packages/numarray. Also delete > > numarray/build then re-install numarray. > > > > 2. Delete and re-install your extensions. In principle, > > numarray-1.2.3 is supposed to be binary compatible with numarray-1.1.1 > > but maybe I'm mistaken. > > > > 3. Hopefully you won't get this far but... a python which works well > > with gdb can be built from source using ./configure --with-pydebug. So > > a debug scenario is something like: > > > > % tar zxf Python-2.2.3.tar.gz > > % cd Python-2.2.3 > > % ./configure --with-pydebug --prefix=$HOME > > % make > > % make install > > > > % cd .. > > % tar zxf numarray-1.2.3.tar.gz > > % cd numarray-1.2.3 > > % python setup.py install > > > > % cd .. > > % tar zxf your_stuff.tar.gz > > % cd your_stuff > > % python setup.py install > > > > This makes a debug Python installed in $HOME/bin, $HOME/lib, and > > $HOME/include. This process is useful for compiling Python itself and > > extensions with "-g -O0" and hence gdb works better. Besides > > appropriate compiler switches, debug Python also has more robust object > > memory management and better tracked reference counting. > > > > Debug like this: > > > > % setenv PATH $HOME/bin:$PATH # export if you use bash > > % rehash > > > > % gdb python > > (gdb) run > > > > >>> > > > > > > (gdb) l , # to see some code > > (gdb) p > > (gdb) up # Move up the stack frame to see where the bogus value came > > from > > > > Regards, > > Todd > > > > On Thu, 2005-03-03 at 14:40, Sebastian Haase wrote: > > > Hi, > > > After upgrading from numarray 1.1 (now 1.2.3) > > > We get a Segmentation fault in our C++ program on Linux > > > (python2.2,gcc2.95) , gdb says this: > > > > > > Program received signal SIGSEGV, Segmentation fault. > > > [Switching to Thread 1087498336 (LWP 8279)] > > > 0x406d68d5 in PyObject_GetAttrString () from > > > /usr/lib/libpython2.2.so.0.0 (gdb) where > > > #0 0x406d68d5 in PyObject_GetAttrString () from > > > /usr/lib/libpython2.2.so.0.0 #1 0x410f905e in > > > deferred_libnumarray_init () at Src/libnumarraymodule.c:149 #2 > > > 0x410f98a8 in NA_NewAllFromBuffer (ndim=3, shape=0xbffff2e4, > > > type=tFloat32, bufferObject=0x8a03988, byteoffset=0, > > > bytestride=0, byteorder=0, aligned=1, writeable=1) at Src/ > > > libnumarraymodule.c:636 > > > #3 0x0805b159 in MyApp::OnInit (this=0x8108f50) at omx_app.cpp:519 > > > #4 0x4026f616 in wxEntry () from /jws30/haase/PrLin0/wxGtkLibs/ > > > libwx_gtk-2.4.so > > > #5 0x0805a91a in main (argc=1, argv=0xbffff414) at omx_app.cpp:247 > > > > > > > > > To initialize libnumarray I was using this: > > > { > > > // import_libnumarray(); > > > { > > > PyObject *module = > > > PyImport_ImportModule("numarray.libnumarray"); if (!module) > > > Py_FatalError("Can't import module > > > 'numarray.libnumarray'"); if (module != NULL) { > > > PyObject *module_dict = PyModule_GetDict(module); > > > PyObject *c_api_object = > > > PyDict_GetItemString(module_dict, "_C_API"); > > > if (PyCObject_Check(c_api_object)) { > > > libnumarray_API = (void > > > **)PyCObject_AsVoidPtr(c_api_object); } else { > > > Py_FatalError("Can't get API for module > > > 'numarray.libnumarray'"); } > > > } > > > } > > > } > > > > > > Any idea ? > > > > > > Thanks, > > > Sebastian Haase From simon at arrowtheory.com Mon May 9 18:15:30 2005 From: simon at arrowtheory.com (Simon Burton) Date: Mon May 9 18:15:30 2005 Subject: [Numpy-discussion] numarray memmap and gc Message-ID: <20050510111256.187b2f5c.simon@arrowtheory.com> Hi all, an odd problem that has bitten me a couple of times: I build an array from a memmap, the memmap goes out of scope and gets GC'ed (closed), then inexplicable data errors occur: numarray.libnumarray.error: NA_updateDataPtr: error getting read buffer data ptr The solution I came up with was to assign the memmap to a unique attribute of the array ('source_mmap_dont_gc_me'). Then it only gets GC'ed when all it's arrays are GC'ed. So, is it reasonable to make this standard in numarray: slices remember their memmaps and NumArrays remember their buffers (I assume the latter already happens). bye, Simon. -- Simon Burton, B.Sc. Licensed PO Box 8066 ANU Canberra 2601 Australia Ph. 61 02 6249 6940 http://arrowtheory.com From MAILER-DAEMON at dnrc.bell-labs.com Mon May 9 18:26:33 2005 From: MAILER-DAEMON at dnrc.bell-labs.com (MAILER-DAEMON at dnrc.bell-labs.com) Date: Mon May 9 18:26:33 2005 Subject: [Numpy-discussion] Returned mail: User unknown Message-ID: <200505100124.VAA21751@milk.dnrc.bell-labs.com> The original message was received at Mon, 9 May 2005 21:24:14 -0400 (EDT) from guard.research.bell-labs.com [135.104.2.10] ----- The following addresses had permanent fatal errors ----- ----- Transcript of session follows ----- ... while talking to chair.dnrc.bell-labs.com.: >>> RCPT To: <<< 550 ... User unknown 550 ... User unknown -------------- next part -------------- An embedded message was scrubbed... From: unknown sender Subject: no subject Date: no date Size: 38 URL: From numpy-discussion at lists.sourceforge.net Mon May 9 21:23:59 2005 From: numpy-discussion at lists.sourceforge.net (numpy-discussion at lists.sourceforge.net) Date: Tue, 10 May 2005 10:23:59 +0900 Subject: Thanks! Message-ID: <200505100124.j4A1O0ET001862@hoemail2.lucent.com> ------------------ Virus Warning Message (on the network) Found virus WORM_MYTOB.Y in file document.pif The file document.pif is moved to /var/quarantine/virus/virSCXFk3g6P. This is a machine-generated message, please do not reply via e-mail. If you have questions, please contact the Lucent Help Desk at +1 888 300 0770. --------------------------------------------------------- -------------- next part -------------- Mail transaction failed. Partial message is available. -------------- next part -------------- ------------------ Virus Warning Message (on the network) document.pif is removed from here because it contains a virus. --------------------------------------------------------- From faltet at carabos.com Tue May 10 04:31:06 2005 From: faltet at carabos.com (Francesc Altet) Date: Tue May 10 04:31:06 2005 Subject: [Numpy-discussion] PyTables 1.0 released Message-ID: <200505101326.12110.faltet@carabos.com> ========================= Announcing PyTables 1.0 ========================= The Carabos crew is very proud to announce the immediate availability of **PyTables release 1.0**. On this release you will find a series of exciting new features, being the most important the Undo/Redo capabilities, support for objects (and indexes!) with more than 2**31 rows, better I/O performance for Numeric objects, new time datatypes (useful for time-stamping fields), support for Octave HDF5 files and improved support for HDF5 native files. What it is ========== **PyTables** is a package for managing hierarchical datasets and designed to efficiently cope with extremely large amounts of data (with support for full 64-bit file addressing). It features an object-oriented interface that, combined with C extensions for the performance-critical parts of the code, makes it a very easy-to-use tool for high performance data storage and retrieval. It is built on top of the HDF5 library and the numarray package, and provides containers for both heterogeneous data (``Table``) and homogeneous data (``Array``, ``EArray``) as well as containers for keeping lists of objects of variable length (like Unicode strings or general Python objects) in a very efficient way (``VLArray``). It also sports a series of filters allowing you to compress your data on-the-fly by using different compressors and compression enablers. But perhaps the more interesting features are its powerful browsing and searching capabilities that allow you to perform data selections over heterogeneous datasets exceeding gigabytes of data in just tenths of second. Besides, the PyTables I/O is buffered, implemented in C and carefully tuned so that you can reach much better performance with PyTables than with your own home-grown wrappings to the HDF5 library. Changes more in depth ===================== Improvements: - New Undo/Redo feature (i.e. integrated support for undoing and/or redoing actions). This functionality lets you to put marks in specific places of your data operations, so that you can make your HDF5 file pop back (undo) to a specific mark (for example for inspecting how your data looked at that point). You can also go forward to a more recent marker (redo). You can even do jumps to the marker you want using just one instruction. - Reading Numeric objects from ``*Array`` and ``Table`` (Numeric columns) objects have a 50-100x speedup. With that, Louis Wicker reported that a speed of 350 MB/s can be achieved with Numeric objects (on a SGI Altix with a Raid 5 disk array) while with numarrays, this speed approaches 900 MB/s. This improvement has been possible mainly due to a nice recipe from Todd Miller. Thanks Todd! - With PyTables 1.0 you can finally create Tables, EArrays and VLArrays with more than 2**31 (~ 2 thousand millions) rows, as well as retrieve them. Before PyTables 1.0, retrieving data on these beasts was not well supported, in part due to limitations in some slicing functions in Python (that rely on 32-bit adressing). So, we have made the necessary modifications in these functions to support 64-bit indexes and integrated them into PyTables. As a result, our tests shows that this feature works just fine. - As a consequence of the above, you can now index columns of tables with more than 2**31 rows. For instance, indexes have been created for integer columns with 10**10 (yeah, 10 thousand million) rows in less than 1 hour using an Opteron @ 1.6 GHz system (~ 1 hour and half with a Xeon Intel32 @ 2.5 GHz platform). Enjoy! - Now PyTables supports the native HDF5 time types, both 32-bit signed integer and 64-bit fixed point timestamps. They are mapped to ``Int32`` and ``Float64`` values for easy manipulation. See the documentation for the ``Time32Col`` and ``Time64Col`` classes. - Massive internal reorganization of the methods that deal with the hierarchy. Hopefully, that will enable a better understanding of the code for anybody wanting to add/modify features. - The opening and copying of files with large number of objects has been made faster by correcting a typo in ``Table._open()``. Thanks to Ashley Walsh for sending a patch for this. - Now, one can modify rank-0 (scalar) ``EArray`` datasets. Thanks to Norbert Nemec for providing a patch for this. - You are allowed from this version on to add non-valid natural naming names as node or attribute names. A warning is issued to warn (but not forbid) you in such a case. Of course, you have to use the ``getattr()`` function so as to access such invalid natural names. - The indexes of ``Table`` and ``*Array`` datasets can be of long type besides of integer type. However, indexes in slices are still restricted to regular integer type. - The concept of ``READ_ONLY`` system attributes has disappeared. You can change them now at your own risk! However, you still cannot remove or rename system attributes. - Now, one can do reads in-between write loops using ``table.row`` instances. This is thanks to a decoupling in I/O buffering: now there is a buffer for reading and other for writing, so that no collisions take place anymore. Fixes #1186892. - Support for Octave HDF5 output format. Even complex arrays are supported. Thanks to Edward C. Jones for reporting this format. Backward-incompatible changes: - The format of indexes has been changed and indexes in files created with PyTables pre-1.0 versions are ignored now. However, ``ptrepack`` can still save your life because it is able to convert your old files into the new indexing format. Also, if you copy the affected tables to other locations (by using ``Leaf.copy()``), it will regenerate your indexes with the new format for you. - The API has changed a little bit (nothing serious) for some methods. See ``RELEASE-NOTES.txt`` for more details. Bug fixes: - Added partial support for native HDF5 chunked datasets. They can be read now, and even extended, but only along the first extensible dimension. This limitation may be removed when multiple extensible dimensions are supported in PyTables. - Formerly, when the name of a column in a table was subsumed in another column name, PyTables crashed while retrieving information of the former column. That has been fixed. - A bug prevented the use of indexed columns of tables that were in other hierarchical level than root. This is solved now. - When a ``Group`` was renamed you were not able to modify its attributes. This has been fixed. - When whether ``Table.modifyColumns()`` or ``Table.modifyRows()`` were called, a subsequent call to ``Table.flush()`` didn't really flush the modified data to disk. This works as intended now. - Fixed some issues when iterating over ``*Array`` objects using the ``List`` or ``Tuple`` flavor. Important note for Python 2.4 and Windows users =============================================== If you are willing to use PyTables with Python 2.4 in Windows platforms, you will need to get the HDF5 library compiled for MSVC 7.1, aka .NET 2003. It can be found at: ftp://ftp.ncsa.uiuc.edu/HDF/HDF5/current/bin/windows/5-164-win-net.ZIP Users of Python 2.3 on Windows will have to download the version of HDF5 compiled with MSVC 6.0 available in: ftp://ftp.ncsa.uiuc.edu/HDF/HDF5/current/bin/windows/5-164-win.ZIP Where can PyTables be applied? ============================== PyTables is not designed to work as a relational database competitor, but rather as a teammate. If you want to work with large datasets of multidimensional data (for example, for multidimensional analysis), or just provide a categorized structure for some portions of your cluttered RDBS, then give PyTables a try. It works well for storing data from data acquisition systems (DAS), simulation software, network data monitoring systems (for example, traffic measurements of IP packets on routers), very large XML files, or for creating a centralized repository for system logs, to name only a few possible uses. What is a table? ================ A table is defined as a collection of records whose values are stored in fixed-length fields. All records have the same structure and all values in each field have the same data type. The terms "fixed-length" and "strict data types" seem to be quite a strange requirement for a language like Python that supports dynamic data types, but they serve a useful function if the goal is to save very large quantities of data (such as is generated by many scientific applications, for example) in an efficient manner that reduces demand on CPU time and I/O resources. What is HDF5? ============= For those people who know nothing about HDF5, it is a general purpose library and file format for storing scientific data made at NCSA. HDF5 can store two primary objects: datasets and groups. A dataset is essentially a multidimensional array of data elements, and a group is a structure for organizing objects in an HDF5 file. Using these two basic constructs, one can create and store almost any kind of scientific data structure, such as images, arrays of vectors, and structured and unstructured grids. You can also mix and match them in HDF5 files according to your needs. Platforms ========= We are using Linux on top of Intel32 as the main development platform, but PyTables should be easy to compile/install on other UNIX machines. This package has also been successfully compiled and tested on a FreeBSD 5.4 with Opteron64 processors, a UltraSparc platform with Solaris 7 and Solaris 8, a SGI Origin3000 with Itanium processors running IRIX 6.5 (using the gcc compiler), Microsoft Windows and MacOSX (10.2 although 10.3 should work fine as well). In particular, it has been thoroughly tested on 64-bit platforms, like Linux-64 on top of an Intel Itanium, AMD Opteron (in 64-bit mode) or PowerPC G5 (in 64-bit mode) where all the tests pass successfully. Regarding Windows platforms, PyTables has been tested with Windows 2000 and Windows XP (using the Microsoft Visual C compiler), but it should also work with other flavors as well. Web site ======== Go to the PyTables web site for more details: http://pytables.sourceforge.net/ To know more about the company behind the PyTables development, see: http://www.carabos.com/ Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. ---- **Enjoy data!** -- The PyTables Team From jmiller at stsci.edu Tue May 10 05:16:11 2005 From: jmiller at stsci.edu (Todd Miller) Date: Tue May 10 05:16:11 2005 Subject: [Numpy-discussion] Re: ANN: numarray-1.2.3 -- segfault in in my C program In-Reply-To: <200505091758.11466.haase@msg.ucsf.edu> References: <1109869619.19608.16.camel@halloween.stsci.edu> <1109948612.19608.210.camel@halloween.stsci.edu> <200503041146.50434.haase@msg.ucsf.edu> <200505091758.11466.haase@msg.ucsf.edu> Message-ID: <1115727212.4479.27.camel@jaytmiller.comcast.net> On Mon, 2005-05-09 at 17:58 -0700, Sebastian Haase wrote: > Hi, > Now I'm back to this problem. In the meantime I upgraded again: > >>> na.__version__ > '1.4.0' > All my python2.2 + numarray + my-extensions work. (Debian Woody) > Only my C++ program segfaults. It looked to me like a problem with > NA_NewAllFromBuffer -- was there any recent change ? Yes there was. You can undo it in your private CVS checkout like this: % cd numarray/Src % cvs update -j 1.3 -1.2 libnumarray.ch % cd .. % python setup.py install --genapi --selftest > (I saw that I was using a bytestride of 0 - but it used to work and giving a 1 > for that argument still segfaults -- is 0 OK ??) I don't know for sure. It seems like it should be... just leads to 0 strides. > Now I noticed in the gdb traceback that I chokes is the function > deferred_libnumarray_init > (( > ((#0 0x406d68d5 in PyObject_GetAttrString () from > ((/usr/lib/libpython2.2.so.0.0 #1 0x410f905e in > ((deferred_libnumarray_init () at Src/libnumarraymodule.c:149 > > Is this function the standard function that always get called to init numarray > (in embedded Python AND using "normal" python) ? Yes and no. deferred_libnumarray_init() is really for stuff that can't be done at libnumarray.so init time. This is because of circular dependencies. The "real" numarray init function is either "import numarray" in Python or "import_array()" in C. After that, certain calls into the C-API trigger the deferred initialization. deferred_libnumarray_init() is never called directly by application code. > Have you tested this with Python2.2 ? Yes, using the numarray unit tests. NewAllFromBuffer is also used directly by the numarray ufuncs, so it is tested to a degree by numarray itself. My gut feel is you're hitting an embedding problem, but my gut is only so accurate. > > (See for more info below == I did follow Todd's suggestions 1 & 2, but hope > not having to recompile python from source.) That's really not hard. A 5-10 minute exercise usually. Regards, Todd From jmiller at stsci.edu Tue May 10 05:16:31 2005 From: jmiller at stsci.edu (Todd Miller) Date: Tue May 10 05:16:31 2005 Subject: [Numpy-discussion] numarray memmap and gc In-Reply-To: <20050510111256.187b2f5c.simon@arrowtheory.com> References: <20050510111256.187b2f5c.simon@arrowtheory.com> Message-ID: <1115727214.4479.29.camel@jaytmiller.comcast.net> On Tue, 2005-05-10 at 11:12 +1000, Simon Burton wrote: > Hi all, > > an odd problem that has bitten me a couple of times: > I build an array from a memmap, the memmap goes out of > scope and gets GC'ed (closed), then inexplicable data errors > occur: > > numarray.libnumarray.error: NA_updateDataPtr: error getting read buffer data ptr > > The solution I came up with was to assign the memmap to a unique attribute > of the array ('source_mmap_dont_gc_me'). Then it only gets GC'ed when all it's arrays are GC'ed. > > So, is it reasonable to make this standard in numarray: slices remember their > memmaps and NumArrays remember their buffers (I assume the latter already happens). I think we should make MemmapSlices remember their parent Memmaps. The NumArray remembers the MemmapSlice (buffer) as long as it lives, and the slice in turn would keep the Memmap alive. I have hit a similar problem with Tkinter buttons, twice, so I know the aggravation involved. I'm nervous about sticking in a circular reference (the Memmap already remembers it's slices) but I suppose the garbage collector was added for exactly this case. I'm not entirely sure how hard it's going to be to add. From haase at msg.ucsf.edu Tue May 10 10:33:36 2005 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue May 10 10:33:36 2005 Subject: [Numpy-discussion] Re: ANN: numarray-1.2.3 -- segfault in in my C program In-Reply-To: <1115727212.4479.27.camel@jaytmiller.comcast.net> References: <1109869619.19608.16.camel@halloween.stsci.edu> <200505091758.11466.haase@msg.ucsf.edu> <1115727212.4479.27.camel@jaytmiller.comcast.net> Message-ID: <200505101031.23907.haase@msg.ucsf.edu> Todd, I would be very thankful if you could compile and run this program: #include #include int main() { Py_Initialize(); PyObject *pymodMain = PyImport_ImportModule("__main__"); PyObject* globals = PyModule_GetDict(pymodMain); PyEval_InitThreads(); // import_libnumarray(); PyObject *module = PyImport_ImportModule("numarray.libnumarray"); if (!module) Py_FatalError("Can't import module 'numarray.libnumarray'"); if (module != NULL) { PyObject *module_dict = PyModule_GetDict(module); PyObject *c_api_object = PyDict_GetItemString(module_dict, "_C_API"); if (PyCObject_Check(c_api_object)) { libnumarray_API = (void **)PyCObject_AsVoidPtr(c_api_object); } else { Py_FatalError("Can't get API for module 'numarray.libnumarray'"); } } float *colMapsRGB = new float[4 * 3 * 256]; int shape[3]= { 4, 3, 256 }; PyObject *obj = PyBuffer_FromReadWriteMemory(colMapsRGB, 4*3 *256 * sizeof(float)); PyObject *obj2 =(PyObject*) NA_NewAllFromBuffer( 3, shape, tFloat32, obj, 0, 0, NA_ByteOrder(), 1, 1/*writable*/) ; return 0; } I compile with: g++-2.95 -o testnumarrayembedded -Wall -finline-functions -g -I/usr/include/ python2.2 -I/jws30/haase/myCVS/numarray/Include testnumarrayembedded.cpp -pthread -lpython2.2 It runs through with setting PYTHONPATH to my old numarray(1.1) directory. And now with numarray CVS it now doesn't even find the library... $ ./testnumarrayembedded Fatal Python error: Can't import module 'numarray.libnumarray' Aborted Still, my non-embedded python shell runs all fine - your CVS-update (-j...) suggestion didn't seem to make a difference. Thanks, Sebastian On Tuesday 10 May 2005 05:13, Todd Miller wrote: > On Mon, 2005-05-09 at 17:58 -0700, Sebastian Haase wrote: > > Hi, > > > > Now I'm back to this problem. In the meantime I upgraded again: > > >>> na.__version__ > > > > '1.4.0' > > All my python2.2 + numarray + my-extensions work. (Debian Woody) > > Only my C++ program segfaults. It looked to me like a problem with > > NA_NewAllFromBuffer -- was there any recent change ? > > Yes there was. You can undo it in your private CVS checkout like this: > > % cd numarray/Src > % cvs update -j 1.3 -1.2 libnumarray.ch > % cd .. > % python setup.py install --genapi --selftest > > > (I saw that I was using a bytestride of 0 - but it used to work and > > giving a 1 for that argument still segfaults -- is 0 OK ??) > > I don't know for sure. It seems like it should be... just leads to 0 > strides. > > > Now I noticed in the gdb traceback that I chokes is the function > > deferred_libnumarray_init > > (( > > ((#0 0x406d68d5 in PyObject_GetAttrString () from > > ((/usr/lib/libpython2.2.so.0.0 #1 0x410f905e in > > ((deferred_libnumarray_init () at Src/libnumarraymodule.c:149 > > > > Is this function the standard function that always get called to init > > numarray (in embedded Python AND using "normal" python) ? > > Yes and no. deferred_libnumarray_init() is really for stuff that can't > be done at libnumarray.so init time. This is because of circular > dependencies. > > The "real" numarray init function is either "import numarray" in Python > or "import_array()" in C. After that, certain calls into the C-API > trigger the deferred initialization. deferred_libnumarray_init() is > never called directly by application code. > > > Have you tested this with Python2.2 ? > > Yes, using the numarray unit tests. NewAllFromBuffer is also used > directly by the numarray ufuncs, so it is tested to a degree by > numarray itself. My gut feel is you're hitting an embedding problem, > but my gut is only so accurate. > > > (See for more info below == I did follow Todd's suggestions 1 & 2, but > > hope not having to recompile python from source.) > > That's really not hard. A 5-10 minute exercise usually. > > Regards, > Todd > > > > ------------------------------------------------------- > This SF.Net email is sponsored by Oracle Space Sweepstakes > Want to be the first software developer in space? > Enter now for the Oracle Space Sweepstakes! > http://ads.osdn.com/?ad_id=7393&alloc_id=16281&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From jmiller at stsci.edu Tue May 10 11:43:34 2005 From: jmiller at stsci.edu (Todd Miller) Date: Tue May 10 11:43:34 2005 Subject: [Fwd: Re: [Numpy-discussion] Re: ANN: numarray-1.2.3 -- segfault in in my C program] Message-ID: <1115750435.27493.45.camel@halloween.stsci.edu> Forgot to reply-all. -------------- next part -------------- An embedded message was scrubbed... From: unknown sender Subject: no subject Date: no date Size: 38 URL: From jmiller at stsci.edu Tue May 10 14:39:50 2005 From: jmiller at stsci.edu (Todd Miller) Date: Tue, 10 May 2005 14:39:50 -0400 Subject: [Numpy-discussion] Re: ANN: numarray-1.2.3 -- segfault in in my C program In-Reply-To: <200505101031.23907.haase@msg.ucsf.edu> References: <1109869619.19608.16.camel@halloween.stsci.edu> <200505091758.11466.haase@msg.ucsf.edu> <1115727212.4479.27.camel@jaytmiller.comcast.net> <200505101031.23907.haase@msg.ucsf.edu> Message-ID: <1115750389.27493.43.camel@halloween.stsci.edu> Hi Sebastian, I tried building and running this but I can't even get to main(). I even tried the attached, which I think is completely decoupled from numarray and there is the same problem. BTW, did you leave out -fPIC and -shared on the command line? To simplify things, you might try: a. removing: > PyObject *pymodMain = PyImport_ImportModule("__main__"); > PyObject* globals = PyModule_GetDict(pymodMain); > PyEval_InitThreads(); b. get rid of threads for now c. downshift from C++ to C Bottom line: I think this is a more fundamental "embedding" setup problem. Regards, Todd On Tue, 2005-05-10 at 13:31, Sebastian Haase wrote: > Todd, > I would be very thankful if you could compile and run this program: > > #include > #include > int main() { > Py_Initialize(); > PyObject *pymodMain = PyImport_ImportModule("__main__"); > PyObject* globals = PyModule_GetDict(pymodMain); > PyEval_InitThreads(); > > // import_libnumarray(); > PyObject *module = PyImport_ImportModule("numarray.libnumarray"); > if (!module) > Py_FatalError("Can't import module 'numarray.libnumarray'"); > if (module != NULL) { > PyObject *module_dict = PyModule_GetDict(module); > PyObject *c_api_object = > PyDict_GetItemString(module_dict, "_C_API"); > if (PyCObject_Check(c_api_object)) { > libnumarray_API = (void **)PyCObject_AsVoidPtr(c_api_object); > } else { > Py_FatalError("Can't get API for module 'numarray.libnumarray'"); > } > } > float *colMapsRGB = new float[4 * 3 * 256]; > int shape[3]= { 4, 3, 256 }; > PyObject *obj = PyBuffer_FromReadWriteMemory(colMapsRGB, 4*3 *256 * > sizeof(float)); > PyObject *obj2 =(PyObject*) NA_NewAllFromBuffer( 3, shape, tFloat32, obj, > 0, 0, NA_ByteOrder(), 1, 1/*writable*/) ; > return 0; > } > > > I compile with: > g++-2.95 -o testnumarrayembedded -Wall -finline-functions -g -I/usr/include/ > python2.2 -I/jws30/haase/myCVS/numarray/Include testnumarrayembedded.cpp > -pthread -lpython2.2 > > > It runs through with setting PYTHONPATH to my old numarray(1.1) directory. > And now with numarray CVS it now doesn't even find the library... > $ ./testnumarrayembedded > Fatal Python error: Can't import module 'numarray.libnumarray' > Aborted > > Still, my non-embedded python shell runs all fine - your CVS-update (-j...) > suggestion didn't seem to make a difference. > > Thanks, > Sebastian > > > > On Tuesday 10 May 2005 05:13, Todd Miller wrote: > > On Mon, 2005-05-09 at 17:58 -0700, Sebastian Haase wrote: > > > Hi, > > > > > > Now I'm back to this problem. In the meantime I upgraded again: > > > >>> na.__version__ > > > > > > '1.4.0' > > > All my python2.2 + numarray + my-extensions work. (Debian Woody) > > > Only my C++ program segfaults. It looked to me like a problem with > > > NA_NewAllFromBuffer -- was there any recent change ? > > > > Yes there was. You can undo it in your private CVS checkout like this: > > > > % cd numarray/Src > > % cvs update -j 1.3 -1.2 libnumarray.ch > > % cd .. > > % python setup.py install --genapi --selftest > > > > > (I saw that I was using a bytestride of 0 - but it used to work and > > > giving a 1 for that argument still segfaults -- is 0 OK ??) > > > > I don't know for sure. It seems like it should be... just leads to 0 > > strides. > > > > > Now I noticed in the gdb traceback that I chokes is the function > > > deferred_libnumarray_init > > > (( > > > ((#0 0x406d68d5 in PyObject_GetAttrString () from > > > ((/usr/lib/libpython2.2.so.0.0 #1 0x410f905e in > > > ((deferred_libnumarray_init () at Src/libnumarraymodule.c:149 > > > > > > Is this function the standard function that always get called to init > > > numarray (in embedded Python AND using "normal" python) ? > > > > Yes and no. deferred_libnumarray_init() is really for stuff that can't > > be done at libnumarray.so init time. This is because of circular > > dependencies. > > > > The "real" numarray init function is either "import numarray" in Python > > or "import_array()" in C. After that, certain calls into the C-API > > trigger the deferred initialization. deferred_libnumarray_init() is > > never called directly by application code. > > > > > Have you tested this with Python2.2 ? > > > > Yes, using the numarray unit tests. NewAllFromBuffer is also used > > directly by the numarray ufuncs, so it is tested to a degree by > > numarray itself. My gut feel is you're hitting an embedding problem, > > but my gut is only so accurate. > > > > > (See for more info below == I did follow Todd's suggestions 1 & 2, but > > > hope not having to recompile python from source.) > > > > That's really not hard. A 5-10 minute exercise usually. > > > > Regards, > > Todd > > > > > > > > ------------------------------------------------------- > > This SF.Net email is sponsored by Oracle Space Sweepstakes > > Want to be the first software developer in space? > > Enter now for the Oracle Space Sweepstakes! > > http://ads.osdn.com/?ad_id=7393&alloc_id=16281&op=click > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > ------------------------------------------------------- > This SF.Net email is sponsored by Oracle Space Sweepstakes > Want to be the first software developer in space? > Enter now for the Oracle Space Sweepstakes! > http://ads.osdn.com/?ad_id=7393&alloc_id=16281&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- -------------- next part -------------- A non-text attachment was scrubbed... Name: testnumarrayembedded.c Type: text/x-c Size: 1403 bytes Desc: not available URL: From Reidar.Strand.Hagen at idi.ntnu.no Tue May 10 12:35:19 2005 From: Reidar.Strand.Hagen at idi.ntnu.no (Reidar Strand Hagen) Date: Tue May 10 12:35:19 2005 Subject: [Numpy-discussion] Expansion of Matrix Message-ID: Hi Sorry if this is an obvious question, but I'm getting close to loosing sleep over it now :) If I have a matrix: 1 2 3 4 5 6 7 8 9 is there a nice general way to expand the matrix to: 1 0 2 3 4 0 5 6 0 0 0 0 7 0 8 9 The actual dataset in question has dimensions 128 * 128 * 128; so big enough for me to worry quite abit about duplication and general speed. vennlig hilsen / regards Reidar Strand Hagen From haase at msg.ucsf.edu Tue May 10 13:23:46 2005 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue May 10 13:23:46 2005 Subject: [Numpy-discussion] Re: ANN: numarray-1.2.3 -- segfault in in my C program In-Reply-To: <1115750389.27493.43.camel@halloween.stsci.edu> References: <1109869619.19608.16.camel@halloween.stsci.edu> <200505101031.23907.haase@msg.ucsf.edu> <1115750389.27493.43.camel@halloween.stsci.edu> Message-ID: <200505101321.02528.haase@msg.ucsf.edu> Todd, Thanks for working on this with me. I did not use -fPIC and/or -shared. What is -fPIC anywhy ? I have been using my code + makefile since numarray 0.3.6 (maybe 0.4). I took my gcc/g++ arguments from examples in the wxWindows/wxPython. Trying your modified(attached) c-version I get the exact same behaviour that I had before IF I DON'T use -shared. (I.e. works with numarray1.1; new numarray gives "Can't import module 'numarray.libnumarray'") (I took -pthread out) With -shared I get a segfault before main() - gdb: Program received signal SIGSEGV, Segmentation fault. 0x00000001 in ?? () How can this be ? How can this change from numarray 1.1 to 1.2 (or 1.2.3) ? Thanks, Sebastian On Tuesday 10 May 2005 11:39, Todd Miller wrote: > Hi Sebastian, > > I tried building and running this but I can't even get to main(). I > even tried the attached, which I think is completely decoupled from > numarray and there is the same problem. > > BTW, did you leave out -fPIC and -shared on the command line? > > To simplify things, you might try: > > a. removing: > > PyObject *pymodMain = PyImport_ImportModule("__main__"); > > PyObject* globals = PyModule_GetDict(pymodMain); > > PyEval_InitThreads(); > > b. get rid of threads for now > > c. downshift from C++ to C > > Bottom line: I think this is a more fundamental "embedding" setup problem. > > Regards, > Todd > > On Tue, 2005-05-10 at 13:31, Sebastian Haase wrote: > > Todd, > > I would be very thankful if you could compile and run this program: > > > > #include > > #include > > int main() { > > Py_Initialize(); > > PyObject *pymodMain = PyImport_ImportModule("__main__"); > > PyObject* globals = PyModule_GetDict(pymodMain); > > PyEval_InitThreads(); > > > > // import_libnumarray(); > > PyObject *module = PyImport_ImportModule("numarray.libnumarray"); > > if (!module) > > Py_FatalError("Can't import module 'numarray.libnumarray'"); > > if (module != NULL) { > > PyObject *module_dict = PyModule_GetDict(module); > > PyObject *c_api_object = > > PyDict_GetItemString(module_dict, "_C_API"); > > if (PyCObject_Check(c_api_object)) { > > libnumarray_API = (void **)PyCObject_AsVoidPtr(c_api_object); > > } else { > > Py_FatalError("Can't get API for module 'numarray.libnumarray'"); > > } > > } > > float *colMapsRGB = new float[4 * 3 * 256]; > > int shape[3]= { 4, 3, 256 }; > > PyObject *obj = PyBuffer_FromReadWriteMemory(colMapsRGB, 4*3 *256 * > > sizeof(float)); > > PyObject *obj2 =(PyObject*) NA_NewAllFromBuffer( 3, shape, tFloat32, > > obj, 0, 0, NA_ByteOrder(), 1, 1/*writable*/) ; > > return 0; > > } > > > > > > I compile with: > > g++-2.95 -o testnumarrayembedded -Wall -finline-functions -g > > -I/usr/include/ python2.2 -I/jws30/haase/myCVS/numarray/Include > > testnumarrayembedded.cpp -pthread -lpython2.2 > > > > > > It runs through with setting PYTHONPATH to my old numarray(1.1) > > directory. And now with numarray CVS it now doesn't even find the > > library... $ ./testnumarrayembedded > > Fatal Python error: Can't import module 'numarray.libnumarray' > > Aborted > > > > Still, my non-embedded python shell runs all fine - your CVS-update > > (-j...) suggestion didn't seem to make a difference. > > > > Thanks, > > Sebastian > > > > On Tuesday 10 May 2005 05:13, Todd Miller wrote: > > > On Mon, 2005-05-09 at 17:58 -0700, Sebastian Haase wrote: > > > > Hi, > > > > > > > > Now I'm back to this problem. In the meantime I upgraded again: > > > > >>> na.__version__ > > > > > > > > '1.4.0' > > > > All my python2.2 + numarray + my-extensions work. (Debian Woody) > > > > Only my C++ program segfaults. It looked to me like a problem with > > > > NA_NewAllFromBuffer -- was there any recent change ? > > > > > > Yes there was. You can undo it in your private CVS checkout like this: > > > > > > % cd numarray/Src > > > % cvs update -j 1.3 -1.2 libnumarray.ch > > > % cd .. > > > % python setup.py install --genapi --selftest > > > > > > > (I saw that I was using a bytestride of 0 - but it used to work and > > > > giving a 1 for that argument still segfaults -- is 0 OK ??) > > > > > > I don't know for sure. It seems like it should be... just leads to 0 > > > strides. > > > > > > > Now I noticed in the gdb traceback that I chokes is the function > > > > deferred_libnumarray_init > > > > (( > > > > ((#0 0x406d68d5 in PyObject_GetAttrString () from > > > > ((/usr/lib/libpython2.2.so.0.0 #1 0x410f905e in > > > > ((deferred_libnumarray_init () at Src/libnumarraymodule.c:149 > > > > > > > > Is this function the standard function that always get called to init > > > > numarray (in embedded Python AND using "normal" python) ? > > > > > > Yes and no. deferred_libnumarray_init() is really for stuff that can't > > > be done at libnumarray.so init time. This is because of circular > > > dependencies. > > > > > > The "real" numarray init function is either "import numarray" in Python > > > or "import_array()" in C. After that, certain calls into the C-API > > > trigger the deferred initialization. deferred_libnumarray_init() is > > > never called directly by application code. > > > > > > > Have you tested this with Python2.2 ? > > > > > > Yes, using the numarray unit tests. NewAllFromBuffer is also used > > > directly by the numarray ufuncs, so it is tested to a degree by > > > numarray itself. My gut feel is you're hitting an embedding problem, > > > but my gut is only so accurate. > > > > > > > (See for more info below == I did follow Todd's suggestions 1 & 2, > > > > but hope not having to recompile python from source.) > > > > > > That's really not hard. A 5-10 minute exercise usually. > > > > > > Regards, > > > Todd From simon at arrowtheory.com Tue May 10 17:29:35 2005 From: simon at arrowtheory.com (Simon Burton) Date: Tue May 10 17:29:35 2005 Subject: [Numpy-discussion] numarray memmap and gc In-Reply-To: <1115727214.4479.29.camel@jaytmiller.comcast.net> References: <20050510111256.187b2f5c.simon@arrowtheory.com> <1115727214.4479.29.camel@jaytmiller.comcast.net> Message-ID: <20050511102540.515675ca.simon@arrowtheory.com> On Tue, 10 May 2005 08:13:34 -0400 Todd Miller wrote: > > I'm nervous about sticking in a circular reference (the Memmap already > remembers it's slices) but I suppose the garbage collector was added for > exactly this case. I'm not entirely sure how hard it's going to be to > add. Yes, this is a good point. But also, memmaps have a close method so users can always explicitly break this cycle if they want. Either way, appropriate warnings should be added to the documentation. bye, Simon. -- Simon Burton, B.Sc. Licensed PO Box 8066 ANU Canberra 2601 Australia Ph. 61 02 6249 6940 http://arrowtheory.com From simon at arrowtheory.com Tue May 10 17:48:36 2005 From: simon at arrowtheory.com (Simon Burton) Date: Tue May 10 17:48:36 2005 Subject: [Numpy-discussion] numarray.ma docs confuse Numeric/numarray Message-ID: <20050511104651.70d2e199.simon@arrowtheory.com> It seems the masked array docs mention Numeric in a few places when numarray was intended. Patch attached. Simon. -- Simon Burton, B.Sc. Licensed PO Box 8066 ANU Canberra 2601 Australia Ph. 61 02 6249 6940 http://arrowtheory.com -------------- next part -------------- A non-text attachment was scrubbed... Name: ma.tex.patch Type: application/octet-stream Size: 1856 bytes Desc: not available URL: From customer-support at ncsecu.org Tue May 10 20:08:39 2005 From: customer-support at ncsecu.org (State Employees Credit Union) Date: Tue May 10 20:08:39 2005 Subject: [Numpy-discussion] SECU - Urgent account verification! Message-ID: <200505110259.j4B2x68T014799@med-track.net> An HTML attachment was scrubbed... URL: From andrea_gavana at tin.it Wed May 11 09:43:09 2005 From: andrea_gavana at tin.it (andrea_gavana at tin.it) Date: Wed May 11 09:43:09 2005 Subject: [Numpy-discussion] About Reshape() Message-ID: <4200063400037BA1@ims9c.cp.tin.it> Hello NG, I am not really a guru with Numeric, and I am trying to port some Matlab code to Python, using Numeric. The code is working, but it is slown down because of the different implementation (ordering) of reshape() in Matlab and Numeric. For example, in Matlab I can do: a = 1:10; reshape(a,5,2) ans = 1 6 2 7 3 8 4 9 5 10 In Numeric: a = range(1, 11) Numeric.reshape(a, (5,2)) array([[ 1, 2], [ 3, 4], [ 5, 6], [ 7, 8], [ 9, 10]]) Hmm... let's try again... Numeric.reshape(a, (2,5)) array([[ 1, 2, 3, 4, 5], [ 6, 7, 8, 9, 10]]) Hmm... the only solution I found: Numeric.transpose(Numeric.reshape(a, (2,5))) array([[ 1, 6], [ 2, 7], [ 3, 8], [ 4, 9], [ 5, 10]]) This continuous use of transpose() is killing the performances of my scripts... is there a workaround? Am I missing something so obvious? Thank you for every suggestion. Andrea. From andrea_gavana at tin.it Wed May 11 13:04:29 2005 From: andrea_gavana at tin.it (andrea_gavana at tin.it) Date: Wed May 11 13:04:29 2005 Subject: [Numpy-discussion] Square All Elements Of A Matrix Message-ID: <4200063400037D9A@ims9c.cp.tin.it> Hello NG, in one script, I generated a matrix "dx" which has a (200000, 10) shape. Well, everything is working fine (matrix multiplication, sum, exponentiation etc...), but I'm having problems in squaring all elements of this matrix. Suppose I have: import Numeric import random dx = Numeric.ones((200000,10), Numeric.Float0, savespace=1) dx[:,:] = random.random() td1 = dx*2.0 ---> This takes about 0.2 seconds td2 = dx**2.0 ---> This takes about 6 seconds Why this difference? And why it is so slow the exponentiation? I have tried the same operation in Matlab and it was almost instantaneous. I don't want to stick with Matlab, noting the great job the Numeric developers have done. Is there something that I am missing? Sorry if it is a basic question. Andrea. From cookedm at physics.mcmaster.ca Wed May 11 13:30:25 2005 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Wed May 11 13:30:25 2005 Subject: [Numpy-discussion] Square All Elements Of A Matrix In-Reply-To: <4200063400037D9A@ims9c.cp.tin.it> (andrea gavana's message of "Wed, 11 May 2005 22:02:35 +0200") References: <4200063400037D9A@ims9c.cp.tin.it> Message-ID: andrea_gavana at tin.it writes: > Hello NG, > > in one script, I generated a matrix "dx" which has a (200000, 10) shape. > Well, everything is working fine (matrix multiplication, sum, exponentiation > etc...), but I'm having problems in squaring all elements of this matrix. > > Suppose I have: > > import Numeric > import random > > dx = Numeric.ones((200000,10), Numeric.Float0, savespace=1) > dx[:,:] = random.random() It's faster to do import RandomArray dx = RandomArray.random((200000,10)).astype(Numeric.Float0) dx.savespace(1) (it won't use the same RNG as random, though) > td1 = dx*2.0 ---> This takes about 0.2 seconds > td2 = dx**2.0 ---> This takes about 6 seconds > > Why this difference? And why it is so slow the exponentiation? I have tried > the same operation in Matlab and it was almost instantaneous. I don't want > to stick with Matlab, noting the great job the Numeric developers > have done. Hmm, I don't see this, with either 23.8 or 24.0b2. What version of Numeric are you using, and on what platform? dx**2 uses the C 'pow' function on each element (so using 2 or 2.0 here doesn't make a difference), so may be your platform pow() is slow. dx*dx should be faster (it takes 80% of the time that dx**2 does on my machine). -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From Fernando.Perez at colorado.edu Wed May 11 13:35:26 2005 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Wed May 11 13:35:26 2005 Subject: [Numpy-discussion] Square All Elements Of A Matrix In-Reply-To: <4200063400037D9A@ims9c.cp.tin.it> References: <4200063400037D9A@ims9c.cp.tin.it> Message-ID: <42826BDC.7030706@colorado.edu> andrea_gavana at tin.it wrote: > Hello NG, > > in one script, I generated a matrix "dx" which has a (200000, 10) shape. > Well, everything is working fine (matrix multiplication, sum, exponentiation > etc...), but I'm having problems in squaring all elements of this matrix. > > Suppose I have: > > import Numeric > import random > > dx = Numeric.ones((200000,10), Numeric.Float0, savespace=1) > dx[:,:] = random.random() > > td1 = dx*2.0 ---> This takes about 0.2 seconds > td2 = dx**2.0 ---> This takes about 6 seconds > > Why this difference? And why it is so slow the exponentiation? I have tried > the same operation in Matlab and it was almost instantaneous. I don't want > to stick with Matlab, noting the great job the Numeric developers have done. > > Is there something that I am missing? I'm not sure what the problem may be, but I certainly don't see your results: In [22]: dx=RandomArray.random((200000, 10)).astype(Numeric.Float0) In [23]: time td1 = dx*2.0 CPU times: user 0.20 s, sys: 0.06 s, total: 0.26 s Wall time: 0.26 In [24]: time td2 = dx**2.0 CPU times: user 0.30 s, sys: 0.06 s, total: 0.36 s Wall time: 0.36 An increase like this (50% in cpu time) is reasonable for computing a power vs. a plain multiplication. This is using Numeric 23.7 on a Fedora3 box (2.6 GHZ P4). Do you know for a fact that you have enough RAM, and that the 6 secs. don't come from paging (the arrays aren't that large, but you might have other stuff in memory forcing paging). Cheers, f From andrea_gavana at tin.it Wed May 11 13:40:01 2005 From: andrea_gavana at tin.it (andrea_gavana at tin.it) Date: Wed May 11 13:40:01 2005 Subject: [Numpy-discussion] Square All Elements Of A Matrix In-Reply-To: Message-ID: <4200063400037DE5@ims9c.cp.tin.it> Hello David >It's faster to do > >import RandomArray > >dx = RandomArray.random((200000,10)).astype(Numeric.Float0) >dx.savespace(1) > >(it won't use the same RNG as random, though) thanks for the tip. >> td1 = dx*2.0 ---> This takes about 0.2 seconds >> td2 = dx**2.0 ---> This takes about 6 seconds >> >Hmm, I don't see this, with either 23.8 or 24.0b2. What version of >Numeric are you using, and on what platform? I have currently Numeric 23.8, with numarray 1.3.1. I am on a Windows XP machine, with 1 GB RAM and 3.0 GHz. My Python version is 2.3.4. >dx**2 uses the C 'pow' function on each element (so using 2 or 2.0 >here doesn't make a difference), so may be your platform pow() is >slow. Probably that is the reason... but why? Is Windows the problem? >dx*dx should be faster (it takes 80% of the time that dx**2 does on my >machine). On my machine, dx*dx is istantaneous (about 0.1 secons). This are my timings: import time; start = time.time(); td = dx*dx; print time.time() - start 0.121000051498 import time; start = time.time(); td = dx**2; print time.time() - start 4.367000103 !!!!!!!! Does anyone have an explanation for this?!?!? Thank you for your suggestions. Andrea. From andrea_gavana at tin.it Wed May 11 13:45:03 2005 From: andrea_gavana at tin.it (andrea_gavana at tin.it) Date: Wed May 11 13:45:03 2005 Subject: [Numpy-discussion] Square All Elements Of A Matrix In-Reply-To: <42826BDC.7030706@colorado.edu> Message-ID: <4200063400037DF0@ims9c.cp.tin.it> Hello Fernando, >I'm not sure what the problem may be, but I certainly don't see your results: > >In [22]: dx=RandomArray.random((200000, 10)).astype(Numeric.Float0) > >In [23]: time td1 = dx*2.0 >CPU times: user 0.20 s, sys: 0.06 s, total: 0.26 s >Wall time: 0.26 > >In [24]: time td2 = dx**2.0 >CPU times: user 0.30 s, sys: 0.06 s, total: 0.36 s >Wall time: 0.36 This is what I get: start = time.time(); td2 = dx**2; print time.time() - start 4.367000103 start = time.time(); td2 = dx*dx; print time.time() - start 0.121000051498 This is on a Windows XP, 1 GB RAM, 3.0 GHz, Python 2.3.4, Numeric 23.8. >This is using Numeric 23.7 on a Fedora3 box (2.6 GHZ P4). Do you know for a fact that >you have enough RAM, and that the 6 secs. don't come from paging (the arrays aren't >that large, but you might have other stuff in memory forcing paging). No, the only thing I have opened is PythonWin (the editor), and I am using currently 100 MB of the RAM. I don't know why this happens... Thanks for your help. Andrea. From cookedm at physics.mcmaster.ca Wed May 11 13:58:20 2005 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Wed May 11 13:58:20 2005 Subject: [Numpy-discussion] Square All Elements Of A Matrix In-Reply-To: <4200063400037DE5@ims9c.cp.tin.it> (andrea gavana's message of "Wed, 11 May 2005 22:38:24 +0200") References: <4200063400037DE5@ims9c.cp.tin.it> Message-ID: andrea_gavana at tin.it writes: > Hello David >>Hmm, I don't see this, with either 23.8 or 24.0b2. What version of >>Numeric are you using, and on what platform? > > I have currently Numeric 23.8, with numarray 1.3.1. I am on a Windows XP > machine, with 1 GB RAM and 3.0 GHz. My Python version is 2.3.4. > >>dx**2 uses the C 'pow' function on each element (so using 2 or 2.0 >>here doesn't make a difference), so may be your platform pow() is >>slow. > > Probably that is the reason... but why? Is Windows the problem? > >>dx*dx should be faster (it takes 80% of the time that dx**2 does on my >>machine). > > On my machine, dx*dx is istantaneous (about 0.1 secons). This are my timings: > > import time; start = time.time(); td = dx*dx; print time.time() - start > 0.121000051498 > > import time; start = time.time(); td = dx**2; print time.time() - start > 4.367000103 > > !!!!!!!! Could you check the speed of td = exp(2*log(dx)) If it's comparable to dx**2, then your pow() library function isn't any good :-( -- it's not special-casing integer powers. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From andrea_gavana at tin.it Wed May 11 14:02:25 2005 From: andrea_gavana at tin.it (andrea_gavana at tin.it) Date: Wed May 11 14:02:25 2005 Subject: [Numpy-discussion] Square All Elements Of A Matrix In-Reply-To: Message-ID: <4200063400037E0A@ims9c.cp.tin.it> Hello David, >Could you check the speed of > >td = exp(2*log(dx)) > >If it's comparable to dx**2, then your pow() library function isn't >any good :-( -- it's not special-casing integer powers. I get: start = time.time(); td = Numeric.exp(2*Numeric.log(dx)); print time.time() - start 0.530999898911 Does this make any sense to anyone? I am a bit puzzled... Thanks however for your kind help/suggestions. Andrea. From jbrandmeyer at earthlink.net Wed May 11 18:29:10 2005 From: jbrandmeyer at earthlink.net (Jonathan Brandmeyer) Date: Wed May 11 18:29:10 2005 Subject: [Numpy-discussion] Square All Elements Of A Matrix In-Reply-To: References: <4200063400037DE5@ims9c.cp.tin.it> Message-ID: <1115861247.680.27.camel@localhost.localdomain> On Wed, 2005-05-11 at 16:57 -0400, David M. Cooke wrote: > td = exp(2*log(dx)) > > If it's comparable to dx**2, then your pow() library function isn't > any good :-( -- it's not special-casing integer powers. I believe that this is the case. IIRC, MinGW uses a wrapper around msvcrt.dll's pow() that handles this special case for this very reason. HTH, -Jonathan From cookedm at physics.mcmaster.ca Thu May 12 11:07:16 2005 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Thu May 12 11:07:16 2005 Subject: [Numpy-discussion] Square All Elements Of A Matrix In-Reply-To: <1115861247.680.27.camel@localhost.localdomain> (Jonathan Brandmeyer's message of "Wed, 11 May 2005 21:27:26 -0400") References: <4200063400037DE5@ims9c.cp.tin.it> <1115861247.680.27.camel@localhost.localdomain> Message-ID: Jonathan Brandmeyer writes: > On Wed, 2005-05-11 at 16:57 -0400, David M. Cooke wrote: > >> td = exp(2*log(dx)) >> >> If it's comparable to dx**2, then your pow() library function isn't >> any good :-( -- it's not special-casing integer powers. > > I believe that this is the case. IIRC, MinGW uses a wrapper around > msvcrt.dll's pow() that handles this special case for this very > reason. Is this is a big problem for others? This is the first report I've seen of the problem on Windows. It may be worth special-casing the power ufunc to handle some non-array powers (x**2, x**3, x**0.5, for instance). -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From andrea_gavana at tin.it Thu May 12 12:36:24 2005 From: andrea_gavana at tin.it (andrea_gavana at tin.it) Date: Thu May 12 12:36:24 2005 Subject: [Numpy-discussion] Square All Elements Of A Matrix In-Reply-To: Message-ID: <4200063400038B09@ims9c.cp.tin.it> Hello David, >Is this is a big problem for others? This is the first report I've >seen of the problem on Windows. It may be worth special-casing the >power ufunc to handle some non-array powers (x**2, x**3, x**0.5, for >instance). It's not very clear to me what you wrote, but I have the same behavior on a Windows 2000 machine I have at work (here is WinXP), with Numeric 23.7 (here is 23.8), with Python 2.3.5 (here is 2.3.4). These 2 machines do not have anything in common, as far as I know. Regarding what you said (if I have correctly understood), I get: >>>start = time.time(); td = dx*dx; print time.time() - start 0.269999980927 >>> start = time.time(); td = dx**2; print time.time() - start 4.5560002327 >>> start = time.time(); td = dx**3; print time.time() - start 4.5759999752 >>> start = time.time(); td = dx**0.5; print time.time() - start 4.19600009918 >>> start = time.time(); td = sqrt(dx); print time.time() - start 0.301000118256 >>> start = time.time(); td = exp(log(dx)); print time.time() - start 0.792000055313 >>> start = time.time(); td = dx**RandomArray.random((1,)); print time.time() - start 4.14599990845 Whatever power I use, the timing is almost the same. Curious however, how sqrt(dx) and dx**0.5 behave really differently. Thanks for your help. Andrea. From haase at msg.ucsf.edu Thu May 12 20:15:56 2005 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Thu May 12 20:15:56 2005 Subject: [Numpy-discussion] tuple vs. list as index Message-ID: <200505121704.41025.haase@msg.ucsf.edu> Hi, Is this really obvious to everybody: >>> type(a) >>> a.shape (12, 13, 2, 5) >>> a[(0,0)].shape (2, 5) >>> a[[0,0]].shape (2, 13, 2, 5) >>> na.take(a,[0,0]).shape (2, 13, 2, 5) Is there some explanation in the documentation ? (Is this the same in Numeric / Numeric3 ?) Thanks, Sebastian Haase From jbrandmeyer at earthlink.net Thu May 12 20:41:38 2005 From: jbrandmeyer at earthlink.net (Jonathan Brandmeyer) Date: Thu May 12 20:41:38 2005 Subject: [Numpy-discussion] Square All Elements Of A Matrix In-Reply-To: References: <4200063400037DE5@ims9c.cp.tin.it> <1115861247.680.27.camel@localhost.localdomain> Message-ID: <1115941592.6032.12.camel@localhost.localdomain> On Thu, 2005-05-12 at 14:05 -0400, David M. Cooke wrote: > Jonathan Brandmeyer writes: > > > On Wed, 2005-05-11 at 16:57 -0400, David M. Cooke wrote: > > > >> td = exp(2*log(dx)) > >> > >> If it's comparable to dx**2, then your pow() library function isn't > >> any good :-( -- it's not special-casing integer powers. > > > > I believe that this is the case. IIRC, MinGW uses a wrapper around > > msvcrt.dll's pow() that handles this special case for this very > > reason. > > Is this is a big problem for others? Its never bugged me enough to do anything about it ;) > This is the first report I've > seen of the problem on Windows. It may be worth special-casing the > power ufunc to handle some non-array powers (x**2, x**3, x**0.5, for > instance). If you go this route, I suggest you just use the implementation in mingw-runtime - its in the public domain and has had some precision issues debugged out of it already. -Jonathan From jmiller at stsci.edu Fri May 13 03:35:05 2005 From: jmiller at stsci.edu (Todd Miller) Date: Fri May 13 03:35:05 2005 Subject: [Numpy-discussion] tuple vs. list as index In-Reply-To: <200505121704.41025.haase@msg.ucsf.edu> References: <200505121704.41025.haase@msg.ucsf.edu> Message-ID: <1115980379.4430.20.camel@jaytmiller.comcast.net> On Thu, 2005-05-12 at 17:04 -0700, Sebastian Haase wrote: > Hi, > Is this really obvious to everybody: > > >>> type(a) > > >>> a.shape > (12, 13, 2, 5) > >>> a[(0,0)].shape > (2, 5) > >>> a[[0,0]].shape > (2, 13, 2, 5) > > >>> na.take(a,[0,0]).shape > (2, 13, 2, 5) > > Is there some explanation in the documentation ? The distinction is not an accident and has been discussed. One thing to keep in mind with numarray indexing is that not all sequences used as indices are created equal. There is already a distinction for arrays used as indices, and an additional distinction for Bool arrays used as indices. Back to basics, the explanation is this: >>> a[(0,0)].shape (2, 5) >>> a[0,0].shape (2, 5) Those two have identical semantics because of the way Python works. Either way, the numarray indexing hooks see (0,0), i.e. not ((0,0),) for the first and (0,0) for the second. >>> a[[0,0]].shape (2, 13, 2, 5) A list, on the other hand, *is* seen as ([ 0,0 ],). And we've chosen to exploit that as a shorthand for array-based-indexing making it equivalent to take() as you've shown. > (Is this the same in Numeric / Numeric3 ?) Unless it was added recently, Numeric has no array-based-indexing. Maybe Travis or someone else can fill us in on how Numeric3 handles this. Regards, Todd From andrea_gavana at tin.it Fri May 13 09:07:18 2005 From: andrea_gavana at tin.it (andrea_gavana at tin.it) Date: Fri May 13 09:07:18 2005 Subject: [Numpy-discussion] Square All Elements Of A Matrix In-Reply-To: <8368.129.238.237.96.1115999760.squirrel@www.mza.com> Message-ID: <4200063400039573@ims9c.cp.tin.it> Hello Tony, >What compiler are you using to build Numeric and Python? No compiler for Python or Numeric. For Python, I use the binaries that you can find at www.python.org. I don't have any idea which compiler they have used. For Numeric, I downloaded the binaries that you can find at SourceForge. Python 2.3.4/2.3.5, Numeric 23.8/23.7, Windows XP/2000 Andrea. From anthony.seward at ieee.org Fri May 13 09:33:10 2005 From: anthony.seward at ieee.org (Anthony Joseph Seward) Date: Fri May 13 09:33:10 2005 Subject: [Numpy-discussion] Square All Elements Of A Matrix In-Reply-To: <4200063400038B09@ims9c.cp.tin.it> References: <4200063400038B09@ims9c.cp.tin.it> Message-ID: <8368.129.238.237.96.1115999760.squirrel@www.mza.com> What compiler are you using to build Numeric and Python? Tony andrea_gavana at tin.it said: > Hello David, > >>Is this is a big problem for others? This is the first report I've >>seen of the problem on Windows. It may be worth special-casing the >>power ufunc to handle some non-array powers (x**2, x**3, x**0.5, for >>instance). > > It's not very clear to me what you wrote, but I have the same behavior on > a Windows 2000 machine I have at work (here is WinXP), with Numeric 23.7 > (here is 23.8), with Python 2.3.5 (here is 2.3.4). These 2 machines do not > have anything in common, as far as I know. > Regarding what you said (if I have correctly understood), I get: > >>>>start = time.time(); td = dx*dx; print time.time() - start > 0.269999980927 > >>>> start = time.time(); td = dx**2; print time.time() - start > 4.5560002327 > >>>> start = time.time(); td = dx**3; print time.time() - start > 4.5759999752 > >>>> start = time.time(); td = dx**0.5; print time.time() - start > 4.19600009918 > >>>> start = time.time(); td = sqrt(dx); print time.time() - start > 0.301000118256 > >>>> start = time.time(); td = exp(log(dx)); print time.time() - start > 0.792000055313 > >>>> start = time.time(); td = dx**RandomArray.random((1,)); print >>>> time.time() > - start > 4.14599990845 > > Whatever power I use, the timing is almost the same. Curious however, how > sqrt(dx) and dx**0.5 behave really differently. > > Thanks for your help. > > Andrea. > > > > ------------------------------------------------------- > This SF.Net email is sponsored by Oracle Space Sweepstakes > Want to be the first software developer in space? > Enter now for the Oracle Space Sweepstakes! > http://ads.osdn.com/?ad_ids93&alloc_id281&op?k > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > From anthony.seward at ieee.org Fri May 13 10:39:18 2005 From: anthony.seward at ieee.org (Anthony Joseph Seward) Date: Fri May 13 10:39:18 2005 Subject: [Numpy-discussion] Square All Elements Of A Matrix In-Reply-To: <4200063400039573@ims9c.cp.tin.it> References: <8368.129.238.237.96.1115999760.squirrel@www.mza.com> <4200063400039573@ims9c.cp.tin.it> Message-ID: <11633.129.238.237.96.1116005870.squirrel@www.mza.com> What is the value of sys.version? For me running the attached script yields: Fedora 3: Python version: 2.3.4 (#1, Feb 2 2005, 12:11:53) [GCC 3.4.2 20041017 (Red Hat 3.4.2-6.fc3)] *: 0.381051 **2: 1.31382 **3: 1.198 **0.5: 1.67896 sqrt(): 0.357029 exp(log): 1.2737 exp(2*log): 2.17751 exp(.5*log): 2.19792 Win2k on vmware 5 under Fedora 3 Python version: 2.3.3 (#51, Feb 16 2004, 04:07:52) [MSC v.1200 32 bit (Intel)] *: 0.391 **2: 1.468 **3: 1.547 **0.5: 1.469 sqrt(): 0.437 exp(log): 1.672 exp(2*log): 2.718 exp(.5*log): 2.734 Odd. andrea_gavana at tin.it said: > Hello Tony, > >>What compiler are you using to build Numeric and Python? > > No compiler for Python or Numeric. For Python, I use the binaries that you > can find at www.python.org. I don't have any idea which compiler they have > used. For Numeric, I downloaded the binaries that you can find at > SourceForge. > > Python 2.3.4/2.3.5, Numeric 23.8/23.7, Windows XP/2000 > > Andrea. > > > > > -- ajs: anthony.seward at ieee.org -------------- next part -------------- A non-text attachment was scrubbed... Name: numeric-stats.py Type: application/x-python Size: 1056 bytes Desc: not available URL: From anthony.seward at ieee.org Fri May 13 10:55:19 2005 From: anthony.seward at ieee.org (Anthony Joseph Seward) Date: Fri May 13 10:55:19 2005 Subject: [Numpy-discussion] Square All Elements Of A Matrix In-Reply-To: <11633.129.238.237.96.1116005870.squirrel@www.mza.com> References: <8368.129.238.237.96.1115999760.squirrel@www.mza.com> <4200063400039573@ims9c.cp.tin.it> <11633.129.238.237.96.1116005870.squirrel@www.mza.com> Message-ID: <5860.129.238.237.96.1116006878.squirrel@www.mza.com> More data on FC3 Python version: 2.3.4 (#1, Feb 2 2005, 12:11:53) [GCC 3.4.2 20041017 (Red Hat 3.4.2-6.fc3)] ========== Numeric ========== *: 0.379871 **2: 1.17747 **3: 1.1788 **0.5: 1.66172 sqrt(): 0.362143 exp(log): 1.2656 exp(2*log): 2.16179 exp(.5*log): 2.22712 ========== numarray ========== *: 0.0414059 **2: 0.914946 **3: 1.14649 **0.5: 1.43333 sqrt(): 0.10821 exp(log): 1.1466 exp(2*log): 1.17661 exp(.5*log): 1.24503 From oliphant at ee.byu.edu Fri May 13 11:21:29 2005 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri May 13 11:21:29 2005 Subject: [Numpy-discussion] tuple vs. list as index In-Reply-To: <1115980379.4430.20.camel@jaytmiller.comcast.net> References: <200505121704.41025.haase@msg.ucsf.edu> <1115980379.4430.20.camel@jaytmiller.comcast.net> Message-ID: <4284EF63.7000204@ee.byu.edu> Todd Miller wrote: >On Thu, 2005-05-12 at 17:04 -0700, Sebastian Haase wrote: > > >>Hi, >>Is this really obvious to everybody: >> >> >> >>>>>type(a) >>>>> >>>>> >> >> >> >>>>>a.shape >>>>> >>>>> >>(12, 13, 2, 5) >> >> >>>>>a[(0,0)].shape >>>>> >>>>> >>(2, 5) >> >> >>>>>a[[0,0]].shape >>>>> >>>>> >>(2, 13, 2, 5) >> >> >> >>>>>na.take(a,[0,0]).shape >>>>> >>>>> >>(2, 13, 2, 5) >> >>Is there some explanation in the documentation ? >> >> > >The distinction is not an accident and has been discussed. One thing to >keep in mind with numarray indexing is that not all sequences used as >indices are created equal. There is already a distinction for arrays >used as indices, and an additional distinction for Bool arrays used as > > >indices. Back to basics, the explanation is this: > > This same distinction is kept in Numeric3 and currently works the same way. Numeric3 should have the same semantics as Numarray. -Travis From andrea_gavana at tin.it Fri May 13 12:06:20 2005 From: andrea_gavana at tin.it (andrea_gavana at tin.it) Date: Fri May 13 12:06:20 2005 Subject: [Numpy-discussion] Square All Elements Of A Matrix In-Reply-To: <11633.129.238.237.96.1116005870.squirrel@www.mza.com> Message-ID: <42000634000396F5@ims9c.cp.tin.it> Hello Tony, >What is the value of sys.version? D:\MyProjects>python Python 2.3.4 (#53, May 25 2004, 21:17:02) [MSC v.1200 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.version '2.3.4 (#53, May 25 2004, 21:17:02) [MSC v.1200 32 bit (Intel)]' >>> >Fedora 3: > *: 0.381051 > **2: 1.31382 > **3: 1.198 > **0.5: 1.67896 > sqrt(): 0.357029 > exp(log): 1.2737 > exp(2*log): 2.17751 > exp(.5*log): 2.19792 > >Win2k on vmware 5 under Fedora 3 > Python version: 2.3.3 (#51, Feb 16 2004, 04:07:52) [MSC v.1200 32 bit >(Intel)] > *: 0.391 > **2: 1.468 > **3: 1.547 > **0.5: 1.469 > sqrt(): 0.437 > exp(log): 1.672 > exp(2*log): 2.718 > exp(.5*log): 2.734 Win XP (with Numeric 23.8): D:\MyProjects>numeric-stats.py Python version: 2.3.4 (#53, May 25 2004, 21:17:02) [MSC v.1200 32 bit (Intel)] *: 0.12 **2: 5.037 **3: 5.168 **0.5: 5.558 sqrt(): 0.14 exp(log): 0.46 exp(2*log): 1.012 exp(.5*log): 1.021 Win XP (with numarray 1.3.1): D:\MyProjects>numeric-stats.py Python version: 2.3.4 (#53, May 25 2004, 21:17:02) [MSC v.1200 32 bit (Intel)] *: 0.321 **2: 1.502 **3: 1.502 **0.5: 1.492 sqrt(): 1.953 exp(log): 1.993 exp(2*log): 2.043 exp(.5*log): 2.033 I get the same results (scaled on RAM) with my Windows 2000 machine at work. Is there some kind of explanation? A possible solution? Does anyone else get similar differences in timings as I do? > >Odd. *Really* Odd. Thank you for your suggestions. Andrea From stephen.walton at csun.edu Fri May 13 14:13:22 2005 From: stephen.walton at csun.edu (Stephen Walton) Date: Fri May 13 14:13:22 2005 Subject: [Numpy-discussion] About Reshape() In-Reply-To: <4200063400037BA1@ims9c.cp.tin.it> References: <4200063400037BA1@ims9c.cp.tin.it> Message-ID: <428517BB.8070103@csun.edu> andrea_gavana at tin.it wrote: >This continuous use of transpose() is killing the performances of my scripts... >is there a workaround? Am I missing something so obvious? > > Not necessarily obvious, but you've hit the old problem that MATLAB stores arrays in column major order (first index varies most rapidly, borrowed from Fortran), while Numeric and numarray store arrays in row major order (borrowed from C). The only way to avoid the transpose() calls is to rework your algorithms to work on arrays stored in row major order. From andrea_gavana at tin.it Fri May 13 14:23:05 2005 From: andrea_gavana at tin.it (andrea_gavana at tin.it) Date: Fri May 13 14:23:05 2005 Subject: [Numpy-discussion] About Reshape() In-Reply-To: <428517BB.8070103@csun.edu> Message-ID: <42000634000397C4@ims9c.cp.tin.it> Hello Stephen, >Not necessarily obvious, but you've hit the old problem that MATLAB >stores arrays in column major order (first index varies most rapidly, >borrowed from Fortran), while Numeric and numarray store arrays in row >major order (borrowed from C). Yes, I have seen the difference ;-) >The only way to avoid the transpose() >calls is to rework your algorithms to work on arrays stored in row major > >order. Yes, this is what I have done. It took some work because I still have my mind highly-badly conditioned by Matlab syntaxes, but it can be done. Thanks for your comments. Andrea. From carlosjosepita at yahoo.com.ar Tue May 17 17:33:18 2005 From: carlosjosepita at yahoo.com.ar (Carlos Pita) Date: Tue May 17 17:33:18 2005 Subject: [Numpy-discussion] Wrapping C array Message-ID: <200505172136.23149.carlosjosepita@yahoo.com.ar> Hi! I need to wrap a pre-existent C array into a numarray. It shouldn't be a copy of the original array because this is an audio buffer with a fixed address and also input/output copying from/to the buffer would be overkilling, so the numarray *must* be lightly wrapping the buffer and not a memcpy of it. But NA_NewArray duplicates the passed buffer so I'm not sure how to achieve the goal stated above or if it's possible at all. Can you help me? Regards, Carlos From Alexandre.Fayolle at logilab.fr Wed May 18 01:04:04 2005 From: Alexandre.Fayolle at logilab.fr (Alexandre) Date: Wed May 18 01:04:04 2005 Subject: [Numpy-discussion] Wrapping C array In-Reply-To: <200505172136.23149.carlosjosepita@yahoo.com.ar> References: <200505172136.23149.carlosjosepita@yahoo.com.ar> Message-ID: <20050518080158.GA18444@crater.logilab.fr> On Tue, May 17, 2005 at 09:36:22PM -0300, Carlos Pita wrote: > Hi! > I need to wrap a pre-existent C array into > a numarray. It shouldn't be a copy of the > original array because this is an audio buffer > with a fixed address and also input/output > copying from/to the buffer would be overkilling, > so the numarray *must* be lightly wrapping the > buffer and not a memcpy of it. But NA_NewArray > duplicates the passed buffer so I'm not sure > how to achieve the goal stated above or if > it's possible at all. > Can you help me? I'm not sure for Numarray, but for Numeric, you can use PyArray_FromDimsAndData(). There should be a similar function in numarray. -- Alexandre Fayolle LOGILAB, Paris (France). http://www.logilab.com http://www.logilab.fr http://www.logilab.org -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From jmiller at stsci.edu Wed May 18 08:31:21 2005 From: jmiller at stsci.edu (Todd Miller) Date: Wed May 18 08:31:21 2005 Subject: [Numpy-discussion] Wrapping C array In-Reply-To: <20050518080158.GA18444@crater.logilab.fr> References: <200505172136.23149.carlosjosepita@yahoo.com.ar> <20050518080158.GA18444@crater.logilab.fr> Message-ID: <1116421435.1460.0.camel@halloween.stsci.edu> On Wed, 2005-05-18 at 04:01, Alexandre wrote: > On Tue, May 17, 2005 at 09:36:22PM -0300, Carlos Pita wrote: > > Hi! > > I need to wrap a pre-existent C array into > > a numarray. It shouldn't be a copy of the > > original array because this is an audio buffer > > with a fixed address and also input/output > > copying from/to the buffer would be overkilling, > > so the numarray *must* be lightly wrapping the > > buffer and not a memcpy of it. But NA_NewArray > > duplicates the passed buffer so I'm not sure > > how to achieve the goal stated above or if > > it's possible at all. > > Can you help me? > > I'm not sure for Numarray, but for Numeric, you can use > PyArray_FromDimsAndData(). There should be a similar function in > numarray. That function does indeed exist for Numeric and has non-copying semantics. Regards, Todd From carlosjosepita at yahoo.com.ar Wed May 18 08:43:34 2005 From: carlosjosepita at yahoo.com.ar (Carlos Pita) Date: Wed May 18 08:43:34 2005 Subject: [Numpy-discussion] Wrapping C array In-Reply-To: <20050518080158.GA18444@crater.logilab.fr> References: <200505172136.23149.carlosjosepita@yahoo.com.ar> <20050518080158.GA18444@crater.logilab.fr> Message-ID: <200505181137.37147.carlosjosepita@yahoo.com.ar> But as NA_NewArray does, FromDimsAndData copies its contents from the passed pointer, which is what I'm willing to avoid. PyObject* PyArray FromDimsAndData(int nd, int *dims, int type, char *data) This function will allocate a numarray of the specified shape and type, and it's contents will be copied from the specified data pointer. The fact is that the audio buffer to be wrapped is a simple, fixed-size and contiguous C array, so I guess it should be easy to access through a numarray. But NA_NewArray and friends all seems to copy contents to a buffer of their own. Thank you. Regards, Carlos > On Tue, May 17, 2005 at 09:36:22PM -0300, Carlos Pita wrote: > > Hi! > > I need to wrap a pre-existent C array into > > a numarray. It shouldn't be a copy of the > > original array because this is an audio buffer > > with a fixed address and also input/output > > copying from/to the buffer would be overkilling, > > so the numarray *must* be lightly wrapping the > > buffer and not a memcpy of it. But NA_NewArray > > duplicates the passed buffer so I'm not sure > > how to achieve the goal stated above or if > > it's possible at all. > > Can you help me? > > I'm not sure for Numarray, but for Numeric, you can use > PyArray_FromDimsAndData(). There should be a similar function in > numarray. > From carlosjosepita at yahoo.com.ar Wed May 18 09:14:10 2005 From: carlosjosepita at yahoo.com.ar (Carlos Pita) Date: Wed May 18 09:14:10 2005 Subject: [Numpy-discussion] Wrapping C array In-Reply-To: <1116421435.1460.0.camel@halloween.stsci.edu> References: <200505172136.23149.carlosjosepita@yahoo.com.ar> <20050518080158.GA18444@crater.logilab.fr> <1116421435.1460.0.camel@halloween.stsci.edu> Message-ID: <200505181318.56247.carlosjosepita@yahoo.com.ar> The documentation for PyArray_FromDimsAndData that I quoted in my previous post seems to suggest that a copy is done. But browsing the web numarray cvs I find out that its implementation first creates a PyBuffer from the passed data: buf = PyBuffer_FromReadWriteMemory(data, nelements * descr->elsize); a = NA_NewAllFromBuffer( nd, dimensions, descr->type_num, buf, 0, descr->elsize, NA_ByteOrder(), 1, 1); Then, from the NA_NewAllFromBuffer doc (in numarray manual): It is possible to create a Python buffer object from an array of C data and then construct a numarray using this function which refers to the C data without making a copy. So you both are right. I guess creating a PyBuffer as above and then calling NA_NewAllFromBuffer is the best way to do what I was asking for in numarray (that is, without recurring to the numeric simulation API). Am I right? Regards, Carlos > On Wed, 2005-05-18 at 04:01, Alexandre wrote: > > On Tue, May 17, 2005 at 09:36:22PM -0300, Carlos Pita wrote: > > > Hi! > > > I need to wrap a pre-existent C array into > > > a numarray. It shouldn't be a copy of the > > > original array because this is an audio buffer > > > with a fixed address and also input/output > > > copying from/to the buffer would be overkilling, > > > so the numarray *must* be lightly wrapping the > > > buffer and not a memcpy of it. But NA_NewArray > > > duplicates the passed buffer so I'm not sure > > > how to achieve the goal stated above or if > > > it's possible at all. > > > Can you help me? > > > > I'm not sure for Numarray, but for Numeric, you can use > > PyArray_FromDimsAndData(). There should be a similar function in > > numarray. > > That function does indeed exist for Numeric and has non-copying > semantics. > > Regards, > Todd > > > > ------------------------------------------------------- > This SF.Net email is sponsored by Oracle Space Sweepstakes > Want to be the first software developer in space? > Enter now for the Oracle Space Sweepstakes! > http://ads.osdn.com/?ad_id=7412&alloc_id=16344&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From jmiller at stsci.edu Wed May 18 09:33:41 2005 From: jmiller at stsci.edu (Todd Miller) Date: Wed May 18 09:33:41 2005 Subject: [Numpy-discussion] Wrapping C array In-Reply-To: <1116421435.1460.0.camel@halloween.stsci.edu> References: <200505172136.23149.carlosjosepita@yahoo.com.ar> <20050518080158.GA18444@crater.logilab.fr> <1116421435.1460.0.camel@halloween.stsci.edu> Message-ID: <1116433645.1460.40.camel@halloween.stsci.edu> On Wed, 2005-05-18 at 09:03, Todd Miller wrote: > On Wed, 2005-05-18 at 04:01, Alexandre wrote: > > On Tue, May 17, 2005 at 09:36:22PM -0300, Carlos Pita wrote: > > > Hi! > > > I need to wrap a pre-existent C array into > > > a numarray. It shouldn't be a copy of the > > > original array because this is an audio buffer > > > with a fixed address and also input/output > > > copying from/to the buffer would be overkilling, > > > so the numarray *must* be lightly wrapping the > > > buffer and not a memcpy of it. But NA_NewArray > > > duplicates the passed buffer so I'm not sure > > > how to achieve the goal stated above or if > > > it's possible at all. > > > Can you help me? > > > > I'm not sure for Numarray, but for Numeric, you can use > > PyArray_FromDimsAndData(). There should be a similar function in > > numarray. > > That function does indeed exist for Numeric and has non-copying > semantics. I misspoke here, meaning to say numarray. More in a subsequent post. Todd From jmiller at stsci.edu Wed May 18 09:39:20 2005 From: jmiller at stsci.edu (Todd Miller) Date: Wed May 18 09:39:20 2005 Subject: [Numpy-discussion] Wrapping C array In-Reply-To: <200505181318.56247.carlosjosepita@yahoo.com.ar> References: <200505172136.23149.carlosjosepita@yahoo.com.ar> <20050518080158.GA18444@crater.logilab.fr> <1116421435.1460.0.camel@halloween.stsci.edu> <200505181318.56247.carlosjosepita@yahoo.com.ar> Message-ID: <1116434185.1460.58.camel@halloween.stsci.edu> On Wed, 2005-05-18 at 12:18, Carlos Pita wrote: > The documentation for PyArray_FromDimsAndData > that I quoted in my previous post seems to suggest > that a copy is done. But browsing the web I'd appreciate it if you could send me a reference to where that is because it is stale. > numarray cvs I find out that its implementation > first creates a PyBuffer from the passed data: > > buf = PyBuffer_FromReadWriteMemory(data, nelements * descr->elsize); > a = NA_NewAllFromBuffer( nd, dimensions, descr->type_num, buf, > 0, descr->elsize, NA_ByteOrder(), 1, 1); > > Then, from the NA_NewAllFromBuffer doc (in numarray > manual): > > It is possible to create a Python buffer object from > an array of C data and then construct a numarray using > this function which refers to the C data without > making a copy. > > So you both are right. I guess creating a > PyBuffer as above and then calling > NA_NewAllFromBuffer is the best way > to do what I was asking for in numarray > (that is, without recurring to the > numeric simulation API). > > Am I right? Not in my opinion, for two reasons: 1. PyArray_FromDimsAndData is easier to use, internalizing the buffer creation step. 2. These days, my recommendation is to use the Numeric compatibility layer because it gives the best portability, to Numeric now and to Numeric3 in the future. Most of the NA_ routines exist to facilitate the implementation of numarray itself and because we didn't always know a compatibility layer was possible or desirable. Of course you're free to use the NA_ routines and we're not planning on removing them, but I think the Numeric compatible routines are a better choice. Regards, Todd From carlosjosepita at yahoo.com.ar Wed May 18 09:56:16 2005 From: carlosjosepita at yahoo.com.ar (Carlos Pita) Date: Wed May 18 09:56:16 2005 Subject: [Numpy-discussion] Wrapping C array In-Reply-To: <1116434185.1460.58.camel@halloween.stsci.edu> References: <200505172136.23149.carlosjosepita@yahoo.com.ar> <200505181318.56247.carlosjosepita@yahoo.com.ar> <1116434185.1460.58.camel@halloween.stsci.edu> Message-ID: <200505181359.14435.carlosjosepita@yahoo.com.ar> > > The documentation for PyArray_FromDimsAndData > > that I quoted in my previous post seems to suggest > > that a copy is done. But browsing the web > > I'd appreciate it if you could send me a reference to where that is > because it is stale. I take the excerpt from the numarray user manual from the python2.3-numarray 1.1.1 debian (sarge) package. But it's updated in the cvs: \begin{cfuncdesc}{PyObject*}{PyArray_FromDimsAndData}{int nd, int *dims, int type, char *data} This function will allocate a numarray of the specified shape and type which will refer to the data buffer specified by \var{data}. The contents of \var{data} will not be copied nor will \var{data} be deallocated upon the deletion of the array. \end{cfuncdesc} It seems to be ok. When it says "nor will \var{data} be deallocated upon the deletion of the array", does it mean that the python buffer object ref count won't be decremented ? Or, more generally, the created numarray will refer directly to the C array segments exposed by the python buffer object or will keep a reference to this object? > Not in my opinion, for two reasons: > > 1. PyArray_FromDimsAndData is easier to use, internalizing the buffer > creation step. > > 2. These days, my recommendation is to use the Numeric compatibility > layer because it gives the best portability, to Numeric now and to I will take your advice. Thank you! Regards, Carlos From jmiller at stsci.edu Wed May 18 10:10:28 2005 From: jmiller at stsci.edu (Todd Miller) Date: Wed May 18 10:10:28 2005 Subject: [Numpy-discussion] Wrapping C array In-Reply-To: <200505181358.31406.carlosjosepita@yahoo.com.ar> References: <200505172136.23149.carlosjosepita@yahoo.com.ar> <200505181318.56247.carlosjosepita@yahoo.com.ar> <1116434185.1460.58.camel@halloween.stsci.edu> <200505181358.31406.carlosjosepita@yahoo.com.ar> Message-ID: <1116436049.1460.65.camel@halloween.stsci.edu> On Wed, 2005-05-18 at 12:58, Carlos Pita wrote: > > > The documentation for PyArray_FromDimsAndData > > > that I quoted in my previous post seems to suggest > > > that a copy is done. But browsing the web > > > > I'd appreciate it if you could send me a reference to where that is > > because it is stale. > > I take the excerpt from the numarray user manual from > the python2.3-numarray 1.1.1 debian (sarge) package. > But it's updated in the cvs: > > \begin{cfuncdesc}{PyObject*}{PyArray_FromDimsAndData}{int nd, int *dims, int type, char *data} > This function will allocate a numarray of the specified shape and type > which will refer to the data buffer specified by \var{data}. The contents > of \var{data} will not be copied nor will \var{data} be deallocated upon > the deletion of the array. > \end{cfuncdesc} > > It seems to be ok. When it says "nor will \var{data} be deallocated upon > the deletion of the array", does it mean that the python buffer object > ref count won't be decremented ? Or, more generally, the created > numarray will refer directly to the C array segments exposed by the > python buffer object or will keep a reference to this object? It means the internal buffer object refers to but does not "own" the array data passed into PyArray_FromDimsAndData(). At deallocation time, the internal buffer object is DECREF'd and disappears but the storage referred to by the data parameter does not. So, I/O buffers or other statically allocated arrays should work. Regards, Todd From edcjones at comcast.net Thu May 19 14:52:15 2005 From: edcjones at comcast.net (Edward C. Jones) Date: Thu May 19 14:52:15 2005 Subject: [Numpy-discussion] numarray.nd_image: max, min not preserved by rotate Message-ID: <428D0A32.1090904@comcast.net> I have up-to-date (5/18/05) Debian unstable on a PC. I compiled Python and numarray from source and I keep them in "/usr/local". Python is version 2.4.1 and numarray is version 1.3.1. The output of the program below is: 0.0 1.0 -0.107813523267 1.10781352327 I would expect a rotated image to have min == 0.0 and max == 1.0. The max and min of arr2 occur near the edge of the rotated half plane. This looks like a bug in nd_image. ------------------------- #! /usr/bin/env python import numarray, numarray.nd_image arr = numarray.zeros((256, 256), 'Float64') arr[128:, :] = 1.0 print arr.min(), arr.max() arr2 = numarray.nd_image.rotate(arr, 30, reshape=False) print arr2.min(), arr2.max() From tom.denniston at gmail.com Thu May 19 20:16:25 2005 From: tom.denniston at gmail.com (Tom Denniston) Date: Thu May 19 20:16:25 2005 Subject: [Numpy-discussion] Strange behavior of the RandomArray module Message-ID: I am running under python 1.4 and windows XP. I get the following results in my python interpretter: -------------------------------------------- n=0 import RandomArray while 1: arr = RandomArray.randint(0, 2509, 252) for x in arr: if x>2508: raise "Found number greater than 2508: " + str(x) + " on iteration " + str(n) n=n+1 Traceback (most recent call last): File "", line 5, in -toplevel- raise "Found number greater than 2508: " + str(x) + " on iteration " + str(n) Found number greater than 2508: 2509 on iteration 91719 ------------------------------------------- Even though the Numeric reference (see below) on RandomArray class claims that the number returned is always strictly less than the max. Does anyone know what might cause this problem? randint(minimum, maximum, shape=ReturnFloat) The randint() function returns an array of the specified shape and containing random (standard) integers greater than or equal to minimum and strictly less than maximum . If no shape is specified, a single number is returned. (exerpt from http://www.pfdubois.com/numpy/html2/numpy-19.html#pgfId-303108) From cookedm at physics.mcmaster.ca Thu May 19 22:35:29 2005 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Thu May 19 22:35:29 2005 Subject: [Numpy-discussion] Strange behavior of the RandomArray module In-Reply-To: References: Message-ID: <8F003FF6-845B-45C2-9E22-944719E03237@physics.mcmaster.ca> On May 19, 2005, at 23:13, Tom Denniston wrote: > I am running under python 1.4 and windows XP. I hope you mean 2.4 :-) > I get the following results in my python interpretter: > > -------------------------------------------- > n=0 > import RandomArray > while 1: > arr = RandomArray.randint(0, 2509, 252) > for x in arr: > if x>2508: > raise "Found number greater than 2508: " + > str(x) + " on iteration " + str(n) > n=n+1 > > > > Traceback (most recent call last): > File "", line 5, in -toplevel- > raise "Found number greater than 2508: " + str(x) + " on iteration > " + str(n) > Found number greater than 2508: 2509 on iteration 91719 > > ------------------------------------------- > > Even though the Numeric reference (see below) on RandomArray class > claims that the number returned is always strictly less than the max. > Does anyone > know what might cause this problem? This looks like it may be the same problem as bug #545336: http://sourceforge.net/tracker/ index.php&func=detail&aid=545336&group_id=1369&atid=101369 I ran the above (generating 1e10 numbers). The above problem happened 280 times, so it comes up at a rate of 2.8e-8 -- about what you'd expect for something going wrong with a last-bit error in rounding floats. There is indeed some float-double casting going on in ranlibmodule.c; that's probably it. I've gone and replaced that in CVS, generating an array of doubles directly instead for random() and randint(). This fixes it for me. [As an aside: note that the other random functions use floats. Really, an array of floats should be returned for those instead of an array of doubles. Next thing on the list :-)] -- |>|\/|< /------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From meesters at uni-mainz.de Fri May 20 05:31:26 2005 From: meesters at uni-mainz.de (Christian Meesters) Date: Fri May 20 05:31:26 2005 Subject: [Numpy-discussion] applying function under certain conditions Message-ID: Hi I have a large 1D numarray array with some huge values and lots of zeros. Now, for better representation I wanted to plot the logarithm. Well, calculating the logarithm is not such a good idea if you have zeros in your array ... However, how do you implement such cases like "only apply a function if a certain condition is true", I'd like to ask whether there are nice short snippets which could show such cases. My own code is (unnecessarily?) long and ugly. I tried to combine 'where' and 'greater', but didn't succeed. I guess I'm just don't see the wood for the trees. How would you write this in my case? TIA Regards, Christian From Alexandre.Fayolle at logilab.fr Fri May 20 05:53:04 2005 From: Alexandre.Fayolle at logilab.fr (Alexandre) Date: Fri May 20 05:53:04 2005 Subject: [Numpy-discussion] applying function under certain conditions In-Reply-To: References: Message-ID: <20050520124958.GI27695@crater.logilab.fr> On Fri, May 20, 2005 at 02:22:25PM +0200, Christian Meesters wrote: > Hi > > I have a large 1D numarray array with some huge values and lots of > zeros. Now, for better representation I wanted to plot the logarithm. > Well, calculating the logarithm is not such a good idea if you have > zeros in your array ... > However, how do you implement such cases like "only apply a function if > a certain condition is true", I'd like to ask whether there are nice > short snippets which could show such cases. My own code is > (unnecessarily?) long and ugly. I tried to combine 'where' and > 'greater', but didn't succeed. I guess I'm just don't see the wood for > the trees. How would you write this in my case? Assuming your data contains only positive or null values,you could try replacing zeros by a very small value before computing the log: log_scaled = log(where(data, data, 1e-50)) This uses the fact that 0 is false and non-zero is true. If you have negative values, you can use something like log_scaled = log(where(data>0, data, 1e-50)) since data>0 will return a boolean array suitable for where. -- Alexandre Fayolle LOGILAB, Paris (France). http://www.logilab.com http://www.logilab.fr http://www.logilab.org -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From perry at stsci.edu Fri May 20 06:28:51 2005 From: perry at stsci.edu (Perry Greenfield) Date: Fri May 20 06:28:51 2005 Subject: [Numpy-discussion] applying function under certain conditions In-Reply-To: <20050520124958.GI27695@crater.logilab.fr> References: <20050520124958.GI27695@crater.logilab.fr> Message-ID: <1e661f1687aebd36a555fa364445dac7@stsci.edu> On May 20, 2005, at 8:49 AM, Alexandre wrote: > On Fri, May 20, 2005 at 02:22:25PM +0200, Christian Meesters wrote: >> Hi >> >> I have a large 1D numarray array with some huge values and lots of >> zeros. Now, for better representation I wanted to plot the logarithm. >> Well, calculating the logarithm is not such a good idea if you have >> zeros in your array ... >> However, how do you implement such cases like "only apply a function >> if >> a certain condition is true", I'd like to ask whether there are nice >> short snippets which could show such cases. My own code is >> (unnecessarily?) long and ugly. I tried to combine 'where' and >> 'greater', but didn't succeed. I guess I'm just don't see the wood for >> the trees. How would you write this in my case? > > Assuming your data contains only positive or null values,you could try > replacing zeros by a very small value before computing the log: > > log_scaled = log(where(data, data, 1e-50)) > > This uses the fact that 0 is false and non-zero is true. > > If you have negative values, you can use something like > > log_scaled = log(where(data>0, data, 1e-50)) > > since data>0 will return a boolean array suitable for where. > > -- > Alexandre Fayolle LOGILAB, Paris (France). > http://www.logilab.com http://www.logilab.fr http://www.logilab.org Another alternative that takes two lines but perhaps is more readable is: data[data<=0] = 1e-50 # if you don't mind modifying the original array log_scaled = log(data) From meesters at uni-mainz.de Fri May 20 06:45:31 2005 From: meesters at uni-mainz.de (Christian Meesters) Date: Fri May 20 06:45:31 2005 Subject: [Numpy-discussion] applying function under certain conditions In-Reply-To: References: Message-ID: Hi Thanks. So, it is as easy as I thought ... Since I do not want to modify the original array in this case (it's only for plotting purposes), I will go for Alexandre's solution here. Thanks a lot, Christian From rowen at cesmail.net Fri May 20 15:40:04 2005 From: rowen at cesmail.net (Russell E. Owen) Date: Fri May 20 15:40:04 2005 Subject: [Numpy-discussion] Re: applying function under certain conditions References: Message-ID: In article , Christian Meesters wrote: > Thanks. So, it is as easy as I thought ... > Since I do not want to modify the original array in this case (it's > only for plotting purposes), I will go for Alexandre's solution here. One more thing to consider...do you really want log? If you are looking for log-like behavior for positive values but graceful handling of 0 and negative values, consider arcsinh, e.g. arcsinh(scale * (x-offset)). -- Russell *Due to the properties I just mentioned, arcsinh is a very useful function for stretching astronomical images. Thanks to Robert Lupton for pointing this out to me. From meesters at uni-mainz.de Fri May 20 23:57:46 2005 From: meesters at uni-mainz.de (Christian Meesters) Date: Fri May 20 23:57:46 2005 Subject: [Numpy-discussion] Re: applying function under certain conditions In-Reply-To: References: Message-ID: <72352ac02b03c4342745c1c66ca6e7ce@uni-mainz.de> Hi Oh yes, that's an idea. I actually really wanted log, though more for historical display reasons than for mathematical reasons. In my particular case it hardly makes a difference for the eye whether I use arcsinh or log, but in the near future I might want to use it. I will keep it in mind. Thanks for pointing this out to me. Have a nice weekend, Christian On 21 May 2005, at 00:36, Russell E. Owen wrote: > In article , > Christian Meesters wrote: > >> Thanks. So, it is as easy as I thought ... >> Since I do not want to modify the original array in this case (it's >> only for plotting purposes), I will go for Alexandre's solution here. > > One more thing to consider...do you really want log? If you are looking > for log-like behavior for positive values but graceful handling of 0 > and > negative values, consider arcsinh, e.g. arcsinh(scale * (x-offset)). > > -- Russell > > *Due to the properties I just mentioned, arcsinh is a very useful > function for stretching astronomical images. Thanks to Robert Lupton > for > pointing this out to me. > > > From a.jaffe at imperial.ac.uk Sun May 22 09:48:32 2005 From: a.jaffe at imperial.ac.uk (Andrew Jaffe) Date: Sun May 22 09:48:32 2005 Subject: [Numpy-discussion] numarray 1.3.1 setup patch for mac OS X veclib Message-ID: Hi all, I've had some trouble getting the numarray-1.3.1 setup scripts to recognize the optimized BLAS/LAPACK, veclib, on OS X. But I think I've tracked down the problems. First, the section of addons.py that checks for the existence of the framework (around line 47) needs to set lapack_dirs = [], I think, since it needs to exist later on. Second, the section after that that sets lapack_compile_args, lapack_link_args and lapack_include_dirs for *all* cases (lines 57-60) overwrites lapack_link_args from the if...elif...else above. Finally, cblas.h isn't included correctly, since the framework option is only used for linking, not compiling. I've attached a patchfile for addons.py below; I switched the two sections and added to the framework if... clause. Use it with > patch addons.py addons.patch I hope this is a reasonable way to propagate this to the developers. Please let me know if it doesn't work for any reason, or if it breaks any other install, which I haven't checked. I assume it works on my machine, since: > otool -L build/lib.darwin-7.9.0-Power_Macintosh-2.3/numarray/linear_algebra/ lapack_lite2.so /System/Library/Frameworks/Python.framework/Versions/2.3/Python (compatibility version 2.3.0, current version 2.3.0) /System/Library/Frameworks/vecLib.framework/Versions/A/vecLib (compatibility version 1.0.0, current version 153.2.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 71.1.3) Andrew p.s. only tested under OS X 10.3.9 (latest version of Panther). -------------- next part -------------- A non-text attachment was scrubbed... Name: addons.patch Type: application/octet-stream Size: 1241 bytes Desc: not available URL: -------------- next part -------------- ______________________________________________________________________ Andrew Jaffe a.jaffe at imperial.ac.uk Astrophysics Group +44 207 594-7526 Blackett Laboratory, Room 1013 FAX 7541 Imperial College, Prince Consort Road London SW7 2AZ ENGLAND http://astro.imperial.ac.uk/~jaffe From a.schmolck at gmx.net Mon May 23 17:13:28 2005 From: a.schmolck at gmx.net (Alexander Schmolck) Date: Mon May 23 17:13:28 2005 Subject: [Numpy-discussion] dot-product of selected columns or rows Message-ID: Suppose I have some iterative algorithm that selects and deselects 'features' represented by a list of indices `active` that correspond to rows (or equivalently columns) in some matrices `M`, `N`, ... What would be the most efficient way to carry out the equivalent of: A = dot(take(M, active), take(C, active),axis=1) assuming that M and C remain constant throughtout the computation and that A could be modified in-place for the relevant indices in `active`. Also am I right to assume that generally the best approach to cheaply create a matrix with a different diagonal from a square and contiguous n*n matrix `S` (that is no longer needed) would be the following in-place operation? S.flat[::n+1] += vec In case it matters: I'm using Numeric and scipy, but not numarray. 'as From jmiller at stsci.edu Wed May 25 10:45:31 2005 From: jmiller at stsci.edu (Todd Miller) Date: Wed May 25 10:45:31 2005 Subject: [Numpy-discussion] ANN: numarray-1.3.2 Message-ID: <1117042956.24369.99.camel@halloween.stsci.edu> Numarray is an array processing package designed to efficiently manipulate large multi-dimensional arrays. Numarray is modelled after Numeric and features c-code generated from python template scripts, the capacity to operate directly on arrays in files, arrays of heterogeneous records, string arrays, and in-place operation on memory mapped files. ========================================================================= Release Notes for numarray-1.3.2 I. ENHANCEMENTS None. 1.3.2 fixes the problem with gcc4 on Mac OS-X 10.4 Tiger II. BUGS FIXED / CLOSED 1193125 Build failure on Mac OS 10.4 (Tiger) 1195370 numarray1.3.1 setup.py broken See http://sourceforge.net/tracker/?atid=450446&group_id=1369&func=browse for more details. ========================================================================= Release Notes for numarray-1.3.1 I. ENHANCEMENTS None. 1.3.1 fixes the problem with gcc-3.4.3 II. BUGS FIXED / CLOSED 1152323 /usr/include/fenv.h:96: error: conflicting types for 'fegete 1185024 numarray-1.2.3 fails to compile with gcc-3.4.3 1187162 Numarray 1.3.0 installation failure See http://sourceforge.net/tracker/?atid=450446&group_id=1369&func=browse for more details. ========================================================================= Release Notes for numarray-1.3.0 Numarray is an array processing package designed to efficiently manipulate large multi-dimensional arrays. Numarray is modelled after Numeric and features c-code generated from python template scripts, the capacity to operate directly on arrays in files, arrays of heterogeneous records, string arrays, and in-place operation on memory mapped files. I. ENHANCEMENTS 1. Migration of NumArray.__del__ to C (tp_dealloc). Overall performance. 2. Removal of dictionary update from array view creation improves performance of view/slice/subarray creation. This should e.g. improve the performance of wxPython sequence protocol access to Nx2 arrays. Subclasses now need to do a.flags |= numarray.generic._UPDATEDICT to ensure that dictionary based attributes are inherited by views. NumArrays no longer do this by default. 2. Modifications to support scipy.special. 3. Removal of an unnecessary getattr() from ufunc calling sequence. Ufunc performance. II. BUGS FIXED / CLOSED 1179355 average() broken in numarray 1.2.3 1167184 Floating point exception in numarray's dot() 1151892 Bug in matrixmultiply with zero size arrays 1160184 RecArray reversal 1156172 Incorect error message for shape incompatability 1155538 Incorrect error message when multiplying arrays See http://sourceforge.net/tracker/?atid=450446&group_id=1369&func=browse for more details. III. CAUTIONS This release should be backward binary compatible with numarray-1.1.1 and 1.2.3. WHERE ----------- Numarray-1.3.0 windows executable installers, source code, and manual is here: http://sourceforge.net/project/showfiles.php?group_id=1369 Numarray is hosted by Source Forge in the same project which hosts Numeric: http://sourceforge.net/projects/numpy/ The web page for Numarray information is at: http://stsdas.stsci.edu/numarray/index.html Trackers for Numarray Bugs, Feature Requests, Support, and Patches are at the Source Forge project for NumPy at: http://sourceforge.net/tracker/?group_id=1369 REQUIREMENTS ------------------------------ numarray-1.3.0 requires Python 2.2.2 or greater. AUTHORS, LICENSE ------------------------------ Numarray was written by Perry Greenfield, Rick White, Todd Miller, JC Hsu, Paul Barrett, Phil Hodge at the Space Telescope Science Institute. We'd like to acknowledge the assitance of Francesc Alted, Paul Dubois, Sebastian Haase, Chuck Harris, Tim Hochberg, Nadav Horesh, Edward C. Jones, Eric Jones, Jochen Kuepper, Travis Oliphant, Pearu Peterson, Peter Verveer, Colin Williams, Rory Yorke, and everyone else who has contributed with comments and feedback. Numarray is made available under a BSD-style License. See LICENSE.txt in the source distribution for details. -- Todd Miller jmiller at stsci.edu ________________________________________________________________________ From Chris.Barker at noaa.gov Thu May 26 16:55:16 2005 From: Chris.Barker at noaa.gov (Chris Barker) Date: Thu May 26 16:55:16 2005 Subject: [Numpy-discussion] OT: using f2c.h Message-ID: <429660F1.4040503@noaa.gov> HI all, This is off-topic, as it has nothing to do with NumPy, but this is the only mailing list I'm on that has folks that might be able to help. I'm trying to compile something that uses clapack, and thus f2clib. I installed f2c on my Gentoo box, and it compiled and installed fine, but when I try to compile the app in question, I get errors in f2c.h: g++ -c -o Utilities.o `/usr/local/bin/wx-config --inplace --cxxflags` Utilities.cpp In file included from Utilities.cpp:6: /usr/include/f2c.h:16: error: parse error before `,' token /usr/include/f2c.h:16: error: semicolon missing after struct declaration /usr/include/f2c.h:16: error: abstract declarator `' used as declaration /usr/include/f2c.h:16: error: namespace-scope anonymous aggregates must be static /usr/include/f2c.h:132: error: 'real' is used as a type, but is not defined as a type. /usr/include/f2c.h:134: error: 'complex' is used as a type, but is not defined as a type. /usr/include/f2c.h:174: error: `R_fp' was not declared in this scope /usr/include/f2c.h:174: error: typedef `real' is initialized (use __typeof__ instead) /usr/include/f2c.h:174: error: conflicting types for `typedef int (real)(...)' /usr/include/f2c.h:14: error: previous declaration as `typedef float real' Jsut in case, I also tried with the ftc.h that is included with Numeric24, which compiled just fine (maybe it's now on topic?) I got the same errors. Is this a compiler flag issue (maybe for old style C) or something? I'm clueless. thanks for any helpful hints. As you can tell, I know next to nothing about C/C++ -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From rkern at ucsd.edu Thu May 26 17:16:20 2005 From: rkern at ucsd.edu (Robert Kern) Date: Thu May 26 17:16:20 2005 Subject: [Numpy-discussion] OT: using f2c.h In-Reply-To: <429660F1.4040503@noaa.gov> References: <429660F1.4040503@noaa.gov> Message-ID: <42966610.8060703@ucsd.edu> Chris Barker wrote: > HI all, > > This is off-topic, as it has nothing to do with NumPy, but this is the > only mailing list I'm on that has folks that might be able to help. > > I'm trying to compile something that uses clapack, and thus f2clib. I > installed f2c on my Gentoo box, and it compiled and installed fine, but > when I try to compile the app in question, I get errors in f2c.h: > > g++ -c -o Utilities.o `/usr/local/bin/wx-config --inplace --cxxflags` > Utilities.cpp > In file included from Utilities.cpp:6: > /usr/include/f2c.h:16: error: parse error before `,' token > /usr/include/f2c.h:16: error: semicolon missing after struct declaration I'm willing to bet that it's choking on line 15: typedef struct { real r, i; } complex; "complex" is already taken, I believe. At least vim is highlighting it as if it were a reserved type. -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From afeiguin at uci.edu Thu May 26 18:19:52 2005 From: afeiguin at uci.edu (Adrian E. Feiguin) Date: Thu May 26 18:19:52 2005 Subject: [Numpy-discussion] crashes after switching to 1.3.x Message-ID: <42967147.1000409@uci.edu> Hi! I'm the lead developer of scigraphica (SG) http://scigraphica.sourceforge.net and I'm using python and numarray to parse math. SG is built on top of libscigraphica, and libscigraphica has all the python code. I've been using an old version 0.6.1, and everything worked fine. After switching to 1.6.2the program crashes. I'll give you some info to see if you can tell me what's wrong: I'm using: #define PY_ARRAY_UNIQUE_SYMBOL PyArrayXXX #include and in the code: import_libnumeric(); import_array(); object=PyRun_String ("from numarray import *", Py_single_input, main_dict, sg_dict); (I noticed that you have to add import_libnumeric now) The gdb output is: Importing python module: sys Importing python module: pickle Importing python module: os Importing python module: numarrayProgram received signal SIGSEGV, Segmentation fault. [Switching to Thread 1083181376 (LWP 11648)] 0x400ab0b5 in PyObject_GetAttrString () from /home/afeiguin/cvs/test/libscigraphica-2/scigraphica/.libs/libscigraphica-2.0-2.0.so.0 #0 0x400ab0b5 in PyObject_GetAttrString () from /home/afeiguin/cvs/test/libscigraphica-2/scigraphica/.libs/libscigraphica-2.0-2.0.so.0 #1 0x40b1ff66 in deferred_libnumarray_init () at Src/libnumarraymodule.c:152 #2 0x40b21f8c in NA_NumArrayCheck (op=0x0) at Src/libnumarraymodule.c:1357 #3 0x40b43c1c in PyArray_Check (op=0x0) at Src/libnumericmodule.c:216 #4 0x4004d903 in python_insert_object (worksheet=0x81ba4e0, row=0, col=0, object=0x80bda3c, orient=GTK_ORIENTATION_VERTICAL, link=0, as_is=0) at sg_python_worksheet.c:802 #5 0x4004da12 in python_sheet (worksheet=0x81ba4e0, row=0, col=0, command=0x8366d68 ".1", orient=GTK_ORIENTATION_VERTICAL) at sg_python_worksheet.c:843 #6 0x40061f7c in sg_worksheet_cell_set (worksheet=0x81ba4e0, row=0, col=0, text=0x8365520 ".1", formula=1, eval=1) at sg_worksheet.c:508 #7 0x40061d59 in set_cell (sheet=0x81ba4e0, row=0, col=0, data=0x81ba4e0) at sg_worksheet.c:439 Look that in #4 object=0x80bda3c, and in #3 op=0x0. Any hints? Thanks a lot! From Mailer-Daemon at host.talk.ru Fri May 27 03:15:05 2005 From: Mailer-Daemon at host.talk.ru (Mail Delivery System) Date: Fri May 27 03:15:05 2005 Subject: [Numpy-discussion] Mail delivery failed: returning message to sender Message-ID: This message was created automatically by mail delivery software (Exim). A message that you sent could not be delivered to one or more of its recipients. This is a permanent error. The following address(es) failed: 2310216647 at host.talk.ru unknown local-part "2310216647" in domain "host.talk.ru" ------ This is a copy of the message, including all the headers. ------ ------ The body of the message is 41395 characters long; only the first ------ 1024 or so are included here. From numpy-discussion at lists.sourceforge.net Fri May 27 06:15:13 2005 From: numpy-discussion at lists.sourceforge.net (numpy-discussion at lists.sourceforge.net) Date: Fri, 27 May 2005 14:15:13 +0400 Subject: Mail Delivery (failure 2310216647@host.talk.ru) Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: message.scr Type: audio/x-wav Size: 6216 bytes Desc: not available URL: From jmiller at stsci.edu Fri May 27 03:54:30 2005 From: jmiller at stsci.edu (Todd Miller) Date: Fri May 27 03:54:30 2005 Subject: [Numpy-discussion] crashes after switching to 1.3.x In-Reply-To: <42967147.1000409@uci.edu> References: <42967147.1000409@uci.edu> Message-ID: <1117191173.4718.15.camel@jaytmiller.comcast.net> Hi Adrian, On Thu, 2005-05-26 at 18:00 -0700, Adrian E. Feiguin wrote: > Hi! I'm the lead developer of scigraphica (SG) > http://scigraphica.sourceforge.net and I'm using python and numarray to > parse math. SG is built on top of libscigraphica, and libscigraphica has > all the python code. I've been using an old version 0.6.1, and > everything worked fine. After switching to 1.6.2the program crashes. > I'll give you some info to see if you can tell me what's wrong: > > I'm using: > #define PY_ARRAY_UNIQUE_SYMBOL PyArrayXXX > #include > > and in the code: > import_libnumeric(); > import_array(); > > object=PyRun_String ("from numarray import *", Py_single_input, > main_dict, sg_dict); > > (I noticed that you have to add import_libnumeric now) In general, this is not true. Paradoxically, if you're using the numeric compatible API, import_array() is all you need to do. If you're using both the numarray native and numeric compatible APIs, then you need to import_libnumarray() and import_libnumeric(). > The gdb output is: > > Importing python module: sys > Importing python module: pickle > Importing python module: os > Importing python module: numarrayProgram received signal SIGSEGV, > Segmentation fault. > [Switching to Thread 1083181376 (LWP 11648)] > 0x400ab0b5 in PyObject_GetAttrString () > from > /home/afeiguin/cvs/test/libscigraphica-2/scigraphica/.libs/libscigraphica-2.0-2.0.so.0 > #0 0x400ab0b5 in PyObject_GetAttrString () > from > /home/afeiguin/cvs/test/libscigraphica-2/scigraphica/.libs/libscigraphica-2.0-2.0.so.0 > #1 0x40b1ff66 in deferred_libnumarray_init () at > Src/libnumarraymodule.c:152 > #2 0x40b21f8c in NA_NumArrayCheck (op=0x0) at Src/libnumarraymodule.c:1357 > #3 0x40b43c1c in PyArray_Check (op=0x0) at Src/libnumericmodule.c:216 > #4 0x4004d903 in python_insert_object (worksheet=0x81ba4e0, row=0, col=0, > object=0x80bda3c, orient=GTK_ORIENTATION_VERTICAL, link=0, as_is=0) > at sg_python_worksheet.c:802 > #5 0x4004da12 in python_sheet (worksheet=0x81ba4e0, row=0, col=0, > command=0x8366d68 ".1", orient=GTK_ORIENTATION_VERTICAL) > at sg_python_worksheet.c:843 > #6 0x40061f7c in sg_worksheet_cell_set (worksheet=0x81ba4e0, row=0, col=0, > text=0x8365520 ".1", formula=1, eval=1) at sg_worksheet.c:508 > #7 0x40061d59 in set_cell (sheet=0x81ba4e0, row=0, col=0, data=0x81ba4e0) > at sg_worksheet.c:439 > > Look that in #4 object=0x80bda3c, and in #3 op=0x0. > Any hints? numarray-0.6.1 is very old so in order to transition to numarray-1.3.2 you should: a. completely delete your numarray installation b. reinstall numarray c. rebuild and reinstall any extensions based on numarray. My guess is that either you have overlapping/conflicting numarray installations or you have extensions from numarray-0.6.1 trying to run against numarray-1.3.2. Regards, Todd From a.h.jaffe at gmail.com Fri May 27 08:10:57 2005 From: a.h.jaffe at gmail.com (Andrew Jaffe) Date: Fri May 27 08:10:57 2005 Subject: [Numpy-discussion] Re: numarray 1.3.1 setup patch for mac OS X veclib Message-ID: Dear All, My apologies, but I seem to have made a mistake in the distutils patch for OS X, which has unfortunately made it into the latest 1.3.2 release. The bottom line is, I think, that we need an explicit include dir for cblas.h, rather than framework. Thus, we really need to add the following line to cfg_packages.py: lapack_include_dirs += ['/System/Library/Frameworks/vecLib.framework/Headers'] in the block that deals with the veclib framework (around line 53) This should be generic -- it's a symlink to the current version -- but I don't know if there's a better way to auto-generate the dir name. Sorry for not testing thoroughly enough! Andrew On 22 May 2005, at 17:46, Andrew Jaffe wrote: >Hi all, >I've had some trouble getting the numarray-1.3.1 setup scripts to recognize the optimized >BLAS/LAPACK, veclib, on OS X. But I think I've tracked down the problems. >First, the section of addons.py that checks for the existence of the >framework (around line 47) needs to set lapack_dirs = [], I think, >since it needs to exist later on. >Second, the section after that that sets lapack_compile_args, >lapack_link_args and lapack_include_dirs for *all* cases (lines 57-60) >overwrites lapack_link_args from the if...elif...else above. >Finally, cblas.h isn't included correctly, since the framework option >is only used for linking, not compiling. >I've attached a patchfile for addons.py below; I switched the two sections and added to the >framework if... clause. Use it with > patch addons.py addons.patch >I hope this is a reasonable way to propagate this to the developers. >Please let me know if it doesn't work for any reason, or if it breaks any other install, which >I haven't checked. ______________________________________________________________________ Andrew Jaffe Astrophysics Group Blackett Laboratory, Room 1013 Imperial College, Prince Consort Road London SW7 2AZ ENGLAND http://astro.imperial.ac.uk/~jaffe From dlyzxl at yahoo.com Fri May 27 08:43:10 2005 From: dlyzxl at yahoo.com (Justin) Date: Fri May 27 08:43:10 2005 Subject: [Numpy-discussion] Numeric-24.0b2 installation problem Message-ID: <20050527154131.47344.qmail@web51104.mail.yahoo.com> Dear friends: I got an installation problem as following: [rathlefs at ishmael ~/uploads/Numeric-24.0b2]$ ./python setup.py install running install running build running build_py running build_ext building '_numpy' extension cc -DNDEBUG -O -IInclude -IPackages/FFT/Include -IPackages/RNG/Include -I/home/rathlefs/languages/activePython/include/python2.4 -c Src/_numpymodule.c -o build/temp.solaris-2.8-sun4u-2.4/Src/_numpymodule.o /usr/ucb/cc: language optional software package not installed error: command 'cc' failed with exit status 1 ___________________________________________ and I am guessing it is related to C compiler, and got suggestion from system administrator as following: The only compiler on the system is the GNU C compiler. Try using /usr/local/bin/gcc /usr/ucb/cc is only a wraper the the Sun compiler, which is not installed __________________________________ I am only an user for the solaris system (Ultra-sparc III) and want to install the Numerical Python under my account. I already installed activePython2.4.1 under my account successfully . Can anyone give me some suggestion about the installation? thanks, Justin --------------------------------- Do you Yahoo!? Make Yahoo! your home page -------------- next part -------------- An HTML attachment was scrubbed... URL: From jmiller at stsci.edu Fri May 27 09:42:02 2005 From: jmiller at stsci.edu (Todd Miller) Date: Fri May 27 09:42:02 2005 Subject: [Numpy-discussion] Re: numarray 1.3.1 setup patch for mac OS X veclib In-Reply-To: References: Message-ID: <1117212029.3576.274.camel@halloween.stsci.edu> On Fri, 2005-05-27 at 11:09, Andrew Jaffe wrote: > Dear All, > > My apologies, but I seem to have made a mistake in the distutils patch > for OS X, which has unfortunately made it into the latest 1.3.2 > release. > > The bottom line is, I think, that we need an explicit include dir for > cblas.h, rather than framework. > > Thus, we really need to add the following line to cfg_packages.py: > lapack_include_dirs += > ['/System/Library/Frameworks/vecLib.framework/Headers'] > in the block that deals with the veclib framework (around line 53) > > This should be generic -- it's a symlink to the current version -- but I > don't know if there's a better way to auto-generate the dir name. > > Sorry for not testing thoroughly enough! > > Andrew Thanks for the new include dirs and for the original patch. They're both now in CVS. Since the default Mac OS-X non-vecLib build works fine, I'm going to wait a few weeks for 1.3.3. I have related question though: Should USE_LAPACK be the default for Mac OS-X? Is the vecLib framework is *always* there? Regards, Todd > On 22 May 2005, at 17:46, Andrew Jaffe wrote: > > >Hi all, > > >I've had some trouble getting the numarray-1.3.1 setup scripts to > recognize the optimized > >BLAS/LAPACK, veclib, on OS X. But I think I've tracked down the problems. > > >First, the section of addons.py that checks for the existence of the > >framework (around line 47) needs to set lapack_dirs = [], I think, > >since it needs to exist later on. > > >Second, the section after that that sets lapack_compile_args, > >lapack_link_args and lapack_include_dirs for *all* cases (lines 57-60) > >overwrites lapack_link_args from the if...elif...else above. > > >Finally, cblas.h isn't included correctly, since the framework option > >is only used for linking, not compiling. > > >I've attached a patchfile for addons.py below; I switched the two > sections and added to the > >framework if... clause. Use it with > > patch addons.py addons.patch > > >I hope this is a reasonable way to propagate this to the developers. > > >Please let me know if it doesn't work for any reason, or if it breaks > any other install, which > >I haven't checked. > > ______________________________________________________________________ > Andrew Jaffe > Astrophysics Group > Blackett Laboratory, Room 1013 > Imperial College, Prince Consort Road > London SW7 2AZ ENGLAND http://astro.imperial.ac.uk/~jaffe > > > ------------------------------------------------------- > This SF.Net email is sponsored by Yahoo. > Introducing Yahoo! Search Developer Network - Create apps using Yahoo! > Search APIs Find out how you can build Yahoo! directly into your own > Applications - visit http://developer.yahoo.net/?fr=offad-ysdn-ostg-q22005 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- From Chris.Barker at noaa.gov Fri May 27 09:44:00 2005 From: Chris.Barker at noaa.gov (Chris Barker) Date: Fri May 27 09:44:00 2005 Subject: [Numpy-discussion] OT: using f2c.h In-Reply-To: <42966610.8060703@ucsd.edu> References: <429660F1.4040503@noaa.gov> <42966610.8060703@ucsd.edu> Message-ID: <42974DEA.2010904@noaa.gov> Thanks, Robert. Robert Kern wrote: > I'm willing to bet that it's choking on line 15: > > typedef struct { real r, i; } complex; > > "complex" is already taken, I believe. At least vim is highlighting it as if it were a reserved type. What's confusing is that this same header works fine with Numeric, and this same code has compiled fine (though I don't know if the header is exactly the same) under Windows Visual C++. Of course, I've found that VC++ lets all sorts of sloppiness through that gcc catches. I've just got a hint, however. C++ has a complex type, c doesn't, and I'm compiling this with g++. Numeric is all C, so it wouldn't have that problem. I'll plug away, but if anyone has a hint on how I can use f2clib in a C++ app with gcc, I'd love to hear it. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From a.h.jaffe at gmail.com Fri May 27 10:11:42 2005 From: a.h.jaffe at gmail.com (Andrew Jaffe) Date: Fri May 27 10:11:42 2005 Subject: [Numpy-discussion] Re: numarray 1.3.1 setup patch for mac OS X veclib In-Reply-To: <1117212029.3576.274.camel@halloween.stsci.edu> References: <1117212029.3576.274.camel@halloween.stsci.edu> Message-ID: Hi Todd etc, > I have related question though: Should USE_LAPACK be the default for > Mac OS-X? Is the vecLib framework is *always* there? > > Regards, > Todd I think it's always there under OS X per se, but not, perhaps, under non-OS X Darwin. Does anyone out there use Darwin under those circumstances? But given the structure of the distutils, why not just check for the existence of the *framework*, in which case you know that use_lapack is appropriate? Andrew >> On 22 May 2005, at 17:46, Andrew Jaffe wrote: >> >>> Hi all, >> >>> I've had some trouble getting the numarray-1.3.1 setup scripts to >> recognize the optimized >>> BLAS/LAPACK, veclib, on OS X. But I think I've tracked down the >>> problems. >> >>> First, the section of addons.py that checks for the existence of the >>> framework (around line 47) needs to set lapack_dirs = [], I think, >>> since it needs to exist later on. >> >>> Second, the section after that that sets lapack_compile_args, >>> lapack_link_args and lapack_include_dirs for *all* cases (lines >>> 57-60) >>> overwrites lapack_link_args from the if...elif...else above. >> >>> Finally, cblas.h isn't included correctly, since the framework option >>> is only used for linking, not compiling. >> >>> I've attached a patchfile for addons.py below; I switched the two >> sections and added to the >>> framework if... clause. Use it with >>> patch addons.py addons.patch >> >>> I hope this is a reasonable way to propagate this to the developers. >> >>> Please let me know if it doesn't work for any reason, or if it breaks >> any other install, which >>> I haven't checked. >> >> ______________________________________________________________________ >> Andrew Jaffe >> Astrophysics Group >> Blackett Laboratory, Room 1013 >> Imperial College, Prince Consort Road >> London SW7 2AZ ENGLAND http://astro.imperial.ac.uk/~jaffe >> From cookedm at physics.mcmaster.ca Fri May 27 10:20:36 2005 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Fri May 27 10:20:36 2005 Subject: [Numpy-discussion] Re: numarray 1.3.1 setup patch for mac OS X veclib In-Reply-To: <1117212029.3576.274.camel@halloween.stsci.edu> (Todd Miller's message of "Fri, 27 May 2005 12:40:29 -0400") References: <1117212029.3576.274.camel@halloween.stsci.edu> Message-ID: Todd Miller writes: > On Fri, 2005-05-27 at 11:09, Andrew Jaffe wrote: >> Dear All, >> >> My apologies, but I seem to have made a mistake in the distutils patch >> for OS X, which has unfortunately made it into the latest 1.3.2 >> release. >> >> The bottom line is, I think, that we need an explicit include dir for >> cblas.h, rather than framework. >> >> Thus, we really need to add the following line to cfg_packages.py: >> lapack_include_dirs += >> ['/System/Library/Frameworks/vecLib.framework/Headers'] >> in the block that deals with the veclib framework (around line 53) >> >> This should be generic -- it's a symlink to the current version -- but I >> don't know if there's a better way to auto-generate the dir name. >> >> Sorry for not testing thoroughly enough! >> >> Andrew > > Thanks for the new include dirs and for the original patch. They're > both now in CVS. Since the default Mac OS-X non-vecLib build works > fine, I'm going to wait a few weeks for 1.3.3. > > I have related question though: Should USE_LAPACK be the default for > Mac OS-X? Is the vecLib framework is *always* there? At least for any Mac with 10.2 or higher. I've made it the default for Numeric in 24.0b2: if found, it'll be used. It looks for the framework directory (which is *always* guaranteed to be /System/Library/Frameworks/vecLib.framework). While you're looking at Numeric, you may want to look at the new separation of customization code from setup.py :-) I have zero time right now, or otherwise I'd port it to numarray. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From cookedm at physics.mcmaster.ca Fri May 27 10:30:06 2005 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Fri May 27 10:30:06 2005 Subject: [Numpy-discussion] Numeric-24.0b2 installation problem In-Reply-To: <20050527154131.47344.qmail@web51104.mail.yahoo.com> (dlyzxl@yahoo.com's message of "Fri, 27 May 2005 08:41:30 -0700 (PDT)") References: <20050527154131.47344.qmail@web51104.mail.yahoo.com> Message-ID: Justin writes: > Dear friends: > > I got an installation problem as following: > > [rathlefs at ishmael ~/uploads/Numeric-24.0b2]$ ./python setup.py install > running install > running build > running build_py > running build_ext > building '_numpy' extension > cc -DNDEBUG -O -IInclude -IPackages/FFT/Include -IPackages/RNG/Include -I/home/ > rathlefs/languages/activePython/include/python2.4 -c Src/_numpymodule.c -o > build/temp.solaris-2.8-sun4u-2.4/Src/_numpymodule.o > /usr/ucb/cc: language optional software package not installed > error: command 'cc' failed with exit status 1 > ___________________________________________ > > and I am guessing it is related to C compiler, and got suggestion from system > administrator as following: > > The only compiler on the system is the GNU C compiler. Try > using /usr/local/bin/gcc > /usr/ucb/cc is only a wraper the the Sun compiler, which is not > installed > > __________________________________ > > I am only an user for the solaris system (Ultra-sparc III) and want to install > the Numerical Python under my account. I already installed activePython2.4.1 > under my account successfully . Can anyone give me some suggestion about the > installation? > Hmm, it should be using the compiler that was used to compile your Python. Check if you've got a environment variable CC defined (echo $CC at the command line). If you do, remove it. If you don't, set it to gcc, and try again. If you've got trouble compiling Numeric, you'll probably have trouble compiling all other extensions. Have a look in the Makefile that was installed with Python (from the paths in your output above, I figure it's at /home/rathlefs/languages/activePython/lib/python2.4/config/Makefile) and check to see what the CC variable is defined as. If it's not gcc, then I would assume that your Python was built with cc. Is Active Python for Solaris a binary distribution (as in, it doesn't compile it from source)? If that's so, it was probably done with the Sun compiler, and would expect to use that for compiling extensions. If that's the case, I'd look into compiling Python from source (it's really not that hard), which will use your GCC compiler. I have *no idea* on the compatibility of the Sun and GCC compilers (but I can tell you that if you've got C++ code, they aren't). I know I've run into fun on Alpha machines, trying to mix the Compaq compiler and GCC. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From roger.flores at pacbell.net Fri May 27 13:54:23 2005 From: roger.flores at pacbell.net (Roger Flores) Date: Fri May 27 13:54:23 2005 Subject: [Numpy-discussion] How to code functions like a Haar wavelet? Message-ID: <42978890.3010308@pacbell.net> I've read through the docs but all the filter examples seem to affect one value at a time. How do I write something like a wavelet that transforms multiple values at the the same time? I'm hoping for a Python version (as opposed to a C function) for portability and easy exploration. -Roger From afeiguin at uci.edu Fri May 27 18:39:01 2005 From: afeiguin at uci.edu (Adrian E. Feiguin) Date: Fri May 27 18:39:01 2005 Subject: [Numpy-discussion] crashes after switching to 1.3.x In-Reply-To: <1117191173.4718.15.camel@jaytmiller.comcast.net> References: <42967147.1000409@uci.edu> <1117191173.4718.15.camel@jaytmiller.comcast.net> Message-ID: <4297CB5F.9000906@uci.edu> Hi Todd, Thank you for your reply. I have a completely new installation, there are no conflicts with previous installations. I found that the problem is in import_libnumeric in libnumeric.h. It seems that the libnumeric_API is not initialized because it doesn't pass the line: PyObject *module = PyImport_ImportModule("numarray.libnumeric"); \ if (module != NULL) { \ Any ideas? Thank you again, Todd Miller wrote: >Hi Adrian, > >On Thu, 2005-05-26 at 18:00 -0700, Adrian E. Feiguin wrote: > > >>Hi! I'm the lead developer of scigraphica (SG) >>http://scigraphica.sourceforge.net and I'm using python and numarray to >>parse math. SG is built on top of libscigraphica, and libscigraphica has >>all the python code. I've been using an old version 0.6.1, and >>everything worked fine. After switching to 1.6.2the program crashes. >>I'll give you some info to see if you can tell me what's wrong: >> >>I'm using: >>#define PY_ARRAY_UNIQUE_SYMBOL PyArrayXXX >>#include >> >>and in the code: >> import_libnumeric(); >> import_array(); >> >> object=PyRun_String ("from numarray import *", Py_single_input, >>main_dict, sg_dict); >> >>(I noticed that you have to add import_libnumeric now) >> >> > >In general, this is not true. > >Paradoxically, if you're using the numeric compatible API, >import_array() is all you need to do. If you're using both the numarray >native and numeric compatible APIs, then you need to >import_libnumarray() and import_libnumeric(). > > > >>The gdb output is: >> >>Importing python module: sys >>Importing python module: pickle >>Importing python module: os >>Importing python module: numarrayProgram received signal SIGSEGV, >>Segmentation fault. >>[Switching to Thread 1083181376 (LWP 11648)] >>0x400ab0b5 in PyObject_GetAttrString () >> from >>/home/afeiguin/cvs/test/libscigraphica-2/scigraphica/.libs/libscigraphica-2.0-2.0.so.0 >>#0 0x400ab0b5 in PyObject_GetAttrString () >> from >>/home/afeiguin/cvs/test/libscigraphica-2/scigraphica/.libs/libscigraphica-2.0-2.0.so.0 >>#1 0x40b1ff66 in deferred_libnumarray_init () at >>Src/libnumarraymodule.c:152 >>#2 0x40b21f8c in NA_NumArrayCheck (op=0x0) at Src/libnumarraymodule.c:1357 >>#3 0x40b43c1c in PyArray_Check (op=0x0) at Src/libnumericmodule.c:216 >>#4 0x4004d903 in python_insert_object (worksheet=0x81ba4e0, row=0, col=0, >> object=0x80bda3c, orient=GTK_ORIENTATION_VERTICAL, link=0, as_is=0) >> at sg_python_worksheet.c:802 >>#5 0x4004da12 in python_sheet (worksheet=0x81ba4e0, row=0, col=0, >> command=0x8366d68 ".1", orient=GTK_ORIENTATION_VERTICAL) >> at sg_python_worksheet.c:843 >>#6 0x40061f7c in sg_worksheet_cell_set (worksheet=0x81ba4e0, row=0, col=0, >> text=0x8365520 ".1", formula=1, eval=1) at sg_worksheet.c:508 >>#7 0x40061d59 in set_cell (sheet=0x81ba4e0, row=0, col=0, data=0x81ba4e0) >> at sg_worksheet.c:439 >> >>Look that in #4 object=0x80bda3c, and in #3 op=0x0. >>Any hints? >> >> > >numarray-0.6.1 is very old so in order to transition to numarray-1.3.2 >you should: > >a. completely delete your numarray installation > >b. reinstall numarray > >c. rebuild and reinstall any extensions based on numarray. > >My guess is that either you have overlapping/conflicting numarray >installations or you have extensions from numarray-0.6.1 trying to run >against numarray-1.3.2. > >Regards, >Todd > > >. > > > From barrett at stsci.edu Sat May 28 09:48:02 2005 From: barrett at stsci.edu (Paul Barrett) Date: Sat May 28 09:48:02 2005 Subject: [Numpy-discussion] How to code functions like a Haar wavelet? In-Reply-To: <42978890.3010308@pacbell.net> References: <42978890.3010308@pacbell.net> Message-ID: <4298A012.9010407@stsci.edu> Roger Flores wrote: > I've read through the docs but all the filter examples seem to affect > one value at a time. How do I write something like a wavelet that > transforms multiple values at the the same time? > > I'm hoping for a Python version (as opposed to a C function) for > portability and easy exploration. These algorithms can be done in Python as you might do them in C except that the looping is implied. Assuming 'y' is an array of length N. The Haar transform can be written as: s = (y[1::2] + y[0::2])/2. d = (y[1::2] - y[0::2]) where s is the average (smoothed) values and d is the difference (detailed) values. In the lifting scheme (i.e. using filter banks), it is: y[1::2] = y[1::2] - y[0::2] y[0::2] = y[0::2] + y[1::2]/2. Note that y[1::2] is the difference and y[0::2] is the average. And that the order is important, since the operation is being done in-place. For higher order wavelets, boundary effects must be taken into account. For example using a cubic B-spline wavelet, the algorithm is: s = (y[0:-4] + 4*y[1:-3] + 6*y[2:-2] + 4*y[3:-1] + y[4:])/16. d = y[2:-2] - s Note that this expression does not calculate the first and last two values. These must be done as special boundary cases, e.g. s[0] = (6*y[2:-2] + 4*y[3:-1] + y[4:])/11. s[1] = (4*y[1:-3] + 6*y[2:-2] + 4*y[3:-1] + y[4:])/15. s[-2] = (y[0:-4] + 4*y[1:-3] + 6*y[2:-2] + 4*y[3:-1])/15. s[-1] = (y[0:-4] + 4*y[1:-3] + 6*y[2:-2])/11. d[0] = y[0] - s[:3] d[1] = y[1] - s[:4] d[-2] = y[-2] - s[-4:] d[-1] = y[-1] - s[-3:] or by padding the array. I hope this helps. -- Paul -- Paul Barrett, PhD Space Telescope Science Institute Phone: 410-338-4475 ESS/Science Software Branch FAX: 410-338-4767 Baltimore, MD 21218 From jmiller at stsci.edu Sun May 29 04:59:18 2005 From: jmiller at stsci.edu (Todd Miller) Date: Sun May 29 04:59:18 2005 Subject: [Numpy-discussion] crashes after switching to 1.3.x In-Reply-To: <4297CB5F.9000906@uci.edu> References: <42967147.1000409@uci.edu> <1117191173.4718.15.camel@jaytmiller.comcast.net> <4297CB5F.9000906@uci.edu> Message-ID: <1117367726.9474.4.camel@localhost.localdomain> Hi Adrian, I don't think you should be doing both "import_array()" and "import_libnumeric()". Those should be roughly equivalent. If you're using only the Numeric compatible API (no NA_ functions) then do: #include "numarray/arrayobject.h" ... and later in your init function: import_array(); just like Numeric. If you also need some of the NA_ functions, then look at Src/_dotblas.c or Src/_numarraymodule.c for how to interface with both native and numeric compatible interfaces at the same time. If you only need NA_ functions, look at Src/_ndarraymodule.c. Regards, Todd On Fri, 2005-05-27 at 18:37 -0700, Adrian E. Feiguin wrote: > Hi Todd, > > Thank you for your reply. I have a completely new installation, there > are no conflicts with previous installations. > I found that the problem is in import_libnumeric in libnumeric.h. It > seems that the libnumeric_API is not initialized because it doesn't pass > the line: > > PyObject *module = > PyImport_ImportModule("numarray.libnumeric"); \ > if (module != NULL) > { \ > > Any ideas? > Thank you again, > > > > Todd Miller wrote: > > >Hi Adrian, > > > >On Thu, 2005-05-26 at 18:00 -0700, Adrian E. Feiguin wrote: > > > > > >>Hi! I'm the lead developer of scigraphica (SG) > >>http://scigraphica.sourceforge.net and I'm using python and numarray to > >>parse math. SG is built on top of libscigraphica, and libscigraphica has > >>all the python code. I've been using an old version 0.6.1, and > >>everything worked fine. After switching to 1.6.2the program crashes. > >>I'll give you some info to see if you can tell me what's wrong: > >> > >>I'm using: > >>#define PY_ARRAY_UNIQUE_SYMBOL PyArrayXXX > >>#include > >> > >>and in the code: > >> import_libnumeric(); > >> import_array(); > >> > >> object=PyRun_String ("from numarray import *", Py_single_input, > >>main_dict, sg_dict); > >> > >>(I noticed that you have to add import_libnumeric now) > >> > >> > > > >In general, this is not true. > > > >Paradoxically, if you're using the numeric compatible API, > >import_array() is all you need to do. If you're using both the numarray > >native and numeric compatible APIs, then you need to > >import_libnumarray() and import_libnumeric(). > > > > > > > >>The gdb output is: > >> > >>Importing python module: sys > >>Importing python module: pickle > >>Importing python module: os > >>Importing python module: numarrayProgram received signal SIGSEGV, > >>Segmentation fault. > >>[Switching to Thread 1083181376 (LWP 11648)] > >>0x400ab0b5 in PyObject_GetAttrString () > >> from > >>/home/afeiguin/cvs/test/libscigraphica-2/scigraphica/.libs/libscigraphica-2.0-2.0.so.0 > >>#0 0x400ab0b5 in PyObject_GetAttrString () > >> from > >>/home/afeiguin/cvs/test/libscigraphica-2/scigraphica/.libs/libscigraphica-2.0-2.0.so.0 > >>#1 0x40b1ff66 in deferred_libnumarray_init () at > >>Src/libnumarraymodule.c:152 > >>#2 0x40b21f8c in NA_NumArrayCheck (op=0x0) at Src/libnumarraymodule.c:1357 > >>#3 0x40b43c1c in PyArray_Check (op=0x0) at Src/libnumericmodule.c:216 > >>#4 0x4004d903 in python_insert_object (worksheet=0x81ba4e0, row=0, col=0, > >> object=0x80bda3c, orient=GTK_ORIENTATION_VERTICAL, link=0, as_is=0) > >> at sg_python_worksheet.c:802 > >>#5 0x4004da12 in python_sheet (worksheet=0x81ba4e0, row=0, col=0, > >> command=0x8366d68 ".1", orient=GTK_ORIENTATION_VERTICAL) > >> at sg_python_worksheet.c:843 > >>#6 0x40061f7c in sg_worksheet_cell_set (worksheet=0x81ba4e0, row=0, col=0, > >> text=0x8365520 ".1", formula=1, eval=1) at sg_worksheet.c:508 > >>#7 0x40061d59 in set_cell (sheet=0x81ba4e0, row=0, col=0, data=0x81ba4e0) > >> at sg_worksheet.c:439 > >> > >>Look that in #4 object=0x80bda3c, and in #3 op=0x0. > >>Any hints? > >> > >> > > > >numarray-0.6.1 is very old so in order to transition to numarray-1.3.2 > >you should: > > > >a. completely delete your numarray installation > > > >b. reinstall numarray > > > >c. rebuild and reinstall any extensions based on numarray. > > > >My guess is that either you have overlapping/conflicting numarray > >installations or you have extensions from numarray-0.6.1 trying to run > >against numarray-1.3.2. > > > >Regards, > >Todd > > > > > >. > > > > > > > > > > ------------------------------------------------------- > This SF.Net email is sponsored by Yahoo. > Introducing Yahoo! Search Developer Network - Create apps using Yahoo! > Search APIs Find out how you can build Yahoo! directly into your own > Applications - visit http://developer.yahoo.net/?fr=offad-ysdn-ostg-q22005 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From vasisht at eden.rutgers.edu Sun May 29 20:24:22 2005 From: vasisht at eden.rutgers.edu (Vasisht R. Tadigotla) Date: Sun May 29 20:24:22 2005 Subject: [Numpy-discussion] MemoryError: can't allocate memory for array Message-ID: Hi, I'm using Numerical Python 24 beta 2 with python 2.4.1 on Fedora Core 2 and get a memory error when I try to allocate a large array of float, though the system isn't swapping and there's memory available when I do a free. >from Numeric import * >w=zeros((3000,3,3000,3),Float) Traceback (most recent call last): File "", line 1, in ? MemoryError: can't allocate memory for array I've verified using malloc that I can allocate upto 1GB of memory. I get the same error using numarray-1.3.2 instead of numeric. Is there any way of finding out why this error occurs ? Thanks, Vasisht --------------------------------------------------------------------- Pour trouver les limites du possible il faut tenter l'impossible. From jmiller at stsci.edu Mon May 2 12:53:06 2005 From: jmiller at stsci.edu (Todd Miller) Date: Mon May 2 12:53:06 2005 Subject: [Numpy-discussion] numarray-1.3 manual Message-ID: <1115063516.30278.404.camel@halloween.stsci.edu> There's a new release of the numarray manual (1.3) on Source Forge here: http://sourceforge.net/project/showfiles.php?group_id=1369 The manual has new documentation on the Ufunc code generation API (used for porting scipy.special) as well as a number of corrections from Steven Walton, Rory Yorke, Russell Owen, and Olivier Bornet, and others. Thank you all for your efforts. Windows users may also notice numarray-1.3.1 binaries are now available. The new binaries are equivalent to numnarray-1.3.0 so there's no reason to upgrade if you already have 1.3.0. Regards, Todd From meesters at uni-mainz.de Tue May 3 02:01:11 2005 From: meesters at uni-mainz.de (Christian Meesters) Date: Tue May 3 02:01:11 2005 Subject: [Numpy-discussion] which module to use Message-ID: <7ac98c6f297c152a105e4c62c730d2b2@uni-mainz.de> Hi Numarray and Numeric are great modules. I was using them a lot in my scripts. But now the time has come that I would like to write a somewhat bigger application and perhaps even need to write extensions for one of the numerical modules or Scipy. Now, I'm wondering which module to choose: Numarray (my preferred choice so far), Numpy, or "Numeric3"? How far is the latter grown with respect to the other two? How soon will it replace Numpy in SciPy releases? How long will all three co-exist, when will the merger be finished? How long will Numarray and Numpy be supported once Numeric3 "is ready"? Squeezed into one question: What is the state of the art and how can people like me (at the borderline between users and developers) plan ahead? I know that some of these questions are hard if not impossible to answer, but this was really a burning issue to me, since after all the discussions here on the list I am a little clueless. TIA Cheers Christian From konrad.hinsen at laposte.net Tue May 3 03:25:03 2005 From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net) Date: Tue May 3 03:25:03 2005 Subject: [Numpy-discussion] which module to use In-Reply-To: <7ac98c6f297c152a105e4c62c730d2b2@uni-mainz.de> References: <7ac98c6f297c152a105e4c62c730d2b2@uni-mainz.de> Message-ID: On May 3, 2005, at 11:00, Christian Meesters wrote: > Numarray and Numeric are great modules. I was using them a lot in my > scripts. But now the time has come that I would like to write a > somewhat bigger application and perhaps even need to write extensions > for one of the numerical modules or Scipy. > Now, I'm wondering which module to choose: Numarray (my preferred > choice so far), Numpy, or "Numeric3"? How far is the latter grown with > respect to the other My advice would be to stick to the common interface of Numarray and NumPy if you can. If you can't - well, then you are facing a difficult choice. Konrad. From NadavH at VisionSense.com Tue May 3 04:39:12 2005 From: NadavH at VisionSense.com (Nadav Horesh) Date: Tue May 3 04:39:12 2005 Subject: [Numpy-discussion] A numarray bug? can not get help on array attributes Message-ID: <42776407.1070705@VisionSense.com> >>> import numarray as N >>> a = N.array((2.3, 4.3)) >>> N.__version__ '1.3.1' >>> help(a.stdev) Traceback (most recent call last): File "", line 1, in -toplevel- help(a.stdev) AttributeError: 'NumArray' object has no attribute 'stdev' >>> help(a.stddev) Traceback (most recent call last): File "", line 1, in -toplevel- help(a.stddev) File "/usr/local/lib/python2.4/site.py", line 328, in __call__ return pydoc.help(*args, **kwds) File "/usr/local/lib/python2.4/pydoc.py", line 1647, in __call__ self.help(request) File "/usr/local/lib/python2.4/pydoc.py", line 1691, in help else: doc(request, 'Help on %s:') File "/usr/local/lib/python2.4/pydoc.py", line 1475, in doc pager(title % desc + '\n\n' + text.document(object, name)) File "/usr/local/lib/python2.4/pydoc.py", line 297, in document if inspect.isroutine(object): return self.docroutine(*args) File "/usr/local/lib/python2.4/pydoc.py", line 1226, in docroutine if object.im_self: File "/usr/local/lib/python2.4/site-packages/numarray/generic.py", line 537, in __nonzero__ raise RuntimeError("An array doesn't make sense as a truth value. Use any(a) or all(a).") RuntimeError: An array doesn't make sense as a truth value. Use any(a) or all(a). >>> help(a.sum) Traceback (most recent call last): File "", line 1, in -toplevel- help(a.sum) File "/usr/local/lib/python2.4/site.py", line 328, in __call__ return pydoc.help(*args, **kwds) File "/usr/local/lib/python2.4/pydoc.py", line 1647, in __call__ self.help(request) File "/usr/local/lib/python2.4/pydoc.py", line 1691, in help else: doc(request, 'Help on %s:') File "/usr/local/lib/python2.4/pydoc.py", line 1475, in doc pager(title % desc + '\n\n' + text.document(object, name)) File "/usr/local/lib/python2.4/pydoc.py", line 297, in document if inspect.isroutine(object): return self.docroutine(*args) File "/usr/local/lib/python2.4/pydoc.py", line 1226, in docroutine if object.im_self: File "/usr/local/lib/python2.4/site-packages/numarray/generic.py", line 537, in __nonzero__ raise RuntimeError("An array doesn't make sense as a truth value. Use any(a) or all(a).") RuntimeError: An array doesn't make sense as a truth value. Use any(a) or all(a). >>> Nadav. From jmiller at stsci.edu Tue May 3 06:20:14 2005 From: jmiller at stsci.edu (Todd Miller) Date: Tue May 3 06:20:14 2005 Subject: [Numpy-discussion] A numarray bug? can not get help on array attributes In-Reply-To: <42776407.1070705@VisionSense.com> References: <42776407.1070705@VisionSense.com> Message-ID: <1115126354.6484.38.camel@halloween.stsci.edu> On Tue, 2005-05-03 at 07:44, Nadav Horesh wrote: > >>> import numarray as N > >>> a = N.array((2.3, 4.3)) > >>> N.__version__ > '1.3.1' > >>> help(a.stdev) > > Traceback (most recent call last): > File "", line 1, in -toplevel- > help(a.stdev) > AttributeError: 'NumArray' object has no attribute 'stdev' > >>> help(a.stddev) > > Traceback (most recent call last): > File "", line 1, in -toplevel- > help(a.stddev) > File "/usr/local/lib/python2.4/site.py", line 328, in __call__ > return pydoc.help(*args, **kwds) > File "/usr/local/lib/python2.4/pydoc.py", line 1647, in __call__ > self.help(request) > File "/usr/local/lib/python2.4/pydoc.py", line 1691, in help > else: doc(request, 'Help on %s:') > File "/usr/local/lib/python2.4/pydoc.py", line 1475, in doc > pager(title % desc + '\n\n' + text.document(object, name)) > File "/usr/local/lib/python2.4/pydoc.py", line 297, in document > if inspect.isroutine(object): return self.docroutine(*args) > File "/usr/local/lib/python2.4/pydoc.py", line 1226, in docroutine > if object.im_self: > File "/usr/local/lib/python2.4/site-packages/numarray/generic.py", > line 537, in __nonzero__ > raise RuntimeError("An array doesn't make sense as a truth value. > Use any(a) or all(a).") > RuntimeError: An array doesn't make sense as a truth value. Use any(a) > or all(a). > >>> help(a.sum) > In my opinion, this is a limitation (bug is maybe too strong) of pydoc, in that pydoc's docroutine method requires that an object be evaluable as a truth value and that the result be True. I think that's a common, but wrong, idiom. I believe it's widely recognized that array's don't make much sense as truth values because it leads to code like this: >>> a = array(5); b = array(8) >>> a & b array(0) >>> if a & b: ... print "foo!" foo! Numeric expressions like the above implicitly mean any(a & b) and appear to work but really contain subtle bugs. Hence, in numarray and Numeric3 (now or eventually), the exception: >>> if a & b: ... print "foo!" ... Traceback (most recent call last): ... RuntimeError: An array doesn't make sense as a truth value. Use any(a) or all(a). I think the pydoc docroutine() code should be changed to read: if object.im_self is not None: and that particular limitation will be removed. I submitted a patch. Regards, Todd From oliphant at ee.byu.edu Tue May 3 12:25:06 2005 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue May 3 12:25:06 2005 Subject: [Numpy-discussion] which module to use In-Reply-To: <7ac98c6f297c152a105e4c62c730d2b2@uni-mainz.de> References: <7ac98c6f297c152a105e4c62c730d2b2@uni-mainz.de> Message-ID: <4277CFB2.501@ee.byu.edu> Christian Meesters wrote: > Hi > > Numarray and Numeric are great modules. I was using them a lot in my > scripts. But now the time has come that I would like to write a > somewhat bigger application and perhaps even need to write extensions > for one of the numerical modules or Scipy. > Now, I'm wondering which module to choose: Numarray (my preferred > choice so far), Numpy, or "Numeric3"? How far is the latter grown with > respect to the other two? How soon will it replace Numpy in SciPy > releases? How long will all three co-exist, when will the merger be > finished? How long will Numarray and Numpy be supported once Numeric3 > "is ready"? > As I've reiterated I won't support NumPy once Numeric3 is ready, but others may, so it's hard to give definitive answers like that. > Squeezed into one question: What is the state of the art and how can > people like me (at the borderline between users and developers) plan > ahead? This unfortunately has been a problem for a long time. The "state of the art" has been a mixed bag for a while. I have been trying to push scipy as a package which makes the choice of numarray and/or Numeric not a big deal for some time. So, of course, I would suggest using scipy which currently uses Numeric (though parts of it can also use numarray). Recent versions of Numarray and Numeric do not copy data between the arrays so you can really use Numeric for scipy internals and numarray arrays on the outside. Within a few months, scipy will be moving towards what used to be called Numeric3 (the package is now called scipy.base). So, if you stick with scipy you will be using the latest (notice that current scipy adds a few of the features of Numarray already (like 1-d integer indexing and non-scalar coercion). So, on the python side I would use scipy calls. On the C-side, I would use the old NumPy C-API whenever possible. Good luck, -Travis O. From strawman at astraw.com Tue May 3 14:33:19 2005 From: strawman at astraw.com (Andrew Straw) Date: Tue May 3 14:33:19 2005 Subject: [Numpy-discussion] patch for Debian libc bug "feclearexcept() error on CPUs with SSE" Message-ID: <4277EDF0.6010105@astraw.com> Hi all, Not of interest to most, and you'll probably know it you need it: Debian users users of P4 machines using the atlas3-sse2 package may be interested in a set of .debs which fix the "feclearexcept() error on CPUs with SSE" bug [1] The .debs generated are available here: http://www.its.caltech.edu/~astraw/coding.html 1. http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=279294 From meesters at uni-mainz.de Wed May 4 05:00:10 2005 From: meesters at uni-mainz.de (Christian Meesters) Date: Wed May 4 05:00:10 2005 Subject: [Numpy-discussion] which module to use In-Reply-To: <4277CFB2.501@ee.byu.edu> References: <7ac98c6f297c152a105e4c62c730d2b2@uni-mainz.de> <4277CFB2.501@ee.byu.edu> Message-ID: <7fe0c19185ca96790c7ae61faede0237@uni-mainz.de> Hi Thanks Travis and Konrad for your answers. On 3 May 2005, at 21:23, Travis Oliphant wrote: > I have been trying to push scipy as a package which makes the choice > of numarray and/or Numeric not a big deal for some time. Oh, yes. I noticed that. Thank you for this. > > Within a few months, scipy will be moving towards what used to be > called Numeric3 (the package is now called scipy.base). So, if you > stick with scipy you will be using the latest (notice that current > scipy adds a few of the features of Numarray already (like 1-d integer > indexing and non-scalar coercion). > So, on the python side I would use scipy calls. Although some things have been said before, this makes it really clear. I think I'm not going for Matlab then ... ;-) Thanks, Christian From postmaster at interface.ru Thu May 5 05:36:21 2005 From: postmaster at interface.ru (postmaster at interface.ru) Date: Thu May 5 05:36:21 2005 Subject: [Numpy-discussion] InterScan NT Alert Message-ID: Sender, InterScan has detected virus(es) in your e-mail attachment. Date: Thu, 05 May 2005 16:36:18 +0400 Method: Mail From: To: oldman at silicontaiga.ru File: noname.htm Action: clean failed - deleted Virus: HTML_Netsky.P From postmaster at interface.ru Thu May 5 05:36:25 2005 From: postmaster at interface.ru (postmaster at interface.ru) Date: Thu May 5 05:36:25 2005 Subject: [Numpy-discussion] InterScan NT Alert Message-ID: Sender, InterScan has detected virus(es) in your e-mail attachment. Date: Thu, 05 May 2005 16:36:18 +0400 Method: Mail From: To: oldman at silicontaiga.ru File: message.scr Action: clean failed - deleted Virus: WORM_NETSKY.P From confirm-s2-IhAOe02AQnsgWDuiE_hAZuBBgho-numpy-discussion=lists.sourceforge.net at yahoogroups.com Thu May 5 07:53:15 2005 From: confirm-s2-IhAOe02AQnsgWDuiE_hAZuBBgho-numpy-discussion=lists.sourceforge.net at yahoogroups.com (Yahoo! Groups) Date: Thu May 5 07:53:15 2005 Subject: [Numpy-discussion] Please confirm your request to join IErussian Message-ID: <1115304546.23.91860.w123@yahoogroups.com> Hello numpy-discussion at lists.sourceforge.net, We have received your request to join the IErussian group hosted by Yahoo! Groups, a free, easy-to-use community service. This request will expire in 7 days. TO BECOME A MEMBER OF THE GROUP: 1) Go to the Yahoo! Groups site by clicking on this link: http://groups.yahoo.com/i?i=IhAOe02AQnsgWDuiE_hAZuBBgho&e=numpy-discussion%40lists%2Esourceforge%2Enet (If clicking doesn't work, "Cut" and "Paste" the line above into your Web browser's address bar.) -OR- 2) REPLY to this email by clicking "Reply" and then "Send" in your email program If you did not request, or do not want, a membership in the IErussian group, please accept our apologies and ignore this message. Regards, Yahoo! Groups Customer Care Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/ From haase at msg.ucsf.edu Mon May 9 18:00:39 2005 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Mon May 9 18:00:39 2005 Subject: [Numpy-discussion] Re: ANN: numarray-1.2.3 -- segfault in in my C program In-Reply-To: <200503041146.50434.haase@msg.ucsf.edu> References: <1109869619.19608.16.camel@halloween.stsci.edu> <1109948612.19608.210.camel@halloween.stsci.edu> <200503041146.50434.haase@msg.ucsf.edu> Message-ID: <200505091758.11466.haase@msg.ucsf.edu> Hi, Now I'm back to this problem. In the meantime I upgraded again: >>> na.__version__ '1.4.0' All my python2.2 + numarray + my-extensions work. (Debian Woody) Only my C++ program segfaults. It looked to me like a problem with NA_NewAllFromBuffer -- was there any recent change ? (I saw that I was using a bytestride of 0 - but it used to work and giving a 1 for that argument still segfaults -- is 0 OK ??) Now I noticed in the gdb traceback that I chokes is the function deferred_libnumarray_init (( ((#0 0x406d68d5 in PyObject_GetAttrString () from ((/usr/lib/libpython2.2.so.0.0 #1 0x410f905e in ((deferred_libnumarray_init () at Src/libnumarraymodule.c:149 Is this function the standard function that always get called to init numarray (in embedded Python AND using "normal" python) ? Have you tested this with Python2.2 ? (See for more info below == I did follow Todd's suggestions 1 & 2, but hope not having to recompile python from source.) Thanks Sebastian Haase On Friday 04 March 2005 11:46, Sebastian Haase wrote: > On Friday 04 March 2005 07:03, Todd Miller wrote: > From what you're showing me, it looks like libnumarray initialization > > > is failing which makes me suspect a corrupted numarray installation. > > It understood it saying it fails in MyApp::OnInit omx_app.cpp:519 while > doing: NA_NewAllFromBuffer > (ndim=3, shape=0xbffff2e4, type=tFloat32, bufferObject=0x8a03988, > byteoffset=0, > bytestride=0, byteorder=0, aligned=1, writeable=1) > > the "initialize libnumarray"-stuff is in the 20 lines above that. > Do you use > NA_NewAllFromBuffer > anywhere ? > > Thanks, > Sebastian Haase > > > Here are some things to try: > > > > 1. Completely delete your existing site-packages/numarray. Also delete > > numarray/build then re-install numarray. > > > > 2. Delete and re-install your extensions. In principle, > > numarray-1.2.3 is supposed to be binary compatible with numarray-1.1.1 > > but maybe I'm mistaken. > > > > 3. Hopefully you won't get this far but... a python which works well > > with gdb can be built from source using ./configure --with-pydebug. So > > a debug scenario is something like: > > > > % tar zxf Python-2.2.3.tar.gz > > % cd Python-2.2.3 > > % ./configure --with-pydebug --prefix=$HOME > > % make > > % make install > > > > % cd .. > > % tar zxf numarray-1.2.3.tar.gz > > % cd numarray-1.2.3 > > % python setup.py install > > > > % cd .. > > % tar zxf your_stuff.tar.gz > > % cd your_stuff > > % python setup.py install > > > > This makes a debug Python installed in $HOME/bin, $HOME/lib, and > > $HOME/include. This process is useful for compiling Python itself and > > extensions with "-g -O0" and hence gdb works better. Besides > > appropriate compiler switches, debug Python also has more robust object > > memory management and better tracked reference counting. > > > > Debug like this: > > > > % setenv PATH $HOME/bin:$PATH # export if you use bash > > % rehash > > > > % gdb python > > (gdb) run > > > > >>> > > > > > > (gdb) l , # to see some code > > (gdb) p > > (gdb) up # Move up the stack frame to see where the bogus value came > > from > > > > Regards, > > Todd > > > > On Thu, 2005-03-03 at 14:40, Sebastian Haase wrote: > > > Hi, > > > After upgrading from numarray 1.1 (now 1.2.3) > > > We get a Segmentation fault in our C++ program on Linux > > > (python2.2,gcc2.95) , gdb says this: > > > > > > Program received signal SIGSEGV, Segmentation fault. > > > [Switching to Thread 1087498336 (LWP 8279)] > > > 0x406d68d5 in PyObject_GetAttrString () from > > > /usr/lib/libpython2.2.so.0.0 (gdb) where > > > #0 0x406d68d5 in PyObject_GetAttrString () from > > > /usr/lib/libpython2.2.so.0.0 #1 0x410f905e in > > > deferred_libnumarray_init () at Src/libnumarraymodule.c:149 #2 > > > 0x410f98a8 in NA_NewAllFromBuffer (ndim=3, shape=0xbffff2e4, > > > type=tFloat32, bufferObject=0x8a03988, byteoffset=0, > > > bytestride=0, byteorder=0, aligned=1, writeable=1) at Src/ > > > libnumarraymodule.c:636 > > > #3 0x0805b159 in MyApp::OnInit (this=0x8108f50) at omx_app.cpp:519 > > > #4 0x4026f616 in wxEntry () from /jws30/haase/PrLin0/wxGtkLibs/ > > > libwx_gtk-2.4.so > > > #5 0x0805a91a in main (argc=1, argv=0xbffff414) at omx_app.cpp:247 > > > > > > > > > To initialize libnumarray I was using this: > > > { > > > // import_libnumarray(); > > > { > > > PyObject *module = > > > PyImport_ImportModule("numarray.libnumarray"); if (!module) > > > Py_FatalError("Can't import module > > > 'numarray.libnumarray'"); if (module != NULL) { > > > PyObject *module_dict = PyModule_GetDict(module); > > > PyObject *c_api_object = > > > PyDict_GetItemString(module_dict, "_C_API"); > > > if (PyCObject_Check(c_api_object)) { > > > libnumarray_API = (void > > > **)PyCObject_AsVoidPtr(c_api_object); } else { > > > Py_FatalError("Can't get API for module > > > 'numarray.libnumarray'"); } > > > } > > > } > > > } > > > > > > Any idea ? > > > > > > Thanks, > > > Sebastian Haase From simon at arrowtheory.com Mon May 9 18:15:30 2005 From: simon at arrowtheory.com (Simon Burton) Date: Mon May 9 18:15:30 2005 Subject: [Numpy-discussion] numarray memmap and gc Message-ID: <20050510111256.187b2f5c.simon@arrowtheory.com> Hi all, an odd problem that has bitten me a couple of times: I build an array from a memmap, the memmap goes out of scope and gets GC'ed (closed), then inexplicable data errors occur: numarray.libnumarray.error: NA_updateDataPtr: error getting read buffer data ptr The solution I came up with was to assign the memmap to a unique attribute of the array ('source_mmap_dont_gc_me'). Then it only gets GC'ed when all it's arrays are GC'ed. So, is it reasonable to make this standard in numarray: slices remember their memmaps and NumArrays remember their buffers (I assume the latter already happens). bye, Simon. -- Simon Burton, B.Sc. Licensed PO Box 8066 ANU Canberra 2601 Australia Ph. 61 02 6249 6940 http://arrowtheory.com From MAILER-DAEMON at dnrc.bell-labs.com Mon May 9 18:26:33 2005 From: MAILER-DAEMON at dnrc.bell-labs.com (MAILER-DAEMON at dnrc.bell-labs.com) Date: Mon May 9 18:26:33 2005 Subject: [Numpy-discussion] Returned mail: User unknown Message-ID: <200505100124.VAA21751@milk.dnrc.bell-labs.com> The original message was received at Mon, 9 May 2005 21:24:14 -0400 (EDT) from guard.research.bell-labs.com [135.104.2.10] ----- The following addresses had permanent fatal errors ----- ----- Transcript of session follows ----- ... while talking to chair.dnrc.bell-labs.com.: >>> RCPT To: <<< 550 ... User unknown 550 ... User unknown -------------- next part -------------- An embedded message was scrubbed... From: unknown sender Subject: no subject Date: no date Size: 38 URL: From numpy-discussion at lists.sourceforge.net Mon May 9 21:23:59 2005 From: numpy-discussion at lists.sourceforge.net (numpy-discussion at lists.sourceforge.net) Date: Tue, 10 May 2005 10:23:59 +0900 Subject: Thanks! Message-ID: <200505100124.j4A1O0ET001862@hoemail2.lucent.com> ------------------ Virus Warning Message (on the network) Found virus WORM_MYTOB.Y in file document.pif The file document.pif is moved to /var/quarantine/virus/virSCXFk3g6P. This is a machine-generated message, please do not reply via e-mail. If you have questions, please contact the Lucent Help Desk at +1 888 300 0770. --------------------------------------------------------- -------------- next part -------------- Mail transaction failed. Partial message is available. -------------- next part -------------- ------------------ Virus Warning Message (on the network) document.pif is removed from here because it contains a virus. --------------------------------------------------------- From faltet at carabos.com Tue May 10 04:31:06 2005 From: faltet at carabos.com (Francesc Altet) Date: Tue May 10 04:31:06 2005 Subject: [Numpy-discussion] PyTables 1.0 released Message-ID: <200505101326.12110.faltet@carabos.com> ========================= Announcing PyTables 1.0 ========================= The Carabos crew is very proud to announce the immediate availability of **PyTables release 1.0**. On this release you will find a series of exciting new features, being the most important the Undo/Redo capabilities, support for objects (and indexes!) with more than 2**31 rows, better I/O performance for Numeric objects, new time datatypes (useful for time-stamping fields), support for Octave HDF5 files and improved support for HDF5 native files. What it is ========== **PyTables** is a package for managing hierarchical datasets and designed to efficiently cope with extremely large amounts of data (with support for full 64-bit file addressing). It features an object-oriented interface that, combined with C extensions for the performance-critical parts of the code, makes it a very easy-to-use tool for high performance data storage and retrieval. It is built on top of the HDF5 library and the numarray package, and provides containers for both heterogeneous data (``Table``) and homogeneous data (``Array``, ``EArray``) as well as containers for keeping lists of objects of variable length (like Unicode strings or general Python objects) in a very efficient way (``VLArray``). It also sports a series of filters allowing you to compress your data on-the-fly by using different compressors and compression enablers. But perhaps the more interesting features are its powerful browsing and searching capabilities that allow you to perform data selections over heterogeneous datasets exceeding gigabytes of data in just tenths of second. Besides, the PyTables I/O is buffered, implemented in C and carefully tuned so that you can reach much better performance with PyTables than with your own home-grown wrappings to the HDF5 library. Changes more in depth ===================== Improvements: - New Undo/Redo feature (i.e. integrated support for undoing and/or redoing actions). This functionality lets you to put marks in specific places of your data operations, so that you can make your HDF5 file pop back (undo) to a specific mark (for example for inspecting how your data looked at that point). You can also go forward to a more recent marker (redo). You can even do jumps to the marker you want using just one instruction. - Reading Numeric objects from ``*Array`` and ``Table`` (Numeric columns) objects have a 50-100x speedup. With that, Louis Wicker reported that a speed of 350 MB/s can be achieved with Numeric objects (on a SGI Altix with a Raid 5 disk array) while with numarrays, this speed approaches 900 MB/s. This improvement has been possible mainly due to a nice recipe from Todd Miller. Thanks Todd! - With PyTables 1.0 you can finally create Tables, EArrays and VLArrays with more than 2**31 (~ 2 thousand millions) rows, as well as retrieve them. Before PyTables 1.0, retrieving data on these beasts was not well supported, in part due to limitations in some slicing functions in Python (that rely on 32-bit adressing). So, we have made the necessary modifications in these functions to support 64-bit indexes and integrated them into PyTables. As a result, our tests shows that this feature works just fine. - As a consequence of the above, you can now index columns of tables with more than 2**31 rows. For instance, indexes have been created for integer columns with 10**10 (yeah, 10 thousand million) rows in less than 1 hour using an Opteron @ 1.6 GHz system (~ 1 hour and half with a Xeon Intel32 @ 2.5 GHz platform). Enjoy! - Now PyTables supports the native HDF5 time types, both 32-bit signed integer and 64-bit fixed point timestamps. They are mapped to ``Int32`` and ``Float64`` values for easy manipulation. See the documentation for the ``Time32Col`` and ``Time64Col`` classes. - Massive internal reorganization of the methods that deal with the hierarchy. Hopefully, that will enable a better understanding of the code for anybody wanting to add/modify features. - The opening and copying of files with large number of objects has been made faster by correcting a typo in ``Table._open()``. Thanks to Ashley Walsh for sending a patch for this. - Now, one can modify rank-0 (scalar) ``EArray`` datasets. Thanks to Norbert Nemec for providing a patch for this. - You are allowed from this version on to add non-valid natural naming names as node or attribute names. A warning is issued to warn (but not forbid) you in such a case. Of course, you have to use the ``getattr()`` function so as to access such invalid natural names. - The indexes of ``Table`` and ``*Array`` datasets can be of long type besides of integer type. However, indexes in slices are still restricted to regular integer type. - The concept of ``READ_ONLY`` system attributes has disappeared. You can change them now at your own risk! However, you still cannot remove or rename system attributes. - Now, one can do reads in-between write loops using ``table.row`` instances. This is thanks to a decoupling in I/O buffering: now there is a buffer for reading and other for writing, so that no collisions take place anymore. Fixes #1186892. - Support for Octave HDF5 output format. Even complex arrays are supported. Thanks to Edward C. Jones for reporting this format. Backward-incompatible changes: - The format of indexes has been changed and indexes in files created with PyTables pre-1.0 versions are ignored now. However, ``ptrepack`` can still save your life because it is able to convert your old files into the new indexing format. Also, if you copy the affected tables to other locations (by using ``Leaf.copy()``), it will regenerate your indexes with the new format for you. - The API has changed a little bit (nothing serious) for some methods. See ``RELEASE-NOTES.txt`` for more details. Bug fixes: - Added partial support for native HDF5 chunked datasets. They can be read now, and even extended, but only along the first extensible dimension. This limitation may be removed when multiple extensible dimensions are supported in PyTables. - Formerly, when the name of a column in a table was subsumed in another column name, PyTables crashed while retrieving information of the former column. That has been fixed. - A bug prevented the use of indexed columns of tables that were in other hierarchical level than root. This is solved now. - When a ``Group`` was renamed you were not able to modify its attributes. This has been fixed. - When whether ``Table.modifyColumns()`` or ``Table.modifyRows()`` were called, a subsequent call to ``Table.flush()`` didn't really flush the modified data to disk. This works as intended now. - Fixed some issues when iterating over ``*Array`` objects using the ``List`` or ``Tuple`` flavor. Important note for Python 2.4 and Windows users =============================================== If you are willing to use PyTables with Python 2.4 in Windows platforms, you will need to get the HDF5 library compiled for MSVC 7.1, aka .NET 2003. It can be found at: ftp://ftp.ncsa.uiuc.edu/HDF/HDF5/current/bin/windows/5-164-win-net.ZIP Users of Python 2.3 on Windows will have to download the version of HDF5 compiled with MSVC 6.0 available in: ftp://ftp.ncsa.uiuc.edu/HDF/HDF5/current/bin/windows/5-164-win.ZIP Where can PyTables be applied? ============================== PyTables is not designed to work as a relational database competitor, but rather as a teammate. If you want to work with large datasets of multidimensional data (for example, for multidimensional analysis), or just provide a categorized structure for some portions of your cluttered RDBS, then give PyTables a try. It works well for storing data from data acquisition systems (DAS), simulation software, network data monitoring systems (for example, traffic measurements of IP packets on routers), very large XML files, or for creating a centralized repository for system logs, to name only a few possible uses. What is a table? ================ A table is defined as a collection of records whose values are stored in fixed-length fields. All records have the same structure and all values in each field have the same data type. The terms "fixed-length" and "strict data types" seem to be quite a strange requirement for a language like Python that supports dynamic data types, but they serve a useful function if the goal is to save very large quantities of data (such as is generated by many scientific applications, for example) in an efficient manner that reduces demand on CPU time and I/O resources. What is HDF5? ============= For those people who know nothing about HDF5, it is a general purpose library and file format for storing scientific data made at NCSA. HDF5 can store two primary objects: datasets and groups. A dataset is essentially a multidimensional array of data elements, and a group is a structure for organizing objects in an HDF5 file. Using these two basic constructs, one can create and store almost any kind of scientific data structure, such as images, arrays of vectors, and structured and unstructured grids. You can also mix and match them in HDF5 files according to your needs. Platforms ========= We are using Linux on top of Intel32 as the main development platform, but PyTables should be easy to compile/install on other UNIX machines. This package has also been successfully compiled and tested on a FreeBSD 5.4 with Opteron64 processors, a UltraSparc platform with Solaris 7 and Solaris 8, a SGI Origin3000 with Itanium processors running IRIX 6.5 (using the gcc compiler), Microsoft Windows and MacOSX (10.2 although 10.3 should work fine as well). In particular, it has been thoroughly tested on 64-bit platforms, like Linux-64 on top of an Intel Itanium, AMD Opteron (in 64-bit mode) or PowerPC G5 (in 64-bit mode) where all the tests pass successfully. Regarding Windows platforms, PyTables has been tested with Windows 2000 and Windows XP (using the Microsoft Visual C compiler), but it should also work with other flavors as well. Web site ======== Go to the PyTables web site for more details: http://pytables.sourceforge.net/ To know more about the company behind the PyTables development, see: http://www.carabos.com/ Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. ---- **Enjoy data!** -- The PyTables Team From jmiller at stsci.edu Tue May 10 05:16:11 2005 From: jmiller at stsci.edu (Todd Miller) Date: Tue May 10 05:16:11 2005 Subject: [Numpy-discussion] Re: ANN: numarray-1.2.3 -- segfault in in my C program In-Reply-To: <200505091758.11466.haase@msg.ucsf.edu> References: <1109869619.19608.16.camel@halloween.stsci.edu> <1109948612.19608.210.camel@halloween.stsci.edu> <200503041146.50434.haase@msg.ucsf.edu> <200505091758.11466.haase@msg.ucsf.edu> Message-ID: <1115727212.4479.27.camel@jaytmiller.comcast.net> On Mon, 2005-05-09 at 17:58 -0700, Sebastian Haase wrote: > Hi, > Now I'm back to this problem. In the meantime I upgraded again: > >>> na.__version__ > '1.4.0' > All my python2.2 + numarray + my-extensions work. (Debian Woody) > Only my C++ program segfaults. It looked to me like a problem with > NA_NewAllFromBuffer -- was there any recent change ? Yes there was. You can undo it in your private CVS checkout like this: % cd numarray/Src % cvs update -j 1.3 -1.2 libnumarray.ch % cd .. % python setup.py install --genapi --selftest > (I saw that I was using a bytestride of 0 - but it used to work and giving a 1 > for that argument still segfaults -- is 0 OK ??) I don't know for sure. It seems like it should be... just leads to 0 strides. > Now I noticed in the gdb traceback that I chokes is the function > deferred_libnumarray_init > (( > ((#0 0x406d68d5 in PyObject_GetAttrString () from > ((/usr/lib/libpython2.2.so.0.0 #1 0x410f905e in > ((deferred_libnumarray_init () at Src/libnumarraymodule.c:149 > > Is this function the standard function that always get called to init numarray > (in embedded Python AND using "normal" python) ? Yes and no. deferred_libnumarray_init() is really for stuff that can't be done at libnumarray.so init time. This is because of circular dependencies. The "real" numarray init function is either "import numarray" in Python or "import_array()" in C. After that, certain calls into the C-API trigger the deferred initialization. deferred_libnumarray_init() is never called directly by application code. > Have you tested this with Python2.2 ? Yes, using the numarray unit tests. NewAllFromBuffer is also used directly by the numarray ufuncs, so it is tested to a degree by numarray itself. My gut feel is you're hitting an embedding problem, but my gut is only so accurate. > > (See for more info below == I did follow Todd's suggestions 1 & 2, but hope > not having to recompile python from source.) That's really not hard. A 5-10 minute exercise usually. Regards, Todd From jmiller at stsci.edu Tue May 10 05:16:31 2005 From: jmiller at stsci.edu (Todd Miller) Date: Tue May 10 05:16:31 2005 Subject: [Numpy-discussion] numarray memmap and gc In-Reply-To: <20050510111256.187b2f5c.simon@arrowtheory.com> References: <20050510111256.187b2f5c.simon@arrowtheory.com> Message-ID: <1115727214.4479.29.camel@jaytmiller.comcast.net> On Tue, 2005-05-10 at 11:12 +1000, Simon Burton wrote: > Hi all, > > an odd problem that has bitten me a couple of times: > I build an array from a memmap, the memmap goes out of > scope and gets GC'ed (closed), then inexplicable data errors > occur: > > numarray.libnumarray.error: NA_updateDataPtr: error getting read buffer data ptr > > The solution I came up with was to assign the memmap to a unique attribute > of the array ('source_mmap_dont_gc_me'). Then it only gets GC'ed when all it's arrays are GC'ed. > > So, is it reasonable to make this standard in numarray: slices remember their > memmaps and NumArrays remember their buffers (I assume the latter already happens). I think we should make MemmapSlices remember their parent Memmaps. The NumArray remembers the MemmapSlice (buffer) as long as it lives, and the slice in turn would keep the Memmap alive. I have hit a similar problem with Tkinter buttons, twice, so I know the aggravation involved. I'm nervous about sticking in a circular reference (the Memmap already remembers it's slices) but I suppose the garbage collector was added for exactly this case. I'm not entirely sure how hard it's going to be to add. From haase at msg.ucsf.edu Tue May 10 10:33:36 2005 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue May 10 10:33:36 2005 Subject: [Numpy-discussion] Re: ANN: numarray-1.2.3 -- segfault in in my C program In-Reply-To: <1115727212.4479.27.camel@jaytmiller.comcast.net> References: <1109869619.19608.16.camel@halloween.stsci.edu> <200505091758.11466.haase@msg.ucsf.edu> <1115727212.4479.27.camel@jaytmiller.comcast.net> Message-ID: <200505101031.23907.haase@msg.ucsf.edu> Todd, I would be very thankful if you could compile and run this program: #include #include int main() { Py_Initialize(); PyObject *pymodMain = PyImport_ImportModule("__main__"); PyObject* globals = PyModule_GetDict(pymodMain); PyEval_InitThreads(); // import_libnumarray(); PyObject *module = PyImport_ImportModule("numarray.libnumarray"); if (!module) Py_FatalError("Can't import module 'numarray.libnumarray'"); if (module != NULL) { PyObject *module_dict = PyModule_GetDict(module); PyObject *c_api_object = PyDict_GetItemString(module_dict, "_C_API"); if (PyCObject_Check(c_api_object)) { libnumarray_API = (void **)PyCObject_AsVoidPtr(c_api_object); } else { Py_FatalError("Can't get API for module 'numarray.libnumarray'"); } } float *colMapsRGB = new float[4 * 3 * 256]; int shape[3]= { 4, 3, 256 }; PyObject *obj = PyBuffer_FromReadWriteMemory(colMapsRGB, 4*3 *256 * sizeof(float)); PyObject *obj2 =(PyObject*) NA_NewAllFromBuffer( 3, shape, tFloat32, obj, 0, 0, NA_ByteOrder(), 1, 1/*writable*/) ; return 0; } I compile with: g++-2.95 -o testnumarrayembedded -Wall -finline-functions -g -I/usr/include/ python2.2 -I/jws30/haase/myCVS/numarray/Include testnumarrayembedded.cpp -pthread -lpython2.2 It runs through with setting PYTHONPATH to my old numarray(1.1) directory. And now with numarray CVS it now doesn't even find the library... $ ./testnumarrayembedded Fatal Python error: Can't import module 'numarray.libnumarray' Aborted Still, my non-embedded python shell runs all fine - your CVS-update (-j...) suggestion didn't seem to make a difference. Thanks, Sebastian On Tuesday 10 May 2005 05:13, Todd Miller wrote: > On Mon, 2005-05-09 at 17:58 -0700, Sebastian Haase wrote: > > Hi, > > > > Now I'm back to this problem. In the meantime I upgraded again: > > >>> na.__version__ > > > > '1.4.0' > > All my python2.2 + numarray + my-extensions work. (Debian Woody) > > Only my C++ program segfaults. It looked to me like a problem with > > NA_NewAllFromBuffer -- was there any recent change ? > > Yes there was. You can undo it in your private CVS checkout like this: > > % cd numarray/Src > % cvs update -j 1.3 -1.2 libnumarray.ch > % cd .. > % python setup.py install --genapi --selftest > > > (I saw that I was using a bytestride of 0 - but it used to work and > > giving a 1 for that argument still segfaults -- is 0 OK ??) > > I don't know for sure. It seems like it should be... just leads to 0 > strides. > > > Now I noticed in the gdb traceback that I chokes is the function > > deferred_libnumarray_init > > (( > > ((#0 0x406d68d5 in PyObject_GetAttrString () from > > ((/usr/lib/libpython2.2.so.0.0 #1 0x410f905e in > > ((deferred_libnumarray_init () at Src/libnumarraymodule.c:149 > > > > Is this function the standard function that always get called to init > > numarray (in embedded Python AND using "normal" python) ? > > Yes and no. deferred_libnumarray_init() is really for stuff that can't > be done at libnumarray.so init time. This is because of circular > dependencies. > > The "real" numarray init function is either "import numarray" in Python > or "import_array()" in C. After that, certain calls into the C-API > trigger the deferred initialization. deferred_libnumarray_init() is > never called directly by application code. > > > Have you tested this with Python2.2 ? > > Yes, using the numarray unit tests. NewAllFromBuffer is also used > directly by the numarray ufuncs, so it is tested to a degree by > numarray itself. My gut feel is you're hitting an embedding problem, > but my gut is only so accurate. > > > (See for more info below == I did follow Todd's suggestions 1 & 2, but > > hope not having to recompile python from source.) > > That's really not hard. A 5-10 minute exercise usually. > > Regards, > Todd > > > > ------------------------------------------------------- > This SF.Net email is sponsored by Oracle Space Sweepstakes > Want to be the first software developer in space? > Enter now for the Oracle Space Sweepstakes! > http://ads.osdn.com/?ad_id=7393&alloc_id=16281&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From jmiller at stsci.edu Tue May 10 11:43:34 2005 From: jmiller at stsci.edu (Todd Miller) Date: Tue May 10 11:43:34 2005 Subject: [Fwd: Re: [Numpy-discussion] Re: ANN: numarray-1.2.3 -- segfault in in my C program] Message-ID: <1115750435.27493.45.camel@halloween.stsci.edu> Forgot to reply-all. -------------- next part -------------- An embedded message was scrubbed... From: unknown sender Subject: no subject Date: no date Size: 38 URL: From jmiller at stsci.edu Tue May 10 14:39:50 2005 From: jmiller at stsci.edu (Todd Miller) Date: Tue, 10 May 2005 14:39:50 -0400 Subject: [Numpy-discussion] Re: ANN: numarray-1.2.3 -- segfault in in my C program In-Reply-To: <200505101031.23907.haase@msg.ucsf.edu> References: <1109869619.19608.16.camel@halloween.stsci.edu> <200505091758.11466.haase@msg.ucsf.edu> <1115727212.4479.27.camel@jaytmiller.comcast.net> <200505101031.23907.haase@msg.ucsf.edu> Message-ID: <1115750389.27493.43.camel@halloween.stsci.edu> Hi Sebastian, I tried building and running this but I can't even get to main(). I even tried the attached, which I think is completely decoupled from numarray and there is the same problem. BTW, did you leave out -fPIC and -shared on the command line? To simplify things, you might try: a. removing: > PyObject *pymodMain = PyImport_ImportModule("__main__"); > PyObject* globals = PyModule_GetDict(pymodMain); > PyEval_InitThreads(); b. get rid of threads for now c. downshift from C++ to C Bottom line: I think this is a more fundamental "embedding" setup problem. Regards, Todd On Tue, 2005-05-10 at 13:31, Sebastian Haase wrote: > Todd, > I would be very thankful if you could compile and run this program: > > #include > #include > int main() { > Py_Initialize(); > PyObject *pymodMain = PyImport_ImportModule("__main__"); > PyObject* globals = PyModule_GetDict(pymodMain); > PyEval_InitThreads(); > > // import_libnumarray(); > PyObject *module = PyImport_ImportModule("numarray.libnumarray"); > if (!module) > Py_FatalError("Can't import module 'numarray.libnumarray'"); > if (module != NULL) { > PyObject *module_dict = PyModule_GetDict(module); > PyObject *c_api_object = > PyDict_GetItemString(module_dict, "_C_API"); > if (PyCObject_Check(c_api_object)) { > libnumarray_API = (void **)PyCObject_AsVoidPtr(c_api_object); > } else { > Py_FatalError("Can't get API for module 'numarray.libnumarray'"); > } > } > float *colMapsRGB = new float[4 * 3 * 256]; > int shape[3]= { 4, 3, 256 }; > PyObject *obj = PyBuffer_FromReadWriteMemory(colMapsRGB, 4*3 *256 * > sizeof(float)); > PyObject *obj2 =(PyObject*) NA_NewAllFromBuffer( 3, shape, tFloat32, obj, > 0, 0, NA_ByteOrder(), 1, 1/*writable*/) ; > return 0; > } > > > I compile with: > g++-2.95 -o testnumarrayembedded -Wall -finline-functions -g -I/usr/include/ > python2.2 -I/jws30/haase/myCVS/numarray/Include testnumarrayembedded.cpp > -pthread -lpython2.2 > > > It runs through with setting PYTHONPATH to my old numarray(1.1) directory. > And now with numarray CVS it now doesn't even find the library... > $ ./testnumarrayembedded > Fatal Python error: Can't import module 'numarray.libnumarray' > Aborted > > Still, my non-embedded python shell runs all fine - your CVS-update (-j...) > suggestion didn't seem to make a difference. > > Thanks, > Sebastian > > > > On Tuesday 10 May 2005 05:13, Todd Miller wrote: > > On Mon, 2005-05-09 at 17:58 -0700, Sebastian Haase wrote: > > > Hi, > > > > > > Now I'm back to this problem. In the meantime I upgraded again: > > > >>> na.__version__ > > > > > > '1.4.0' > > > All my python2.2 + numarray + my-extensions work. (Debian Woody) > > > Only my C++ program segfaults. It looked to me like a problem with > > > NA_NewAllFromBuffer -- was there any recent change ? > > > > Yes there was. You can undo it in your private CVS checkout like this: > > > > % cd numarray/Src > > % cvs update -j 1.3 -1.2 libnumarray.ch > > % cd .. > > % python setup.py install --genapi --selftest > > > > > (I saw that I was using a bytestride of 0 - but it used to work and > > > giving a 1 for that argument still segfaults -- is 0 OK ??) > > > > I don't know for sure. It seems like it should be... just leads to 0 > > strides. > > > > > Now I noticed in the gdb traceback that I chokes is the function > > > deferred_libnumarray_init > > > (( > > > ((#0 0x406d68d5 in PyObject_GetAttrString () from > > > ((/usr/lib/libpython2.2.so.0.0 #1 0x410f905e in > > > ((deferred_libnumarray_init () at Src/libnumarraymodule.c:149 > > > > > > Is this function the standard function that always get called to init > > > numarray (in embedded Python AND using "normal" python) ? > > > > Yes and no. deferred_libnumarray_init() is really for stuff that can't > > be done at libnumarray.so init time. This is because of circular > > dependencies. > > > > The "real" numarray init function is either "import numarray" in Python > > or "import_array()" in C. After that, certain calls into the C-API > > trigger the deferred initialization. deferred_libnumarray_init() is > > never called directly by application code. > > > > > Have you tested this with Python2.2 ? > > > > Yes, using the numarray unit tests. NewAllFromBuffer is also used > > directly by the numarray ufuncs, so it is tested to a degree by > > numarray itself. My gut feel is you're hitting an embedding problem, > > but my gut is only so accurate. > > > > > (See for more info below == I did follow Todd's suggestions 1 & 2, but > > > hope not having to recompile python from source.) > > > > That's really not hard. A 5-10 minute exercise usually. > > > > Regards, > > Todd > > > > > > > > ------------------------------------------------------- > > This SF.Net email is sponsored by Oracle Space Sweepstakes > > Want to be the first software developer in space? > > Enter now for the Oracle Space Sweepstakes! > > http://ads.osdn.com/?ad_id=7393&alloc_id=16281&op=click > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > ------------------------------------------------------- > This SF.Net email is sponsored by Oracle Space Sweepstakes > Want to be the first software developer in space? > Enter now for the Oracle Space Sweepstakes! > http://ads.osdn.com/?ad_id=7393&alloc_id=16281&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- -------------- next part -------------- A non-text attachment was scrubbed... Name: testnumarrayembedded.c Type: text/x-c Size: 1403 bytes Desc: not available URL: From Reidar.Strand.Hagen at idi.ntnu.no Tue May 10 12:35:19 2005 From: Reidar.Strand.Hagen at idi.ntnu.no (Reidar Strand Hagen) Date: Tue May 10 12:35:19 2005 Subject: [Numpy-discussion] Expansion of Matrix Message-ID: Hi Sorry if this is an obvious question, but I'm getting close to loosing sleep over it now :) If I have a matrix: 1 2 3 4 5 6 7 8 9 is there a nice general way to expand the matrix to: 1 0 2 3 4 0 5 6 0 0 0 0 7 0 8 9 The actual dataset in question has dimensions 128 * 128 * 128; so big enough for me to worry quite abit about duplication and general speed. vennlig hilsen / regards Reidar Strand Hagen From haase at msg.ucsf.edu Tue May 10 13:23:46 2005 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue May 10 13:23:46 2005 Subject: [Numpy-discussion] Re: ANN: numarray-1.2.3 -- segfault in in my C program In-Reply-To: <1115750389.27493.43.camel@halloween.stsci.edu> References: <1109869619.19608.16.camel@halloween.stsci.edu> <200505101031.23907.haase@msg.ucsf.edu> <1115750389.27493.43.camel@halloween.stsci.edu> Message-ID: <200505101321.02528.haase@msg.ucsf.edu> Todd, Thanks for working on this with me. I did not use -fPIC and/or -shared. What is -fPIC anywhy ? I have been using my code + makefile since numarray 0.3.6 (maybe 0.4). I took my gcc/g++ arguments from examples in the wxWindows/wxPython. Trying your modified(attached) c-version I get the exact same behaviour that I had before IF I DON'T use -shared. (I.e. works with numarray1.1; new numarray gives "Can't import module 'numarray.libnumarray'") (I took -pthread out) With -shared I get a segfault before main() - gdb: Program received signal SIGSEGV, Segmentation fault. 0x00000001 in ?? () How can this be ? How can this change from numarray 1.1 to 1.2 (or 1.2.3) ? Thanks, Sebastian On Tuesday 10 May 2005 11:39, Todd Miller wrote: > Hi Sebastian, > > I tried building and running this but I can't even get to main(). I > even tried the attached, which I think is completely decoupled from > numarray and there is the same problem. > > BTW, did you leave out -fPIC and -shared on the command line? > > To simplify things, you might try: > > a. removing: > > PyObject *pymodMain = PyImport_ImportModule("__main__"); > > PyObject* globals = PyModule_GetDict(pymodMain); > > PyEval_InitThreads(); > > b. get rid of threads for now > > c. downshift from C++ to C > > Bottom line: I think this is a more fundamental "embedding" setup problem. > > Regards, > Todd > > On Tue, 2005-05-10 at 13:31, Sebastian Haase wrote: > > Todd, > > I would be very thankful if you could compile and run this program: > > > > #include > > #include > > int main() { > > Py_Initialize(); > > PyObject *pymodMain = PyImport_ImportModule("__main__"); > > PyObject* globals = PyModule_GetDict(pymodMain); > > PyEval_InitThreads(); > > > > // import_libnumarray(); > > PyObject *module = PyImport_ImportModule("numarray.libnumarray"); > > if (!module) > > Py_FatalError("Can't import module 'numarray.libnumarray'"); > > if (module != NULL) { > > PyObject *module_dict = PyModule_GetDict(module); > > PyObject *c_api_object = > > PyDict_GetItemString(module_dict, "_C_API"); > > if (PyCObject_Check(c_api_object)) { > > libnumarray_API = (void **)PyCObject_AsVoidPtr(c_api_object); > > } else { > > Py_FatalError("Can't get API for module 'numarray.libnumarray'"); > > } > > } > > float *colMapsRGB = new float[4 * 3 * 256]; > > int shape[3]= { 4, 3, 256 }; > > PyObject *obj = PyBuffer_FromReadWriteMemory(colMapsRGB, 4*3 *256 * > > sizeof(float)); > > PyObject *obj2 =(PyObject*) NA_NewAllFromBuffer( 3, shape, tFloat32, > > obj, 0, 0, NA_ByteOrder(), 1, 1/*writable*/) ; > > return 0; > > } > > > > > > I compile with: > > g++-2.95 -o testnumarrayembedded -Wall -finline-functions -g > > -I/usr/include/ python2.2 -I/jws30/haase/myCVS/numarray/Include > > testnumarrayembedded.cpp -pthread -lpython2.2 > > > > > > It runs through with setting PYTHONPATH to my old numarray(1.1) > > directory. And now with numarray CVS it now doesn't even find the > > library... $ ./testnumarrayembedded > > Fatal Python error: Can't import module 'numarray.libnumarray' > > Aborted > > > > Still, my non-embedded python shell runs all fine - your CVS-update > > (-j...) suggestion didn't seem to make a difference. > > > > Thanks, > > Sebastian > > > > On Tuesday 10 May 2005 05:13, Todd Miller wrote: > > > On Mon, 2005-05-09 at 17:58 -0700, Sebastian Haase wrote: > > > > Hi, > > > > > > > > Now I'm back to this problem. In the meantime I upgraded again: > > > > >>> na.__version__ > > > > > > > > '1.4.0' > > > > All my python2.2 + numarray + my-extensions work. (Debian Woody) > > > > Only my C++ program segfaults. It looked to me like a problem with > > > > NA_NewAllFromBuffer -- was there any recent change ? > > > > > > Yes there was. You can undo it in your private CVS checkout like this: > > > > > > % cd numarray/Src > > > % cvs update -j 1.3 -1.2 libnumarray.ch > > > % cd .. > > > % python setup.py install --genapi --selftest > > > > > > > (I saw that I was using a bytestride of 0 - but it used to work and > > > > giving a 1 for that argument still segfaults -- is 0 OK ??) > > > > > > I don't know for sure. It seems like it should be... just leads to 0 > > > strides. > > > > > > > Now I noticed in the gdb traceback that I chokes is the function > > > > deferred_libnumarray_init > > > > (( > > > > ((#0 0x406d68d5 in PyObject_GetAttrString () from > > > > ((/usr/lib/libpython2.2.so.0.0 #1 0x410f905e in > > > > ((deferred_libnumarray_init () at Src/libnumarraymodule.c:149 > > > > > > > > Is this function the standard function that always get called to init > > > > numarray (in embedded Python AND using "normal" python) ? > > > > > > Yes and no. deferred_libnumarray_init() is really for stuff that can't > > > be done at libnumarray.so init time. This is because of circular > > > dependencies. > > > > > > The "real" numarray init function is either "import numarray" in Python > > > or "import_array()" in C. After that, certain calls into the C-API > > > trigger the deferred initialization. deferred_libnumarray_init() is > > > never called directly by application code. > > > > > > > Have you tested this with Python2.2 ? > > > > > > Yes, using the numarray unit tests. NewAllFromBuffer is also used > > > directly by the numarray ufuncs, so it is tested to a degree by > > > numarray itself. My gut feel is you're hitting an embedding problem, > > > but my gut is only so accurate. > > > > > > > (See for more info below == I did follow Todd's suggestions 1 & 2, > > > > but hope not having to recompile python from source.) > > > > > > That's really not hard. A 5-10 minute exercise usually. > > > > > > Regards, > > > Todd From simon at arrowtheory.com Tue May 10 17:29:35 2005 From: simon at arrowtheory.com (Simon Burton) Date: Tue May 10 17:29:35 2005 Subject: [Numpy-discussion] numarray memmap and gc In-Reply-To: <1115727214.4479.29.camel@jaytmiller.comcast.net> References: <20050510111256.187b2f5c.simon@arrowtheory.com> <1115727214.4479.29.camel@jaytmiller.comcast.net> Message-ID: <20050511102540.515675ca.simon@arrowtheory.com> On Tue, 10 May 2005 08:13:34 -0400 Todd Miller wrote: > > I'm nervous about sticking in a circular reference (the Memmap already > remembers it's slices) but I suppose the garbage collector was added for > exactly this case. I'm not entirely sure how hard it's going to be to > add. Yes, this is a good point. But also, memmaps have a close method so users can always explicitly break this cycle if they want. Either way, appropriate warnings should be added to the documentation. bye, Simon. -- Simon Burton, B.Sc. Licensed PO Box 8066 ANU Canberra 2601 Australia Ph. 61 02 6249 6940 http://arrowtheory.com From simon at arrowtheory.com Tue May 10 17:48:36 2005 From: simon at arrowtheory.com (Simon Burton) Date: Tue May 10 17:48:36 2005 Subject: [Numpy-discussion] numarray.ma docs confuse Numeric/numarray Message-ID: <20050511104651.70d2e199.simon@arrowtheory.com> It seems the masked array docs mention Numeric in a few places when numarray was intended. Patch attached. Simon. -- Simon Burton, B.Sc. Licensed PO Box 8066 ANU Canberra 2601 Australia Ph. 61 02 6249 6940 http://arrowtheory.com -------------- next part -------------- A non-text attachment was scrubbed... Name: ma.tex.patch Type: application/octet-stream Size: 1856 bytes Desc: not available URL: From customer-support at ncsecu.org Tue May 10 20:08:39 2005 From: customer-support at ncsecu.org (State Employees Credit Union) Date: Tue May 10 20:08:39 2005 Subject: [Numpy-discussion] SECU - Urgent account verification! Message-ID: <200505110259.j4B2x68T014799@med-track.net> An HTML attachment was scrubbed... URL: From andrea_gavana at tin.it Wed May 11 09:43:09 2005 From: andrea_gavana at tin.it (andrea_gavana at tin.it) Date: Wed May 11 09:43:09 2005 Subject: [Numpy-discussion] About Reshape() Message-ID: <4200063400037BA1@ims9c.cp.tin.it> Hello NG, I am not really a guru with Numeric, and I am trying to port some Matlab code to Python, using Numeric. The code is working, but it is slown down because of the different implementation (ordering) of reshape() in Matlab and Numeric. For example, in Matlab I can do: a = 1:10; reshape(a,5,2) ans = 1 6 2 7 3 8 4 9 5 10 In Numeric: a = range(1, 11) Numeric.reshape(a, (5,2)) array([[ 1, 2], [ 3, 4], [ 5, 6], [ 7, 8], [ 9, 10]]) Hmm... let's try again... Numeric.reshape(a, (2,5)) array([[ 1, 2, 3, 4, 5], [ 6, 7, 8, 9, 10]]) Hmm... the only solution I found: Numeric.transpose(Numeric.reshape(a, (2,5))) array([[ 1, 6], [ 2, 7], [ 3, 8], [ 4, 9], [ 5, 10]]) This continuous use of transpose() is killing the performances of my scripts... is there a workaround? Am I missing something so obvious? Thank you for every suggestion. Andrea. From andrea_gavana at tin.it Wed May 11 13:04:29 2005 From: andrea_gavana at tin.it (andrea_gavana at tin.it) Date: Wed May 11 13:04:29 2005 Subject: [Numpy-discussion] Square All Elements Of A Matrix Message-ID: <4200063400037D9A@ims9c.cp.tin.it> Hello NG, in one script, I generated a matrix "dx" which has a (200000, 10) shape. Well, everything is working fine (matrix multiplication, sum, exponentiation etc...), but I'm having problems in squaring all elements of this matrix. Suppose I have: import Numeric import random dx = Numeric.ones((200000,10), Numeric.Float0, savespace=1) dx[:,:] = random.random() td1 = dx*2.0 ---> This takes about 0.2 seconds td2 = dx**2.0 ---> This takes about 6 seconds Why this difference? And why it is so slow the exponentiation? I have tried the same operation in Matlab and it was almost instantaneous. I don't want to stick with Matlab, noting the great job the Numeric developers have done. Is there something that I am missing? Sorry if it is a basic question. Andrea. From cookedm at physics.mcmaster.ca Wed May 11 13:30:25 2005 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Wed May 11 13:30:25 2005 Subject: [Numpy-discussion] Square All Elements Of A Matrix In-Reply-To: <4200063400037D9A@ims9c.cp.tin.it> (andrea gavana's message of "Wed, 11 May 2005 22:02:35 +0200") References: <4200063400037D9A@ims9c.cp.tin.it> Message-ID: andrea_gavana at tin.it writes: > Hello NG, > > in one script, I generated a matrix "dx" which has a (200000, 10) shape. > Well, everything is working fine (matrix multiplication, sum, exponentiation > etc...), but I'm having problems in squaring all elements of this matrix. > > Suppose I have: > > import Numeric > import random > > dx = Numeric.ones((200000,10), Numeric.Float0, savespace=1) > dx[:,:] = random.random() It's faster to do import RandomArray dx = RandomArray.random((200000,10)).astype(Numeric.Float0) dx.savespace(1) (it won't use the same RNG as random, though) > td1 = dx*2.0 ---> This takes about 0.2 seconds > td2 = dx**2.0 ---> This takes about 6 seconds > > Why this difference? And why it is so slow the exponentiation? I have tried > the same operation in Matlab and it was almost instantaneous. I don't want > to stick with Matlab, noting the great job the Numeric developers > have done. Hmm, I don't see this, with either 23.8 or 24.0b2. What version of Numeric are you using, and on what platform? dx**2 uses the C 'pow' function on each element (so using 2 or 2.0 here doesn't make a difference), so may be your platform pow() is slow. dx*dx should be faster (it takes 80% of the time that dx**2 does on my machine). -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From Fernando.Perez at colorado.edu Wed May 11 13:35:26 2005 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Wed May 11 13:35:26 2005 Subject: [Numpy-discussion] Square All Elements Of A Matrix In-Reply-To: <4200063400037D9A@ims9c.cp.tin.it> References: <4200063400037D9A@ims9c.cp.tin.it> Message-ID: <42826BDC.7030706@colorado.edu> andrea_gavana at tin.it wrote: > Hello NG, > > in one script, I generated a matrix "dx" which has a (200000, 10) shape. > Well, everything is working fine (matrix multiplication, sum, exponentiation > etc...), but I'm having problems in squaring all elements of this matrix. > > Suppose I have: > > import Numeric > import random > > dx = Numeric.ones((200000,10), Numeric.Float0, savespace=1) > dx[:,:] = random.random() > > td1 = dx*2.0 ---> This takes about 0.2 seconds > td2 = dx**2.0 ---> This takes about 6 seconds > > Why this difference? And why it is so slow the exponentiation? I have tried > the same operation in Matlab and it was almost instantaneous. I don't want > to stick with Matlab, noting the great job the Numeric developers have done. > > Is there something that I am missing? I'm not sure what the problem may be, but I certainly don't see your results: In [22]: dx=RandomArray.random((200000, 10)).astype(Numeric.Float0) In [23]: time td1 = dx*2.0 CPU times: user 0.20 s, sys: 0.06 s, total: 0.26 s Wall time: 0.26 In [24]: time td2 = dx**2.0 CPU times: user 0.30 s, sys: 0.06 s, total: 0.36 s Wall time: 0.36 An increase like this (50% in cpu time) is reasonable for computing a power vs. a plain multiplication. This is using Numeric 23.7 on a Fedora3 box (2.6 GHZ P4). Do you know for a fact that you have enough RAM, and that the 6 secs. don't come from paging (the arrays aren't that large, but you might have other stuff in memory forcing paging). Cheers, f From andrea_gavana at tin.it Wed May 11 13:40:01 2005 From: andrea_gavana at tin.it (andrea_gavana at tin.it) Date: Wed May 11 13:40:01 2005 Subject: [Numpy-discussion] Square All Elements Of A Matrix In-Reply-To: Message-ID: <4200063400037DE5@ims9c.cp.tin.it> Hello David >It's faster to do > >import RandomArray > >dx = RandomArray.random((200000,10)).astype(Numeric.Float0) >dx.savespace(1) > >(it won't use the same RNG as random, though) thanks for the tip. >> td1 = dx*2.0 ---> This takes about 0.2 seconds >> td2 = dx**2.0 ---> This takes about 6 seconds >> >Hmm, I don't see this, with either 23.8 or 24.0b2. What version of >Numeric are you using, and on what platform? I have currently Numeric 23.8, with numarray 1.3.1. I am on a Windows XP machine, with 1 GB RAM and 3.0 GHz. My Python version is 2.3.4. >dx**2 uses the C 'pow' function on each element (so using 2 or 2.0 >here doesn't make a difference), so may be your platform pow() is >slow. Probably that is the reason... but why? Is Windows the problem? >dx*dx should be faster (it takes 80% of the time that dx**2 does on my >machine). On my machine, dx*dx is istantaneous (about 0.1 secons). This are my timings: import time; start = time.time(); td = dx*dx; print time.time() - start 0.121000051498 import time; start = time.time(); td = dx**2; print time.time() - start 4.367000103 !!!!!!!! Does anyone have an explanation for this?!?!? Thank you for your suggestions. Andrea. From andrea_gavana at tin.it Wed May 11 13:45:03 2005 From: andrea_gavana at tin.it (andrea_gavana at tin.it) Date: Wed May 11 13:45:03 2005 Subject: [Numpy-discussion] Square All Elements Of A Matrix In-Reply-To: <42826BDC.7030706@colorado.edu> Message-ID: <4200063400037DF0@ims9c.cp.tin.it> Hello Fernando, >I'm not sure what the problem may be, but I certainly don't see your results: > >In [22]: dx=RandomArray.random((200000, 10)).astype(Numeric.Float0) > >In [23]: time td1 = dx*2.0 >CPU times: user 0.20 s, sys: 0.06 s, total: 0.26 s >Wall time: 0.26 > >In [24]: time td2 = dx**2.0 >CPU times: user 0.30 s, sys: 0.06 s, total: 0.36 s >Wall time: 0.36 This is what I get: start = time.time(); td2 = dx**2; print time.time() - start 4.367000103 start = time.time(); td2 = dx*dx; print time.time() - start 0.121000051498 This is on a Windows XP, 1 GB RAM, 3.0 GHz, Python 2.3.4, Numeric 23.8. >This is using Numeric 23.7 on a Fedora3 box (2.6 GHZ P4). Do you know for a fact that >you have enough RAM, and that the 6 secs. don't come from paging (the arrays aren't >that large, but you might have other stuff in memory forcing paging). No, the only thing I have opened is PythonWin (the editor), and I am using currently 100 MB of the RAM. I don't know why this happens... Thanks for your help. Andrea. From cookedm at physics.mcmaster.ca Wed May 11 13:58:20 2005 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Wed May 11 13:58:20 2005 Subject: [Numpy-discussion] Square All Elements Of A Matrix In-Reply-To: <4200063400037DE5@ims9c.cp.tin.it> (andrea gavana's message of "Wed, 11 May 2005 22:38:24 +0200") References: <4200063400037DE5@ims9c.cp.tin.it> Message-ID: andrea_gavana at tin.it writes: > Hello David >>Hmm, I don't see this, with either 23.8 or 24.0b2. What version of >>Numeric are you using, and on what platform? > > I have currently Numeric 23.8, with numarray 1.3.1. I am on a Windows XP > machine, with 1 GB RAM and 3.0 GHz. My Python version is 2.3.4. > >>dx**2 uses the C 'pow' function on each element (so using 2 or 2.0 >>here doesn't make a difference), so may be your platform pow() is >>slow. > > Probably that is the reason... but why? Is Windows the problem? > >>dx*dx should be faster (it takes 80% of the time that dx**2 does on my >>machine). > > On my machine, dx*dx is istantaneous (about 0.1 secons). This are my timings: > > import time; start = time.time(); td = dx*dx; print time.time() - start > 0.121000051498 > > import time; start = time.time(); td = dx**2; print time.time() - start > 4.367000103 > > !!!!!!!! Could you check the speed of td = exp(2*log(dx)) If it's comparable to dx**2, then your pow() library function isn't any good :-( -- it's not special-casing integer powers. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From andrea_gavana at tin.it Wed May 11 14:02:25 2005 From: andrea_gavana at tin.it (andrea_gavana at tin.it) Date: Wed May 11 14:02:25 2005 Subject: [Numpy-discussion] Square All Elements Of A Matrix In-Reply-To: Message-ID: <4200063400037E0A@ims9c.cp.tin.it> Hello David, >Could you check the speed of > >td = exp(2*log(dx)) > >If it's comparable to dx**2, then your pow() library function isn't >any good :-( -- it's not special-casing integer powers. I get: start = time.time(); td = Numeric.exp(2*Numeric.log(dx)); print time.time() - start 0.530999898911 Does this make any sense to anyone? I am a bit puzzled... Thanks however for your kind help/suggestions. Andrea. From jbrandmeyer at earthlink.net Wed May 11 18:29:10 2005 From: jbrandmeyer at earthlink.net (Jonathan Brandmeyer) Date: Wed May 11 18:29:10 2005 Subject: [Numpy-discussion] Square All Elements Of A Matrix In-Reply-To: References: <4200063400037DE5@ims9c.cp.tin.it> Message-ID: <1115861247.680.27.camel@localhost.localdomain> On Wed, 2005-05-11 at 16:57 -0400, David M. Cooke wrote: > td = exp(2*log(dx)) > > If it's comparable to dx**2, then your pow() library function isn't > any good :-( -- it's not special-casing integer powers. I believe that this is the case. IIRC, MinGW uses a wrapper around msvcrt.dll's pow() that handles this special case for this very reason. HTH, -Jonathan From cookedm at physics.mcmaster.ca Thu May 12 11:07:16 2005 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Thu May 12 11:07:16 2005 Subject: [Numpy-discussion] Square All Elements Of A Matrix In-Reply-To: <1115861247.680.27.camel@localhost.localdomain> (Jonathan Brandmeyer's message of "Wed, 11 May 2005 21:27:26 -0400") References: <4200063400037DE5@ims9c.cp.tin.it> <1115861247.680.27.camel@localhost.localdomain> Message-ID: Jonathan Brandmeyer writes: > On Wed, 2005-05-11 at 16:57 -0400, David M. Cooke wrote: > >> td = exp(2*log(dx)) >> >> If it's comparable to dx**2, then your pow() library function isn't >> any good :-( -- it's not special-casing integer powers. > > I believe that this is the case. IIRC, MinGW uses a wrapper around > msvcrt.dll's pow() that handles this special case for this very > reason. Is this is a big problem for others? This is the first report I've seen of the problem on Windows. It may be worth special-casing the power ufunc to handle some non-array powers (x**2, x**3, x**0.5, for instance). -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From andrea_gavana at tin.it Thu May 12 12:36:24 2005 From: andrea_gavana at tin.it (andrea_gavana at tin.it) Date: Thu May 12 12:36:24 2005 Subject: [Numpy-discussion] Square All Elements Of A Matrix In-Reply-To: Message-ID: <4200063400038B09@ims9c.cp.tin.it> Hello David, >Is this is a big problem for others? This is the first report I've >seen of the problem on Windows. It may be worth special-casing the >power ufunc to handle some non-array powers (x**2, x**3, x**0.5, for >instance). It's not very clear to me what you wrote, but I have the same behavior on a Windows 2000 machine I have at work (here is WinXP), with Numeric 23.7 (here is 23.8), with Python 2.3.5 (here is 2.3.4). These 2 machines do not have anything in common, as far as I know. Regarding what you said (if I have correctly understood), I get: >>>start = time.time(); td = dx*dx; print time.time() - start 0.269999980927 >>> start = time.time(); td = dx**2; print time.time() - start 4.5560002327 >>> start = time.time(); td = dx**3; print time.time() - start 4.5759999752 >>> start = time.time(); td = dx**0.5; print time.time() - start 4.19600009918 >>> start = time.time(); td = sqrt(dx); print time.time() - start 0.301000118256 >>> start = time.time(); td = exp(log(dx)); print time.time() - start 0.792000055313 >>> start = time.time(); td = dx**RandomArray.random((1,)); print time.time() - start 4.14599990845 Whatever power I use, the timing is almost the same. Curious however, how sqrt(dx) and dx**0.5 behave really differently. Thanks for your help. Andrea. From haase at msg.ucsf.edu Thu May 12 20:15:56 2005 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Thu May 12 20:15:56 2005 Subject: [Numpy-discussion] tuple vs. list as index Message-ID: <200505121704.41025.haase@msg.ucsf.edu> Hi, Is this really obvious to everybody: >>> type(a) >>> a.shape (12, 13, 2, 5) >>> a[(0,0)].shape (2, 5) >>> a[[0,0]].shape (2, 13, 2, 5) >>> na.take(a,[0,0]).shape (2, 13, 2, 5) Is there some explanation in the documentation ? (Is this the same in Numeric / Numeric3 ?) Thanks, Sebastian Haase From jbrandmeyer at earthlink.net Thu May 12 20:41:38 2005 From: jbrandmeyer at earthlink.net (Jonathan Brandmeyer) Date: Thu May 12 20:41:38 2005 Subject: [Numpy-discussion] Square All Elements Of A Matrix In-Reply-To: References: <4200063400037DE5@ims9c.cp.tin.it> <1115861247.680.27.camel@localhost.localdomain> Message-ID: <1115941592.6032.12.camel@localhost.localdomain> On Thu, 2005-05-12 at 14:05 -0400, David M. Cooke wrote: > Jonathan Brandmeyer writes: > > > On Wed, 2005-05-11 at 16:57 -0400, David M. Cooke wrote: > > > >> td = exp(2*log(dx)) > >> > >> If it's comparable to dx**2, then your pow() library function isn't > >> any good :-( -- it's not special-casing integer powers. > > > > I believe that this is the case. IIRC, MinGW uses a wrapper around > > msvcrt.dll's pow() that handles this special case for this very > > reason. > > Is this is a big problem for others? Its never bugged me enough to do anything about it ;) > This is the first report I've > seen of the problem on Windows. It may be worth special-casing the > power ufunc to handle some non-array powers (x**2, x**3, x**0.5, for > instance). If you go this route, I suggest you just use the implementation in mingw-runtime - its in the public domain and has had some precision issues debugged out of it already. -Jonathan From jmiller at stsci.edu Fri May 13 03:35:05 2005 From: jmiller at stsci.edu (Todd Miller) Date: Fri May 13 03:35:05 2005 Subject: [Numpy-discussion] tuple vs. list as index In-Reply-To: <200505121704.41025.haase@msg.ucsf.edu> References: <200505121704.41025.haase@msg.ucsf.edu> Message-ID: <1115980379.4430.20.camel@jaytmiller.comcast.net> On Thu, 2005-05-12 at 17:04 -0700, Sebastian Haase wrote: > Hi, > Is this really obvious to everybody: > > >>> type(a) > > >>> a.shape > (12, 13, 2, 5) > >>> a[(0,0)].shape > (2, 5) > >>> a[[0,0]].shape > (2, 13, 2, 5) > > >>> na.take(a,[0,0]).shape > (2, 13, 2, 5) > > Is there some explanation in the documentation ? The distinction is not an accident and has been discussed. One thing to keep in mind with numarray indexing is that not all sequences used as indices are created equal. There is already a distinction for arrays used as indices, and an additional distinction for Bool arrays used as indices. Back to basics, the explanation is this: >>> a[(0,0)].shape (2, 5) >>> a[0,0].shape (2, 5) Those two have identical semantics because of the way Python works. Either way, the numarray indexing hooks see (0,0), i.e. not ((0,0),) for the first and (0,0) for the second. >>> a[[0,0]].shape (2, 13, 2, 5) A list, on the other hand, *is* seen as ([ 0,0 ],). And we've chosen to exploit that as a shorthand for array-based-indexing making it equivalent to take() as you've shown. > (Is this the same in Numeric / Numeric3 ?) Unless it was added recently, Numeric has no array-based-indexing. Maybe Travis or someone else can fill us in on how Numeric3 handles this. Regards, Todd From andrea_gavana at tin.it Fri May 13 09:07:18 2005 From: andrea_gavana at tin.it (andrea_gavana at tin.it) Date: Fri May 13 09:07:18 2005 Subject: [Numpy-discussion] Square All Elements Of A Matrix In-Reply-To: <8368.129.238.237.96.1115999760.squirrel@www.mza.com> Message-ID: <4200063400039573@ims9c.cp.tin.it> Hello Tony, >What compiler are you using to build Numeric and Python? No compiler for Python or Numeric. For Python, I use the binaries that you can find at www.python.org. I don't have any idea which compiler they have used. For Numeric, I downloaded the binaries that you can find at SourceForge. Python 2.3.4/2.3.5, Numeric 23.8/23.7, Windows XP/2000 Andrea. From anthony.seward at ieee.org Fri May 13 09:33:10 2005 From: anthony.seward at ieee.org (Anthony Joseph Seward) Date: Fri May 13 09:33:10 2005 Subject: [Numpy-discussion] Square All Elements Of A Matrix In-Reply-To: <4200063400038B09@ims9c.cp.tin.it> References: <4200063400038B09@ims9c.cp.tin.it> Message-ID: <8368.129.238.237.96.1115999760.squirrel@www.mza.com> What compiler are you using to build Numeric and Python? Tony andrea_gavana at tin.it said: > Hello David, > >>Is this is a big problem for others? This is the first report I've >>seen of the problem on Windows. It may be worth special-casing the >>power ufunc to handle some non-array powers (x**2, x**3, x**0.5, for >>instance). > > It's not very clear to me what you wrote, but I have the same behavior on > a Windows 2000 machine I have at work (here is WinXP), with Numeric 23.7 > (here is 23.8), with Python 2.3.5 (here is 2.3.4). These 2 machines do not > have anything in common, as far as I know. > Regarding what you said (if I have correctly understood), I get: > >>>>start = time.time(); td = dx*dx; print time.time() - start > 0.269999980927 > >>>> start = time.time(); td = dx**2; print time.time() - start > 4.5560002327 > >>>> start = time.time(); td = dx**3; print time.time() - start > 4.5759999752 > >>>> start = time.time(); td = dx**0.5; print time.time() - start > 4.19600009918 > >>>> start = time.time(); td = sqrt(dx); print time.time() - start > 0.301000118256 > >>>> start = time.time(); td = exp(log(dx)); print time.time() - start > 0.792000055313 > >>>> start = time.time(); td = dx**RandomArray.random((1,)); print >>>> time.time() > - start > 4.14599990845 > > Whatever power I use, the timing is almost the same. Curious however, how > sqrt(dx) and dx**0.5 behave really differently. > > Thanks for your help. > > Andrea. > > > > ------------------------------------------------------- > This SF.Net email is sponsored by Oracle Space Sweepstakes > Want to be the first software developer in space? > Enter now for the Oracle Space Sweepstakes! > http://ads.osdn.com/?ad_ids93&alloc_id281&op?k > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > From anthony.seward at ieee.org Fri May 13 10:39:18 2005 From: anthony.seward at ieee.org (Anthony Joseph Seward) Date: Fri May 13 10:39:18 2005 Subject: [Numpy-discussion] Square All Elements Of A Matrix In-Reply-To: <4200063400039573@ims9c.cp.tin.it> References: <8368.129.238.237.96.1115999760.squirrel@www.mza.com> <4200063400039573@ims9c.cp.tin.it> Message-ID: <11633.129.238.237.96.1116005870.squirrel@www.mza.com> What is the value of sys.version? For me running the attached script yields: Fedora 3: Python version: 2.3.4 (#1, Feb 2 2005, 12:11:53) [GCC 3.4.2 20041017 (Red Hat 3.4.2-6.fc3)] *: 0.381051 **2: 1.31382 **3: 1.198 **0.5: 1.67896 sqrt(): 0.357029 exp(log): 1.2737 exp(2*log): 2.17751 exp(.5*log): 2.19792 Win2k on vmware 5 under Fedora 3 Python version: 2.3.3 (#51, Feb 16 2004, 04:07:52) [MSC v.1200 32 bit (Intel)] *: 0.391 **2: 1.468 **3: 1.547 **0.5: 1.469 sqrt(): 0.437 exp(log): 1.672 exp(2*log): 2.718 exp(.5*log): 2.734 Odd. andrea_gavana at tin.it said: > Hello Tony, > >>What compiler are you using to build Numeric and Python? > > No compiler for Python or Numeric. For Python, I use the binaries that you > can find at www.python.org. I don't have any idea which compiler they have > used. For Numeric, I downloaded the binaries that you can find at > SourceForge. > > Python 2.3.4/2.3.5, Numeric 23.8/23.7, Windows XP/2000 > > Andrea. > > > > > -- ajs: anthony.seward at ieee.org -------------- next part -------------- A non-text attachment was scrubbed... Name: numeric-stats.py Type: application/x-python Size: 1056 bytes Desc: not available URL: From anthony.seward at ieee.org Fri May 13 10:55:19 2005 From: anthony.seward at ieee.org (Anthony Joseph Seward) Date: Fri May 13 10:55:19 2005 Subject: [Numpy-discussion] Square All Elements Of A Matrix In-Reply-To: <11633.129.238.237.96.1116005870.squirrel@www.mza.com> References: <8368.129.238.237.96.1115999760.squirrel@www.mza.com> <4200063400039573@ims9c.cp.tin.it> <11633.129.238.237.96.1116005870.squirrel@www.mza.com> Message-ID: <5860.129.238.237.96.1116006878.squirrel@www.mza.com> More data on FC3 Python version: 2.3.4 (#1, Feb 2 2005, 12:11:53) [GCC 3.4.2 20041017 (Red Hat 3.4.2-6.fc3)] ========== Numeric ========== *: 0.379871 **2: 1.17747 **3: 1.1788 **0.5: 1.66172 sqrt(): 0.362143 exp(log): 1.2656 exp(2*log): 2.16179 exp(.5*log): 2.22712 ========== numarray ========== *: 0.0414059 **2: 0.914946 **3: 1.14649 **0.5: 1.43333 sqrt(): 0.10821 exp(log): 1.1466 exp(2*log): 1.17661 exp(.5*log): 1.24503 From oliphant at ee.byu.edu Fri May 13 11:21:29 2005 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Fri May 13 11:21:29 2005 Subject: [Numpy-discussion] tuple vs. list as index In-Reply-To: <1115980379.4430.20.camel@jaytmiller.comcast.net> References: <200505121704.41025.haase@msg.ucsf.edu> <1115980379.4430.20.camel@jaytmiller.comcast.net> Message-ID: <4284EF63.7000204@ee.byu.edu> Todd Miller wrote: >On Thu, 2005-05-12 at 17:04 -0700, Sebastian Haase wrote: > > >>Hi, >>Is this really obvious to everybody: >> >> >> >>>>>type(a) >>>>> >>>>> >> >> >> >>>>>a.shape >>>>> >>>>> >>(12, 13, 2, 5) >> >> >>>>>a[(0,0)].shape >>>>> >>>>> >>(2, 5) >> >> >>>>>a[[0,0]].shape >>>>> >>>>> >>(2, 13, 2, 5) >> >> >> >>>>>na.take(a,[0,0]).shape >>>>> >>>>> >>(2, 13, 2, 5) >> >>Is there some explanation in the documentation ? >> >> > >The distinction is not an accident and has been discussed. One thing to >keep in mind with numarray indexing is that not all sequences used as >indices are created equal. There is already a distinction for arrays >used as indices, and an additional distinction for Bool arrays used as > > >indices. Back to basics, the explanation is this: > > This same distinction is kept in Numeric3 and currently works the same way. Numeric3 should have the same semantics as Numarray. -Travis From andrea_gavana at tin.it Fri May 13 12:06:20 2005 From: andrea_gavana at tin.it (andrea_gavana at tin.it) Date: Fri May 13 12:06:20 2005 Subject: [Numpy-discussion] Square All Elements Of A Matrix In-Reply-To: <11633.129.238.237.96.1116005870.squirrel@www.mza.com> Message-ID: <42000634000396F5@ims9c.cp.tin.it> Hello Tony, >What is the value of sys.version? D:\MyProjects>python Python 2.3.4 (#53, May 25 2004, 21:17:02) [MSC v.1200 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.version '2.3.4 (#53, May 25 2004, 21:17:02) [MSC v.1200 32 bit (Intel)]' >>> >Fedora 3: > *: 0.381051 > **2: 1.31382 > **3: 1.198 > **0.5: 1.67896 > sqrt(): 0.357029 > exp(log): 1.2737 > exp(2*log): 2.17751 > exp(.5*log): 2.19792 > >Win2k on vmware 5 under Fedora 3 > Python version: 2.3.3 (#51, Feb 16 2004, 04:07:52) [MSC v.1200 32 bit >(Intel)] > *: 0.391 > **2: 1.468 > **3: 1.547 > **0.5: 1.469 > sqrt(): 0.437 > exp(log): 1.672 > exp(2*log): 2.718 > exp(.5*log): 2.734 Win XP (with Numeric 23.8): D:\MyProjects>numeric-stats.py Python version: 2.3.4 (#53, May 25 2004, 21:17:02) [MSC v.1200 32 bit (Intel)] *: 0.12 **2: 5.037 **3: 5.168 **0.5: 5.558 sqrt(): 0.14 exp(log): 0.46 exp(2*log): 1.012 exp(.5*log): 1.021 Win XP (with numarray 1.3.1): D:\MyProjects>numeric-stats.py Python version: 2.3.4 (#53, May 25 2004, 21:17:02) [MSC v.1200 32 bit (Intel)] *: 0.321 **2: 1.502 **3: 1.502 **0.5: 1.492 sqrt(): 1.953 exp(log): 1.993 exp(2*log): 2.043 exp(.5*log): 2.033 I get the same results (scaled on RAM) with my Windows 2000 machine at work. Is there some kind of explanation? A possible solution? Does anyone else get similar differences in timings as I do? > >Odd. *Really* Odd. Thank you for your suggestions. Andrea From stephen.walton at csun.edu Fri May 13 14:13:22 2005 From: stephen.walton at csun.edu (Stephen Walton) Date: Fri May 13 14:13:22 2005 Subject: [Numpy-discussion] About Reshape() In-Reply-To: <4200063400037BA1@ims9c.cp.tin.it> References: <4200063400037BA1@ims9c.cp.tin.it> Message-ID: <428517BB.8070103@csun.edu> andrea_gavana at tin.it wrote: >This continuous use of transpose() is killing the performances of my scripts... >is there a workaround? Am I missing something so obvious? > > Not necessarily obvious, but you've hit the old problem that MATLAB stores arrays in column major order (first index varies most rapidly, borrowed from Fortran), while Numeric and numarray store arrays in row major order (borrowed from C). The only way to avoid the transpose() calls is to rework your algorithms to work on arrays stored in row major order. From andrea_gavana at tin.it Fri May 13 14:23:05 2005 From: andrea_gavana at tin.it (andrea_gavana at tin.it) Date: Fri May 13 14:23:05 2005 Subject: [Numpy-discussion] About Reshape() In-Reply-To: <428517BB.8070103@csun.edu> Message-ID: <42000634000397C4@ims9c.cp.tin.it> Hello Stephen, >Not necessarily obvious, but you've hit the old problem that MATLAB >stores arrays in column major order (first index varies most rapidly, >borrowed from Fortran), while Numeric and numarray store arrays in row >major order (borrowed from C). Yes, I have seen the difference ;-) >The only way to avoid the transpose() >calls is to rework your algorithms to work on arrays stored in row major > >order. Yes, this is what I have done. It took some work because I still have my mind highly-badly conditioned by Matlab syntaxes, but it can be done. Thanks for your comments. Andrea. From carlosjosepita at yahoo.com.ar Tue May 17 17:33:18 2005 From: carlosjosepita at yahoo.com.ar (Carlos Pita) Date: Tue May 17 17:33:18 2005 Subject: [Numpy-discussion] Wrapping C array Message-ID: <200505172136.23149.carlosjosepita@yahoo.com.ar> Hi! I need to wrap a pre-existent C array into a numarray. It shouldn't be a copy of the original array because this is an audio buffer with a fixed address and also input/output copying from/to the buffer would be overkilling, so the numarray *must* be lightly wrapping the buffer and not a memcpy of it. But NA_NewArray duplicates the passed buffer so I'm not sure how to achieve the goal stated above or if it's possible at all. Can you help me? Regards, Carlos From Alexandre.Fayolle at logilab.fr Wed May 18 01:04:04 2005 From: Alexandre.Fayolle at logilab.fr (Alexandre) Date: Wed May 18 01:04:04 2005 Subject: [Numpy-discussion] Wrapping C array In-Reply-To: <200505172136.23149.carlosjosepita@yahoo.com.ar> References: <200505172136.23149.carlosjosepita@yahoo.com.ar> Message-ID: <20050518080158.GA18444@crater.logilab.fr> On Tue, May 17, 2005 at 09:36:22PM -0300, Carlos Pita wrote: > Hi! > I need to wrap a pre-existent C array into > a numarray. It shouldn't be a copy of the > original array because this is an audio buffer > with a fixed address and also input/output > copying from/to the buffer would be overkilling, > so the numarray *must* be lightly wrapping the > buffer and not a memcpy of it. But NA_NewArray > duplicates the passed buffer so I'm not sure > how to achieve the goal stated above or if > it's possible at all. > Can you help me? I'm not sure for Numarray, but for Numeric, you can use PyArray_FromDimsAndData(). There should be a similar function in numarray. -- Alexandre Fayolle LOGILAB, Paris (France). http://www.logilab.com http://www.logilab.fr http://www.logilab.org -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From jmiller at stsci.edu Wed May 18 08:31:21 2005 From: jmiller at stsci.edu (Todd Miller) Date: Wed May 18 08:31:21 2005 Subject: [Numpy-discussion] Wrapping C array In-Reply-To: <20050518080158.GA18444@crater.logilab.fr> References: <200505172136.23149.carlosjosepita@yahoo.com.ar> <20050518080158.GA18444@crater.logilab.fr> Message-ID: <1116421435.1460.0.camel@halloween.stsci.edu> On Wed, 2005-05-18 at 04:01, Alexandre wrote: > On Tue, May 17, 2005 at 09:36:22PM -0300, Carlos Pita wrote: > > Hi! > > I need to wrap a pre-existent C array into > > a numarray. It shouldn't be a copy of the > > original array because this is an audio buffer > > with a fixed address and also input/output > > copying from/to the buffer would be overkilling, > > so the numarray *must* be lightly wrapping the > > buffer and not a memcpy of it. But NA_NewArray > > duplicates the passed buffer so I'm not sure > > how to achieve the goal stated above or if > > it's possible at all. > > Can you help me? > > I'm not sure for Numarray, but for Numeric, you can use > PyArray_FromDimsAndData(). There should be a similar function in > numarray. That function does indeed exist for Numeric and has non-copying semantics. Regards, Todd From carlosjosepita at yahoo.com.ar Wed May 18 08:43:34 2005 From: carlosjosepita at yahoo.com.ar (Carlos Pita) Date: Wed May 18 08:43:34 2005 Subject: [Numpy-discussion] Wrapping C array In-Reply-To: <20050518080158.GA18444@crater.logilab.fr> References: <200505172136.23149.carlosjosepita@yahoo.com.ar> <20050518080158.GA18444@crater.logilab.fr> Message-ID: <200505181137.37147.carlosjosepita@yahoo.com.ar> But as NA_NewArray does, FromDimsAndData copies its contents from the passed pointer, which is what I'm willing to avoid. PyObject* PyArray FromDimsAndData(int nd, int *dims, int type, char *data) This function will allocate a numarray of the specified shape and type, and it's contents will be copied from the specified data pointer. The fact is that the audio buffer to be wrapped is a simple, fixed-size and contiguous C array, so I guess it should be easy to access through a numarray. But NA_NewArray and friends all seems to copy contents to a buffer of their own. Thank you. Regards, Carlos > On Tue, May 17, 2005 at 09:36:22PM -0300, Carlos Pita wrote: > > Hi! > > I need to wrap a pre-existent C array into > > a numarray. It shouldn't be a copy of the > > original array because this is an audio buffer > > with a fixed address and also input/output > > copying from/to the buffer would be overkilling, > > so the numarray *must* be lightly wrapping the > > buffer and not a memcpy of it. But NA_NewArray > > duplicates the passed buffer so I'm not sure > > how to achieve the goal stated above or if > > it's possible at all. > > Can you help me? > > I'm not sure for Numarray, but for Numeric, you can use > PyArray_FromDimsAndData(). There should be a similar function in > numarray. > From carlosjosepita at yahoo.com.ar Wed May 18 09:14:10 2005 From: carlosjosepita at yahoo.com.ar (Carlos Pita) Date: Wed May 18 09:14:10 2005 Subject: [Numpy-discussion] Wrapping C array In-Reply-To: <1116421435.1460.0.camel@halloween.stsci.edu> References: <200505172136.23149.carlosjosepita@yahoo.com.ar> <20050518080158.GA18444@crater.logilab.fr> <1116421435.1460.0.camel@halloween.stsci.edu> Message-ID: <200505181318.56247.carlosjosepita@yahoo.com.ar> The documentation for PyArray_FromDimsAndData that I quoted in my previous post seems to suggest that a copy is done. But browsing the web numarray cvs I find out that its implementation first creates a PyBuffer from the passed data: buf = PyBuffer_FromReadWriteMemory(data, nelements * descr->elsize); a = NA_NewAllFromBuffer( nd, dimensions, descr->type_num, buf, 0, descr->elsize, NA_ByteOrder(), 1, 1); Then, from the NA_NewAllFromBuffer doc (in numarray manual): It is possible to create a Python buffer object from an array of C data and then construct a numarray using this function which refers to the C data without making a copy. So you both are right. I guess creating a PyBuffer as above and then calling NA_NewAllFromBuffer is the best way to do what I was asking for in numarray (that is, without recurring to the numeric simulation API). Am I right? Regards, Carlos > On Wed, 2005-05-18 at 04:01, Alexandre wrote: > > On Tue, May 17, 2005 at 09:36:22PM -0300, Carlos Pita wrote: > > > Hi! > > > I need to wrap a pre-existent C array into > > > a numarray. It shouldn't be a copy of the > > > original array because this is an audio buffer > > > with a fixed address and also input/output > > > copying from/to the buffer would be overkilling, > > > so the numarray *must* be lightly wrapping the > > > buffer and not a memcpy of it. But NA_NewArray > > > duplicates the passed buffer so I'm not sure > > > how to achieve the goal stated above or if > > > it's possible at all. > > > Can you help me? > > > > I'm not sure for Numarray, but for Numeric, you can use > > PyArray_FromDimsAndData(). There should be a similar function in > > numarray. > > That function does indeed exist for Numeric and has non-copying > semantics. > > Regards, > Todd > > > > ------------------------------------------------------- > This SF.Net email is sponsored by Oracle Space Sweepstakes > Want to be the first software developer in space? > Enter now for the Oracle Space Sweepstakes! > http://ads.osdn.com/?ad_id=7412&alloc_id=16344&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From jmiller at stsci.edu Wed May 18 09:33:41 2005 From: jmiller at stsci.edu (Todd Miller) Date: Wed May 18 09:33:41 2005 Subject: [Numpy-discussion] Wrapping C array In-Reply-To: <1116421435.1460.0.camel@halloween.stsci.edu> References: <200505172136.23149.carlosjosepita@yahoo.com.ar> <20050518080158.GA18444@crater.logilab.fr> <1116421435.1460.0.camel@halloween.stsci.edu> Message-ID: <1116433645.1460.40.camel@halloween.stsci.edu> On Wed, 2005-05-18 at 09:03, Todd Miller wrote: > On Wed, 2005-05-18 at 04:01, Alexandre wrote: > > On Tue, May 17, 2005 at 09:36:22PM -0300, Carlos Pita wrote: > > > Hi! > > > I need to wrap a pre-existent C array into > > > a numarray. It shouldn't be a copy of the > > > original array because this is an audio buffer > > > with a fixed address and also input/output > > > copying from/to the buffer would be overkilling, > > > so the numarray *must* be lightly wrapping the > > > buffer and not a memcpy of it. But NA_NewArray > > > duplicates the passed buffer so I'm not sure > > > how to achieve the goal stated above or if > > > it's possible at all. > > > Can you help me? > > > > I'm not sure for Numarray, but for Numeric, you can use > > PyArray_FromDimsAndData(). There should be a similar function in > > numarray. > > That function does indeed exist for Numeric and has non-copying > semantics. I misspoke here, meaning to say numarray. More in a subsequent post. Todd From jmiller at stsci.edu Wed May 18 09:39:20 2005 From: jmiller at stsci.edu (Todd Miller) Date: Wed May 18 09:39:20 2005 Subject: [Numpy-discussion] Wrapping C array In-Reply-To: <200505181318.56247.carlosjosepita@yahoo.com.ar> References: <200505172136.23149.carlosjosepita@yahoo.com.ar> <20050518080158.GA18444@crater.logilab.fr> <1116421435.1460.0.camel@halloween.stsci.edu> <200505181318.56247.carlosjosepita@yahoo.com.ar> Message-ID: <1116434185.1460.58.camel@halloween.stsci.edu> On Wed, 2005-05-18 at 12:18, Carlos Pita wrote: > The documentation for PyArray_FromDimsAndData > that I quoted in my previous post seems to suggest > that a copy is done. But browsing the web I'd appreciate it if you could send me a reference to where that is because it is stale. > numarray cvs I find out that its implementation > first creates a PyBuffer from the passed data: > > buf = PyBuffer_FromReadWriteMemory(data, nelements * descr->elsize); > a = NA_NewAllFromBuffer( nd, dimensions, descr->type_num, buf, > 0, descr->elsize, NA_ByteOrder(), 1, 1); > > Then, from the NA_NewAllFromBuffer doc (in numarray > manual): > > It is possible to create a Python buffer object from > an array of C data and then construct a numarray using > this function which refers to the C data without > making a copy. > > So you both are right. I guess creating a > PyBuffer as above and then calling > NA_NewAllFromBuffer is the best way > to do what I was asking for in numarray > (that is, without recurring to the > numeric simulation API). > > Am I right? Not in my opinion, for two reasons: 1. PyArray_FromDimsAndData is easier to use, internalizing the buffer creation step. 2. These days, my recommendation is to use the Numeric compatibility layer because it gives the best portability, to Numeric now and to Numeric3 in the future. Most of the NA_ routines exist to facilitate the implementation of numarray itself and because we didn't always know a compatibility layer was possible or desirable. Of course you're free to use the NA_ routines and we're not planning on removing them, but I think the Numeric compatible routines are a better choice. Regards, Todd From carlosjosepita at yahoo.com.ar Wed May 18 09:56:16 2005 From: carlosjosepita at yahoo.com.ar (Carlos Pita) Date: Wed May 18 09:56:16 2005 Subject: [Numpy-discussion] Wrapping C array In-Reply-To: <1116434185.1460.58.camel@halloween.stsci.edu> References: <200505172136.23149.carlosjosepita@yahoo.com.ar> <200505181318.56247.carlosjosepita@yahoo.com.ar> <1116434185.1460.58.camel@halloween.stsci.edu> Message-ID: <200505181359.14435.carlosjosepita@yahoo.com.ar> > > The documentation for PyArray_FromDimsAndData > > that I quoted in my previous post seems to suggest > > that a copy is done. But browsing the web > > I'd appreciate it if you could send me a reference to where that is > because it is stale. I take the excerpt from the numarray user manual from the python2.3-numarray 1.1.1 debian (sarge) package. But it's updated in the cvs: \begin{cfuncdesc}{PyObject*}{PyArray_FromDimsAndData}{int nd, int *dims, int type, char *data} This function will allocate a numarray of the specified shape and type which will refer to the data buffer specified by \var{data}. The contents of \var{data} will not be copied nor will \var{data} be deallocated upon the deletion of the array. \end{cfuncdesc} It seems to be ok. When it says "nor will \var{data} be deallocated upon the deletion of the array", does it mean that the python buffer object ref count won't be decremented ? Or, more generally, the created numarray will refer directly to the C array segments exposed by the python buffer object or will keep a reference to this object? > Not in my opinion, for two reasons: > > 1. PyArray_FromDimsAndData is easier to use, internalizing the buffer > creation step. > > 2. These days, my recommendation is to use the Numeric compatibility > layer because it gives the best portability, to Numeric now and to I will take your advice. Thank you! Regards, Carlos From jmiller at stsci.edu Wed May 18 10:10:28 2005 From: jmiller at stsci.edu (Todd Miller) Date: Wed May 18 10:10:28 2005 Subject: [Numpy-discussion] Wrapping C array In-Reply-To: <200505181358.31406.carlosjosepita@yahoo.com.ar> References: <200505172136.23149.carlosjosepita@yahoo.com.ar> <200505181318.56247.carlosjosepita@yahoo.com.ar> <1116434185.1460.58.camel@halloween.stsci.edu> <200505181358.31406.carlosjosepita@yahoo.com.ar> Message-ID: <1116436049.1460.65.camel@halloween.stsci.edu> On Wed, 2005-05-18 at 12:58, Carlos Pita wrote: > > > The documentation for PyArray_FromDimsAndData > > > that I quoted in my previous post seems to suggest > > > that a copy is done. But browsing the web > > > > I'd appreciate it if you could send me a reference to where that is > > because it is stale. > > I take the excerpt from the numarray user manual from > the python2.3-numarray 1.1.1 debian (sarge) package. > But it's updated in the cvs: > > \begin{cfuncdesc}{PyObject*}{PyArray_FromDimsAndData}{int nd, int *dims, int type, char *data} > This function will allocate a numarray of the specified shape and type > which will refer to the data buffer specified by \var{data}. The contents > of \var{data} will not be copied nor will \var{data} be deallocated upon > the deletion of the array. > \end{cfuncdesc} > > It seems to be ok. When it says "nor will \var{data} be deallocated upon > the deletion of the array", does it mean that the python buffer object > ref count won't be decremented ? Or, more generally, the created > numarray will refer directly to the C array segments exposed by the > python buffer object or will keep a reference to this object? It means the internal buffer object refers to but does not "own" the array data passed into PyArray_FromDimsAndData(). At deallocation time, the internal buffer object is DECREF'd and disappears but the storage referred to by the data parameter does not. So, I/O buffers or other statically allocated arrays should work. Regards, Todd From edcjones at comcast.net Thu May 19 14:52:15 2005 From: edcjones at comcast.net (Edward C. Jones) Date: Thu May 19 14:52:15 2005 Subject: [Numpy-discussion] numarray.nd_image: max, min not preserved by rotate Message-ID: <428D0A32.1090904@comcast.net> I have up-to-date (5/18/05) Debian unstable on a PC. I compiled Python and numarray from source and I keep them in "/usr/local". Python is version 2.4.1 and numarray is version 1.3.1. The output of the program below is: 0.0 1.0 -0.107813523267 1.10781352327 I would expect a rotated image to have min == 0.0 and max == 1.0. The max and min of arr2 occur near the edge of the rotated half plane. This looks like a bug in nd_image. ------------------------- #! /usr/bin/env python import numarray, numarray.nd_image arr = numarray.zeros((256, 256), 'Float64') arr[128:, :] = 1.0 print arr.min(), arr.max() arr2 = numarray.nd_image.rotate(arr, 30, reshape=False) print arr2.min(), arr2.max() From tom.denniston at gmail.com Thu May 19 20:16:25 2005 From: tom.denniston at gmail.com (Tom Denniston) Date: Thu May 19 20:16:25 2005 Subject: [Numpy-discussion] Strange behavior of the RandomArray module Message-ID: I am running under python 1.4 and windows XP. I get the following results in my python interpretter: -------------------------------------------- n=0 import RandomArray while 1: arr = RandomArray.randint(0, 2509, 252) for x in arr: if x>2508: raise "Found number greater than 2508: " + str(x) + " on iteration " + str(n) n=n+1 Traceback (most recent call last): File "", line 5, in -toplevel- raise "Found number greater than 2508: " + str(x) + " on iteration " + str(n) Found number greater than 2508: 2509 on iteration 91719 ------------------------------------------- Even though the Numeric reference (see below) on RandomArray class claims that the number returned is always strictly less than the max. Does anyone know what might cause this problem? randint(minimum, maximum, shape=ReturnFloat) The randint() function returns an array of the specified shape and containing random (standard) integers greater than or equal to minimum and strictly less than maximum . If no shape is specified, a single number is returned. (exerpt from http://www.pfdubois.com/numpy/html2/numpy-19.html#pgfId-303108) From cookedm at physics.mcmaster.ca Thu May 19 22:35:29 2005 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Thu May 19 22:35:29 2005 Subject: [Numpy-discussion] Strange behavior of the RandomArray module In-Reply-To: References: Message-ID: <8F003FF6-845B-45C2-9E22-944719E03237@physics.mcmaster.ca> On May 19, 2005, at 23:13, Tom Denniston wrote: > I am running under python 1.4 and windows XP. I hope you mean 2.4 :-) > I get the following results in my python interpretter: > > -------------------------------------------- > n=0 > import RandomArray > while 1: > arr = RandomArray.randint(0, 2509, 252) > for x in arr: > if x>2508: > raise "Found number greater than 2508: " + > str(x) + " on iteration " + str(n) > n=n+1 > > > > Traceback (most recent call last): > File "", line 5, in -toplevel- > raise "Found number greater than 2508: " + str(x) + " on iteration > " + str(n) > Found number greater than 2508: 2509 on iteration 91719 > > ------------------------------------------- > > Even though the Numeric reference (see below) on RandomArray class > claims that the number returned is always strictly less than the max. > Does anyone > know what might cause this problem? This looks like it may be the same problem as bug #545336: http://sourceforge.net/tracker/ index.php&func=detail&aid=545336&group_id=1369&atid=101369 I ran the above (generating 1e10 numbers). The above problem happened 280 times, so it comes up at a rate of 2.8e-8 -- about what you'd expect for something going wrong with a last-bit error in rounding floats. There is indeed some float-double casting going on in ranlibmodule.c; that's probably it. I've gone and replaced that in CVS, generating an array of doubles directly instead for random() and randint(). This fixes it for me. [As an aside: note that the other random functions use floats. Really, an array of floats should be returned for those instead of an array of doubles. Next thing on the list :-)] -- |>|\/|< /------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From meesters at uni-mainz.de Fri May 20 05:31:26 2005 From: meesters at uni-mainz.de (Christian Meesters) Date: Fri May 20 05:31:26 2005 Subject: [Numpy-discussion] applying function under certain conditions Message-ID: Hi I have a large 1D numarray array with some huge values and lots of zeros. Now, for better representation I wanted to plot the logarithm. Well, calculating the logarithm is not such a good idea if you have zeros in your array ... However, how do you implement such cases like "only apply a function if a certain condition is true", I'd like to ask whether there are nice short snippets which could show such cases. My own code is (unnecessarily?) long and ugly. I tried to combine 'where' and 'greater', but didn't succeed. I guess I'm just don't see the wood for the trees. How would you write this in my case? TIA Regards, Christian From Alexandre.Fayolle at logilab.fr Fri May 20 05:53:04 2005 From: Alexandre.Fayolle at logilab.fr (Alexandre) Date: Fri May 20 05:53:04 2005 Subject: [Numpy-discussion] applying function under certain conditions In-Reply-To: References: Message-ID: <20050520124958.GI27695@crater.logilab.fr> On Fri, May 20, 2005 at 02:22:25PM +0200, Christian Meesters wrote: > Hi > > I have a large 1D numarray array with some huge values and lots of > zeros. Now, for better representation I wanted to plot the logarithm. > Well, calculating the logarithm is not such a good idea if you have > zeros in your array ... > However, how do you implement such cases like "only apply a function if > a certain condition is true", I'd like to ask whether there are nice > short snippets which could show such cases. My own code is > (unnecessarily?) long and ugly. I tried to combine 'where' and > 'greater', but didn't succeed. I guess I'm just don't see the wood for > the trees. How would you write this in my case? Assuming your data contains only positive or null values,you could try replacing zeros by a very small value before computing the log: log_scaled = log(where(data, data, 1e-50)) This uses the fact that 0 is false and non-zero is true. If you have negative values, you can use something like log_scaled = log(where(data>0, data, 1e-50)) since data>0 will return a boolean array suitable for where. -- Alexandre Fayolle LOGILAB, Paris (France). http://www.logilab.com http://www.logilab.fr http://www.logilab.org -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From perry at stsci.edu Fri May 20 06:28:51 2005 From: perry at stsci.edu (Perry Greenfield) Date: Fri May 20 06:28:51 2005 Subject: [Numpy-discussion] applying function under certain conditions In-Reply-To: <20050520124958.GI27695@crater.logilab.fr> References: <20050520124958.GI27695@crater.logilab.fr> Message-ID: <1e661f1687aebd36a555fa364445dac7@stsci.edu> On May 20, 2005, at 8:49 AM, Alexandre wrote: > On Fri, May 20, 2005 at 02:22:25PM +0200, Christian Meesters wrote: >> Hi >> >> I have a large 1D numarray array with some huge values and lots of >> zeros. Now, for better representation I wanted to plot the logarithm. >> Well, calculating the logarithm is not such a good idea if you have >> zeros in your array ... >> However, how do you implement such cases like "only apply a function >> if >> a certain condition is true", I'd like to ask whether there are nice >> short snippets which could show such cases. My own code is >> (unnecessarily?) long and ugly. I tried to combine 'where' and >> 'greater', but didn't succeed. I guess I'm just don't see the wood for >> the trees. How would you write this in my case? > > Assuming your data contains only positive or null values,you could try > replacing zeros by a very small value before computing the log: > > log_scaled = log(where(data, data, 1e-50)) > > This uses the fact that 0 is false and non-zero is true. > > If you have negative values, you can use something like > > log_scaled = log(where(data>0, data, 1e-50)) > > since data>0 will return a boolean array suitable for where. > > -- > Alexandre Fayolle LOGILAB, Paris (France). > http://www.logilab.com http://www.logilab.fr http://www.logilab.org Another alternative that takes two lines but perhaps is more readable is: data[data<=0] = 1e-50 # if you don't mind modifying the original array log_scaled = log(data) From meesters at uni-mainz.de Fri May 20 06:45:31 2005 From: meesters at uni-mainz.de (Christian Meesters) Date: Fri May 20 06:45:31 2005 Subject: [Numpy-discussion] applying function under certain conditions In-Reply-To: References: Message-ID: Hi Thanks. So, it is as easy as I thought ... Since I do not want to modify the original array in this case (it's only for plotting purposes), I will go for Alexandre's solution here. Thanks a lot, Christian From rowen at cesmail.net Fri May 20 15:40:04 2005 From: rowen at cesmail.net (Russell E. Owen) Date: Fri May 20 15:40:04 2005 Subject: [Numpy-discussion] Re: applying function under certain conditions References: Message-ID: In article , Christian Meesters wrote: > Thanks. So, it is as easy as I thought ... > Since I do not want to modify the original array in this case (it's > only for plotting purposes), I will go for Alexandre's solution here. One more thing to consider...do you really want log? If you are looking for log-like behavior for positive values but graceful handling of 0 and negative values, consider arcsinh, e.g. arcsinh(scale * (x-offset)). -- Russell *Due to the properties I just mentioned, arcsinh is a very useful function for stretching astronomical images. Thanks to Robert Lupton for pointing this out to me. From meesters at uni-mainz.de Fri May 20 23:57:46 2005 From: meesters at uni-mainz.de (Christian Meesters) Date: Fri May 20 23:57:46 2005 Subject: [Numpy-discussion] Re: applying function under certain conditions In-Reply-To: References: Message-ID: <72352ac02b03c4342745c1c66ca6e7ce@uni-mainz.de> Hi Oh yes, that's an idea. I actually really wanted log, though more for historical display reasons than for mathematical reasons. In my particular case it hardly makes a difference for the eye whether I use arcsinh or log, but in the near future I might want to use it. I will keep it in mind. Thanks for pointing this out to me. Have a nice weekend, Christian On 21 May 2005, at 00:36, Russell E. Owen wrote: > In article , > Christian Meesters wrote: > >> Thanks. So, it is as easy as I thought ... >> Since I do not want to modify the original array in this case (it's >> only for plotting purposes), I will go for Alexandre's solution here. > > One more thing to consider...do you really want log? If you are looking > for log-like behavior for positive values but graceful handling of 0 > and > negative values, consider arcsinh, e.g. arcsinh(scale * (x-offset)). > > -- Russell > > *Due to the properties I just mentioned, arcsinh is a very useful > function for stretching astronomical images. Thanks to Robert Lupton > for > pointing this out to me. > > > From a.jaffe at imperial.ac.uk Sun May 22 09:48:32 2005 From: a.jaffe at imperial.ac.uk (Andrew Jaffe) Date: Sun May 22 09:48:32 2005 Subject: [Numpy-discussion] numarray 1.3.1 setup patch for mac OS X veclib Message-ID: Hi all, I've had some trouble getting the numarray-1.3.1 setup scripts to recognize the optimized BLAS/LAPACK, veclib, on OS X. But I think I've tracked down the problems. First, the section of addons.py that checks for the existence of the framework (around line 47) needs to set lapack_dirs = [], I think, since it needs to exist later on. Second, the section after that that sets lapack_compile_args, lapack_link_args and lapack_include_dirs for *all* cases (lines 57-60) overwrites lapack_link_args from the if...elif...else above. Finally, cblas.h isn't included correctly, since the framework option is only used for linking, not compiling. I've attached a patchfile for addons.py below; I switched the two sections and added to the framework if... clause. Use it with > patch addons.py addons.patch I hope this is a reasonable way to propagate this to the developers. Please let me know if it doesn't work for any reason, or if it breaks any other install, which I haven't checked. I assume it works on my machine, since: > otool -L build/lib.darwin-7.9.0-Power_Macintosh-2.3/numarray/linear_algebra/ lapack_lite2.so /System/Library/Frameworks/Python.framework/Versions/2.3/Python (compatibility version 2.3.0, current version 2.3.0) /System/Library/Frameworks/vecLib.framework/Versions/A/vecLib (compatibility version 1.0.0, current version 153.2.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 71.1.3) Andrew p.s. only tested under OS X 10.3.9 (latest version of Panther). -------------- next part -------------- A non-text attachment was scrubbed... Name: addons.patch Type: application/octet-stream Size: 1241 bytes Desc: not available URL: -------------- next part -------------- ______________________________________________________________________ Andrew Jaffe a.jaffe at imperial.ac.uk Astrophysics Group +44 207 594-7526 Blackett Laboratory, Room 1013 FAX 7541 Imperial College, Prince Consort Road London SW7 2AZ ENGLAND http://astro.imperial.ac.uk/~jaffe From a.schmolck at gmx.net Mon May 23 17:13:28 2005 From: a.schmolck at gmx.net (Alexander Schmolck) Date: Mon May 23 17:13:28 2005 Subject: [Numpy-discussion] dot-product of selected columns or rows Message-ID: Suppose I have some iterative algorithm that selects and deselects 'features' represented by a list of indices `active` that correspond to rows (or equivalently columns) in some matrices `M`, `N`, ... What would be the most efficient way to carry out the equivalent of: A = dot(take(M, active), take(C, active),axis=1) assuming that M and C remain constant throughtout the computation and that A could be modified in-place for the relevant indices in `active`. Also am I right to assume that generally the best approach to cheaply create a matrix with a different diagonal from a square and contiguous n*n matrix `S` (that is no longer needed) would be the following in-place operation? S.flat[::n+1] += vec In case it matters: I'm using Numeric and scipy, but not numarray. 'as From jmiller at stsci.edu Wed May 25 10:45:31 2005 From: jmiller at stsci.edu (Todd Miller) Date: Wed May 25 10:45:31 2005 Subject: [Numpy-discussion] ANN: numarray-1.3.2 Message-ID: <1117042956.24369.99.camel@halloween.stsci.edu> Numarray is an array processing package designed to efficiently manipulate large multi-dimensional arrays. Numarray is modelled after Numeric and features c-code generated from python template scripts, the capacity to operate directly on arrays in files, arrays of heterogeneous records, string arrays, and in-place operation on memory mapped files. ========================================================================= Release Notes for numarray-1.3.2 I. ENHANCEMENTS None. 1.3.2 fixes the problem with gcc4 on Mac OS-X 10.4 Tiger II. BUGS FIXED / CLOSED 1193125 Build failure on Mac OS 10.4 (Tiger) 1195370 numarray1.3.1 setup.py broken See http://sourceforge.net/tracker/?atid=450446&group_id=1369&func=browse for more details. ========================================================================= Release Notes for numarray-1.3.1 I. ENHANCEMENTS None. 1.3.1 fixes the problem with gcc-3.4.3 II. BUGS FIXED / CLOSED 1152323 /usr/include/fenv.h:96: error: conflicting types for 'fegete 1185024 numarray-1.2.3 fails to compile with gcc-3.4.3 1187162 Numarray 1.3.0 installation failure See http://sourceforge.net/tracker/?atid=450446&group_id=1369&func=browse for more details. ========================================================================= Release Notes for numarray-1.3.0 Numarray is an array processing package designed to efficiently manipulate large multi-dimensional arrays. Numarray is modelled after Numeric and features c-code generated from python template scripts, the capacity to operate directly on arrays in files, arrays of heterogeneous records, string arrays, and in-place operation on memory mapped files. I. ENHANCEMENTS 1. Migration of NumArray.__del__ to C (tp_dealloc). Overall performance. 2. Removal of dictionary update from array view creation improves performance of view/slice/subarray creation. This should e.g. improve the performance of wxPython sequence protocol access to Nx2 arrays. Subclasses now need to do a.flags |= numarray.generic._UPDATEDICT to ensure that dictionary based attributes are inherited by views. NumArrays no longer do this by default. 2. Modifications to support scipy.special. 3. Removal of an unnecessary getattr() from ufunc calling sequence. Ufunc performance. II. BUGS FIXED / CLOSED 1179355 average() broken in numarray 1.2.3 1167184 Floating point exception in numarray's dot() 1151892 Bug in matrixmultiply with zero size arrays 1160184 RecArray reversal 1156172 Incorect error message for shape incompatability 1155538 Incorrect error message when multiplying arrays See http://sourceforge.net/tracker/?atid=450446&group_id=1369&func=browse for more details. III. CAUTIONS This release should be backward binary compatible with numarray-1.1.1 and 1.2.3. WHERE ----------- Numarray-1.3.0 windows executable installers, source code, and manual is here: http://sourceforge.net/project/showfiles.php?group_id=1369 Numarray is hosted by Source Forge in the same project which hosts Numeric: http://sourceforge.net/projects/numpy/ The web page for Numarray information is at: http://stsdas.stsci.edu/numarray/index.html Trackers for Numarray Bugs, Feature Requests, Support, and Patches are at the Source Forge project for NumPy at: http://sourceforge.net/tracker/?group_id=1369 REQUIREMENTS ------------------------------ numarray-1.3.0 requires Python 2.2.2 or greater. AUTHORS, LICENSE ------------------------------ Numarray was written by Perry Greenfield, Rick White, Todd Miller, JC Hsu, Paul Barrett, Phil Hodge at the Space Telescope Science Institute. We'd like to acknowledge the assitance of Francesc Alted, Paul Dubois, Sebastian Haase, Chuck Harris, Tim Hochberg, Nadav Horesh, Edward C. Jones, Eric Jones, Jochen Kuepper, Travis Oliphant, Pearu Peterson, Peter Verveer, Colin Williams, Rory Yorke, and everyone else who has contributed with comments and feedback. Numarray is made available under a BSD-style License. See LICENSE.txt in the source distribution for details. -- Todd Miller jmiller at stsci.edu ________________________________________________________________________ From Chris.Barker at noaa.gov Thu May 26 16:55:16 2005 From: Chris.Barker at noaa.gov (Chris Barker) Date: Thu May 26 16:55:16 2005 Subject: [Numpy-discussion] OT: using f2c.h Message-ID: <429660F1.4040503@noaa.gov> HI all, This is off-topic, as it has nothing to do with NumPy, but this is the only mailing list I'm on that has folks that might be able to help. I'm trying to compile something that uses clapack, and thus f2clib. I installed f2c on my Gentoo box, and it compiled and installed fine, but when I try to compile the app in question, I get errors in f2c.h: g++ -c -o Utilities.o `/usr/local/bin/wx-config --inplace --cxxflags` Utilities.cpp In file included from Utilities.cpp:6: /usr/include/f2c.h:16: error: parse error before `,' token /usr/include/f2c.h:16: error: semicolon missing after struct declaration /usr/include/f2c.h:16: error: abstract declarator `' used as declaration /usr/include/f2c.h:16: error: namespace-scope anonymous aggregates must be static /usr/include/f2c.h:132: error: 'real' is used as a type, but is not defined as a type. /usr/include/f2c.h:134: error: 'complex' is used as a type, but is not defined as a type. /usr/include/f2c.h:174: error: `R_fp' was not declared in this scope /usr/include/f2c.h:174: error: typedef `real' is initialized (use __typeof__ instead) /usr/include/f2c.h:174: error: conflicting types for `typedef int (real)(...)' /usr/include/f2c.h:14: error: previous declaration as `typedef float real' Jsut in case, I also tried with the ftc.h that is included with Numeric24, which compiled just fine (maybe it's now on topic?) I got the same errors. Is this a compiler flag issue (maybe for old style C) or something? I'm clueless. thanks for any helpful hints. As you can tell, I know next to nothing about C/C++ -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From rkern at ucsd.edu Thu May 26 17:16:20 2005 From: rkern at ucsd.edu (Robert Kern) Date: Thu May 26 17:16:20 2005 Subject: [Numpy-discussion] OT: using f2c.h In-Reply-To: <429660F1.4040503@noaa.gov> References: <429660F1.4040503@noaa.gov> Message-ID: <42966610.8060703@ucsd.edu> Chris Barker wrote: > HI all, > > This is off-topic, as it has nothing to do with NumPy, but this is the > only mailing list I'm on that has folks that might be able to help. > > I'm trying to compile something that uses clapack, and thus f2clib. I > installed f2c on my Gentoo box, and it compiled and installed fine, but > when I try to compile the app in question, I get errors in f2c.h: > > g++ -c -o Utilities.o `/usr/local/bin/wx-config --inplace --cxxflags` > Utilities.cpp > In file included from Utilities.cpp:6: > /usr/include/f2c.h:16: error: parse error before `,' token > /usr/include/f2c.h:16: error: semicolon missing after struct declaration I'm willing to bet that it's choking on line 15: typedef struct { real r, i; } complex; "complex" is already taken, I believe. At least vim is highlighting it as if it were a reserved type. -- Robert Kern rkern at ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From afeiguin at uci.edu Thu May 26 18:19:52 2005 From: afeiguin at uci.edu (Adrian E. Feiguin) Date: Thu May 26 18:19:52 2005 Subject: [Numpy-discussion] crashes after switching to 1.3.x Message-ID: <42967147.1000409@uci.edu> Hi! I'm the lead developer of scigraphica (SG) http://scigraphica.sourceforge.net and I'm using python and numarray to parse math. SG is built on top of libscigraphica, and libscigraphica has all the python code. I've been using an old version 0.6.1, and everything worked fine. After switching to 1.6.2the program crashes. I'll give you some info to see if you can tell me what's wrong: I'm using: #define PY_ARRAY_UNIQUE_SYMBOL PyArrayXXX #include and in the code: import_libnumeric(); import_array(); object=PyRun_String ("from numarray import *", Py_single_input, main_dict, sg_dict); (I noticed that you have to add import_libnumeric now) The gdb output is: Importing python module: sys Importing python module: pickle Importing python module: os Importing python module: numarrayProgram received signal SIGSEGV, Segmentation fault. [Switching to Thread 1083181376 (LWP 11648)] 0x400ab0b5 in PyObject_GetAttrString () from /home/afeiguin/cvs/test/libscigraphica-2/scigraphica/.libs/libscigraphica-2.0-2.0.so.0 #0 0x400ab0b5 in PyObject_GetAttrString () from /home/afeiguin/cvs/test/libscigraphica-2/scigraphica/.libs/libscigraphica-2.0-2.0.so.0 #1 0x40b1ff66 in deferred_libnumarray_init () at Src/libnumarraymodule.c:152 #2 0x40b21f8c in NA_NumArrayCheck (op=0x0) at Src/libnumarraymodule.c:1357 #3 0x40b43c1c in PyArray_Check (op=0x0) at Src/libnumericmodule.c:216 #4 0x4004d903 in python_insert_object (worksheet=0x81ba4e0, row=0, col=0, object=0x80bda3c, orient=GTK_ORIENTATION_VERTICAL, link=0, as_is=0) at sg_python_worksheet.c:802 #5 0x4004da12 in python_sheet (worksheet=0x81ba4e0, row=0, col=0, command=0x8366d68 ".1", orient=GTK_ORIENTATION_VERTICAL) at sg_python_worksheet.c:843 #6 0x40061f7c in sg_worksheet_cell_set (worksheet=0x81ba4e0, row=0, col=0, text=0x8365520 ".1", formula=1, eval=1) at sg_worksheet.c:508 #7 0x40061d59 in set_cell (sheet=0x81ba4e0, row=0, col=0, data=0x81ba4e0) at sg_worksheet.c:439 Look that in #4 object=0x80bda3c, and in #3 op=0x0. Any hints? Thanks a lot! From Mailer-Daemon at host.talk.ru Fri May 27 03:15:05 2005 From: Mailer-Daemon at host.talk.ru (Mail Delivery System) Date: Fri May 27 03:15:05 2005 Subject: [Numpy-discussion] Mail delivery failed: returning message to sender Message-ID: This message was created automatically by mail delivery software (Exim). A message that you sent could not be delivered to one or more of its recipients. This is a permanent error. The following address(es) failed: 2310216647 at host.talk.ru unknown local-part "2310216647" in domain "host.talk.ru" ------ This is a copy of the message, including all the headers. ------ ------ The body of the message is 41395 characters long; only the first ------ 1024 or so are included here. From numpy-discussion at lists.sourceforge.net Fri May 27 06:15:13 2005 From: numpy-discussion at lists.sourceforge.net (numpy-discussion at lists.sourceforge.net) Date: Fri, 27 May 2005 14:15:13 +0400 Subject: Mail Delivery (failure 2310216647@host.talk.ru) Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: message.scr Type: audio/x-wav Size: 6216 bytes Desc: not available URL: From jmiller at stsci.edu Fri May 27 03:54:30 2005 From: jmiller at stsci.edu (Todd Miller) Date: Fri May 27 03:54:30 2005 Subject: [Numpy-discussion] crashes after switching to 1.3.x In-Reply-To: <42967147.1000409@uci.edu> References: <42967147.1000409@uci.edu> Message-ID: <1117191173.4718.15.camel@jaytmiller.comcast.net> Hi Adrian, On Thu, 2005-05-26 at 18:00 -0700, Adrian E. Feiguin wrote: > Hi! I'm the lead developer of scigraphica (SG) > http://scigraphica.sourceforge.net and I'm using python and numarray to > parse math. SG is built on top of libscigraphica, and libscigraphica has > all the python code. I've been using an old version 0.6.1, and > everything worked fine. After switching to 1.6.2the program crashes. > I'll give you some info to see if you can tell me what's wrong: > > I'm using: > #define PY_ARRAY_UNIQUE_SYMBOL PyArrayXXX > #include > > and in the code: > import_libnumeric(); > import_array(); > > object=PyRun_String ("from numarray import *", Py_single_input, > main_dict, sg_dict); > > (I noticed that you have to add import_libnumeric now) In general, this is not true. Paradoxically, if you're using the numeric compatible API, import_array() is all you need to do. If you're using both the numarray native and numeric compatible APIs, then you need to import_libnumarray() and import_libnumeric(). > The gdb output is: > > Importing python module: sys > Importing python module: pickle > Importing python module: os > Importing python module: numarrayProgram received signal SIGSEGV, > Segmentation fault. > [Switching to Thread 1083181376 (LWP 11648)] > 0x400ab0b5 in PyObject_GetAttrString () > from > /home/afeiguin/cvs/test/libscigraphica-2/scigraphica/.libs/libscigraphica-2.0-2.0.so.0 > #0 0x400ab0b5 in PyObject_GetAttrString () > from > /home/afeiguin/cvs/test/libscigraphica-2/scigraphica/.libs/libscigraphica-2.0-2.0.so.0 > #1 0x40b1ff66 in deferred_libnumarray_init () at > Src/libnumarraymodule.c:152 > #2 0x40b21f8c in NA_NumArrayCheck (op=0x0) at Src/libnumarraymodule.c:1357 > #3 0x40b43c1c in PyArray_Check (op=0x0) at Src/libnumericmodule.c:216 > #4 0x4004d903 in python_insert_object (worksheet=0x81ba4e0, row=0, col=0, > object=0x80bda3c, orient=GTK_ORIENTATION_VERTICAL, link=0, as_is=0) > at sg_python_worksheet.c:802 > #5 0x4004da12 in python_sheet (worksheet=0x81ba4e0, row=0, col=0, > command=0x8366d68 ".1", orient=GTK_ORIENTATION_VERTICAL) > at sg_python_worksheet.c:843 > #6 0x40061f7c in sg_worksheet_cell_set (worksheet=0x81ba4e0, row=0, col=0, > text=0x8365520 ".1", formula=1, eval=1) at sg_worksheet.c:508 > #7 0x40061d59 in set_cell (sheet=0x81ba4e0, row=0, col=0, data=0x81ba4e0) > at sg_worksheet.c:439 > > Look that in #4 object=0x80bda3c, and in #3 op=0x0. > Any hints? numarray-0.6.1 is very old so in order to transition to numarray-1.3.2 you should: a. completely delete your numarray installation b. reinstall numarray c. rebuild and reinstall any extensions based on numarray. My guess is that either you have overlapping/conflicting numarray installations or you have extensions from numarray-0.6.1 trying to run against numarray-1.3.2. Regards, Todd From a.h.jaffe at gmail.com Fri May 27 08:10:57 2005 From: a.h.jaffe at gmail.com (Andrew Jaffe) Date: Fri May 27 08:10:57 2005 Subject: [Numpy-discussion] Re: numarray 1.3.1 setup patch for mac OS X veclib Message-ID: Dear All, My apologies, but I seem to have made a mistake in the distutils patch for OS X, which has unfortunately made it into the latest 1.3.2 release. The bottom line is, I think, that we need an explicit include dir for cblas.h, rather than framework. Thus, we really need to add the following line to cfg_packages.py: lapack_include_dirs += ['/System/Library/Frameworks/vecLib.framework/Headers'] in the block that deals with the veclib framework (around line 53) This should be generic -- it's a symlink to the current version -- but I don't know if there's a better way to auto-generate the dir name. Sorry for not testing thoroughly enough! Andrew On 22 May 2005, at 17:46, Andrew Jaffe wrote: >Hi all, >I've had some trouble getting the numarray-1.3.1 setup scripts to recognize the optimized >BLAS/LAPACK, veclib, on OS X. But I think I've tracked down the problems. >First, the section of addons.py that checks for the existence of the >framework (around line 47) needs to set lapack_dirs = [], I think, >since it needs to exist later on. >Second, the section after that that sets lapack_compile_args, >lapack_link_args and lapack_include_dirs for *all* cases (lines 57-60) >overwrites lapack_link_args from the if...elif...else above. >Finally, cblas.h isn't included correctly, since the framework option >is only used for linking, not compiling. >I've attached a patchfile for addons.py below; I switched the two sections and added to the >framework if... clause. Use it with > patch addons.py addons.patch >I hope this is a reasonable way to propagate this to the developers. >Please let me know if it doesn't work for any reason, or if it breaks any other install, which >I haven't checked. ______________________________________________________________________ Andrew Jaffe Astrophysics Group Blackett Laboratory, Room 1013 Imperial College, Prince Consort Road London SW7 2AZ ENGLAND http://astro.imperial.ac.uk/~jaffe From dlyzxl at yahoo.com Fri May 27 08:43:10 2005 From: dlyzxl at yahoo.com (Justin) Date: Fri May 27 08:43:10 2005 Subject: [Numpy-discussion] Numeric-24.0b2 installation problem Message-ID: <20050527154131.47344.qmail@web51104.mail.yahoo.com> Dear friends: I got an installation problem as following: [rathlefs at ishmael ~/uploads/Numeric-24.0b2]$ ./python setup.py install running install running build running build_py running build_ext building '_numpy' extension cc -DNDEBUG -O -IInclude -IPackages/FFT/Include -IPackages/RNG/Include -I/home/rathlefs/languages/activePython/include/python2.4 -c Src/_numpymodule.c -o build/temp.solaris-2.8-sun4u-2.4/Src/_numpymodule.o /usr/ucb/cc: language optional software package not installed error: command 'cc' failed with exit status 1 ___________________________________________ and I am guessing it is related to C compiler, and got suggestion from system administrator as following: The only compiler on the system is the GNU C compiler. Try using /usr/local/bin/gcc /usr/ucb/cc is only a wraper the the Sun compiler, which is not installed __________________________________ I am only an user for the solaris system (Ultra-sparc III) and want to install the Numerical Python under my account. I already installed activePython2.4.1 under my account successfully . Can anyone give me some suggestion about the installation? thanks, Justin --------------------------------- Do you Yahoo!? Make Yahoo! your home page -------------- next part -------------- An HTML attachment was scrubbed... URL: From jmiller at stsci.edu Fri May 27 09:42:02 2005 From: jmiller at stsci.edu (Todd Miller) Date: Fri May 27 09:42:02 2005 Subject: [Numpy-discussion] Re: numarray 1.3.1 setup patch for mac OS X veclib In-Reply-To: References: Message-ID: <1117212029.3576.274.camel@halloween.stsci.edu> On Fri, 2005-05-27 at 11:09, Andrew Jaffe wrote: > Dear All, > > My apologies, but I seem to have made a mistake in the distutils patch > for OS X, which has unfortunately made it into the latest 1.3.2 > release. > > The bottom line is, I think, that we need an explicit include dir for > cblas.h, rather than framework. > > Thus, we really need to add the following line to cfg_packages.py: > lapack_include_dirs += > ['/System/Library/Frameworks/vecLib.framework/Headers'] > in the block that deals with the veclib framework (around line 53) > > This should be generic -- it's a symlink to the current version -- but I > don't know if there's a better way to auto-generate the dir name. > > Sorry for not testing thoroughly enough! > > Andrew Thanks for the new include dirs and for the original patch. They're both now in CVS. Since the default Mac OS-X non-vecLib build works fine, I'm going to wait a few weeks for 1.3.3. I have related question though: Should USE_LAPACK be the default for Mac OS-X? Is the vecLib framework is *always* there? Regards, Todd > On 22 May 2005, at 17:46, Andrew Jaffe wrote: > > >Hi all, > > >I've had some trouble getting the numarray-1.3.1 setup scripts to > recognize the optimized > >BLAS/LAPACK, veclib, on OS X. But I think I've tracked down the problems. > > >First, the section of addons.py that checks for the existence of the > >framework (around line 47) needs to set lapack_dirs = [], I think, > >since it needs to exist later on. > > >Second, the section after that that sets lapack_compile_args, > >lapack_link_args and lapack_include_dirs for *all* cases (lines 57-60) > >overwrites lapack_link_args from the if...elif...else above. > > >Finally, cblas.h isn't included correctly, since the framework option > >is only used for linking, not compiling. > > >I've attached a patchfile for addons.py below; I switched the two > sections and added to the > >framework if... clause. Use it with > > patch addons.py addons.patch > > >I hope this is a reasonable way to propagate this to the developers. > > >Please let me know if it doesn't work for any reason, or if it breaks > any other install, which > >I haven't checked. > > ______________________________________________________________________ > Andrew Jaffe > Astrophysics Group > Blackett Laboratory, Room 1013 > Imperial College, Prince Consort Road > London SW7 2AZ ENGLAND http://astro.imperial.ac.uk/~jaffe > > > ------------------------------------------------------- > This SF.Net email is sponsored by Yahoo. > Introducing Yahoo! Search Developer Network - Create apps using Yahoo! > Search APIs Find out how you can build Yahoo! directly into your own > Applications - visit http://developer.yahoo.net/?fr=offad-ysdn-ostg-q22005 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- From Chris.Barker at noaa.gov Fri May 27 09:44:00 2005 From: Chris.Barker at noaa.gov (Chris Barker) Date: Fri May 27 09:44:00 2005 Subject: [Numpy-discussion] OT: using f2c.h In-Reply-To: <42966610.8060703@ucsd.edu> References: <429660F1.4040503@noaa.gov> <42966610.8060703@ucsd.edu> Message-ID: <42974DEA.2010904@noaa.gov> Thanks, Robert. Robert Kern wrote: > I'm willing to bet that it's choking on line 15: > > typedef struct { real r, i; } complex; > > "complex" is already taken, I believe. At least vim is highlighting it as if it were a reserved type. What's confusing is that this same header works fine with Numeric, and this same code has compiled fine (though I don't know if the header is exactly the same) under Windows Visual C++. Of course, I've found that VC++ lets all sorts of sloppiness through that gcc catches. I've just got a hint, however. C++ has a complex type, c doesn't, and I'm compiling this with g++. Numeric is all C, so it wouldn't have that problem. I'll plug away, but if anyone has a hint on how I can use f2clib in a C++ app with gcc, I'd love to hear it. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From a.h.jaffe at gmail.com Fri May 27 10:11:42 2005 From: a.h.jaffe at gmail.com (Andrew Jaffe) Date: Fri May 27 10:11:42 2005 Subject: [Numpy-discussion] Re: numarray 1.3.1 setup patch for mac OS X veclib In-Reply-To: <1117212029.3576.274.camel@halloween.stsci.edu> References: <1117212029.3576.274.camel@halloween.stsci.edu> Message-ID: Hi Todd etc, > I have related question though: Should USE_LAPACK be the default for > Mac OS-X? Is the vecLib framework is *always* there? > > Regards, > Todd I think it's always there under OS X per se, but not, perhaps, under non-OS X Darwin. Does anyone out there use Darwin under those circumstances? But given the structure of the distutils, why not just check for the existence of the *framework*, in which case you know that use_lapack is appropriate? Andrew >> On 22 May 2005, at 17:46, Andrew Jaffe wrote: >> >>> Hi all, >> >>> I've had some trouble getting the numarray-1.3.1 setup scripts to >> recognize the optimized >>> BLAS/LAPACK, veclib, on OS X. But I think I've tracked down the >>> problems. >> >>> First, the section of addons.py that checks for the existence of the >>> framework (around line 47) needs to set lapack_dirs = [], I think, >>> since it needs to exist later on. >> >>> Second, the section after that that sets lapack_compile_args, >>> lapack_link_args and lapack_include_dirs for *all* cases (lines >>> 57-60) >>> overwrites lapack_link_args from the if...elif...else above. >> >>> Finally, cblas.h isn't included correctly, since the framework option >>> is only used for linking, not compiling. >> >>> I've attached a patchfile for addons.py below; I switched the two >> sections and added to the >>> framework if... clause. Use it with >>> patch addons.py addons.patch >> >>> I hope this is a reasonable way to propagate this to the developers. >> >>> Please let me know if it doesn't work for any reason, or if it breaks >> any other install, which >>> I haven't checked. >> >> ______________________________________________________________________ >> Andrew Jaffe >> Astrophysics Group >> Blackett Laboratory, Room 1013 >> Imperial College, Prince Consort Road >> London SW7 2AZ ENGLAND http://astro.imperial.ac.uk/~jaffe >> From cookedm at physics.mcmaster.ca Fri May 27 10:20:36 2005 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Fri May 27 10:20:36 2005 Subject: [Numpy-discussion] Re: numarray 1.3.1 setup patch for mac OS X veclib In-Reply-To: <1117212029.3576.274.camel@halloween.stsci.edu> (Todd Miller's message of "Fri, 27 May 2005 12:40:29 -0400") References: <1117212029.3576.274.camel@halloween.stsci.edu> Message-ID: Todd Miller writes: > On Fri, 2005-05-27 at 11:09, Andrew Jaffe wrote: >> Dear All, >> >> My apologies, but I seem to have made a mistake in the distutils patch >> for OS X, which has unfortunately made it into the latest 1.3.2 >> release. >> >> The bottom line is, I think, that we need an explicit include dir for >> cblas.h, rather than framework. >> >> Thus, we really need to add the following line to cfg_packages.py: >> lapack_include_dirs += >> ['/System/Library/Frameworks/vecLib.framework/Headers'] >> in the block that deals with the veclib framework (around line 53) >> >> This should be generic -- it's a symlink to the current version -- but I >> don't know if there's a better way to auto-generate the dir name. >> >> Sorry for not testing thoroughly enough! >> >> Andrew > > Thanks for the new include dirs and for the original patch. They're > both now in CVS. Since the default Mac OS-X non-vecLib build works > fine, I'm going to wait a few weeks for 1.3.3. > > I have related question though: Should USE_LAPACK be the default for > Mac OS-X? Is the vecLib framework is *always* there? At least for any Mac with 10.2 or higher. I've made it the default for Numeric in 24.0b2: if found, it'll be used. It looks for the framework directory (which is *always* guaranteed to be /System/Library/Frameworks/vecLib.framework). While you're looking at Numeric, you may want to look at the new separation of customization code from setup.py :-) I have zero time right now, or otherwise I'd port it to numarray. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From cookedm at physics.mcmaster.ca Fri May 27 10:30:06 2005 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Fri May 27 10:30:06 2005 Subject: [Numpy-discussion] Numeric-24.0b2 installation problem In-Reply-To: <20050527154131.47344.qmail@web51104.mail.yahoo.com> (dlyzxl@yahoo.com's message of "Fri, 27 May 2005 08:41:30 -0700 (PDT)") References: <20050527154131.47344.qmail@web51104.mail.yahoo.com> Message-ID: Justin writes: > Dear friends: > > I got an installation problem as following: > > [rathlefs at ishmael ~/uploads/Numeric-24.0b2]$ ./python setup.py install > running install > running build > running build_py > running build_ext > building '_numpy' extension > cc -DNDEBUG -O -IInclude -IPackages/FFT/Include -IPackages/RNG/Include -I/home/ > rathlefs/languages/activePython/include/python2.4 -c Src/_numpymodule.c -o > build/temp.solaris-2.8-sun4u-2.4/Src/_numpymodule.o > /usr/ucb/cc: language optional software package not installed > error: command 'cc' failed with exit status 1 > ___________________________________________ > > and I am guessing it is related to C compiler, and got suggestion from system > administrator as following: > > The only compiler on the system is the GNU C compiler. Try > using /usr/local/bin/gcc > /usr/ucb/cc is only a wraper the the Sun compiler, which is not > installed > > __________________________________ > > I am only an user for the solaris system (Ultra-sparc III) and want to install > the Numerical Python under my account. I already installed activePython2.4.1 > under my account successfully . Can anyone give me some suggestion about the > installation? > Hmm, it should be using the compiler that was used to compile your Python. Check if you've got a environment variable CC defined (echo $CC at the command line). If you do, remove it. If you don't, set it to gcc, and try again. If you've got trouble compiling Numeric, you'll probably have trouble compiling all other extensions. Have a look in the Makefile that was installed with Python (from the paths in your output above, I figure it's at /home/rathlefs/languages/activePython/lib/python2.4/config/Makefile) and check to see what the CC variable is defined as. If it's not gcc, then I would assume that your Python was built with cc. Is Active Python for Solaris a binary distribution (as in, it doesn't compile it from source)? If that's so, it was probably done with the Sun compiler, and would expect to use that for compiling extensions. If that's the case, I'd look into compiling Python from source (it's really not that hard), which will use your GCC compiler. I have *no idea* on the compatibility of the Sun and GCC compilers (but I can tell you that if you've got C++ code, they aren't). I know I've run into fun on Alpha machines, trying to mix the Compaq compiler and GCC. -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From roger.flores at pacbell.net Fri May 27 13:54:23 2005 From: roger.flores at pacbell.net (Roger Flores) Date: Fri May 27 13:54:23 2005 Subject: [Numpy-discussion] How to code functions like a Haar wavelet? Message-ID: <42978890.3010308@pacbell.net> I've read through the docs but all the filter examples seem to affect one value at a time. How do I write something like a wavelet that transforms multiple values at the the same time? I'm hoping for a Python version (as opposed to a C function) for portability and easy exploration. -Roger From afeiguin at uci.edu Fri May 27 18:39:01 2005 From: afeiguin at uci.edu (Adrian E. Feiguin) Date: Fri May 27 18:39:01 2005 Subject: [Numpy-discussion] crashes after switching to 1.3.x In-Reply-To: <1117191173.4718.15.camel@jaytmiller.comcast.net> References: <42967147.1000409@uci.edu> <1117191173.4718.15.camel@jaytmiller.comcast.net> Message-ID: <4297CB5F.9000906@uci.edu> Hi Todd, Thank you for your reply. I have a completely new installation, there are no conflicts with previous installations. I found that the problem is in import_libnumeric in libnumeric.h. It seems that the libnumeric_API is not initialized because it doesn't pass the line: PyObject *module = PyImport_ImportModule("numarray.libnumeric"); \ if (module != NULL) { \ Any ideas? Thank you again, Todd Miller wrote: >Hi Adrian, > >On Thu, 2005-05-26 at 18:00 -0700, Adrian E. Feiguin wrote: > > >>Hi! I'm the lead developer of scigraphica (SG) >>http://scigraphica.sourceforge.net and I'm using python and numarray to >>parse math. SG is built on top of libscigraphica, and libscigraphica has >>all the python code. I've been using an old version 0.6.1, and >>everything worked fine. After switching to 1.6.2the program crashes. >>I'll give you some info to see if you can tell me what's wrong: >> >>I'm using: >>#define PY_ARRAY_UNIQUE_SYMBOL PyArrayXXX >>#include >> >>and in the code: >> import_libnumeric(); >> import_array(); >> >> object=PyRun_String ("from numarray import *", Py_single_input, >>main_dict, sg_dict); >> >>(I noticed that you have to add import_libnumeric now) >> >> > >In general, this is not true. > >Paradoxically, if you're using the numeric compatible API, >import_array() is all you need to do. If you're using both the numarray >native and numeric compatible APIs, then you need to >import_libnumarray() and import_libnumeric(). > > > >>The gdb output is: >> >>Importing python module: sys >>Importing python module: pickle >>Importing python module: os >>Importing python module: numarrayProgram received signal SIGSEGV, >>Segmentation fault. >>[Switching to Thread 1083181376 (LWP 11648)] >>0x400ab0b5 in PyObject_GetAttrString () >> from >>/home/afeiguin/cvs/test/libscigraphica-2/scigraphica/.libs/libscigraphica-2.0-2.0.so.0 >>#0 0x400ab0b5 in PyObject_GetAttrString () >> from >>/home/afeiguin/cvs/test/libscigraphica-2/scigraphica/.libs/libscigraphica-2.0-2.0.so.0 >>#1 0x40b1ff66 in deferred_libnumarray_init () at >>Src/libnumarraymodule.c:152 >>#2 0x40b21f8c in NA_NumArrayCheck (op=0x0) at Src/libnumarraymodule.c:1357 >>#3 0x40b43c1c in PyArray_Check (op=0x0) at Src/libnumericmodule.c:216 >>#4 0x4004d903 in python_insert_object (worksheet=0x81ba4e0, row=0, col=0, >> object=0x80bda3c, orient=GTK_ORIENTATION_VERTICAL, link=0, as_is=0) >> at sg_python_worksheet.c:802 >>#5 0x4004da12 in python_sheet (worksheet=0x81ba4e0, row=0, col=0, >> command=0x8366d68 ".1", orient=GTK_ORIENTATION_VERTICAL) >> at sg_python_worksheet.c:843 >>#6 0x40061f7c in sg_worksheet_cell_set (worksheet=0x81ba4e0, row=0, col=0, >> text=0x8365520 ".1", formula=1, eval=1) at sg_worksheet.c:508 >>#7 0x40061d59 in set_cell (sheet=0x81ba4e0, row=0, col=0, data=0x81ba4e0) >> at sg_worksheet.c:439 >> >>Look that in #4 object=0x80bda3c, and in #3 op=0x0. >>Any hints? >> >> > >numarray-0.6.1 is very old so in order to transition to numarray-1.3.2 >you should: > >a. completely delete your numarray installation > >b. reinstall numarray > >c. rebuild and reinstall any extensions based on numarray. > >My guess is that either you have overlapping/conflicting numarray >installations or you have extensions from numarray-0.6.1 trying to run >against numarray-1.3.2. > >Regards, >Todd > > >. > > > From barrett at stsci.edu Sat May 28 09:48:02 2005 From: barrett at stsci.edu (Paul Barrett) Date: Sat May 28 09:48:02 2005 Subject: [Numpy-discussion] How to code functions like a Haar wavelet? In-Reply-To: <42978890.3010308@pacbell.net> References: <42978890.3010308@pacbell.net> Message-ID: <4298A012.9010407@stsci.edu> Roger Flores wrote: > I've read through the docs but all the filter examples seem to affect > one value at a time. How do I write something like a wavelet that > transforms multiple values at the the same time? > > I'm hoping for a Python version (as opposed to a C function) for > portability and easy exploration. These algorithms can be done in Python as you might do them in C except that the looping is implied. Assuming 'y' is an array of length N. The Haar transform can be written as: s = (y[1::2] + y[0::2])/2. d = (y[1::2] - y[0::2]) where s is the average (smoothed) values and d is the difference (detailed) values. In the lifting scheme (i.e. using filter banks), it is: y[1::2] = y[1::2] - y[0::2] y[0::2] = y[0::2] + y[1::2]/2. Note that y[1::2] is the difference and y[0::2] is the average. And that the order is important, since the operation is being done in-place. For higher order wavelets, boundary effects must be taken into account. For example using a cubic B-spline wavelet, the algorithm is: s = (y[0:-4] + 4*y[1:-3] + 6*y[2:-2] + 4*y[3:-1] + y[4:])/16. d = y[2:-2] - s Note that this expression does not calculate the first and last two values. These must be done as special boundary cases, e.g. s[0] = (6*y[2:-2] + 4*y[3:-1] + y[4:])/11. s[1] = (4*y[1:-3] + 6*y[2:-2] + 4*y[3:-1] + y[4:])/15. s[-2] = (y[0:-4] + 4*y[1:-3] + 6*y[2:-2] + 4*y[3:-1])/15. s[-1] = (y[0:-4] + 4*y[1:-3] + 6*y[2:-2])/11. d[0] = y[0] - s[:3] d[1] = y[1] - s[:4] d[-2] = y[-2] - s[-4:] d[-1] = y[-1] - s[-3:] or by padding the array. I hope this helps. -- Paul -- Paul Barrett, PhD Space Telescope Science Institute Phone: 410-338-4475 ESS/Science Software Branch FAX: 410-338-4767 Baltimore, MD 21218 From jmiller at stsci.edu Sun May 29 04:59:18 2005 From: jmiller at stsci.edu (Todd Miller) Date: Sun May 29 04:59:18 2005 Subject: [Numpy-discussion] crashes after switching to 1.3.x In-Reply-To: <4297CB5F.9000906@uci.edu> References: <42967147.1000409@uci.edu> <1117191173.4718.15.camel@jaytmiller.comcast.net> <4297CB5F.9000906@uci.edu> Message-ID: <1117367726.9474.4.camel@localhost.localdomain> Hi Adrian, I don't think you should be doing both "import_array()" and "import_libnumeric()". Those should be roughly equivalent. If you're using only the Numeric compatible API (no NA_ functions) then do: #include "numarray/arrayobject.h" ... and later in your init function: import_array(); just like Numeric. If you also need some of the NA_ functions, then look at Src/_dotblas.c or Src/_numarraymodule.c for how to interface with both native and numeric compatible interfaces at the same time. If you only need NA_ functions, look at Src/_ndarraymodule.c. Regards, Todd On Fri, 2005-05-27 at 18:37 -0700, Adrian E. Feiguin wrote: > Hi Todd, > > Thank you for your reply. I have a completely new installation, there > are no conflicts with previous installations. > I found that the problem is in import_libnumeric in libnumeric.h. It > seems that the libnumeric_API is not initialized because it doesn't pass > the line: > > PyObject *module = > PyImport_ImportModule("numarray.libnumeric"); \ > if (module != NULL) > { \ > > Any ideas? > Thank you again, > > > > Todd Miller wrote: > > >Hi Adrian, > > > >On Thu, 2005-05-26 at 18:00 -0700, Adrian E. Feiguin wrote: > > > > > >>Hi! I'm the lead developer of scigraphica (SG) > >>http://scigraphica.sourceforge.net and I'm using python and numarray to > >>parse math. SG is built on top of libscigraphica, and libscigraphica has > >>all the python code. I've been using an old version 0.6.1, and > >>everything worked fine. After switching to 1.6.2the program crashes. > >>I'll give you some info to see if you can tell me what's wrong: > >> > >>I'm using: > >>#define PY_ARRAY_UNIQUE_SYMBOL PyArrayXXX > >>#include > >> > >>and in the code: > >> import_libnumeric(); > >> import_array(); > >> > >> object=PyRun_String ("from numarray import *", Py_single_input, > >>main_dict, sg_dict); > >> > >>(I noticed that you have to add import_libnumeric now) > >> > >> > > > >In general, this is not true. > > > >Paradoxically, if you're using the numeric compatible API, > >import_array() is all you need to do. If you're using both the numarray > >native and numeric compatible APIs, then you need to > >import_libnumarray() and import_libnumeric(). > > > > > > > >>The gdb output is: > >> > >>Importing python module: sys > >>Importing python module: pickle > >>Importing python module: os > >>Importing python module: numarrayProgram received signal SIGSEGV, > >>Segmentation fault. > >>[Switching to Thread 1083181376 (LWP 11648)] > >>0x400ab0b5 in PyObject_GetAttrString () > >> from > >>/home/afeiguin/cvs/test/libscigraphica-2/scigraphica/.libs/libscigraphica-2.0-2.0.so.0 > >>#0 0x400ab0b5 in PyObject_GetAttrString () > >> from > >>/home/afeiguin/cvs/test/libscigraphica-2/scigraphica/.libs/libscigraphica-2.0-2.0.so.0 > >>#1 0x40b1ff66 in deferred_libnumarray_init () at > >>Src/libnumarraymodule.c:152 > >>#2 0x40b21f8c in NA_NumArrayCheck (op=0x0) at Src/libnumarraymodule.c:1357 > >>#3 0x40b43c1c in PyArray_Check (op=0x0) at Src/libnumericmodule.c:216 > >>#4 0x4004d903 in python_insert_object (worksheet=0x81ba4e0, row=0, col=0, > >> object=0x80bda3c, orient=GTK_ORIENTATION_VERTICAL, link=0, as_is=0) > >> at sg_python_worksheet.c:802 > >>#5 0x4004da12 in python_sheet (worksheet=0x81ba4e0, row=0, col=0, > >> command=0x8366d68 ".1", orient=GTK_ORIENTATION_VERTICAL) > >> at sg_python_worksheet.c:843 > >>#6 0x40061f7c in sg_worksheet_cell_set (worksheet=0x81ba4e0, row=0, col=0, > >> text=0x8365520 ".1", formula=1, eval=1) at sg_worksheet.c:508 > >>#7 0x40061d59 in set_cell (sheet=0x81ba4e0, row=0, col=0, data=0x81ba4e0) > >> at sg_worksheet.c:439 > >> > >>Look that in #4 object=0x80bda3c, and in #3 op=0x0. > >>Any hints? > >> > >> > > > >numarray-0.6.1 is very old so in order to transition to numarray-1.3.2 > >you should: > > > >a. completely delete your numarray installation > > > >b. reinstall numarray > > > >c. rebuild and reinstall any extensions based on numarray. > > > >My guess is that either you have overlapping/conflicting numarray > >installations or you have extensions from numarray-0.6.1 trying to run > >against numarray-1.3.2. > > > >Regards, > >Todd > > > > > >. > > > > > > > > > > ------------------------------------------------------- > This SF.Net email is sponsored by Yahoo. > Introducing Yahoo! Search Developer Network - Create apps using Yahoo! > Search APIs Find out how you can build Yahoo! directly into your own > Applications - visit http://developer.yahoo.net/?fr=offad-ysdn-ostg-q22005 > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From vasisht at eden.rutgers.edu Sun May 29 20:24:22 2005 From: vasisht at eden.rutgers.edu (Vasisht R. Tadigotla) Date: Sun May 29 20:24:22 2005 Subject: [Numpy-discussion] MemoryError: can't allocate memory for array Message-ID: Hi, I'm using Numerical Python 24 beta 2 with python 2.4.1 on Fedora Core 2 and get a memory error when I try to allocate a large array of float, though the system isn't swapping and there's memory available when I do a free. >from Numeric import * >w=zeros((3000,3,3000,3),Float) Traceback (most recent call last): File "", line 1, in ? MemoryError: can't allocate memory for array I've verified using malloc that I can allocate upto 1GB of memory. I get the same error using numarray-1.3.2 instead of numeric. Is there any way of finding out why this error occurs ? Thanks, Vasisht --------------------------------------------------------------------- Pour trouver les limites du possible il faut tenter l'impossible.