From Gerald.Rosenfellner at vai.at Thu Dec 2 02:34:01 2004 From: Gerald.Rosenfellner at vai.at (Rosenfellner Gerald VAI.ISC 1) Date: Thu Dec 2 02:34:01 2004 Subject: [Numpy-discussion] Problem with trace function? Message-ID: <2489378D03C2CB45A7CEB4F53F40F93F028BD45C@lnzvt-mxs0001.vaig.vads.cc> Hi all, I have some trouble with the new numarray library. Most of the time the numarray code prints out some negative value whereas the Numeric code yields positive values. This behaviour justs shows up when using large arrays. Can anybody point out what I need to change in the numarray version to get the Numeric behaviour? System: WinXP, python2.3.4, Numeric23.5, numarray1.1.1 ----- NUMARRAY-VERSION: from numarray import greater, reshape, trace from numarray.random_array import standard_normal n=1000 A = greater(standard_normal(n*n), 0.9) A = reshape(A, (n, n)) print trace(A) ----- NUMERIC-Version: from Numeric import greater, reshape, trace from RandomArray import standard_normal n=1000 A = greater(standard_normal(n*n), 0.9) A = reshape(A, (n, n)) print trace(A) ----- Thank you for your help. Gerald -------------- next part -------------- An HTML attachment was scrubbed... URL: From ryorke at telkomsa.net Thu Dec 2 09:57:05 2004 From: ryorke at telkomsa.net (Rory Yorke) Date: Thu Dec 2 09:57:05 2004 Subject: [Numpy-discussion] Re: Problem with trace function? In-Reply-To: <2489378D03C2CB45A7CEB4F53F40F93F028BD45C@lnzvt-mxs0001.vaig.vads.cc> References: <2489378D03C2CB45A7CEB4F53F40F93F028BD45C@lnzvt-mxs0001.vaig.vads.cc> Message-ID: <20041202173122.GA3899@localhost.localdomain> > Most of the time the numarray code prints out some negative value > whereas the Numeric code yields positive values. numarray.greater() returns a Bool array, while Numeric.greater() returns a 32-bit integer array. Try this code: import numarray as na import Numeric as nu print na.greater([0.1,0.9],0.5).type() print nu.greater([0.1,0.9],0.5).typecode() ## prints an 'ell' for n in range(126,131): A=na.ones((n,n)).astype(na.Bool) B=nu.ones((n,n)).astype('1') ## that's a one print n,na.trace(A),nu.trace(B) The problem is overflow in the addition of 8 bit signed integers; it occurs in both numarray and Numeric, given the right array type. The snippet below should give you some idea of how to find what you want: from numarray import * from numarray.random_array import standard_normal A=standard_normal((1000,1000)) > 0.9 print len(nonzero(diagonal(A))[0]) print trace(A.astype(Int32)) Cheers, Rory From florian.proff.schulze at gmx.net Thu Dec 2 12:51:05 2004 From: florian.proff.schulze at gmx.net (Florian Schulze) Date: Thu Dec 2 12:51:05 2004 Subject: [Numpy-discussion] View on same data with different type Message-ID: Hi! I would like to be able to access the same array (memory location) with arrays of different size and with different typecodes. Let's say I have an array of 8 UInt8 and I want to view it as 2 UInt32. I want to be able to change the content of either array and the change should be visible in both arrays. To speak in C notation, I want an *UInt8 and a *Uint32 to the same memory location. Is that possible with Numeric or numarray, maybe even with slices of the same data? The reason I want this is that I want to prevent copying memory around. It would be even cooler if this would work with mmaped arrays, though then it's enough when it would work with read only mmaps. BTW, why isn't it allowed to create overlapping mmap slices? Regards, Florian Schulze From lkemohawk at yahoo.com Thu Dec 2 16:45:06 2004 From: lkemohawk at yahoo.com (kevin lester) Date: Thu Dec 2 16:45:06 2004 Subject: [Numpy-discussion] suppress_small=1 ? Message-ID: <20041203004421.79709.qmail@web53905.mail.yahoo.com> Hi, How can I control my output presentation. There are times when using: print array_str(a,suppress_small=1), does not print a float as intended, but remains to output in exponential form. I am using a win32 os and mostly using PythonWin shell but also tried using wxPython's shell for same results. I tried several other aspects such as sys.float_output_suppress_small to no avail. Another question I have is: performing an operation on an array produces a new array index subscript. Is it possible to re-assign the original index subscript to an array after performing the operation. (tunnel vision afflicted...lol) Thank you very much, Kevin __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From florian.proff.schulze at gmx.net Fri Dec 3 06:54:02 2004 From: florian.proff.schulze at gmx.net (Florian Schulze) Date: Fri Dec 3 06:54:02 2004 Subject: [Numpy-discussion] Re: View on same data with different type References: Message-ID: On Thu, 02 Dec 2004 21:49:10 +0100, Florian Schulze wrote: > Hi! > > I would like to be able to access the same array (memory location) with > arrays of different size and with different typecodes. Let's say I have > an array of 8 UInt8 and I want to view it as 2 UInt32. I want to be able > to change the content of either array and the change should be visible > in both arrays. To speak in C notation, I want an *UInt8 and a *Uint32 > to the same memory location. > Is that possible with Numeric or numarray, maybe even with slices of the > same data? > The reason I want this is that I want to prevent copying memory around. > It would be even cooler if this would work with mmaped arrays, though > then it's enough when it would work with read only mmaps. BTW, why isn't > it allowed to create overlapping mmap slices? > > Regards, > Florian Schulze Hi again! Todd Miller explained me how to do it (though it wasn't working as is, I figured it out). As I think it is interesting to other people, I post it here: >>> import numarray >>> a = numarray.arrayrange(12, type='i1') # We can now create an array pointing to the same memory location with different type: >>> b = numarray.array(sequence=a._data, shape=(3,), type='i4') # The key here is to use a._data. You have to specify the shape, or else it doesn't work. >>> a array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11], type=Int8) >>> b array([ 50462976, 117835012, 185207048]) >>> a[3] = 10 >>> a array([ 0, 1, 2, 10, 4, 5, 6, 7, 8, 9, 10, 11], type=Int8) >>> b array([167903488, 117835012, 185207048]) # Here you see how the assignment to a also changed b. >>> a._data >>> b._data # Here you see that both arrays really point to the same memory location. I tried it with numarray.records.array and it's basically the same, though you have to use 'buffer' instead of 'sequence'. I will have to try it out with mmap, but Todd told me it should be no problem. He also told me it could also be done with views by calling .view() on the array and then set the other values (_type, _shape, _itemsize, [_strides, _bytestride]) but I think this is more error prone and I didn't test it. Thanks again to Todd Miller. Regards, Florian Schulze From jmiller at stsci.edu Fri Dec 3 07:46:02 2004 From: jmiller at stsci.edu (Todd Miller) Date: Fri Dec 3 07:46:02 2004 Subject: [Fwd: Re: [Numpy-discussion] View on same data with different type] Message-ID: <1102088728.28877.350.camel@halloween.stsci.edu> Once again I forgot to reply-all... Florian apparently figured out whatever flaws were in my original message attached here. One thing people interested in this kind of data aliasing should keep in mind is that all of the hidden variables I mentioned (in the attached) exist to make things like array slices and record arrays work properly. Just reusing the _data, by itself, is not sufficient to work *in general* because not all of the _data is always used. It does work for the relatively simple case of a new array, but might fail if the array came from a field of a recarray, a numerical array slice, a transposed numerical array, etc. Be careful. Regards, Todd -------------- next part -------------- An embedded message was scrubbed... From: unknown sender Subject: no subject Date: no date Size: 38 URL: From jmiller at stsci.edu Thu Dec 2 16:25:43 2004 From: jmiller at stsci.edu (Todd Miller) Date: 02 Dec 2004 16:25:43 -0500 Subject: [Numpy-discussion] View on same data with different type In-Reply-To: References: Message-ID: <1102022743.28877.240.camel@halloween.stsci.edu> On Thu, 2004-12-02 at 15:49, Florian Schulze wrote: > Hi! > > I would like to be able to access the same array (memory location) with > arrays of different size and with different typecodes. Let's say I have an > array of 8 UInt8 and I want to view it as 2 UInt32. I want to be able to > change the content of either array and the change should be visible in > both arrays. To speak in C notation, I want an *UInt8 and a *Uint32 to the > same memory location. > Is that possible with Numeric or numarray, maybe even with slices of the > same data? In numarray, it is definitely possible to do what you want (in Python) by accessing private attributes of the array: _data, _byteoffset, _shape, _strides, _bytestride, _itemsize. Given an array A, one approach would be to create an array B using buffer sharing and the NumArray constructor like this: >>> A = arange(100, type='Int8') >>> B = NumArray(shape=(25,), buffer=A._data, type='Int32') Another approach would be to use a view like this: >>> B = A.view() >>> B._type = Int32 >>> B._shape = (25,) >>> B._itemsize = 4 # >>> B._strides = (4,) # >>> B._bytestride = 4 > The reason I want this is that I want to prevent copying memory around. It > would be even cooler if this would work with mmaped arrays, though then > it's enough when it would work with read only mmaps. BTW, why isn't it > allowed to create overlapping mmap slices? IIRC, since mmap slices are resizable, permitting overlaps would have complicated things... so without a compelling need, I/we decided to KISS. The kind of re-typing I showed above works for mmaps too... there's no need to overlap since the same slice can be used in two different arrays. Todd --=-fvu/cVmG8DY6+oRhVZR/-- From vivekrao4 at yahoo.com Fri Dec 3 19:06:03 2004 From: vivekrao4 at yahoo.com (Vivek Rao) Date: Fri Dec 3 19:06:03 2004 Subject: [Numpy-discussion] Numarray 1.1.1 with Python 2.4 #60 crashes on Windows XP Pro Message-ID: <20041204030532.93719.qmail@web53410.mail.yahoo.com> I have installed numarray 1.1.1 for Python 2.4. When I go to C:\Python24\Lib\site-packages\numarray and type 'python numtest.py' a crash occurs. Debugging it Microsoft Visual Studio, the error message is "unhandled exception in python.exe (Python24.dll): 0xC0000005 Access Violation" The Python version I am using is Python 2.4 (#60, Nov 30 2004, 11:49:19) [MSC v.1310 32 bit (Intel)] on win32 . Is this a known problem? Vivek Rao From faltet at carabos.com Sat Dec 4 03:08:01 2004 From: faltet at carabos.com (Francesc Altet) Date: Sat Dec 4 03:08:01 2004 Subject: [Numpy-discussion] ANN: PyTables 0.9.1 released Message-ID: <200412041207.04631.faltet@carabos.com> Announcing PyTables 0.9.1 ------------------------- This release is mainly a maintenance version. In it, some bugs has been fixed and a few improvements has been made. One important thing is that chunk sizes in EArrays has been re-tuned to get much better performance and compression rations. Besides, it has been tested against the latest Python 2.4 and all test units seems to pass fine. What it is ---------- PyTables is a solid hierarchical database package designed to efficiently manage extremely large amounts of data (with support for full 64-bit file addressing). It features an object-oriented interface that, combined with C extensions for the performance-critical parts of the code, makes it a very easy-to-use tool for high performance data storage and retrieval. It is built on top of the HDF5 library and the numarray package, and provides containers for both heterogeneous data (Tables) and homogeneous data (Array, EArray) as well as containers for keeping lists of objects of variable length (like Unicode strings or general Python objects) in a very efficient way (VLArray). It also sports a series of filters allowing you to compress your data on-the-fly by using different compressors and compression enablers. But perhaps the more interesting features are its powerful browsing and searching capabilities that allow you to perform data selections over heterogeneous datasets exceeding gigabytes of data in just tenths of second. Besides, all the PyTables I/O is buffered, implemented in C and carefully tuned so that you can reach much better performance with PyTables than with your own home-grown wrappings to the HDF5 library. Changes more in depth --------------------- Improvements: - The chunksize computation for EArrays has been re-tuned to allow better performance and *much* better compression rations. - New --unpackshort and --quantize flags has been added to nctoh5 script. --unpackshort unpack short integer variables to float variables using scale_factor and add_offset netCDF variable attributes. --quantize quantize data to improve compression using least_significant_digit netCDF variable attribute (not active by default). See http://www.cdc.noaa.gov/cdc/conventions/cdc_netcdf_standard.shtml for further explanation of what this attribute means. Thanks to Jeff Whitaker for providing this. - Table.itersequence has received a new parameter called "sort". This allows to disable the sorting of the sequence in case the user wants so. Backward-incompatible changes: - Now, the AttributeSet class throw an AttributeError on __getattr__ for nonexistent attributes in it. Formerly, the routine returned None, which is pretty much against convention in Python and breaks the built-in hasattr() function. Thanks to Norbert Nemec for noting this and offering a patch. - VLArray.read() has changed its behaviour. Now, it always returns a list, as stated in documentation, even when the number of elements to return is 0 or 1. This is much more consistent when representing the actual number of elements on a certain VLArray row. API additions: - A Row.getTable() has been added. It is an accessor for the associated Table object. - A File.copyAttrs() has been added. It allows copying attributes from one leaf to other. Properly speaking, this was already there, but not documented :-/ Bug fixes: - Now, the copy of hierarchies works even when there are scalar Arrays (i.e. Arrays which shape is ()) on it. Thanks to Norbert Nemec for providing a patch. - Solved a memory leak regarding the Filters instance associated with the File object, that was not released after closing the file. Now, there are no known leaks on PyTables itself. - Fixed a bug in Table.append() when the table was indexed. The problem was that if table was in auto-indexing mode, some rows were lost in the indexation process and hence, not indexed correctly. - Improved security of nodes name checking. Closes #1074335 Important note for Python 2.4 and Windows users ----------------------------------------------- If you are willing to use PyTables with Python 2.4 in Windows platforms, you will need to get the HDF5 library compiled for MSVC 7.1, aka .NET (and possible LZO and UCL as well, if you want support for LZO and UCL at all). It can be found at: ftp://ftp.ncsa.uiuc.edu/HDF/HDF5/current/bin/windows/5-163-winxp-net2003.zip Where can PyTables be applied? ------------------------------ PyTables is not designed to work as a relational database competitor, but rather as a teammate. If you want to work with large datasets of multidimensional data (for example, for multidimensional analysis), or just provide a categorized structure for some portions of your cluttered RDBS, then give PyTables a try. It works well for storing data from data acquisition systems (DAS), simulation software, network data monitoring systems (for example, traffic measurements of IP packets on routers), very large XML files, or for creating a centralized repository for system logs, to name only a few possible uses. What is a table? ---------------- A table is defined as a collection of records whose values are stored in fixed-length fields. All records have the same structure and all values in each field have the same data type. The terms "fixed-length" and "strict data types" seem to be quite a strange requirement for a language like Python that supports dynamic data types, but they serve a useful function if the goal is to save very large quantities of data (such as is generated by many scientific applications, for example) in an efficient manner that reduces demand on CPU time and I/O resources. What is HDF5? ------------- For those people who know nothing about HDF5, it is a general purpose library and file format for storing scientific data made at NCSA. HDF5 can store two primary objects: datasets and groups. A dataset is essentially a multidimensional array of data elements, and a group is a structure for organizing objects in an HDF5 file. Using these two basic constructs, one can create and store almost any kind of scientific data structure, such as images, arrays of vectors, and structured and unstructured grids. You can also mix and match them in HDF5 files according to your needs. Platforms --------- I'm using Linux (Intel 32-bit) as the main development platform, but PyTables should be easy to compile/install on many other UNIX machines. This package has also passed all the tests on a UltraSparc platform with Solaris 7 and Solaris 8. It also compiles and passes all the tests on a SGI Origin2000 with MIPS R12000 processors, with the MIPSPro compiler and running IRIX 6.5. It also runs fine on Linux 64-bit platforms, like AMD Opteron running GNU/Linux 2.4.21 Server, Intel Itanium (IA64) running GNU/Linux 2.4.21 or PowerPC G5 with Linux 2.6.x in 64bit mode. It has also been tested in MacOSX platforms (10.2 but should also work on newer versions). Regarding Windows platforms, PyTables has been tested with Windows 2000 and Windows XP (using the Microsoft Visual C compiler), but it should also work with other flavors as well. Web site -------- Go to the PyTables web site for more details: http://pytables.sourceforge.net/ To know more about the company behind the PyTables development, see: http://www.carabos.com/ Share your experience --------------------- Let me know of any bugs, suggestions, gripes, kudos, etc. you may have. Enjoy data! -- Francesc Altet Who's your data daddy? ?PyTables From jmiller at stsci.edu Sat Dec 4 05:49:05 2004 From: jmiller at stsci.edu (Todd Miller) Date: Sat Dec 4 05:49:05 2004 Subject: [Numpy-discussion] Numarray 1.1.1 with Python 2.4 #60 crashes on Windows XP Pro In-Reply-To: <20041204030532.93719.qmail@web53410.mail.yahoo.com> References: <20041204030532.93719.qmail@web53410.mail.yahoo.com> Message-ID: <1102167843.3372.3.camel@jaytmiller.comcast.net> On Fri, 2004-12-03 at 19:05 -0800, Vivek Rao wrote: > I have installed numarray 1.1.1 for Python 2.4. When I > go to C:\Python24\Lib\site-packages\numarray and type > 'python numtest.py' a crash occurs. Debugging it > Microsoft Visual Studio, the error message is > > "unhandled exception in python.exe (Python24.dll): > 0xC0000005 Access Violation" > > The Python version I am using is > > Python 2.4 (#60, Nov 30 2004, 11:49:19) [MSC v.1310 32 > bit (Intel)] on win32 . > > Is this a known problem? Yes and no. In my experience, it's best not to run modules that are part of a Python package from within the package. To run the selftest, I recommend doing something like this: > cd C:\ > python >>> import numarray.testall as testall >>> testall.test() Regards, Todd From rbastian at club-internet.fr Sat Dec 4 06:54:06 2004 From: rbastian at club-internet.fr (=?iso-8859-15?q?Ren=E9=20Bastian?=) Date: Sat Dec 4 06:54:06 2004 Subject: [Numpy-discussion] segm fault Message-ID: <04120407473000.00776@rbastian> Hi, i use numarray 1.1.1 with python2.3b1 import RandomArray2 as R w=R.uniform(-1., +1., 44100) produces "Segmentation fault" i never meet that. thanks for your help -- Ren? Bastian http://www.musiques-rb.org : Musique en Python From jmiller at stsci.edu Sat Dec 4 07:46:03 2004 From: jmiller at stsci.edu (Todd Miller) Date: Sat Dec 4 07:46:03 2004 Subject: [Numpy-discussion] segm fault In-Reply-To: <04120407473000.00776@rbastian> References: <04120407473000.00776@rbastian> Message-ID: <1102175126.3372.13.camel@jaytmiller.comcast.net> On Sat, 2004-12-04 at 07:47 +0100, Ren? Bastian wrote: > Hi, > > i use numarray 1.1.1 with python2.3b1 > > import RandomArray2 as R > w=R.uniform(-1., +1., 44100) > > produces "Segmentation fault" > > i never meet that. > thanks for your help > This looks like either you've got remanents of a really old numarray installed or you're trying to run from within the numarray package directory. In the numarray-speak of today, you'd write the import as: import numarray.random_array as R If you're running from the numarray package directory, don't. If you think maybe you've got old numarray stuff installed, figure out where RandomArray2 is coming from and delete it; use numarray.random_array. Regards, Todd From jmiller at stsci.edu Sat Dec 4 07:47:00 2004 From: jmiller at stsci.edu (Todd Miller) Date: Sat Dec 4 07:47:00 2004 Subject: [Numpy-discussion] segm fault In-Reply-To: <04120407473000.00776@rbastian> References: <04120407473000.00776@rbastian> Message-ID: <1102175192.3372.15.camel@jaytmiller.comcast.net> On Sat, 2004-12-04 at 07:47 +0100, Ren? Bastian wrote: > Hi, > > i use numarray 1.1.1 with python2.3b1 And... upgrade to Python-2.3.4 or Python-2.4. From stephen.walton at csun.edu Mon Dec 6 16:07:18 2004 From: stephen.walton at csun.edu (Stephen Walton) Date: Mon Dec 6 16:07:18 2004 Subject: [Numpy-discussion] numarray vs IPython Message-ID: <1102377944.8589.27.camel@freyer.sfo.csun.edu> When I run numarray.testall.test() from IPython, I get a slew of error messages instead of the successful test output I get when the same version is run from the stock Python interpreter. Anyone else see this? From ariciputi at pito.com Tue Dec 7 00:55:08 2004 From: ariciputi at pito.com (Andrea Riciputi) Date: Tue Dec 7 00:55:08 2004 Subject: [Numpy-discussion] numarray vs IPython In-Reply-To: <1102377944.8589.27.camel@freyer.sfo.csun.edu> References: <1102377944.8589.27.camel@freyer.sfo.csun.edu> Message-ID: <9E1F0936-482D-11D9-8F12-000A95C0BC68@pito.com> The same here (OS X, Python 2.3.4, ipython 0.6.5). The problem is due to the way in which numarray performs its tests. It relies on the Python module doctest that execute the script in doc strings. I think that since in numarray doc strings the expected output is written in the standard Python way (i.e. >>> foo) while ipython writes out something like "Out[n]: foo" the checks fail. HTH, Andrea. On 7 Dec 2004, at 01:05, Stephen Walton wrote: > When I run numarray.testall.test() from IPython, I get a slew of error > messages instead of the successful test output I get when the same > version is run from the stock Python interpreter. Anyone else see > this? From verveer at embl-heidelberg.de Tue Dec 7 01:13:20 2004 From: verveer at embl-heidelberg.de (Peter Verveer) Date: Tue Dec 7 01:13:20 2004 Subject: [Numpy-discussion] numarray vs IPython In-Reply-To: <9E1F0936-482D-11D9-8F12-000A95C0BC68@pito.com> References: <1102377944.8589.27.camel@freyer.sfo.csun.edu> <9E1F0936-482D-11D9-8F12-000A95C0BC68@pito.com> Message-ID: <2628F06A-4830-11D9-B2CB-000A95C92C8E@embl-heidelberg.de> This could be true, I noticed that the tests for the nd_image package work fine, and those are written using unittest, which does not rely on the exact textual form of the expected output. On 7 Dec 2004, at 09:54, Andrea Riciputi wrote: > The same here (OS X, Python 2.3.4, ipython 0.6.5). The problem is due > to the way in which numarray performs its tests. It relies on the > Python module doctest that execute the script in doc strings. I think > that since in numarray doc strings the expected output is written in > the standard Python way (i.e. >>> foo) while ipython writes out > something like "Out[n]: foo" the checks fail. > > HTH, > Andrea. > > > On 7 Dec 2004, at 01:05, Stephen Walton wrote: > >> When I run numarray.testall.test() from IPython, I get a slew of error >> messages instead of the successful test output I get when the same >> version is run from the stock Python interpreter. Anyone else see >> this? > > > > ------------------------------------------------------- > SF email is sponsored by - The IT Product Guide > Read honest & candid reviews on hundreds of IT Products from real > users. > Discover which products truly live up to the hype. Start reading now. > http://productguide.itmanagersjournal.com/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From Fernando.Perez at colorado.edu Tue Dec 7 11:06:02 2004 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Tue Dec 7 11:06:02 2004 Subject: [Numpy-discussion] numarray vs IPython In-Reply-To: <9E1F0936-482D-11D9-8F12-000A95C0BC68@pito.com> References: <1102377944.8589.27.camel@freyer.sfo.csun.edu> <9E1F0936-482D-11D9-8F12-000A95C0BC68@pito.com> Message-ID: <41B5FEE9.7050704@colorado.edu> Andrea Riciputi schrieb: > The same here (OS X, Python 2.3.4, ipython 0.6.5). The problem is due > to the way in which numarray performs its tests. It relies on the > Python module doctest that execute the script in doc strings. I think > that since in numarray doc strings the expected output is written in > the standard Python way (i.e. >>> foo) while ipython writes out > something like "Out[n]: foo" the checks fail. Thanks for that info, this is most likely the culprit. Here's a simple test to confirm things: start ipython as 'ipython --classic', which disables the fancy prompt system and makes it produce regular python prompts. Run the tests there and see what happens. Since numarray is more and more likely to be run by ipython users, it might be worth adding a bit of robustness to these tests when under ipython. Here are some suggestions, starting with the simplest: 1. Run a first 'empty' test just to do prompt detection. If this one fails, do: try: __IPYTHON__ except NameError: print 'Big time problem, not caused by ipython' else: print 'Run ipython --classic for doctests to work' or some fancier version of the above. 2. Even better, patch doctest to recognize prompts based on a regexp. The python-mode.el guys did this for (x)emacs support, and it now works very well with ipython. The python-mode CVS code can be studied for the proper regexps. By default this regexp can be set to recognize the normal ipython prompts. 3. If you do 2, and you want to get real fancy, the numarray doctest could build the regexp automatically, from the ipython prompt configuration strings. These are always available at runtime: In [1]: __IPYTHON__.rc.prom __IPYTHON__.rc.prompt_in1 __IPYTHON__.rc.prompt_in2 __IPYTHON__.rc.prompt_out In [1]: __IPYTHON__.rc.prompt_in1 Out[1]: 'In [\\#]:' In [2]: __IPYTHON__.rc.prompt_in2 Out[2]: ' .\\D.:' In [3]: __IPYTHON__.rc.prompt_out Out[3]: 'Out[\\#]:' So the situation looks not too bad. There's a quick fix (--classic) for now, and if someone feels up to the task of patching doctest, a pretty elegant solution can be implemented. Regards, f From jmiller at stsci.edu Tue Dec 7 12:41:02 2004 From: jmiller at stsci.edu (Todd Miller) Date: Tue Dec 7 12:41:02 2004 Subject: [Numpy-discussion] numarray vs IPython In-Reply-To: <41B5FEE9.7050704@colorado.edu> References: <1102377944.8589.27.camel@freyer.sfo.csun.edu> <9E1F0936-482D-11D9-8F12-000A95C0BC68@pito.com> <41B5FEE9.7050704@colorado.edu> Message-ID: <1102452001.17245.276.camel@halloween.stsci.edu> I wanted to let you guys know that there's a "post-it note" stuck to my monitor to remind me to look at this. A quick glance indicated that --classic does not fix the problem... but that's all I know so far and all I'll get to in the immediate future. Regards, Todd On Tue, 2004-12-07 at 14:05, Fernando Perez wrote: > Andrea Riciputi schrieb: > > The same here (OS X, Python 2.3.4, ipython 0.6.5). The problem is due > > to the way in which numarray performs its tests. It relies on the > > Python module doctest that execute the script in doc strings. I think > > that since in numarray doc strings the expected output is written in > > the standard Python way (i.e. >>> foo) while ipython writes out > > something like "Out[n]: foo" the checks fail. > > Thanks for that info, this is most likely the culprit. Here's a simple test > to confirm things: start ipython as 'ipython --classic', which disables the > fancy prompt system and makes it produce regular python prompts. Run the > tests there and see what happens. > > Since numarray is more and more likely to be run by ipython users, it might be > worth adding a bit of robustness to these tests when under ipython. Here are > some suggestions, starting with the simplest: > > 1. Run a first 'empty' test just to do prompt detection. If this one fails, do: > > try: > __IPYTHON__ > except NameError: > print 'Big time problem, not caused by ipython' > else: > print 'Run ipython --classic for doctests to work' > > or some fancier version of the above. > > 2. Even better, patch doctest to recognize prompts based on a regexp. The > python-mode.el guys did this for (x)emacs support, and it now works very well > with ipython. The python-mode CVS code can be studied for the proper regexps. > By default this regexp can be set to recognize the normal ipython prompts. > > 3. If you do 2, and you want to get real fancy, the numarray doctest could > build the regexp automatically, from the ipython prompt configuration strings. > These are always available at runtime: > > In [1]: __IPYTHON__.rc.prom > __IPYTHON__.rc.prompt_in1 __IPYTHON__.rc.prompt_in2 __IPYTHON__.rc.prompt_out > > In [1]: __IPYTHON__.rc.prompt_in1 > Out[1]: 'In [\\#]:' > > In [2]: __IPYTHON__.rc.prompt_in2 > Out[2]: ' .\\D.:' > > In [3]: __IPYTHON__.rc.prompt_out > Out[3]: 'Out[\\#]:' > > > So the situation looks not too bad. There's a quick fix (--classic) for now, > and if someone feels up to the task of patching doctest, a pretty elegant > solution can be implemented. > > Regards, > > f > > > ------------------------------------------------------- > SF email is sponsored by - The IT Product Guide > Read honest & candid reviews on hundreds of IT Products from real users. > Discover which products truly live up to the hype. Start reading now. > http://productguide.itmanagersjournal.com/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- From Fernando.Perez at colorado.edu Tue Dec 7 12:45:04 2004 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Tue Dec 7 12:45:04 2004 Subject: [Numpy-discussion] numarray vs IPython In-Reply-To: <1102452001.17245.276.camel@halloween.stsci.edu> References: <1102377944.8589.27.camel@freyer.sfo.csun.edu> <9E1F0936-482D-11D9-8F12-000A95C0BC68@pito.com> <41B5FEE9.7050704@colorado.edu> <1102452001.17245.276.camel@halloween.stsci.edu> Message-ID: <41B6162A.1000403@colorado.edu> Todd Miller schrieb: > I wanted to let you guys know that there's a "post-it note" stuck to my > monitor to remind me to look at this. A quick glance indicated that > --classic does not fix the problem... but that's all I know so far and > all I'll get to in the immediate future. Bummer. I would have thought that --classic should have been enough. Sorry not to be able to test myself, but I don't have numarray installed at all. Thanks for looking into it, and please let me know if I can help in any way. Regards f From klimek at grc.nasa.gov Tue Dec 7 15:19:17 2004 From: klimek at grc.nasa.gov (Bob Klimek) Date: Tue Dec 7 15:19:17 2004 Subject: [Numpy-discussion] binary thinning Message-ID: <41B63AA8.5060205@grc.nasa.gov> Hi, I've been playing around with the binary_hit_or_miss function in the nd_image and it seems to work well in general. I understand that binary thinning can be generated by using hit_or_miss, however I can't seem to get it to work right. Anyone have a code snippet that does the job? I follow the procedure at: http://homepages.inf.ed.ac.uk/rbf/HIPR2/thin.htm My implementation is as follows: ## from HIPR2 web site ## [0,0,0] ## [x,1,x] ## [1,1,1] struct1 = numarray.array( [[0, 0, 0], [0, 1, 0], [1, 1, 1]]) struct2 = numarray.array( [[1, 1, 1], [0, 0, 0], [0, 0, 0]]) ## [x,0,0] ## [1,1,0] ## [x,1,x] struct3 = numarray.array( [[0, 0, 0], [1, 1, 0], [0, 1, 0]]) struct4 = numarray.array( [[0, 1, 1], [0, 0, 1], [0, 0, 0]]) b = [] for i in range(4): temp = ND.binary_hit_or_miss(a, struct1, struct2) b.append(temp) temp = ND.binary_hit_or_miss(a, struct3, struct4) b.append(temp) struct1 = ND.rotate(struct1, 90) struct2 = ND.rotate(struct2, 90) struct3 = ND.rotate(struct3, 90) struct4 = ND.rotate(struct4, 90) result = b[0] for i in range(7): result = numarray.logical_or(result, b[i+1]) result = a - result Bob From verveer at embl-heidelberg.de Wed Dec 8 03:12:04 2004 From: verveer at embl-heidelberg.de (Peter Verveer) Date: Wed Dec 8 03:12:04 2004 Subject: [Numpy-discussion] binary thinning In-Reply-To: <41B63AA8.5060205@grc.nasa.gov> References: <41B63AA8.5060205@grc.nasa.gov> Message-ID: Hi Bob, I noticed two problems: 1) you construct rotations of your structures with the nd_image.rotate function. That will not work properly because this function uses spline interpolation, which not necessarily produces exact results. In this case you need to use another, exact rotation method, see my implementation below. 2) I think (correct me if I am wrong) that the application of the 8 structures is sequential, not parallel as you do it. Below I give my implementation, which does not produce the exact same result as shown on the webpage you refer to, although the result seems to be a proper skeleton. This may be because the order in which the structures are applied matters (I think). Could you maybe test that by permutating the order in which the structures are applied? I would be interested if this is true and if one can get the same result as on the webpage by finding the right sequence... Cheers, Peter import numarray from numarray import nd_image image = numarray.array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0], [0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0], [0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0], [0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0], [0, 0, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0], [0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0], [0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0], [0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0], [0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0], [0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0], [0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]) hit1 = numarray.array([[0, 0, 0], [0, 1, 0], [1, 1, 1]]) miss1 = numarray.array([[1, 1, 1], [0, 0, 0], [0, 0, 0]]) hit2 = numarray.array([[0, 0, 0], [1, 1, 0], [0, 1, 0]]) miss2 = numarray.array([[0, 1, 1], [0, 0, 1], [0, 0, 0]]) hit_list = [hit1, hit2] miss_list = [miss1, miss2] for ii in range(6): hit_list.append(numarray.transpose(hit_list[-2])[::-1, ...]) miss_list.append(numarray.transpose(miss_list[-2])[::-1, ...]) while 1: last = image for hit, miss in zip(hit_list, miss_list): hm = nd_image.binary_hit_or_miss(image, hit, miss) image = numarray.logical_and(image, numarray.logical_not(hm)) if numarray.abs(last - image).max() == 0: break print image On 8 Dec 2004, at 00:20, Bob Klimek wrote: > Hi, > > I've been playing around with the binary_hit_or_miss function in the > nd_image and it seems to work well in general. I understand that > binary thinning can be generated by using hit_or_miss, however I can't > seem to get it to work right. Anyone have a code snippet that does the > job? > > I follow the procedure at: > > http://homepages.inf.ed.ac.uk/rbf/HIPR2/thin.htm > > My implementation is as follows: > > ## from HIPR2 web site > ## [0,0,0] > ## [x,1,x] > ## [1,1,1] > struct1 = numarray.array( > [[0, 0, 0], > [0, 1, 0], > [1, 1, 1]]) > struct2 = numarray.array( > [[1, 1, 1], > [0, 0, 0], > [0, 0, 0]]) > > ## [x,0,0] > ## [1,1,0] > ## [x,1,x] > struct3 = numarray.array( > [[0, 0, 0], > [1, 1, 0], > [0, 1, 0]]) > struct4 = numarray.array( > [[0, 1, 1], > [0, 0, 1], > [0, 0, 0]]) > > b = [] > for i in range(4): > temp = ND.binary_hit_or_miss(a, struct1, struct2) > b.append(temp) > temp = ND.binary_hit_or_miss(a, struct3, struct4) > b.append(temp) > > struct1 = ND.rotate(struct1, 90) > struct2 = ND.rotate(struct2, 90) > struct3 = ND.rotate(struct3, 90) > struct4 = ND.rotate(struct4, 90) > > result = b[0] > for i in range(7): > result = numarray.logical_or(result, b[i+1]) > > result = a - result > > > Bob > > > > ------------------------------------------------------- > SF email is sponsored by - The IT Product Guide > Read honest & candid reviews on hundreds of IT Products from real > users. > Discover which products truly live up to the hype. Start reading now. > http://productguide.itmanagersjournal.com/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From klimek at grc.nasa.gov Wed Dec 8 15:23:06 2004 From: klimek at grc.nasa.gov (Bob Klimek) Date: Wed Dec 8 15:23:06 2004 Subject: [Numpy-discussion] binary thinning In-Reply-To: References: <41B63AA8.5060205@grc.nasa.gov> Message-ID: <41B78D26.1060007@grc.nasa.gov> Peter Verveer wrote: > Hi Bob, > > I noticed two problems: > > 1) you construct rotations of your structures with the nd_image.rotate > function. That will not work properly because this function uses > spline interpolation, which not necessarily produces exact results. In > this case you need to use another, exact rotation method, see my > implementation below. > > 2) I think (correct me if I am wrong) that the application of the 8 > structures is sequential, not parallel as you do it. > > Below I give my implementation, which does not produce the exact same > result as shown on the webpage you refer to, although the result seems > to be a proper skeleton. This may be because the order in which the > structures are applied matters (I think). Could you maybe test that by > permutating the order in which the structures are applied? I would be > interested if this is true and if one can get the same result as on > the webpage by finding the right sequence... Peter, Thanks much for getting back to me on this. Your sample code was very helpful! :-) I rewrote my function based on yours and I'm getting the same answer. So far I haven't been able to figure out why is it different from the web site and so far I haven't tried changing the order of the structuring elements. I'll try tomorrow. I'll let you know if I come up with anything new. Regards, Bob From cjw at sympatico.ca Thu Dec 9 14:39:03 2004 From: cjw at sympatico.ca (Colin J. Williams) Date: Thu Dec 9 14:39:03 2004 Subject: [Numpy-discussion] NumArray - take method Message-ID: <41B8D3E3.9060302@sympatico.ca> This undocumented method has puzzling behaviour. [Dbg]>>> a array([[0, 1, 2], [3, 4, 5], [6, 7, 8]]) [Dbg]>>> b= a[1:1] [Dbg]>>> b array([]) [Dbg]>>> b.shape (0, 3) [Dbg]>>> b.rank 2 I had expected a rank of zero and an indeterminate shape. Colin W. From tkorvola at e.math.helsinki.fi Sat Dec 11 05:26:12 2004 From: tkorvola at e.math.helsinki.fi (Timo Korvola) Date: Sat Dec 11 05:26:12 2004 Subject: [Numpy-discussion] NumArray - take method In-Reply-To: <41B8D3E3.9060302@sympatico.ca> (Colin J. Williams's message of "Thu, 09 Dec 2004 17:38:27 -0500") References: <41B8D3E3.9060302@sympatico.ca> Message-ID: "Colin J. Williams" writes: > I had expected a rank of zero and an indeterminate shape. Rank zero would imply shape () and that the result is a scalar. Empty arrays have positive rank but one or more of the dimensions are zero. It seems that there is a bug in repr as it does not distinguish between empty arrays of different shapes. The current behaviour is consistent in the sense that indexing by a slice does not change rank, not even if the slice has length zero or one (a slice of length one is thus different from an integer). Only the indexed dimension changes. -- Timo Korvola From cjw at sympatico.ca Sat Dec 11 08:20:00 2004 From: cjw at sympatico.ca (Colin J. Williams) Date: Sat Dec 11 08:20:00 2004 Subject: [Numpy-discussion] NumArray - take method In-Reply-To: References: <41B8D3E3.9060302@sympatico.ca> Message-ID: <41BB1E2B.8030705@sympatico.ca> Timo Korvola wrote: >"Colin J. Williams" writes: > > >>I had expected a rank of zero and an indeterminate shape. >> >> > >Rank zero would imply shape () and that the result is a scalar. Empty >arrays have positive rank but one or more of the dimensions are zero. >It seems that there is a bug in repr as it does not distinguish >between empty arrays of different shapes. > > Perhaps the problem is in str: >>> b= a[:,1:1] >>> b.shape (3, 0) >>> str(b) '[]' >>> repr(b) 'array([])' >The current behaviour is consistent in the sense that indexing by a >slice does not change rank, not even if the slice has length zero or >one (a slice of length one is thus different from an integer). Only >the indexed dimension changes. > > > Yes, you are right. The new docs are a great step forward but it would be good to include the take method. Colin W. From ivilata at carabos.com Mon Dec 13 03:40:04 2004 From: ivilata at carabos.com (Ivan Vilata i Balaguer) Date: Mon Dec 13 03:40:04 2004 Subject: [Numpy-discussion] Using new numeric instances in NumArray and RecArray. Message-ID: <20041213113929.GR4752@orm-embar.terramar.selidor.net> Hi all! I am trying to define new numeric types to use them in NumArray and RecArray objects. My intention is to define them as equivalent to other types, but also make possible to distinguish them. I tried "MyInt32 = IntegralType(4, 'MyType')", which registered OK, is equivalent to Int32, but distinguishable from it. However, I am not allowe to create a NumArray based on it: "a = array([1,2], MyInt32)" fails with "RuntimeError: _numarray_init: can't get typeno for type". Is it possible to integrate new types seamlessly into NumArray and RecArray objects? Is this the right approach, or am I getting that wrong? Well, thanks a lot for your cooperation. PS: Please CC: me with your replies, since I am not registered in the list. Thanks again. -- Ivan Vilata i Balaguer >qo< http://www.carabos.com/ C?rabos Coop. V. V V Enjoy Data "" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From jmiller at stsci.edu Mon Dec 13 04:54:00 2004 From: jmiller at stsci.edu (Todd Miller) Date: Mon Dec 13 04:54:00 2004 Subject: [Numpy-discussion] Using new numeric instances in NumArray and RecArray. In-Reply-To: <20041213113929.GR4752@orm-embar.terramar.selidor.net> References: <20041213113929.GR4752@orm-embar.terramar.selidor.net> Message-ID: <1102942407.4631.9.camel@jaytmiller.comcast.net> On Mon, 2004-12-13 at 12:39 +0100, Ivan Vilata i Balaguer wrote: > Hi all! I am trying to define new numeric types to use them in > NumArray and RecArray objects. My intention is to define them as > equivalent to other types, but also make possible to distinguish them. > I tried "MyInt32 = IntegralType(4, 'MyType')", which registered OK, is > equivalent to Int32, but distinguishable from it. However, I am not > allowe to create a NumArray based on it: "a = array([1,2], MyInt32)" > fails with "RuntimeError: _numarray_init: can't get typeno for type". > > Is it possible to integrate new types seamlessly into NumArray and > RecArray objects? I don't know. No one else has asked for it yet so you're in unfamiliar territory. It does seem (to me anyway) like it would complicate an already complex system. What are you trying to do? > Is this the right approach, or am I getting that > wrong? The system as it stands is not designed for that. Regards, Todd From ivilata at carabos.com Mon Dec 13 08:24:11 2004 From: ivilata at carabos.com (Ivan Vilata i Balaguer) Date: Mon Dec 13 08:24:11 2004 Subject: [Numpy-discussion] Using new numeric instances in NumArray and RecArray. In-Reply-To: <1102942407.4631.9.camel@jaytmiller.comcast.net> References: <20041213113929.GR4752@orm-embar.terramar.selidor.net> <1102942407.4631.9.camel@jaytmiller.comcast.net> Message-ID: <20041213162320.GT4752@orm-embar.terramar.selidor.net> On Mon, Dec 13, 2004 at 07:53:27AM -0500, Todd Miller wrote: > On Mon, 2004-12-13 at 12:39 +0100, Ivan Vilata i Balaguer wrote: > [...] > > Is it possible to integrate new types seamlessly into NumArray and > > RecArray objects? > > I don't know. No one else has asked for it yet so you're in unfamiliar > territory. It does seem (to me anyway) like it would complicate an > already complex system. What are you trying to do? I'm trying to define new types that can be handled as normal types in numarray, but being different (distinguishable) from their equivalent types. For instance, an array of MyInt32 should work exactly as an Int32 array, but I would need to know it is a MyInt32 array and not an Int32 one. However, I you believe that the system is not prepared for that (or it would require too much work) I will try to identify it by some external means. Thank you anyway for your prompt answer. Best regards, Ivan -- Ivan Vilata i Balaguer >qo< http://www.carabos.com/ C?rabos Coop. V. V V Enjoy Data "" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From jmiller at stsci.edu Mon Dec 13 08:49:03 2004 From: jmiller at stsci.edu (Todd Miller) Date: Mon Dec 13 08:49:03 2004 Subject: [Numpy-discussion] Using new numeric instances in NumArray and RecArray. In-Reply-To: <20041213162320.GT4752@orm-embar.terramar.selidor.net> References: <20041213113929.GR4752@orm-embar.terramar.selidor.net> <1102942407.4631.9.camel@jaytmiller.comcast.net> <20041213162320.GT4752@orm-embar.terramar.selidor.net> Message-ID: <1102956503.18595.65.camel@halloween.stsci.edu> On Mon, 2004-12-13 at 11:23, Ivan Vilata i Balaguer wrote: > I'm trying to define new types that can be handled as normal types in > numarray, but being different (distinguishable) from their equivalent > types. For instance, an array of MyInt32 should work exactly as an > Int32 array, but I would need to know it is a MyInt32 array and not an > Int32 one. > > However, I you believe that the system is not prepared for that > (or it would require too much work) I will try to identify it by some > external means. Thank you anyway for your prompt answer. I think the system is simply not designed for that. For what you want, I think Int32 would need to be a class, not an instance, so that it could be subclassed while retaining a relatively simple C implementation. As you noted, the C infrastructure works using a type mapping table which maps type *instances* onto type numbers, so it won't work as it is now. Regards, Todd From haase at msg.ucsf.edu Mon Dec 13 09:10:01 2004 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Mon Dec 13 09:10:01 2004 Subject: [Numpy-discussion] Using new numeric instances in NumArray and RecArray. In-Reply-To: <1102956503.18595.65.camel@halloween.stsci.edu> References: <20041213113929.GR4752@orm-embar.terramar.selidor.net> <20041213162320.GT4752@orm-embar.terramar.selidor.net> <1102956503.18595.65.camel@halloween.stsci.edu> Message-ID: <200412130909.44454.haase@msg.ucsf.edu> Hi Ivan, I might be ignorant because if have never heard about the "same-but-distinguishable" concept (except from general oo-deriving of course). But I might actually have a similar situation. I have a 3d-image file-format which I wanted to handle mostly just by it's pixel-data content, that is: I return it as numarray. But to allow me to access the extra (header) information that comes with that file-format I "stick" a new attribute into the numarray instance. (this must sound very surprising to e.g. C++-people put is standard python as I understand) So here is some outline of my code: ''' class Mrc: # our "mrc"-file-format def __init__(self): open-file, make memmap for header+ data data.Mrc = self # this is the "sticking-in part" return data ''' This is very much paraphrased. I think __init__ doesn't have a return (so I use a seprarate function for that). And (after much debugging) I now use "weakref" for the "sticking-in part" to avoid circular references. Tell me if I guessed you case right (at least somewhat) ;-) Regards, Sebastian Haase On Monday 13 December 2004 08:48 am, Todd Miller wrote: > On Mon, 2004-12-13 at 11:23, Ivan Vilata i Balaguer wrote: > > I'm trying to define new types that can be handled as normal types in > > numarray, but being different (distinguishable) from their equivalent > > types. For instance, an array of MyInt32 should work exactly as an > > Int32 array, but I would need to know it is a MyInt32 array and not an > > Int32 one. > > > > However, I you believe that the system is not prepared for that > > (or it would require too much work) I will try to identify it by some > > external means. Thank you anyway for your prompt answer. > > I think the system is simply not designed for that. For what you want, > I think Int32 would need to be a class, not an instance, so that it > could be subclassed while retaining a relatively simple C > implementation. As you noted, the C infrastructure works using a type > mapping table which maps type *instances* onto type numbers, so it won't > work as it is now. > > Regards, > Todd > > > > ------------------------------------------------------- > SF email is sponsored by - The IT Product Guide > Read honest & candid reviews on hundreds of IT Products from real users. > Discover which products truly live up to the hype. Start reading now. > http://productguide.itmanagersjournal.com/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From ivilata at carabos.com Mon Dec 13 13:58:19 2004 From: ivilata at carabos.com (Ivan Vilata i Balaguer) Date: Mon Dec 13 13:58:19 2004 Subject: [Numpy-discussion] Using new numeric instances in NumArray and RecArray. In-Reply-To: <1102956503.18595.65.camel@halloween.stsci.edu> References: <20041213113929.GR4752@orm-embar.terramar.selidor.net> <1102942407.4631.9.camel@jaytmiller.comcast.net> <20041213162320.GT4752@orm-embar.terramar.selidor.net> <1102956503.18595.65.camel@halloween.stsci.edu> Message-ID: <20041213215740.GX4752@orm-embar.terramar.selidor.net> On Mon, Dec 13, 2004 at 11:48:23AM -0500, Todd Miller wrote: > [...] > I think the system is simply not designed for that. For what you want, > I think Int32 would need to be a class, not an instance, so that it > could be subclassed while retaining a relatively simple C > implementation. As you noted, the C infrastructure works using a type > mapping table which maps type *instances* onto type numbers, so it won't > work as it is now. Uh, what a pity. I thougth that creating new instances with IntegralType and FloatingType would be more useful. I will try with subclassing NumArray and RecArray as I was discussing with Colin. Thanks a lot for your help and useful information. Regards, Ivan -- Ivan Vilata i Balaguer >qo< http://www.carabos.com/ C?rabos Coop. V. V V Enjoy Data "" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From ivilata at carabos.com Mon Dec 13 14:21:06 2004 From: ivilata at carabos.com (Ivan Vilata i Balaguer) Date: Mon Dec 13 14:21:06 2004 Subject: [Numpy-discussion] Using new numeric instances in NumArray and RecArray. In-Reply-To: <41BDCF5D.5000609@sympatico.ca> References: <20041213113929.GR4752@orm-embar.terramar.selidor.net> <41BD9207.6060309@sympatico.ca> <20041213164000.GU4752@orm-embar.terramar.selidor.net> <41BDCF5D.5000609@sympatico.ca> Message-ID: <20041213222035.GY4752@orm-embar.terramar.selidor.net> On Mon, Dec 13, 2004 at 12:20:29PM -0500, Colin J. Williams wrote: > [...] > The complexity would depend partly on whether TimeStuff deals only with > data of the MyTime32 type. > > If all other data types (Int8, Float, Complex etc.) are barred, then the > needed overriding would probably be simpler. > [...] I have the feeling that MyInt32-only support should be enough for the NumArray subclass (it's a uniform array). However, the RecArray subclass should be able to handle all existing types in addition to MyInt32. > > I guess that another factor to consider is whether this is a one-off > project or something which will be in production for years to come. If > the former, it might be simpler to do the time related things outside of > numarray. On the other hand, if execution time is a significant factor, > then it could be worthwhile to modify the ufuncs - that is not a thing I > would like to tackle. Hehe, it will be production code with time-critical requirements. :) However, it now seems to me that subclassing NumArray and RecArray will be the right thing (not the easiest, though) to do. Thanks a lot Colin and Todd for your helpful comments. I'll try to do it and will tell you how it went. Bye! -- Ivan Vilata i Balaguer >qo< http://www.carabos.com/ C?rabos Coop. V. V V Enjoy Data "" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From ivilata at carabos.com Tue Dec 14 09:51:00 2004 From: ivilata at carabos.com (Ivan Vilata i Balaguer) Date: Tue Dec 14 09:51:00 2004 Subject: [Numpy-discussion] Re: Using new numeric instances in NumArray and RecArray. In-Reply-To: <20041213113929.GR4752@orm-embar.terramar.selidor.net> References: <20041213113929.GR4752@orm-embar.terramar.selidor.net> Message-ID: <20041214174949.GB20214@orm-embar.terramar.selidor.net> On Mon, Dec 13, 2004 at 12:39:29PM +0100, Ivan Vilata i Balaguer wrote: > > Hi all! I am trying to define new numeric types to use them in > NumArray and RecArray objects. > [...] Hello again. I have started to code a subclass of RecArray to map custom instance types (MyInt32) to standard Numarray instance types (Int32). The task seems plausible once numarray.typeDict, numarray.records.numfmt and numarray.records.revfmt are tailored to support the new types. Translating a format specification like formats='1MyInt32,(2,3)f8' to formats='1Int32,(2,3)f8' looks hard but it is a question of time. However, I have come across the fact that the constructors for NumArray and RecArray are seldom used directly. Instead, functions like numarray.array() and numarray.records.array() are used. These make heavy transformations on their arguments and then create new {Num,Rec}Array objects. This fact makes using classes different to {Num,Rec}Array very difficult, even more when using different signatures for their constructors. One solution is redefining the aforementioned functions to use custom classes. However, this looks cumbersome and not a simple task, but it seems to me the only way to keep the numarray interface. So I will try to implement those new types around Numarray instead of at the same level. Well, this was my report on my attempt to define new Numarray types! Thank you again for your suggestions. Best wishes, Ivan import disclaimer -- Ivan Vilata i Balaguer >qo< http://www.carabos.com/ C?rabos Coop. V. V V Enjoy Data "" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From luszczek at cs.utk.edu Tue Dec 14 09:53:03 2004 From: luszczek at cs.utk.edu (Piotr Luszczek) Date: Tue Dec 14 09:53:03 2004 Subject: [Numpy-discussion] started work on new version of LAPACK Message-ID: <41BF285D.2080809@cs.utk.edu> Hello all, on behalf of the LAPACK/ScaLAPACK team I'd like to inform you that we have just received funding to work on a new version of LAPACK and ScaLAPACK. Right now we are soliciting input from various groups that use the software. Numpy community is an important user and we would like to hear from you. I have subscribed to this list so I will receive any responses and pass them along to the rest the LAPACK/ScaLAPACK team. Here is a list to give you a brief idea of the issues we are trying to address: - programming language of new implementation (we will keep the old code as it is) - support for threading (OpenMP, pthreads, etc.) - 64-bit addressing - new/improved routines - IEEE 794 (floating-point) support Please let us know what you think. Peter (Piotr Luszczek) From rays at san.rr.com Wed Dec 15 20:13:01 2004 From: rays at san.rr.com (RJ) Date: Wed Dec 15 20:13:01 2004 Subject: [Numpy-discussion] copying ctypes arrays to numarray? Message-ID: <5.2.1.1.2.20041215195942.07e24590@pop-server.san.rr.com> I'm posting here to see if numarray has a method that could work like array.buffer_info(). numarray.info() returns a text output that can't be used like the array method is to memmove() between ctypes and Python arrays without parsing, apparently. ctypes thread with Thomas Heller below. His main question: "Maybe you should ask to make sure that there's no way to copy between objects implementing the buffer protocol with some Python function that I do not know about?" From bogus@does.not.exist.com Wed Dec 15 14:25:33 2004 From: bogus@does.not.exist.com () Date: Wed, 15 Dec 2004 20:25:33 +0100 Subject: [ctypes-users] copying/slicing ctypes arrays, (c_ulong *n)() In-Reply-To: <5.2.0.4.2.20041215100117.0dff97b0@blue-cove.com> (Ray Schumacher's message of "Wed, 15 Dec 2004 10:57:45 -0800") References: <5.2.1.1.2.20041214202542.05f02920@pop-server.san.rr.com> <5.2.1.1.2.20041214202542.05f02920@pop-server.san.rr.com> <5.2.0.4.2.20041215100117.0dff97b0@blue-cove.com> Message-ID: From florian.proff.schulze at gmx.net Thu Dec 16 01:15:02 2004 From: florian.proff.schulze at gmx.net (Florian Schulze) Date: Thu Dec 16 01:15:02 2004 Subject: [Numpy-discussion] Re: copying ctypes arrays to numarray? References: <5.2.1.1.2.20041215195942.07e24590@pop-server.san.rr.com> Message-ID: On Wed, 15 Dec 2004 20:16:07 -0800, RJ wrote: > I'm posting here to see if numarray has a method that could work like > array.buffer_info(). numarray.info() returns a text output that can't > be used like the array method is to memmove() between ctypes and Python > arrays without parsing, apparently. > > ctypes thread with Thomas Heller below. His main question: "Maybe you > should ask to make sure that there's no way to copy between > objects implementing the buffer protocol with some Python function that > I do not know about?" I just tried some things: >>> import ctypes >>> a = (ctypes.c_int * 5)() >>> a[0] = 1; a[1] = 2; a[2] = 3; a[3] = 4; a[4] = 5 >>> list(a) [1, 2, 3, 4, 5] >>> import numarray >>> buf = numarray.zeros(shape=20, type='i4') >>> buf array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) >>> buf[2:7] = a >>> buf array([0, 0, 1, 2, 3, 4, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) >>> temp = numarray.array(sequence=buffer(a), shape=5, type='i4') >>> temp array([1, 2, 3, 4, 5]) >>> temp._data >>> buffer(a) >>> a[2] = 10 >>> temp array([ 1, 2, 10, 4, 5]) The first block just creates the ctypes data. The second block uses slice assignment to copy the data from the ctypes array into the numarray.array. The third block uses the buffer interface to create a numarray.array which points to the same memory location as the ctypes array. The forth block illustrates that it's really the same memory. You have to benchmark which one of the two solutions is the better one for you. Regards, Florian Schulze From klimek at grc.nasa.gov Thu Dec 16 10:07:02 2004 From: klimek at grc.nasa.gov (Bob Klimek) Date: Thu Dec 16 10:07:02 2004 Subject: [Numpy-discussion] binary thinning In-Reply-To: References: <41B63AA8.5060205@grc.nasa.gov> Message-ID: <41C1CF1F.4000807@grc.nasa.gov> Peter Verveer wrote: > Hi Bob, > > Below I give my implementation, which does not produce the exact same > result as shown on the webpage you refer to, although the result seems > to be a proper skeleton. This may be because the order in which the > structures are applied matters (I think). Could you maybe test that by > permutating the order in which the structures are applied? I would be > interested if this is true and if one can get the same result as on > the webpage by finding the right sequence... Peter, I'm finally able to get back with some answers; unfortunately, for a number of reasons my time spent on this is quite fragmented. First, to answer your question about changing the order in which structures are applied. It turns out it does matter. I tried rotating the structures clockwise and counter-clockwise, pre-rotating the structure before doing the four rotations, and I permutating the order, and in some of those cases the results are different, although usually only slightly different - a couple pixels here, a few there. Second, no matter what I tried, I could not duplicate the picture in the webpage. Third, besides the test image (off the webpage) I also tested a skeleton function on few other images using several packages available to me, including an old Matrox MIL, ImageJ, and Matlab. Each one of them produced different results. I'm not sure that one is more correct than another, they're probably all correct. In general the nd_image skeleton produces results similar to Matlab. Peter, would you be interested in adding a few binary morphology functions to nd_image? So far I have working versions of borderkill, borderkeep, reconstruct, thinning, thickening, skeleton, skiz, and convex hull. Even though they were all produced with just what's there right now (erosion, dilation, and hit-or-miss) and a few numarray functions, it took a long time to figure out and could be helpful to the next guy. I'd be happy to send you what I got and/or post it. Regards, Bob From verveer at embl-heidelberg.de Thu Dec 16 10:43:03 2004 From: verveer at embl-heidelberg.de (Peter Verveer) Date: Thu Dec 16 10:43:03 2004 Subject: [Numpy-discussion] binary thinning In-Reply-To: <41C1CF1F.4000807@grc.nasa.gov> References: <41B63AA8.5060205@grc.nasa.gov> <41C1CF1F.4000807@grc.nasa.gov> Message-ID: <35A3C83E-4F92-11D9-8D51-000D932805AC@embl-heidelberg.de> Hi Bob, Thanks for using the nd_image code. It is helpful to get some user feedback. I think I did not tell you in my last email, but your last questions actually allowed me to find a few minor bugs and fix them... > First, to answer your question about changing the order in which > structures are applied. It turns out it does matter. I tried rotating > the structures clockwise and counter-clockwise, pre-rotating the > structure before doing the four rotations, and I permutating the > order, and in some of those cases the results are different, although > usually only slightly different - a couple pixels here, a few there. This seems reasonable to me, I would expect that the order matters. > Second, no matter what I tried, I could not duplicate the picture in > the webpage. > > Third, besides the test image (off the webpage) I also tested a > skeleton function on few other images using several packages available > to me, including an old Matrox MIL, ImageJ, and Matlab. Each one of > them produced different results. I'm not sure that one is more correct > than another, they're probably all correct. In general the nd_image > skeleton produces results similar to Matlab. It seems then that there is no accepted single solution for the skeleton, so I guess the code from nd_image is okay then. > Peter, would you be interested in adding a few binary morphology > functions to nd_image? So far I have working versions of borderkill, > borderkeep, reconstruct, thinning, thickening, skeleton, skiz, and > convex hull. Even though they were all produced with just what's there > right now (erosion, dilation, and hit-or-miss) and a few numarray > functions, it took a long time to figure out and could be helpful to > the next guy. I'd be happy to send you what I got and/or post it. One issue is that nd_image functions are supposed to be generic and work in any dimension. So I kept the binary morphology low-level on purpose, so far. But on the other hand, I don't see why we could not add some stuff even if it only works in 2D (which I assume is the case with your code.) Why don't you send me what you have and I have a look in the next weeks? Cheers, Peter From rays at san.rr.com Thu Dec 16 13:28:01 2004 From: rays at san.rr.com (rays) Date: Thu Dec 16 13:28:01 2004 Subject: [Numpy-discussion] copying ctypes arrays to numarray? In-Reply-To: <5.2.1.1.2.20041215195942.07e24590@pop-server.san.rr.com> Message-ID: <5.2.0.4.2.20041216132555.079aac28@pop-server.san.rr.com> I'm re-posting this here to see if anyone can shed light on numarray.info() behavior, and why assignment from a (ctypes.c_long * 2000)() to a numarray is so much slower than a memmove(). I was surprised that assignment wasn't faster and that numarray assignment was consistently ~.5% faster than Numeric. The N vs. n memmove() flip-flopped for fastest, with array.array always slower. After some discussion, I made a better direct comparison of ctypes and memmove, map etc., using all the methods suggested so far. I think it's fairly valid.(?) Run on a 2.4GHz WinXP, Py2.3, a number of times. Parsing numarray.info() was a royal pain, it writes directly to stdout! I.e.: >>> inf = src.info() class: shape: (32,) strides: (8,) byteoffset: 0 bytestride: 8 itemsize: 8 aligned: 1 contiguous: 1 data: byteorder: little byteswap: 0 type: Float64 >>> inf >>> type(inf) Since the ctype does support slicing, I was considering leaving the data in the device driver buffer (~1MB circular) and poking into it, but memmove is so much faster than slicing the ctype, I'm doing memmove()s to numarray. I presume that I should check for numarray.iscontiguous( ) or is_c_array( ) first to be safe... Results and code below. Thanks to all for the help, Ray >python test.py Array address 11443208 n address 17039424 N address 21102656 for loop 2027.0 us ea map 1781.0 us ea slice 1704.0 us ea assign N 1244.0 us ea assign n 1242.0 us ea memmove 4.3831 us ea memmove N 3.4773 us ea memmove n 3.4803 us ea _____________________________________________________ ## test.py import array import numarray import ctypes import time import string import StringIO import sys buf = (ctypes.c_long * 2000)() Array = array.array("l", [0]*10000) n = numarray.zeros((1000000), numarray.Int32) N = numarray.zeros((1000000), numarray.Int32) #!!!!!!!!!!!!!!! arrrrgggg! !!!!!!!!!!!!!!!! # n.info() writes directly to stdout! stdout = sys.stdout fileo = StringIO.StringIO() sys.stdout = fileo n.info() ninfo = fileo.getvalue( ) fileo.close() sys.stdout = stdout stdout = sys.stdout fileo = StringIO.StringIO() sys.stdout = fileo N.info() Ninfo = fileo.getvalue( ) fileo.close() sys.stdout = stdout #!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! print 'Array address', Array.buffer_info()[0] ninfo = string.split(ninfo) nAddress = int(ninfo[20], 16) print 'n address', nAddress Ninfo = string.split(Ninfo) NAddress = int(Ninfo[20], 16) print 'N address', NAddress t0 = time.clock() for loop in range(1000): for i in range(2000): n[loop+i] = buf[i] print 'for loop ', round((time.clock()-t0)*1000), 'us ea' t0 = time.clock() for loop in range(1000): n[loop:loop+2000] = map(None, buf) print 'map ', round((time.clock()-t0)*1000), 'us ea' t0 = time.clock() for loop in range(1000): n[loop:loop+2000] = buf[0:2000] print 'slice ', round((time.clock()-t0)*1000), 'us ea' t0 = time.clock() for loop in range(10000): N[loop:loop+2000] = buf print 'assign N', round((time.clock()-t0)*100), 'us ea' t0 = time.clock() for loop in range(10000): n[loop:loop+2000] = buf print 'assign n', round((time.clock()-t0)*100), 'us ea' t0 = time.clock() for loop in range(10000): ctypes.memmove(10+Array.buffer_info()[0], buf, 2000) print 'memmove ', round((time.clock()-t0)*1, 4), 'us ea' t0 = time.clock() for loop in range(10000): ctypes.memmove(10+NAddress, buf, 2000) print 'memmove N', round((time.clock()-t0)*1, 4), 'us ea' t0 = time.clock() for loop in range(10000): ctypes.memmove(10+nAddress, buf, 2000) print 'memmove n', round((time.clock()-t0)*1, 4), 'us ea' -------------- next part -------------- An HTML attachment was scrubbed... URL: From jmiller at stsci.edu Thu Dec 16 13:56:01 2004 From: jmiller at stsci.edu (Todd Miller) Date: Thu Dec 16 13:56:01 2004 Subject: [Numpy-discussion] copying ctypes arrays to numarray? In-Reply-To: <5.2.0.4.2.20041216132555.079aac28@pop-server.san.rr.com> References: <5.2.0.4.2.20041216132555.079aac28@pop-server.san.rr.com> Message-ID: <1103234122.3167.43.camel@jaytmiller.comcast.net> I agree that numarray's .info() would be better as a string, but I've been leery of changing it. Before we go to far with this, try: repr(src._data) to get a string like: "" Note: this technique depends on the buffer object used as the basis for the array. Here the repr of numarray's memory object is giving useful information. Other buffers, e.g. a string, might not. It looks to me like you forgot to import Numeric so I don't think there's any numarray:Numeric comparison going on... either that or I just don't understand what you're to compare. To be safe, check the .is_c_array() method. Regards, Todd On Thu, 2004-12-16 at 13:33 -0800, rays wrote: > I'm re-posting this here to see if anyone can shed light on > numarray.info() behavior, and why assignment from a (ctypes.c_long * > 2000)() to a numarray is so much slower than a memmove(). > > I was surprised that assignment wasn't faster and that numarray > assignment was consistently ~.5% faster than Numeric. > The N vs. n memmove() flip-flopped for fastest, with array.array > always slower. > > After some discussion, I made a better direct comparison of ctypes and > memmove, map etc., using all the methods suggested so far. I think > it's fairly valid.(?) Run on a 2.4GHz WinXP, Py2.3, a number of times. > > Parsing numarray.info() was a royal pain, it writes directly to > stdout! > I.e.: > >>> inf = src.info() > class: > shape: (32,) > strides: (8,) > byteoffset: 0 > bytestride: 8 > itemsize: 8 > aligned: 1 > contiguous: 1 > data: aliasing object > 00000000> > byteorder: little > byteswap: 0 > type: Float64 > >>> inf > >>> type(inf) > > > > Since the ctype does support slicing, I was considering leaving the > data in the device driver buffer (~1MB circular) and poking into it, > but memmove is so much faster than slicing the ctype, I'm doing > memmove()s to numarray. > I presume that I should check for numarray.iscontiguous( ) or > is_c_array( ) first to be safe... > > Results and code below. > > Thanks to all for the help, > Ray > > > > >python test.py > Array address 11443208 > n address 17039424 > N address 21102656 > for loop 2027.0 us ea > map 1781.0 us ea > slice 1704.0 us ea > assign N 1244.0 us ea > assign n 1242.0 us ea > memmove 4.3831 us ea > memmove N 3.4773 us ea > memmove n 3.4803 us ea > > _____________________________________________________ > > ## test.py > import array > import numarray > import ctypes > import time > import string > import StringIO > import sys > > buf = (ctypes.c_long * 2000)() > Array = array.array("l", [0]*10000) > n = numarray.zeros((1000000), numarray.Int32) > N = numarray.zeros((1000000), numarray.Int32) > > #!!!!!!!!!!!!!!! arrrrgggg! !!!!!!!!!!!!!!!! > # n.info() writes directly to stdout! > stdout = sys.stdout > fileo = StringIO.StringIO() > sys.stdout = fileo > n.info() > ninfo = fileo.getvalue( ) > fileo.close() > sys.stdout = stdout > > stdout = sys.stdout > fileo = StringIO.StringIO() > sys.stdout = fileo > N.info() > Ninfo = fileo.getvalue( ) > fileo.close() > sys.stdout = stdout > #!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! > > print 'Array address', Array.buffer_info()[0] > > ninfo = string.split(ninfo) > nAddress = int(ninfo[20], 16) > print 'n address', nAddress > Ninfo = string.split(Ninfo) > NAddress = int(Ninfo[20], 16) > print 'N address', NAddress > > t0 = time.clock() > for loop in range(1000): > for i in range(2000): > n[loop+i] = buf[i] > print 'for loop ', round((time.clock()-t0)*1000), 'us ea' > > t0 = time.clock() > for loop in range(1000): > n[loop:loop+2000] = map(None, buf) > print 'map ', round((time.clock()-t0)*1000), 'us ea' > > t0 = time.clock() > for loop in range(1000): > n[loop:loop+2000] = buf[0:2000] > print 'slice ', round((time.clock()-t0)*1000), 'us ea' > > t0 = time.clock() > for loop in range(10000): > N[loop:loop+2000] = buf > print 'assign N', round((time.clock()-t0)*100), 'us ea' > > t0 = time.clock() > for loop in range(10000): > n[loop:loop+2000] = buf > print 'assign n', round((time.clock()-t0)*100), 'us ea' > > t0 = time.clock() > for loop in range(10000): > ctypes.memmove(10+Array.buffer_info()[0], > buf, > 2000) > print 'memmove ', round((time.clock()-t0)*1, 4), 'us ea' > > t0 = time.clock() > for loop in range(10000): > ctypes.memmove(10+NAddress, > buf, > 2000) > print 'memmove N', round((time.clock()-t0)*1, 4), 'us ea' > > t0 = time.clock() > for loop in range(10000): > ctypes.memmove(10+nAddress, > buf, > 2000) > print 'memmove n', round((time.clock()-t0)*1, 4), 'us ea' > > > > > > > From rays at san.rr.com Thu Dec 16 17:05:29 2004 From: rays at san.rr.com (RJ) Date: Thu Dec 16 17:05:29 2004 Subject: [Numpy-discussion] copying ctypes arrays to numarray? In-Reply-To: <1103234122.3167.43.camel@jaytmiller.comcast.net> References: <5.2.0.4.2.20041216132555.079aac28@pop-server.san.rr.com> <5.2.0.4.2.20041216132555.079aac28@pop-server.san.rr.com> Message-ID: <5.2.1.1.2.20041216165316.00b1d4c0@pop-server.san.rr.com> Doh. Thanks Todd, At 04:55 PM 12/16/2004 -0500, Todd Miller wrote: > try: repr(src._data) ... >Other buffers, e.g. a string, might not. Which is fine for this app, so I will. >It looks to me like you forgot to import Numeric Yep. I screwed up the import statements - I'll re-do it tomorrow. In that light, it is interesting that the assign N were always longer than the n ones... >To be safe, check the .is_c_array() method. Will do. I am also a little concerned about the address changing in a long-running process (literally, weeks), so I might check that too. Also, why is assignment from a (ctypes.c_long * 2000)() to a numarray so much slower than a memmove()? Does the assign make a temp buffer? Ray -------------- next part -------------- An HTML attachment was scrubbed... URL: From konrad.hinsen at laposte.net Fri Dec 17 05:30:02 2004 From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net) Date: Fri Dec 17 05:30:02 2004 Subject: [Numpy-discussion] ufuncs on user-defined types in numarray Message-ID: I am working on making ScientificPython (http://dirac.cnrs-orleans.fr/ScientificPython/) compatible with numarray. One problem I stumbled across is that one feature of Numeric seems to be absent from numarray: the possibility to call ufuncs with arguments of non-numeric types, leading to a corresponding method call. As an illustration, the following example works fine with Numeric, but crashes with numarray: from Numeric import sqrt #from numarray import sqrt class DummyValue: def __init__(self, string): self.string = string def __str__(self): return self.string def sqrt(self): return DummyValue('sqrt(%s)' % self.string) x = DummyValue('x') print sqrt(x) Is this a bug or a feature? Is there another way to make ufuncs work with user-defined classes and types? Konrad. -- --------------------------------------------------------------------- Konrad Hinsen Laboratoire L?on Brillouin, CEA Saclay, 91191 Gif-sur-Yvette Cedex, France Tel.: +33-1 69 08 79 25 Fax: +33-1 69 08 82 61 E-Mail: hinsen at llb.saclay.cea.fr --------------------------------------------------------------------- From jmiller at stsci.edu Fri Dec 17 06:54:03 2004 From: jmiller at stsci.edu (Todd Miller) Date: Fri Dec 17 06:54:03 2004 Subject: [Numpy-discussion] copying ctypes arrays to numarray? In-Reply-To: <5.2.1.1.2.20041216165316.00b1d4c0@pop-server.san.rr.com> References: <5.2.0.4.2.20041216132555.079aac28@pop-server.san.rr.com> <5.2.0.4.2.20041216132555.079aac28@pop-server.san.rr.com> <5.2.1.1.2.20041216165316.00b1d4c0@pop-server.san.rr.com> Message-ID: <1103295187.31241.52.camel@halloween.stsci.edu> On Thu, 2004-12-16 at 20:04, RJ wrote: > Doh. > Thanks Todd, > > At 04:55 PM 12/16/2004 -0500, Todd Miller wrote: > > try: repr(src._data) > ... > > Other buffers, e.g. a string, might not. > > Which is fine for this app, so I will. > > > It looks to me like you forgot to import Numeric > > Yep. I screwed up the import statements - I'll re-do it tomorrow. In > that light, it is interesting that the assign N were always longer > than the n ones... > > > To be safe, check the .is_c_array() method. > > Will do. I am also a little concerned about the address changing in a > long-running process (literally, weeks), so I might check that too. As long as you let the NumArray allocate its own memory, it will allocate a numarray.memory object and those don't move. If you resize the array, that's a different story... > Also, why is assignment from a (ctypes.c_long * 2000)() to a numarray > so much slower than a memmove()? Does the assign make a temp buffer? For the assignment to numarray, I think the ctype is opaque and as a result Python uses the sequence protocol; this is a huge amount of work compared to a tight C loop moving bytes. I think every 4 bytes the ctype is doing a method call and building a Python long, numarray is doing a method call and extracting a Python long, then the Python long is destructed. I may be missing something, but the comparisons of for loop, map, and slice look like they're only doing 1000 moves but the array assigments are doing 10000, so I think there's another factor of 10 missing in that comparison. From jmiller at stsci.edu Fri Dec 17 07:10:12 2004 From: jmiller at stsci.edu (Todd Miller) Date: Fri Dec 17 07:10:12 2004 Subject: [Numpy-discussion] ufuncs on user-defined types in numarray In-Reply-To: References: Message-ID: <1103296191.31241.66.camel@halloween.stsci.edu> On Fri, 2004-12-17 at 08:31, konrad.hinsen at laposte.net wrote: > I am working on making ScientificPython > (http://dirac.cnrs-orleans.fr/ScientificPython/) compatible with > numarray. One problem I stumbled across is that one feature of Numeric > seems to be absent from numarray: the possibility to call ufuncs with > arguments of non-numeric types, leading to a corresponding method call. > > As an illustration, the following example works fine with Numeric, but > crashes with numarray: > > from Numeric import sqrt > #from numarray import sqrt > > class DummyValue: > > def __init__(self, string): > self.string = string > > def __str__(self): > return self.string > > def sqrt(self): > return DummyValue('sqrt(%s)' % self.string) > > x = DummyValue('x') > print sqrt(x) > > > Is this a bug or a feature? It's a missing feature. > Is there another way to make ufuncs work with user-defined classes and types? Not that I know of. How critical a feature is it? Do you have a work around? numarray ufunc handling is in flux as I'm adding numarray support to scipy. At some point, there's going to be a major ufunc consolidation, and perhaps I can add this additional feature then. Regards, Todd From konrad.hinsen at laposte.net Fri Dec 17 08:22:02 2004 From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net) Date: Fri Dec 17 08:22:02 2004 Subject: [Numpy-discussion] ufuncs on user-defined types in numarray In-Reply-To: <1103296191.31241.66.camel@halloween.stsci.edu> References: <1103296191.31241.66.camel@halloween.stsci.edu> Message-ID: On Dec 17, 2004, at 16:09, Todd Miller wrote: >> Is there another way to make ufuncs work with user-defined classes >> and types? > > Not that I know of. How critical a feature is it? Do you have a work > around? It is critical for some applications, though not for many. The most important application in ScientificPython is automatic derivatives. These are implemented as a number-like class whose instances store the values of a variable and of its derivatives. Such objects can be fed to any routine that does mathematical calculations, which is what makes it so powerful. Without this mechanism, all function applications would have to be written as method calls, which is less habitual, but more importantly creates code that doesn't work with plain numbers. > numarray ufunc handling is in flux as I'm adding numarray support to > scipy. At some point, there's going to be a major ufunc > consolidation, > and perhaps I can add this additional feature then. Sounds good. I'll postpone the ScientificPython update until then. Konrad. -- --------------------------------------------------------------------- Konrad Hinsen Laboratoire L?on Brillouin, CEA Saclay, 91191 Gif-sur-Yvette Cedex, France Tel.: +33-1 69 08 79 25 Fax: +33-1 69 08 82 61 E-Mail: hinsen at llb.saclay.cea.fr --------------------------------------------------------------------- From rays at san.rr.com Fri Dec 17 08:30:00 2004 From: rays at san.rr.com (RJ) Date: Fri Dec 17 08:30:00 2004 Subject: [Numpy-discussion] copying ctypes arrays to numarray? In-Reply-To: <1103295187.31241.52.camel@halloween.stsci.edu> References: <5.2.1.1.2.20041216165316.00b1d4c0@pop-server.san.rr.com> <5.2.0.4.2.20041216132555.079aac28@pop-server.san.rr.com> <5.2.0.4.2.20041216132555.079aac28@pop-server.san.rr.com> <5.2.1.1.2.20041216165316.00b1d4c0@pop-server.san.rr.com> Message-ID: <5.2.1.1.2.20041217082410.05e4bec0@pop-server.san.rr.com> At 09:53 AM 12/17/2004 -0500, Todd Miller wrote: >I may be missing something, but the comparisons of for loop, map, and >slice look like they're only doing 1000 moves but the array assigments >are doing 10000, so I think there's another factor of 10 missing in >that comparison. The multiplier in the print statement compensates to still give microseconds per move; I couldn't bear to wait for 10,000 slow iterations. I'll re-do the test shortly in any case. Ray Secret anti-spam filter-passing text. Include with reply: qwertyuiop -------------- next part -------------- An HTML attachment was scrubbed... URL: From jmiller at stsci.edu Sat Dec 18 05:53:01 2004 From: jmiller at stsci.edu (Todd Miller) Date: Sat Dec 18 05:53:01 2004 Subject: [Numpy-discussion] Re: simple numarray bug In-Reply-To: <41C395E3.5070903@caltech.edu> References: <41C395E3.5070903@caltech.edu> Message-ID: <1103377939.3258.14.camel@jaytmiller.comcast.net> On Fri, 2004-12-17 at 18:28 -0800, Andrew Straw wrote: > Hi Todd, > Hi Andrew, > (Nice meeting you at SciPy '04!) Yes, nice meeting you. > I think this is a simple bug which I just filed a bug report on SF for: > http://sourceforge.net/tracker/index.php?func=detail&aid=1087378&group_id=1369&atid=101369 > > This email is just a "heads up" that I created the bug report. Thanks for the bug report and heads up. Source Forge automatically gives me e-mail for numarray bug reports so "heads ups" are not necessary. However, I often fall behind real time response so if you make a report and nothing happens for a while gentle nagging is OK. I committed a "fix" for this to CVS... There is now a LinearAlgebraError subclass of Exception aliased as LinAlgError. Regards, Todd From gazzar at email.com Sat Dec 18 22:38:02 2004 From: gazzar at email.com (Gary Ruben) Date: Sat Dec 18 22:38:02 2004 Subject: [Numpy-discussion] ufuncs on user-defined types in numarray Message-ID: <20041219063737.35BB14BDAB@ws1-1.us4.outblaze.com> Thanks Konrad, I didn't realise this was possible. I should use it on my ErrorVal module . Last time I tried (quite a while ago), numarray's object array support wasn't up to allowing me to port ErrorVal from Numeric. I don't think it matters since I don't know of anyone, besides me, who has tried using my module :-( but I thought I'd mention it for Todd's benefit since I'm one who could benefit from implementing it in numarray. Gary R. ----- Original Message ----- From: "Todd Miller" To: konrad.hinsen at laposte.net Subject: Re: [Numpy-discussion] ufuncs on user-defined types in numarray Date: 17 Dec 2004 10:09:51 -0500 > > On Fri, 2004-12-17 at 08:31, konrad.hinsen at laposte.net wrote: > > I am working on making ScientificPython > > (http://dirac.cnrs-orleans.fr/ScientificPython/) compatible with > > numarray. One problem I stumbled across is that one feature of > > Numeric seems to be absent from numarray: the possibility to call > > ufuncs with arguments of non-numeric types, leading to a > > corresponding method call. > > > > As an illustration, the following example works fine with > > Numeric, but crashes with numarray: > > > > from Numeric import sqrt > > #from numarray import sqrt > > > > class DummyValue: > > > > def __init__(self, string): > > self.string = string > > > > def __str__(self): > > return self.string > > > > def sqrt(self): > > return DummyValue('sqrt(%s)' % self.string) > > > > x = DummyValue('x') > > print sqrt(x) > > > > > > Is this a bug or a feature? > > It's a missing feature. > > > Is there another way to make ufuncs work with user-defined classes and types? > > Not that I know of. How critical a feature is it? Do you have a work around? > > numarray ufunc handling is in flux as I'm adding numarray support to > scipy. At some point, there's going to be a major ufunc consolidation, > and perhaps I can add this additional feature then. > > Regards, > Todd > > > > ------------------------------------------------------- > SF email is sponsored by - The IT Product Guide > Read honest & candid reviews on hundreds of IT Products from real users. > Discover which products truly live up to the hype. Start reading now. > http://productguide.itmanagersjournal.com/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- ___________________________________________________________ Sign-up for Ads Free at Mail.com http://promo.mail.com/adsfreejump.htm From konrad.hinsen at laposte.net Sun Dec 19 00:52:02 2004 From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net) Date: Sun Dec 19 00:52:02 2004 Subject: [Numpy-discussion] ufuncs on user-defined types in numarray In-Reply-To: <20041219063737.35BB14BDAB@ws1-1.us4.outblaze.com> References: <20041219063737.35BB14BDAB@ws1-1.us4.outblaze.com> Message-ID: <2E538670-519B-11D9-909D-000A95AB5F10@laposte.net> On 19.12.2004, at 07:37, Gary Ruben wrote: > I didn't realise this was possible. I should use it on my ErrorVal > module . Last time I > tried (quite a Indeed! You could then even have arrays of ErrorVal objects with no extra effort, at least as long as ErrorVal doesn't implement the sequence protocol (because then all the Numeric functions will silently convert them to arrays). Your module looks interesting. What does it do internally, interval arithmetic? Konrad. From jbrandmeyer at earthlink.net Mon Dec 20 10:12:10 2004 From: jbrandmeyer at earthlink.net (Jonathan Brandmeyer) Date: Mon Dec 20 10:12:10 2004 Subject: [Numpy-discussion] Numeric 23.6 for Python 2.4 Message-ID: <2845281.1103566316940.JavaMail.root@waldorf.psp.pas.earthlink.net> Does the Numeric team intend to provide a build of Numeric 23.6 for Python 2.4? If not, I have managed to build a release using the free-as-in-beer version of MSVC 7.1, aka MS Visual C++ Toolkit 2003. I basically followed the directions here: http://www.vrplumber.com/programming/mstoolkit/index.html and altered setup.py to use Numeric's own math libraries rather than the atlas libs. The build has been tested only by running the supplied Test/test.py program after installing the package. For the moment, you can download the file from http://www4.ncsu.edu/~jdbrandm/Numeric-23.6.win32-py2.4.exe but I would prefer it to be hosted on the Sourceforge download page for the project. Its md5sum is 2df692d7b8136b68e3b511ed27cbbdd8 Thanks, -Jonathan Brandmeyer P.S. I apologize if the message's formatting is borked up; I'm sending this from a webmail interface. From jmiller at stsci.edu Mon Dec 20 11:34:07 2004 From: jmiller at stsci.edu (Todd Miller) Date: Mon Dec 20 11:34:07 2004 Subject: [Numpy-discussion] ufuncs on user-defined types in numarray In-Reply-To: <20041219063737.35BB14BDAB@ws1-1.us4.outblaze.com> References: <20041219063737.35BB14BDAB@ws1-1.us4.outblaze.com> Message-ID: <1103571033.3339.69.camel@jaytmiller.comcast.net> Hi Gary and Konrad, I filed a Numarray Enhancement Request on Source Forge for this feature. It's not clear if, when, or how it will be implemented but it won't be completely forgotten either. Regards, Todd On Sun, 2004-12-19 at 16:37 +1000, Gary Ruben wrote: > Thanks Konrad, > I didn't realise this was possible. I should use it on my ErrorVal module . Last time I tried (quite a while ago), numarray's object array support wasn't up to allowing me to port ErrorVal from Numeric. I don't think it matters since I don't know of anyone, besides me, who has tried using my module :-( but I thought I'd mention it for Todd's benefit since I'm one who could benefit from implementing it in numarray. > Gary R. > > ----- Original Message ----- > From: "Todd Miller" > To: konrad.hinsen at laposte.net > Subject: Re: [Numpy-discussion] ufuncs on user-defined types in numarray > Date: 17 Dec 2004 10:09:51 -0500 > > > > > On Fri, 2004-12-17 at 08:31, konrad.hinsen at laposte.net wrote: > > > I am working on making ScientificPython > > > (http://dirac.cnrs-orleans.fr/ScientificPython/) compatible with > > > numarray. One problem I stumbled across is that one feature of > > > Numeric seems to be absent from numarray: the possibility to call > > > ufuncs with arguments of non-numeric types, leading to a > > > corresponding method call. > > > > > > As an illustration, the following example works fine with > > > Numeric, but crashes with numarray: > > > > > > from Numeric import sqrt > > > #from numarray import sqrt > > > > > > class DummyValue: > > > > > > def __init__(self, string): > > > self.string = string > > > > > > def __str__(self): > > > return self.string > > > > > > def sqrt(self): > > > return DummyValue('sqrt(%s)' % self.string) > > > > > > x = DummyValue('x') > > > print sqrt(x) > > > > > > > > > Is this a bug or a feature? > > > > It's a missing feature. > > > > > Is there another way to make ufuncs work with user-defined classes and types? > > > > Not that I know of. How critical a feature is it? Do you have a work around? > > > > numarray ufunc handling is in flux as I'm adding numarray support to > > scipy. At some point, there's going to be a major ufunc consolidation, > > and perhaps I can add this additional feature then. > > > > Regards, > > Todd > > > > > > > > ------------------------------------------------------- > > SF email is sponsored by - The IT Product Guide > > Read honest & candid reviews on hundreds of IT Products from real users. > > Discover which products truly live up to the hype. Start reading now. > > http://productguide.itmanagersjournal.com/ > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From gazzar at email.com Mon Dec 20 17:13:22 2004 From: gazzar at email.com (Gary Ruben) Date: Mon Dec 20 17:13:22 2004 Subject: [Numpy-discussion] ufuncs on user-defined types in numarray Message-ID: <20041221011222.EE0076EEF6@ws1-5.us4.outblaze.com> First, thanks Todd for elevating this to 'requested feature' status. Konrad, I had no idea what interval arithmetic is, so I looked it up. I'm not sure if what I've done is considered interval arithmetic - I suspect it is, but that I've only implemented it partially. The code isn't that clever. It doesn't care about propagating rounding errors correctly at the error limits. It just implements a brute-force approach to calculating operations on the error limits of a data-point value along with the central value. This is already more accurate than what most people do, which is to linearise the error limits and treat them as infinitessimals. The central and outer values are grouped in an object and any operations are applied, without any consideration of rounding errors, to all three values for every operation. Gary ----- Original Message ----- > > I didn't realise this was possible. I should use it on my > > ErrorVal module . > > Last time I tried (quite a > > Indeed! You could then even have arrays of ErrorVal objects with no > extra effort, at least as long as ErrorVal doesn't implement the > sequence protocol (because then all the Numeric functions will > silently convert them to arrays). > > Your module looks interesting. What does it do internally, interval > arithmetic? > > Konrad. -- ___________________________________________________________ Sign-up for Ads Free at Mail.com http://promo.mail.com/adsfreejump.htm From faltet at carabos.com Tue Dec 21 05:08:01 2004 From: faltet at carabos.com (Francesc Altet) Date: Tue Dec 21 05:08:01 2004 Subject: [Numpy-discussion] Striding on NumArray objects Message-ID: <200412211406.57699.faltet@carabos.com> Hi, I'm a bit lost with the next example: In [28]: from numarray import * In [29]: a=arange(10) In [30]: a.iscontiguous() Out[30]: 1 In [31]: b=a[::2] In [32]: b.iscontiguous() Out[32]: 0 That seems to suggest that b shares the same data buffer than a. Indeed: In [36]: a._data Out[36]: In [37]: b._data Out[37]: At this point, I believe that _bytestride should be different on both arrays, but: In [33]: a._bytestride Out[33]: 4 In [34]: b._bytestride Out[34]: 4 while I expected to find b._bytestride equal to 8. Is that an error or a lack of understanding on my part? Cheers, -- Francesc Altet ? >qo< ? http://www.carabos.com/ C?rabos Coop. V. ? V ?V ? Enjoy Data ? ? ? ? ? ? ? ? ? ? "" From tim.hochberg at cox.net Tue Dec 21 06:18:06 2004 From: tim.hochberg at cox.net (Tim Hochberg) Date: Tue Dec 21 06:18:06 2004 Subject: [Numpy-discussion] Striding on NumArray objects In-Reply-To: <200412211406.57699.faltet@carabos.com> References: <200412211406.57699.faltet@carabos.com> Message-ID: <41C8307B.4080505@cox.net> Francesc Altet wrote: >Hi, > >I'm a bit lost with the next example: > >In [28]: from numarray import * >In [29]: a=arange(10) >In [30]: a.iscontiguous() >Out[30]: 1 >In [31]: b=a[::2] >In [32]: b.iscontiguous() >Out[32]: 0 > >That seems to suggest that b shares the same data buffer than a. Indeed: > >In [36]: a._data >Out[36]: >In [37]: b._data >Out[37]: > > OK so far. >At this point, I believe that _bytestride should be different on both >arrays, but: > >In [33]: a._bytestride >Out[33]: 4 >In [34]: b._bytestride >Out[34]: 4 > >while I expected to find b._bytestride equal to 8. > >Is that an error or a lack of understanding on my part? > > What you are looking for is _strides. Since, in general, arrays can be multidimensional, the stride(s) need to be specified as a sequence. _bytestride appears to be equivalent to itemsize(); that is, it tells how many bytes are required to represent one item of the array. Here's what strides looks like in your case: >>> a._strides (4,) >>> b._strides (8,) Hope that helps, -tim >Cheers, > > > From jmiller at stsci.edu Tue Dec 21 07:06:03 2004 From: jmiller at stsci.edu (Todd Miller) Date: Tue Dec 21 07:06:03 2004 Subject: [Numpy-discussion] Striding on NumArray objects In-Reply-To: <200412211406.57699.faltet@carabos.com> References: <200412211406.57699.faltet@carabos.com> Message-ID: <1103641503.30020.49.camel@halloween.stsci.edu> On Tue, 2004-12-21 at 08:06, Francesc Altet wrote: > Hi, > > I'm a bit lost with the next example: > > In [28]: from numarray import * > In [29]: a=arange(10) > In [30]: a.iscontiguous() > Out[30]: 1 > In [31]: b=a[::2] > In [32]: b.iscontiguous() > Out[32]: 0 > > That seems to suggest that b shares the same data buffer than a. Indeed: > > In [36]: a._data > Out[36]: > In [37]: b._data > Out[37]: > > At this point, I believe that _bytestride should be different on both > arrays, but: > > In [33]: a._bytestride > Out[33]: 4 > In [34]: b._bytestride > Out[34]: 4 > > while I expected to find b._bytestride equal to 8. > > Is that an error or a lack of understanding on my part? Hi Francesc, This is a difficult question for me, but I think the answer is that the use of _bytestride is very limited. _bytestride is used to compute the "natural" strides of an almost contiguous array, e.g. a field of a recarray. That is, given a bytestride and a shape, the strides of a field of a contiguous RecArray are implied. However, once we start slicing (say in more than one dimension), _strides contains more and more information and is no longer implied by just the shape and bytestride but also by the history of slicing. From that perspective, it's not clear what _bytestride can be relied upon for in general or that it needs to be (or can be) kept up to date during slicing. FWIW, looking into this uncovered a related bug in numarray.strings where I tried to use _bytestride to do a simple iteration over all the elements of an array... that doesn't work. Regards, Todd From faltet at carabos.com Tue Dec 21 07:29:03 2004 From: faltet at carabos.com (Francesc Altet) Date: Tue Dec 21 07:29:03 2004 Subject: [Numpy-discussion] Striding on NumArray objects In-Reply-To: <1103641503.30020.49.camel@halloween.stsci.edu> References: <200412211406.57699.faltet@carabos.com> <1103641503.30020.49.camel@halloween.stsci.edu> Message-ID: <200412211628.10968.faltet@carabos.com> Hi Todd, A Dimarts 21 Desembre 2004 16:05, vareu escriure: > This is a difficult question for me, but I think the answer is that the > use of _bytestride is very limited. _bytestride is used to compute the > "natural" strides of an almost contiguous array, e.g. a field of a > recarray. That is, given a bytestride and a shape, the strides of a > field of a contiguous RecArray are implied. Yes, I was trying to use _bytestride mainly in RecArray contexts. It just happens that I've used a NumArray object to check the _bytestride behaviour. > However, once we start slicing (say in more than one dimension), > _strides contains more and more information and is no longer implied by > just the shape and bytestride but also by the history of slicing. From > that perspective, it's not clear what _bytestride can be relied upon > for in general or that it needs to be (or can be) kept up to date during > slicing. Well, for the time being it seems that RecArray does not support multidimensionalty, so I can just use _bytestride as a shortcut of _strides[0] here. Although now that I think more about this, using itemsize() as Tim suggested, would be the best to compute strides in RecArrays; however this is supposing that fields in RecArrays are contiguous. This would be always the case? > FWIW, looking into this uncovered a related bug in numarray.strings > where I tried to use _bytestride to do a simple iteration over all the > elements of an array... that doesn't work. Good! Regards, -- Francesc Altet ? >qo< ? http://www.carabos.com/ C?rabos Coop. V. ? V ?V ? Enjoy Data ? ? ? ? ? ? ? ? ? ? "" From jmiller at stsci.edu Tue Dec 21 12:22:03 2004 From: jmiller at stsci.edu (Todd Miller) Date: Tue Dec 21 12:22:03 2004 Subject: [Numpy-discussion] Striding on NumArray objects Message-ID: <1103660500.30020.327.camel@halloween.stsci.edu> > > This is a difficult question for me, but I think the answer is that the > > use of _bytestride is very limited. _bytestride is used to compute the > > "natural" strides of an almost contiguous array, e.g. a field of a > > recarray. That is, given a bytestride and a shape, the strides of a > > field of a contiguous RecArray are implied. > > Yes, I was trying to use _bytestride mainly in RecArray contexts. It just > happens that I've used a NumArray object to check the _bytestride behaviour. > > > However, once we start slicing (say in more than one dimension), > > _strides contains more and more information and is no longer implied by > > just the shape and bytestride but also by the history of slicing. From > > that perspective, it's not clear what _bytestride can be relied upon > > for in general or that it needs to be (or can be) kept up to date during > > slicing. > > Well, for the time being it seems that RecArray does not support > multidimensionalty, so I can just use _bytestride as a shortcut of > _strides[0] here. I wouldn't. I had a veritable stroke of genius and talked to JC Hsu... He suggested doing concrete examples and also indicated that full multi-dimensional RecArray support is almost all (except for repr'ing) there. To be clear, in general, _bytestride != _itemsize. For a RecArray field (of a contiguous RecArray) where each row contains just one element, generally _strides[-1] == _bytestride != _itemsize: >>> r = rec.array(buffer=None, formats="i2,(3,4)f4,a3", shape=10) >>> r.field(0).info() class: shape: (10,) strides: (53,) byteoffset: 0 bytestride: 53 itemsize: 2 aligned: 0 contiguous: 0 data: byteorder: little byteswap: 0 type: Int16 For a RecArray field where each row contains more than one element, _strides[-1] == itemsize: >>> r.field(1).info() class: shape: (10, 3, 4) strides: (53, 16, 4) byteoffset: 2 bytestride: 53 itemsize: 4 aligned: 0 contiguous: 0 data: byteorder: little byteswap: 0 type: Float32 The situation gets more complicated still when someone starts slicing the field or RecArray: >>> x[::3].info() class: shape: (4, 3, 4) strides: (159, 16, 4) byteoffset: 2 bytestride: 53 itemsize: 4 aligned: 0 contiguous: 0 data: byteorder: little byteswap: 0 type: Float32 Multi-dimensional RecArrays add their own twist: >>> r = rec.array(buffer=None, formats="i2,(3,4)f4,a3",shape=(10,10,10)) >>> r.field(1).info() class: shape: (10, 10, 10, 3, 4) strides: (5300, 480, 48, 16, 4) byteoffset: 2 bytestride: 53 itemsize: 4 aligned: 0 contiguous: 0 data: byteorder: little byteswap: 0 type: Float32 HTH, Todd From faltet at carabos.com Wed Dec 22 04:42:04 2004 From: faltet at carabos.com (Francesc Altet) Date: Wed Dec 22 04:42:04 2004 Subject: [Numpy-discussion] Striding on NumArray objects In-Reply-To: <1103660500.30020.327.camel@halloween.stsci.edu> References: <1103660500.30020.327.camel@halloween.stsci.edu> Message-ID: <200412221341.08144.faltet@carabos.com> A Dimarts 21 Desembre 2004 21:21, Todd Miller va escriure: > I had a veritable stroke of genius and talked to JC Hsu... He suggested > doing concrete examples and also indicated that full multi-dimensional > RecArray support is almost all (except for repr'ing) there. Multidimensional RecArrays? Nice!. Is part of the code already in CVS? > To be clear, in general, _bytestride != _itemsize. For a RecArray > field (of a contiguous RecArray) where each row contains just one > element, generally _strides[-1] == _bytestride != _itemsize: Thanks for the examples. That has clarified *a lot* where to find the interesting data: it's clear now that what I need is _strides[0] (because I need to access elements in a column sequentially). And that even works for forthcoming multidimensional RecArrays which is good. Regards, -- Francesc Altet From juenglin at cs.pdx.edu Sun Dec 26 11:33:02 2004 From: juenglin at cs.pdx.edu (Ralf Juengling) Date: Sun Dec 26 11:33:02 2004 Subject: [Numpy-discussion] log bug? Message-ID: <1104089215.3979.6.camel@localhost.localdomain> Hi, I'm not sure this behavior is intended: >>> from numarray import * >>> log(array([0.0], type=Float64)) Warning: Encountered divide by zero(s) in log array([ inf]) >>> -inf would be a more useful result. ralf From jmiller at stsci.edu Sun Dec 26 14:16:05 2004 From: jmiller at stsci.edu (Todd Miller) Date: Sun Dec 26 14:16:05 2004 Subject: [Numpy-discussion] log bug? In-Reply-To: <1104089215.3979.6.camel@localhost.localdomain> References: <1104089215.3979.6.camel@localhost.localdomain> Message-ID: <1104099312.3332.1.camel@jaytmiller.comcast.net> On Sun, 2004-12-26 at 11:26 -0800, Ralf Juengling wrote: > Hi, > > I'm not sure this behavior is intended: > > >>> from numarray import * > >>> log(array([0.0], type=Float64)) > Warning: Encountered divide by zero(s) in log > array([ inf]) > >>> > > > -inf would be a more useful result. > > > ralf Hi Ralf, This is fixed in CVS now. Thanks, Todd From jdhunter at ace.bsd.uchicago.edu Fri Dec 31 05:46:05 2004 From: jdhunter at ace.bsd.uchicago.edu (John Hunter) Date: Fri Dec 31 05:46:05 2004 Subject: [Numpy-discussion] ANN: matplotlib-0.70 Message-ID: matplotlib is a 2D graphics package that produces plots from python scripts, the python shell, or embeds them in your favorite python GUI -- wx, gtk, tk, fltk currently supported with qt in the works. Unlike many python plotting alternatives is written in python, so it is easy to extend. matplotlib is used in the finance industry, web application servers, and many scientific and enginneering disciplines. With a large community of users and developers, matplotlib is approaching the goal of having a full featured, high quality, 2D plotting library for python. A lot of development has gone into matplotlib since the last major release, which I'll summarize here. For details, see the notes for the incremental releases at http://matplotlib.sf.net/whats_new.html. Major changes since matplotlib-0.60 - The alpha version of the users guide - http://matplotlib.sf.net/users_guide.pdf. There are still a number of sections to be completed, but it's a start! - The matlab namespace renamed pylab - if you are upgrading from a version older than 0.64, please remove site-packages/matplotlib before upgrading. See http://matplotlib.sourceforge.net/matlab_to_pylab.py - New plot types: contour plots (contour), polar charts (polar), horizontal bar charts (barh), pie charts (pie), sparse matrix visualization (spy and spy2). Eg, http://matplotlib.sf.net/screenshots.html#polar_demo - Full ipython http://ipython.scipy.org integration in the "pylab" mode for interactive control of matplotlib plots from the python shell. - A significantly improved interactive toolbar for panning, zooming, zoom to rect - see http://matplotlib.sf.net/tutorial.html#toolbar2. - New backends: FLTK, Cairo, GTKCairo - Text - rotated mathtext, mathtext for postscript, text bounding boxes - Colormaps - 14 colormaps built-in http://matplotlib.sf.net/screenshots.html#pcolor_demo - Images - performance optimizations for 4x faster large image handling, PIL support, interpolation and colorbar improvements, imread - Event handling for capturing mouse clicks, movements, keypresses, etc. - same pylab interface works across GUIs. See examples/keypress_demo.py, examples/picker_demo.py, examples/coords_demo.py - set and get matlab style property introspection - http://matplotlib.sf.net/examples/set_and_get.py - improved dates handling for dates and date string formatting from 0000-9999, eg http://matplotlib.sf.net/screenshots.html#finance_work - Be sure to check out the 120 examples at http://matplotlib.sf.net/examples Home page : http://matplotlib.sourceforge.net Downloads : http://sourceforge.net/projects/matplotlib Screenshots : http://matplotlib.sourceforge.net/screenshots.html Tutorial : http://matplotlib.sourceforge.net/tutorial.html Credits : http://matplotlib.sourceforge.net/credits.html John Hunter From Gerald.Rosenfellner at vai.at Thu Dec 2 02:34:01 2004 From: Gerald.Rosenfellner at vai.at (Rosenfellner Gerald VAI.ISC 1) Date: Thu Dec 2 02:34:01 2004 Subject: [Numpy-discussion] Problem with trace function? Message-ID: <2489378D03C2CB45A7CEB4F53F40F93F028BD45C@lnzvt-mxs0001.vaig.vads.cc> Hi all, I have some trouble with the new numarray library. Most of the time the numarray code prints out some negative value whereas the Numeric code yields positive values. This behaviour justs shows up when using large arrays. Can anybody point out what I need to change in the numarray version to get the Numeric behaviour? System: WinXP, python2.3.4, Numeric23.5, numarray1.1.1 ----- NUMARRAY-VERSION: from numarray import greater, reshape, trace from numarray.random_array import standard_normal n=1000 A = greater(standard_normal(n*n), 0.9) A = reshape(A, (n, n)) print trace(A) ----- NUMERIC-Version: from Numeric import greater, reshape, trace from RandomArray import standard_normal n=1000 A = greater(standard_normal(n*n), 0.9) A = reshape(A, (n, n)) print trace(A) ----- Thank you for your help. Gerald -------------- next part -------------- An HTML attachment was scrubbed... URL: From ryorke at telkomsa.net Thu Dec 2 09:57:05 2004 From: ryorke at telkomsa.net (Rory Yorke) Date: Thu Dec 2 09:57:05 2004 Subject: [Numpy-discussion] Re: Problem with trace function? In-Reply-To: <2489378D03C2CB45A7CEB4F53F40F93F028BD45C@lnzvt-mxs0001.vaig.vads.cc> References: <2489378D03C2CB45A7CEB4F53F40F93F028BD45C@lnzvt-mxs0001.vaig.vads.cc> Message-ID: <20041202173122.GA3899@localhost.localdomain> > Most of the time the numarray code prints out some negative value > whereas the Numeric code yields positive values. numarray.greater() returns a Bool array, while Numeric.greater() returns a 32-bit integer array. Try this code: import numarray as na import Numeric as nu print na.greater([0.1,0.9],0.5).type() print nu.greater([0.1,0.9],0.5).typecode() ## prints an 'ell' for n in range(126,131): A=na.ones((n,n)).astype(na.Bool) B=nu.ones((n,n)).astype('1') ## that's a one print n,na.trace(A),nu.trace(B) The problem is overflow in the addition of 8 bit signed integers; it occurs in both numarray and Numeric, given the right array type. The snippet below should give you some idea of how to find what you want: from numarray import * from numarray.random_array import standard_normal A=standard_normal((1000,1000)) > 0.9 print len(nonzero(diagonal(A))[0]) print trace(A.astype(Int32)) Cheers, Rory From florian.proff.schulze at gmx.net Thu Dec 2 12:51:05 2004 From: florian.proff.schulze at gmx.net (Florian Schulze) Date: Thu Dec 2 12:51:05 2004 Subject: [Numpy-discussion] View on same data with different type Message-ID: Hi! I would like to be able to access the same array (memory location) with arrays of different size and with different typecodes. Let's say I have an array of 8 UInt8 and I want to view it as 2 UInt32. I want to be able to change the content of either array and the change should be visible in both arrays. To speak in C notation, I want an *UInt8 and a *Uint32 to the same memory location. Is that possible with Numeric or numarray, maybe even with slices of the same data? The reason I want this is that I want to prevent copying memory around. It would be even cooler if this would work with mmaped arrays, though then it's enough when it would work with read only mmaps. BTW, why isn't it allowed to create overlapping mmap slices? Regards, Florian Schulze From lkemohawk at yahoo.com Thu Dec 2 16:45:06 2004 From: lkemohawk at yahoo.com (kevin lester) Date: Thu Dec 2 16:45:06 2004 Subject: [Numpy-discussion] suppress_small=1 ? Message-ID: <20041203004421.79709.qmail@web53905.mail.yahoo.com> Hi, How can I control my output presentation. There are times when using: print array_str(a,suppress_small=1), does not print a float as intended, but remains to output in exponential form. I am using a win32 os and mostly using PythonWin shell but also tried using wxPython's shell for same results. I tried several other aspects such as sys.float_output_suppress_small to no avail. Another question I have is: performing an operation on an array produces a new array index subscript. Is it possible to re-assign the original index subscript to an array after performing the operation. (tunnel vision afflicted...lol) Thank you very much, Kevin __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From florian.proff.schulze at gmx.net Fri Dec 3 06:54:02 2004 From: florian.proff.schulze at gmx.net (Florian Schulze) Date: Fri Dec 3 06:54:02 2004 Subject: [Numpy-discussion] Re: View on same data with different type References: Message-ID: On Thu, 02 Dec 2004 21:49:10 +0100, Florian Schulze wrote: > Hi! > > I would like to be able to access the same array (memory location) with > arrays of different size and with different typecodes. Let's say I have > an array of 8 UInt8 and I want to view it as 2 UInt32. I want to be able > to change the content of either array and the change should be visible > in both arrays. To speak in C notation, I want an *UInt8 and a *Uint32 > to the same memory location. > Is that possible with Numeric or numarray, maybe even with slices of the > same data? > The reason I want this is that I want to prevent copying memory around. > It would be even cooler if this would work with mmaped arrays, though > then it's enough when it would work with read only mmaps. BTW, why isn't > it allowed to create overlapping mmap slices? > > Regards, > Florian Schulze Hi again! Todd Miller explained me how to do it (though it wasn't working as is, I figured it out). As I think it is interesting to other people, I post it here: >>> import numarray >>> a = numarray.arrayrange(12, type='i1') # We can now create an array pointing to the same memory location with different type: >>> b = numarray.array(sequence=a._data, shape=(3,), type='i4') # The key here is to use a._data. You have to specify the shape, or else it doesn't work. >>> a array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11], type=Int8) >>> b array([ 50462976, 117835012, 185207048]) >>> a[3] = 10 >>> a array([ 0, 1, 2, 10, 4, 5, 6, 7, 8, 9, 10, 11], type=Int8) >>> b array([167903488, 117835012, 185207048]) # Here you see how the assignment to a also changed b. >>> a._data >>> b._data # Here you see that both arrays really point to the same memory location. I tried it with numarray.records.array and it's basically the same, though you have to use 'buffer' instead of 'sequence'. I will have to try it out with mmap, but Todd told me it should be no problem. He also told me it could also be done with views by calling .view() on the array and then set the other values (_type, _shape, _itemsize, [_strides, _bytestride]) but I think this is more error prone and I didn't test it. Thanks again to Todd Miller. Regards, Florian Schulze From jmiller at stsci.edu Fri Dec 3 07:46:02 2004 From: jmiller at stsci.edu (Todd Miller) Date: Fri Dec 3 07:46:02 2004 Subject: [Fwd: Re: [Numpy-discussion] View on same data with different type] Message-ID: <1102088728.28877.350.camel@halloween.stsci.edu> Once again I forgot to reply-all... Florian apparently figured out whatever flaws were in my original message attached here. One thing people interested in this kind of data aliasing should keep in mind is that all of the hidden variables I mentioned (in the attached) exist to make things like array slices and record arrays work properly. Just reusing the _data, by itself, is not sufficient to work *in general* because not all of the _data is always used. It does work for the relatively simple case of a new array, but might fail if the array came from a field of a recarray, a numerical array slice, a transposed numerical array, etc. Be careful. Regards, Todd -------------- next part -------------- An embedded message was scrubbed... From: unknown sender Subject: no subject Date: no date Size: 38 URL: From jmiller at stsci.edu Thu Dec 2 16:25:43 2004 From: jmiller at stsci.edu (Todd Miller) Date: 02 Dec 2004 16:25:43 -0500 Subject: [Numpy-discussion] View on same data with different type In-Reply-To: References: Message-ID: <1102022743.28877.240.camel@halloween.stsci.edu> On Thu, 2004-12-02 at 15:49, Florian Schulze wrote: > Hi! > > I would like to be able to access the same array (memory location) with > arrays of different size and with different typecodes. Let's say I have an > array of 8 UInt8 and I want to view it as 2 UInt32. I want to be able to > change the content of either array and the change should be visible in > both arrays. To speak in C notation, I want an *UInt8 and a *Uint32 to the > same memory location. > Is that possible with Numeric or numarray, maybe even with slices of the > same data? In numarray, it is definitely possible to do what you want (in Python) by accessing private attributes of the array: _data, _byteoffset, _shape, _strides, _bytestride, _itemsize. Given an array A, one approach would be to create an array B using buffer sharing and the NumArray constructor like this: >>> A = arange(100, type='Int8') >>> B = NumArray(shape=(25,), buffer=A._data, type='Int32') Another approach would be to use a view like this: >>> B = A.view() >>> B._type = Int32 >>> B._shape = (25,) >>> B._itemsize = 4 # >>> B._strides = (4,) # >>> B._bytestride = 4 > The reason I want this is that I want to prevent copying memory around. It > would be even cooler if this would work with mmaped arrays, though then > it's enough when it would work with read only mmaps. BTW, why isn't it > allowed to create overlapping mmap slices? IIRC, since mmap slices are resizable, permitting overlaps would have complicated things... so without a compelling need, I/we decided to KISS. The kind of re-typing I showed above works for mmaps too... there's no need to overlap since the same slice can be used in two different arrays. Todd --=-fvu/cVmG8DY6+oRhVZR/-- From vivekrao4 at yahoo.com Fri Dec 3 19:06:03 2004 From: vivekrao4 at yahoo.com (Vivek Rao) Date: Fri Dec 3 19:06:03 2004 Subject: [Numpy-discussion] Numarray 1.1.1 with Python 2.4 #60 crashes on Windows XP Pro Message-ID: <20041204030532.93719.qmail@web53410.mail.yahoo.com> I have installed numarray 1.1.1 for Python 2.4. When I go to C:\Python24\Lib\site-packages\numarray and type 'python numtest.py' a crash occurs. Debugging it Microsoft Visual Studio, the error message is "unhandled exception in python.exe (Python24.dll): 0xC0000005 Access Violation" The Python version I am using is Python 2.4 (#60, Nov 30 2004, 11:49:19) [MSC v.1310 32 bit (Intel)] on win32 . Is this a known problem? Vivek Rao From faltet at carabos.com Sat Dec 4 03:08:01 2004 From: faltet at carabos.com (Francesc Altet) Date: Sat Dec 4 03:08:01 2004 Subject: [Numpy-discussion] ANN: PyTables 0.9.1 released Message-ID: <200412041207.04631.faltet@carabos.com> Announcing PyTables 0.9.1 ------------------------- This release is mainly a maintenance version. In it, some bugs has been fixed and a few improvements has been made. One important thing is that chunk sizes in EArrays has been re-tuned to get much better performance and compression rations. Besides, it has been tested against the latest Python 2.4 and all test units seems to pass fine. What it is ---------- PyTables is a solid hierarchical database package designed to efficiently manage extremely large amounts of data (with support for full 64-bit file addressing). It features an object-oriented interface that, combined with C extensions for the performance-critical parts of the code, makes it a very easy-to-use tool for high performance data storage and retrieval. It is built on top of the HDF5 library and the numarray package, and provides containers for both heterogeneous data (Tables) and homogeneous data (Array, EArray) as well as containers for keeping lists of objects of variable length (like Unicode strings or general Python objects) in a very efficient way (VLArray). It also sports a series of filters allowing you to compress your data on-the-fly by using different compressors and compression enablers. But perhaps the more interesting features are its powerful browsing and searching capabilities that allow you to perform data selections over heterogeneous datasets exceeding gigabytes of data in just tenths of second. Besides, all the PyTables I/O is buffered, implemented in C and carefully tuned so that you can reach much better performance with PyTables than with your own home-grown wrappings to the HDF5 library. Changes more in depth --------------------- Improvements: - The chunksize computation for EArrays has been re-tuned to allow better performance and *much* better compression rations. - New --unpackshort and --quantize flags has been added to nctoh5 script. --unpackshort unpack short integer variables to float variables using scale_factor and add_offset netCDF variable attributes. --quantize quantize data to improve compression using least_significant_digit netCDF variable attribute (not active by default). See http://www.cdc.noaa.gov/cdc/conventions/cdc_netcdf_standard.shtml for further explanation of what this attribute means. Thanks to Jeff Whitaker for providing this. - Table.itersequence has received a new parameter called "sort". This allows to disable the sorting of the sequence in case the user wants so. Backward-incompatible changes: - Now, the AttributeSet class throw an AttributeError on __getattr__ for nonexistent attributes in it. Formerly, the routine returned None, which is pretty much against convention in Python and breaks the built-in hasattr() function. Thanks to Norbert Nemec for noting this and offering a patch. - VLArray.read() has changed its behaviour. Now, it always returns a list, as stated in documentation, even when the number of elements to return is 0 or 1. This is much more consistent when representing the actual number of elements on a certain VLArray row. API additions: - A Row.getTable() has been added. It is an accessor for the associated Table object. - A File.copyAttrs() has been added. It allows copying attributes from one leaf to other. Properly speaking, this was already there, but not documented :-/ Bug fixes: - Now, the copy of hierarchies works even when there are scalar Arrays (i.e. Arrays which shape is ()) on it. Thanks to Norbert Nemec for providing a patch. - Solved a memory leak regarding the Filters instance associated with the File object, that was not released after closing the file. Now, there are no known leaks on PyTables itself. - Fixed a bug in Table.append() when the table was indexed. The problem was that if table was in auto-indexing mode, some rows were lost in the indexation process and hence, not indexed correctly. - Improved security of nodes name checking. Closes #1074335 Important note for Python 2.4 and Windows users ----------------------------------------------- If you are willing to use PyTables with Python 2.4 in Windows platforms, you will need to get the HDF5 library compiled for MSVC 7.1, aka .NET (and possible LZO and UCL as well, if you want support for LZO and UCL at all). It can be found at: ftp://ftp.ncsa.uiuc.edu/HDF/HDF5/current/bin/windows/5-163-winxp-net2003.zip Where can PyTables be applied? ------------------------------ PyTables is not designed to work as a relational database competitor, but rather as a teammate. If you want to work with large datasets of multidimensional data (for example, for multidimensional analysis), or just provide a categorized structure for some portions of your cluttered RDBS, then give PyTables a try. It works well for storing data from data acquisition systems (DAS), simulation software, network data monitoring systems (for example, traffic measurements of IP packets on routers), very large XML files, or for creating a centralized repository for system logs, to name only a few possible uses. What is a table? ---------------- A table is defined as a collection of records whose values are stored in fixed-length fields. All records have the same structure and all values in each field have the same data type. The terms "fixed-length" and "strict data types" seem to be quite a strange requirement for a language like Python that supports dynamic data types, but they serve a useful function if the goal is to save very large quantities of data (such as is generated by many scientific applications, for example) in an efficient manner that reduces demand on CPU time and I/O resources. What is HDF5? ------------- For those people who know nothing about HDF5, it is a general purpose library and file format for storing scientific data made at NCSA. HDF5 can store two primary objects: datasets and groups. A dataset is essentially a multidimensional array of data elements, and a group is a structure for organizing objects in an HDF5 file. Using these two basic constructs, one can create and store almost any kind of scientific data structure, such as images, arrays of vectors, and structured and unstructured grids. You can also mix and match them in HDF5 files according to your needs. Platforms --------- I'm using Linux (Intel 32-bit) as the main development platform, but PyTables should be easy to compile/install on many other UNIX machines. This package has also passed all the tests on a UltraSparc platform with Solaris 7 and Solaris 8. It also compiles and passes all the tests on a SGI Origin2000 with MIPS R12000 processors, with the MIPSPro compiler and running IRIX 6.5. It also runs fine on Linux 64-bit platforms, like AMD Opteron running GNU/Linux 2.4.21 Server, Intel Itanium (IA64) running GNU/Linux 2.4.21 or PowerPC G5 with Linux 2.6.x in 64bit mode. It has also been tested in MacOSX platforms (10.2 but should also work on newer versions). Regarding Windows platforms, PyTables has been tested with Windows 2000 and Windows XP (using the Microsoft Visual C compiler), but it should also work with other flavors as well. Web site -------- Go to the PyTables web site for more details: http://pytables.sourceforge.net/ To know more about the company behind the PyTables development, see: http://www.carabos.com/ Share your experience --------------------- Let me know of any bugs, suggestions, gripes, kudos, etc. you may have. Enjoy data! -- Francesc Altet Who's your data daddy? ?PyTables From jmiller at stsci.edu Sat Dec 4 05:49:05 2004 From: jmiller at stsci.edu (Todd Miller) Date: Sat Dec 4 05:49:05 2004 Subject: [Numpy-discussion] Numarray 1.1.1 with Python 2.4 #60 crashes on Windows XP Pro In-Reply-To: <20041204030532.93719.qmail@web53410.mail.yahoo.com> References: <20041204030532.93719.qmail@web53410.mail.yahoo.com> Message-ID: <1102167843.3372.3.camel@jaytmiller.comcast.net> On Fri, 2004-12-03 at 19:05 -0800, Vivek Rao wrote: > I have installed numarray 1.1.1 for Python 2.4. When I > go to C:\Python24\Lib\site-packages\numarray and type > 'python numtest.py' a crash occurs. Debugging it > Microsoft Visual Studio, the error message is > > "unhandled exception in python.exe (Python24.dll): > 0xC0000005 Access Violation" > > The Python version I am using is > > Python 2.4 (#60, Nov 30 2004, 11:49:19) [MSC v.1310 32 > bit (Intel)] on win32 . > > Is this a known problem? Yes and no. In my experience, it's best not to run modules that are part of a Python package from within the package. To run the selftest, I recommend doing something like this: > cd C:\ > python >>> import numarray.testall as testall >>> testall.test() Regards, Todd From rbastian at club-internet.fr Sat Dec 4 06:54:06 2004 From: rbastian at club-internet.fr (=?iso-8859-15?q?Ren=E9=20Bastian?=) Date: Sat Dec 4 06:54:06 2004 Subject: [Numpy-discussion] segm fault Message-ID: <04120407473000.00776@rbastian> Hi, i use numarray 1.1.1 with python2.3b1 import RandomArray2 as R w=R.uniform(-1., +1., 44100) produces "Segmentation fault" i never meet that. thanks for your help -- Ren? Bastian http://www.musiques-rb.org : Musique en Python From jmiller at stsci.edu Sat Dec 4 07:46:03 2004 From: jmiller at stsci.edu (Todd Miller) Date: Sat Dec 4 07:46:03 2004 Subject: [Numpy-discussion] segm fault In-Reply-To: <04120407473000.00776@rbastian> References: <04120407473000.00776@rbastian> Message-ID: <1102175126.3372.13.camel@jaytmiller.comcast.net> On Sat, 2004-12-04 at 07:47 +0100, Ren? Bastian wrote: > Hi, > > i use numarray 1.1.1 with python2.3b1 > > import RandomArray2 as R > w=R.uniform(-1., +1., 44100) > > produces "Segmentation fault" > > i never meet that. > thanks for your help > This looks like either you've got remanents of a really old numarray installed or you're trying to run from within the numarray package directory. In the numarray-speak of today, you'd write the import as: import numarray.random_array as R If you're running from the numarray package directory, don't. If you think maybe you've got old numarray stuff installed, figure out where RandomArray2 is coming from and delete it; use numarray.random_array. Regards, Todd From jmiller at stsci.edu Sat Dec 4 07:47:00 2004 From: jmiller at stsci.edu (Todd Miller) Date: Sat Dec 4 07:47:00 2004 Subject: [Numpy-discussion] segm fault In-Reply-To: <04120407473000.00776@rbastian> References: <04120407473000.00776@rbastian> Message-ID: <1102175192.3372.15.camel@jaytmiller.comcast.net> On Sat, 2004-12-04 at 07:47 +0100, Ren? Bastian wrote: > Hi, > > i use numarray 1.1.1 with python2.3b1 And... upgrade to Python-2.3.4 or Python-2.4. From stephen.walton at csun.edu Mon Dec 6 16:07:18 2004 From: stephen.walton at csun.edu (Stephen Walton) Date: Mon Dec 6 16:07:18 2004 Subject: [Numpy-discussion] numarray vs IPython Message-ID: <1102377944.8589.27.camel@freyer.sfo.csun.edu> When I run numarray.testall.test() from IPython, I get a slew of error messages instead of the successful test output I get when the same version is run from the stock Python interpreter. Anyone else see this? From ariciputi at pito.com Tue Dec 7 00:55:08 2004 From: ariciputi at pito.com (Andrea Riciputi) Date: Tue Dec 7 00:55:08 2004 Subject: [Numpy-discussion] numarray vs IPython In-Reply-To: <1102377944.8589.27.camel@freyer.sfo.csun.edu> References: <1102377944.8589.27.camel@freyer.sfo.csun.edu> Message-ID: <9E1F0936-482D-11D9-8F12-000A95C0BC68@pito.com> The same here (OS X, Python 2.3.4, ipython 0.6.5). The problem is due to the way in which numarray performs its tests. It relies on the Python module doctest that execute the script in doc strings. I think that since in numarray doc strings the expected output is written in the standard Python way (i.e. >>> foo) while ipython writes out something like "Out[n]: foo" the checks fail. HTH, Andrea. On 7 Dec 2004, at 01:05, Stephen Walton wrote: > When I run numarray.testall.test() from IPython, I get a slew of error > messages instead of the successful test output I get when the same > version is run from the stock Python interpreter. Anyone else see > this? From verveer at embl-heidelberg.de Tue Dec 7 01:13:20 2004 From: verveer at embl-heidelberg.de (Peter Verveer) Date: Tue Dec 7 01:13:20 2004 Subject: [Numpy-discussion] numarray vs IPython In-Reply-To: <9E1F0936-482D-11D9-8F12-000A95C0BC68@pito.com> References: <1102377944.8589.27.camel@freyer.sfo.csun.edu> <9E1F0936-482D-11D9-8F12-000A95C0BC68@pito.com> Message-ID: <2628F06A-4830-11D9-B2CB-000A95C92C8E@embl-heidelberg.de> This could be true, I noticed that the tests for the nd_image package work fine, and those are written using unittest, which does not rely on the exact textual form of the expected output. On 7 Dec 2004, at 09:54, Andrea Riciputi wrote: > The same here (OS X, Python 2.3.4, ipython 0.6.5). The problem is due > to the way in which numarray performs its tests. It relies on the > Python module doctest that execute the script in doc strings. I think > that since in numarray doc strings the expected output is written in > the standard Python way (i.e. >>> foo) while ipython writes out > something like "Out[n]: foo" the checks fail. > > HTH, > Andrea. > > > On 7 Dec 2004, at 01:05, Stephen Walton wrote: > >> When I run numarray.testall.test() from IPython, I get a slew of error >> messages instead of the successful test output I get when the same >> version is run from the stock Python interpreter. Anyone else see >> this? > > > > ------------------------------------------------------- > SF email is sponsored by - The IT Product Guide > Read honest & candid reviews on hundreds of IT Products from real > users. > Discover which products truly live up to the hype. Start reading now. > http://productguide.itmanagersjournal.com/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From Fernando.Perez at colorado.edu Tue Dec 7 11:06:02 2004 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Tue Dec 7 11:06:02 2004 Subject: [Numpy-discussion] numarray vs IPython In-Reply-To: <9E1F0936-482D-11D9-8F12-000A95C0BC68@pito.com> References: <1102377944.8589.27.camel@freyer.sfo.csun.edu> <9E1F0936-482D-11D9-8F12-000A95C0BC68@pito.com> Message-ID: <41B5FEE9.7050704@colorado.edu> Andrea Riciputi schrieb: > The same here (OS X, Python 2.3.4, ipython 0.6.5). The problem is due > to the way in which numarray performs its tests. It relies on the > Python module doctest that execute the script in doc strings. I think > that since in numarray doc strings the expected output is written in > the standard Python way (i.e. >>> foo) while ipython writes out > something like "Out[n]: foo" the checks fail. Thanks for that info, this is most likely the culprit. Here's a simple test to confirm things: start ipython as 'ipython --classic', which disables the fancy prompt system and makes it produce regular python prompts. Run the tests there and see what happens. Since numarray is more and more likely to be run by ipython users, it might be worth adding a bit of robustness to these tests when under ipython. Here are some suggestions, starting with the simplest: 1. Run a first 'empty' test just to do prompt detection. If this one fails, do: try: __IPYTHON__ except NameError: print 'Big time problem, not caused by ipython' else: print 'Run ipython --classic for doctests to work' or some fancier version of the above. 2. Even better, patch doctest to recognize prompts based on a regexp. The python-mode.el guys did this for (x)emacs support, and it now works very well with ipython. The python-mode CVS code can be studied for the proper regexps. By default this regexp can be set to recognize the normal ipython prompts. 3. If you do 2, and you want to get real fancy, the numarray doctest could build the regexp automatically, from the ipython prompt configuration strings. These are always available at runtime: In [1]: __IPYTHON__.rc.prom __IPYTHON__.rc.prompt_in1 __IPYTHON__.rc.prompt_in2 __IPYTHON__.rc.prompt_out In [1]: __IPYTHON__.rc.prompt_in1 Out[1]: 'In [\\#]:' In [2]: __IPYTHON__.rc.prompt_in2 Out[2]: ' .\\D.:' In [3]: __IPYTHON__.rc.prompt_out Out[3]: 'Out[\\#]:' So the situation looks not too bad. There's a quick fix (--classic) for now, and if someone feels up to the task of patching doctest, a pretty elegant solution can be implemented. Regards, f From jmiller at stsci.edu Tue Dec 7 12:41:02 2004 From: jmiller at stsci.edu (Todd Miller) Date: Tue Dec 7 12:41:02 2004 Subject: [Numpy-discussion] numarray vs IPython In-Reply-To: <41B5FEE9.7050704@colorado.edu> References: <1102377944.8589.27.camel@freyer.sfo.csun.edu> <9E1F0936-482D-11D9-8F12-000A95C0BC68@pito.com> <41B5FEE9.7050704@colorado.edu> Message-ID: <1102452001.17245.276.camel@halloween.stsci.edu> I wanted to let you guys know that there's a "post-it note" stuck to my monitor to remind me to look at this. A quick glance indicated that --classic does not fix the problem... but that's all I know so far and all I'll get to in the immediate future. Regards, Todd On Tue, 2004-12-07 at 14:05, Fernando Perez wrote: > Andrea Riciputi schrieb: > > The same here (OS X, Python 2.3.4, ipython 0.6.5). The problem is due > > to the way in which numarray performs its tests. It relies on the > > Python module doctest that execute the script in doc strings. I think > > that since in numarray doc strings the expected output is written in > > the standard Python way (i.e. >>> foo) while ipython writes out > > something like "Out[n]: foo" the checks fail. > > Thanks for that info, this is most likely the culprit. Here's a simple test > to confirm things: start ipython as 'ipython --classic', which disables the > fancy prompt system and makes it produce regular python prompts. Run the > tests there and see what happens. > > Since numarray is more and more likely to be run by ipython users, it might be > worth adding a bit of robustness to these tests when under ipython. Here are > some suggestions, starting with the simplest: > > 1. Run a first 'empty' test just to do prompt detection. If this one fails, do: > > try: > __IPYTHON__ > except NameError: > print 'Big time problem, not caused by ipython' > else: > print 'Run ipython --classic for doctests to work' > > or some fancier version of the above. > > 2. Even better, patch doctest to recognize prompts based on a regexp. The > python-mode.el guys did this for (x)emacs support, and it now works very well > with ipython. The python-mode CVS code can be studied for the proper regexps. > By default this regexp can be set to recognize the normal ipython prompts. > > 3. If you do 2, and you want to get real fancy, the numarray doctest could > build the regexp automatically, from the ipython prompt configuration strings. > These are always available at runtime: > > In [1]: __IPYTHON__.rc.prom > __IPYTHON__.rc.prompt_in1 __IPYTHON__.rc.prompt_in2 __IPYTHON__.rc.prompt_out > > In [1]: __IPYTHON__.rc.prompt_in1 > Out[1]: 'In [\\#]:' > > In [2]: __IPYTHON__.rc.prompt_in2 > Out[2]: ' .\\D.:' > > In [3]: __IPYTHON__.rc.prompt_out > Out[3]: 'Out[\\#]:' > > > So the situation looks not too bad. There's a quick fix (--classic) for now, > and if someone feels up to the task of patching doctest, a pretty elegant > solution can be implemented. > > Regards, > > f > > > ------------------------------------------------------- > SF email is sponsored by - The IT Product Guide > Read honest & candid reviews on hundreds of IT Products from real users. > Discover which products truly live up to the hype. Start reading now. > http://productguide.itmanagersjournal.com/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- From Fernando.Perez at colorado.edu Tue Dec 7 12:45:04 2004 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Tue Dec 7 12:45:04 2004 Subject: [Numpy-discussion] numarray vs IPython In-Reply-To: <1102452001.17245.276.camel@halloween.stsci.edu> References: <1102377944.8589.27.camel@freyer.sfo.csun.edu> <9E1F0936-482D-11D9-8F12-000A95C0BC68@pito.com> <41B5FEE9.7050704@colorado.edu> <1102452001.17245.276.camel@halloween.stsci.edu> Message-ID: <41B6162A.1000403@colorado.edu> Todd Miller schrieb: > I wanted to let you guys know that there's a "post-it note" stuck to my > monitor to remind me to look at this. A quick glance indicated that > --classic does not fix the problem... but that's all I know so far and > all I'll get to in the immediate future. Bummer. I would have thought that --classic should have been enough. Sorry not to be able to test myself, but I don't have numarray installed at all. Thanks for looking into it, and please let me know if I can help in any way. Regards f From klimek at grc.nasa.gov Tue Dec 7 15:19:17 2004 From: klimek at grc.nasa.gov (Bob Klimek) Date: Tue Dec 7 15:19:17 2004 Subject: [Numpy-discussion] binary thinning Message-ID: <41B63AA8.5060205@grc.nasa.gov> Hi, I've been playing around with the binary_hit_or_miss function in the nd_image and it seems to work well in general. I understand that binary thinning can be generated by using hit_or_miss, however I can't seem to get it to work right. Anyone have a code snippet that does the job? I follow the procedure at: http://homepages.inf.ed.ac.uk/rbf/HIPR2/thin.htm My implementation is as follows: ## from HIPR2 web site ## [0,0,0] ## [x,1,x] ## [1,1,1] struct1 = numarray.array( [[0, 0, 0], [0, 1, 0], [1, 1, 1]]) struct2 = numarray.array( [[1, 1, 1], [0, 0, 0], [0, 0, 0]]) ## [x,0,0] ## [1,1,0] ## [x,1,x] struct3 = numarray.array( [[0, 0, 0], [1, 1, 0], [0, 1, 0]]) struct4 = numarray.array( [[0, 1, 1], [0, 0, 1], [0, 0, 0]]) b = [] for i in range(4): temp = ND.binary_hit_or_miss(a, struct1, struct2) b.append(temp) temp = ND.binary_hit_or_miss(a, struct3, struct4) b.append(temp) struct1 = ND.rotate(struct1, 90) struct2 = ND.rotate(struct2, 90) struct3 = ND.rotate(struct3, 90) struct4 = ND.rotate(struct4, 90) result = b[0] for i in range(7): result = numarray.logical_or(result, b[i+1]) result = a - result Bob From verveer at embl-heidelberg.de Wed Dec 8 03:12:04 2004 From: verveer at embl-heidelberg.de (Peter Verveer) Date: Wed Dec 8 03:12:04 2004 Subject: [Numpy-discussion] binary thinning In-Reply-To: <41B63AA8.5060205@grc.nasa.gov> References: <41B63AA8.5060205@grc.nasa.gov> Message-ID: Hi Bob, I noticed two problems: 1) you construct rotations of your structures with the nd_image.rotate function. That will not work properly because this function uses spline interpolation, which not necessarily produces exact results. In this case you need to use another, exact rotation method, see my implementation below. 2) I think (correct me if I am wrong) that the application of the 8 structures is sequential, not parallel as you do it. Below I give my implementation, which does not produce the exact same result as shown on the webpage you refer to, although the result seems to be a proper skeleton. This may be because the order in which the structures are applied matters (I think). Could you maybe test that by permutating the order in which the structures are applied? I would be interested if this is true and if one can get the same result as on the webpage by finding the right sequence... Cheers, Peter import numarray from numarray import nd_image image = numarray.array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0], [0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0], [0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0], [0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0], [0, 0, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0], [0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0], [0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0], [0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0], [0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0], [0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0], [0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]) hit1 = numarray.array([[0, 0, 0], [0, 1, 0], [1, 1, 1]]) miss1 = numarray.array([[1, 1, 1], [0, 0, 0], [0, 0, 0]]) hit2 = numarray.array([[0, 0, 0], [1, 1, 0], [0, 1, 0]]) miss2 = numarray.array([[0, 1, 1], [0, 0, 1], [0, 0, 0]]) hit_list = [hit1, hit2] miss_list = [miss1, miss2] for ii in range(6): hit_list.append(numarray.transpose(hit_list[-2])[::-1, ...]) miss_list.append(numarray.transpose(miss_list[-2])[::-1, ...]) while 1: last = image for hit, miss in zip(hit_list, miss_list): hm = nd_image.binary_hit_or_miss(image, hit, miss) image = numarray.logical_and(image, numarray.logical_not(hm)) if numarray.abs(last - image).max() == 0: break print image On 8 Dec 2004, at 00:20, Bob Klimek wrote: > Hi, > > I've been playing around with the binary_hit_or_miss function in the > nd_image and it seems to work well in general. I understand that > binary thinning can be generated by using hit_or_miss, however I can't > seem to get it to work right. Anyone have a code snippet that does the > job? > > I follow the procedure at: > > http://homepages.inf.ed.ac.uk/rbf/HIPR2/thin.htm > > My implementation is as follows: > > ## from HIPR2 web site > ## [0,0,0] > ## [x,1,x] > ## [1,1,1] > struct1 = numarray.array( > [[0, 0, 0], > [0, 1, 0], > [1, 1, 1]]) > struct2 = numarray.array( > [[1, 1, 1], > [0, 0, 0], > [0, 0, 0]]) > > ## [x,0,0] > ## [1,1,0] > ## [x,1,x] > struct3 = numarray.array( > [[0, 0, 0], > [1, 1, 0], > [0, 1, 0]]) > struct4 = numarray.array( > [[0, 1, 1], > [0, 0, 1], > [0, 0, 0]]) > > b = [] > for i in range(4): > temp = ND.binary_hit_or_miss(a, struct1, struct2) > b.append(temp) > temp = ND.binary_hit_or_miss(a, struct3, struct4) > b.append(temp) > > struct1 = ND.rotate(struct1, 90) > struct2 = ND.rotate(struct2, 90) > struct3 = ND.rotate(struct3, 90) > struct4 = ND.rotate(struct4, 90) > > result = b[0] > for i in range(7): > result = numarray.logical_or(result, b[i+1]) > > result = a - result > > > Bob > > > > ------------------------------------------------------- > SF email is sponsored by - The IT Product Guide > Read honest & candid reviews on hundreds of IT Products from real > users. > Discover which products truly live up to the hype. Start reading now. > http://productguide.itmanagersjournal.com/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From klimek at grc.nasa.gov Wed Dec 8 15:23:06 2004 From: klimek at grc.nasa.gov (Bob Klimek) Date: Wed Dec 8 15:23:06 2004 Subject: [Numpy-discussion] binary thinning In-Reply-To: References: <41B63AA8.5060205@grc.nasa.gov> Message-ID: <41B78D26.1060007@grc.nasa.gov> Peter Verveer wrote: > Hi Bob, > > I noticed two problems: > > 1) you construct rotations of your structures with the nd_image.rotate > function. That will not work properly because this function uses > spline interpolation, which not necessarily produces exact results. In > this case you need to use another, exact rotation method, see my > implementation below. > > 2) I think (correct me if I am wrong) that the application of the 8 > structures is sequential, not parallel as you do it. > > Below I give my implementation, which does not produce the exact same > result as shown on the webpage you refer to, although the result seems > to be a proper skeleton. This may be because the order in which the > structures are applied matters (I think). Could you maybe test that by > permutating the order in which the structures are applied? I would be > interested if this is true and if one can get the same result as on > the webpage by finding the right sequence... Peter, Thanks much for getting back to me on this. Your sample code was very helpful! :-) I rewrote my function based on yours and I'm getting the same answer. So far I haven't been able to figure out why is it different from the web site and so far I haven't tried changing the order of the structuring elements. I'll try tomorrow. I'll let you know if I come up with anything new. Regards, Bob From cjw at sympatico.ca Thu Dec 9 14:39:03 2004 From: cjw at sympatico.ca (Colin J. Williams) Date: Thu Dec 9 14:39:03 2004 Subject: [Numpy-discussion] NumArray - take method Message-ID: <41B8D3E3.9060302@sympatico.ca> This undocumented method has puzzling behaviour. [Dbg]>>> a array([[0, 1, 2], [3, 4, 5], [6, 7, 8]]) [Dbg]>>> b= a[1:1] [Dbg]>>> b array([]) [Dbg]>>> b.shape (0, 3) [Dbg]>>> b.rank 2 I had expected a rank of zero and an indeterminate shape. Colin W. From tkorvola at e.math.helsinki.fi Sat Dec 11 05:26:12 2004 From: tkorvola at e.math.helsinki.fi (Timo Korvola) Date: Sat Dec 11 05:26:12 2004 Subject: [Numpy-discussion] NumArray - take method In-Reply-To: <41B8D3E3.9060302@sympatico.ca> (Colin J. Williams's message of "Thu, 09 Dec 2004 17:38:27 -0500") References: <41B8D3E3.9060302@sympatico.ca> Message-ID: "Colin J. Williams" writes: > I had expected a rank of zero and an indeterminate shape. Rank zero would imply shape () and that the result is a scalar. Empty arrays have positive rank but one or more of the dimensions are zero. It seems that there is a bug in repr as it does not distinguish between empty arrays of different shapes. The current behaviour is consistent in the sense that indexing by a slice does not change rank, not even if the slice has length zero or one (a slice of length one is thus different from an integer). Only the indexed dimension changes. -- Timo Korvola From cjw at sympatico.ca Sat Dec 11 08:20:00 2004 From: cjw at sympatico.ca (Colin J. Williams) Date: Sat Dec 11 08:20:00 2004 Subject: [Numpy-discussion] NumArray - take method In-Reply-To: References: <41B8D3E3.9060302@sympatico.ca> Message-ID: <41BB1E2B.8030705@sympatico.ca> Timo Korvola wrote: >"Colin J. Williams" writes: > > >>I had expected a rank of zero and an indeterminate shape. >> >> > >Rank zero would imply shape () and that the result is a scalar. Empty >arrays have positive rank but one or more of the dimensions are zero. >It seems that there is a bug in repr as it does not distinguish >between empty arrays of different shapes. > > Perhaps the problem is in str: >>> b= a[:,1:1] >>> b.shape (3, 0) >>> str(b) '[]' >>> repr(b) 'array([])' >The current behaviour is consistent in the sense that indexing by a >slice does not change rank, not even if the slice has length zero or >one (a slice of length one is thus different from an integer). Only >the indexed dimension changes. > > > Yes, you are right. The new docs are a great step forward but it would be good to include the take method. Colin W. From ivilata at carabos.com Mon Dec 13 03:40:04 2004 From: ivilata at carabos.com (Ivan Vilata i Balaguer) Date: Mon Dec 13 03:40:04 2004 Subject: [Numpy-discussion] Using new numeric instances in NumArray and RecArray. Message-ID: <20041213113929.GR4752@orm-embar.terramar.selidor.net> Hi all! I am trying to define new numeric types to use them in NumArray and RecArray objects. My intention is to define them as equivalent to other types, but also make possible to distinguish them. I tried "MyInt32 = IntegralType(4, 'MyType')", which registered OK, is equivalent to Int32, but distinguishable from it. However, I am not allowe to create a NumArray based on it: "a = array([1,2], MyInt32)" fails with "RuntimeError: _numarray_init: can't get typeno for type". Is it possible to integrate new types seamlessly into NumArray and RecArray objects? Is this the right approach, or am I getting that wrong? Well, thanks a lot for your cooperation. PS: Please CC: me with your replies, since I am not registered in the list. Thanks again. -- Ivan Vilata i Balaguer >qo< http://www.carabos.com/ C?rabos Coop. V. V V Enjoy Data "" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From jmiller at stsci.edu Mon Dec 13 04:54:00 2004 From: jmiller at stsci.edu (Todd Miller) Date: Mon Dec 13 04:54:00 2004 Subject: [Numpy-discussion] Using new numeric instances in NumArray and RecArray. In-Reply-To: <20041213113929.GR4752@orm-embar.terramar.selidor.net> References: <20041213113929.GR4752@orm-embar.terramar.selidor.net> Message-ID: <1102942407.4631.9.camel@jaytmiller.comcast.net> On Mon, 2004-12-13 at 12:39 +0100, Ivan Vilata i Balaguer wrote: > Hi all! I am trying to define new numeric types to use them in > NumArray and RecArray objects. My intention is to define them as > equivalent to other types, but also make possible to distinguish them. > I tried "MyInt32 = IntegralType(4, 'MyType')", which registered OK, is > equivalent to Int32, but distinguishable from it. However, I am not > allowe to create a NumArray based on it: "a = array([1,2], MyInt32)" > fails with "RuntimeError: _numarray_init: can't get typeno for type". > > Is it possible to integrate new types seamlessly into NumArray and > RecArray objects? I don't know. No one else has asked for it yet so you're in unfamiliar territory. It does seem (to me anyway) like it would complicate an already complex system. What are you trying to do? > Is this the right approach, or am I getting that > wrong? The system as it stands is not designed for that. Regards, Todd From ivilata at carabos.com Mon Dec 13 08:24:11 2004 From: ivilata at carabos.com (Ivan Vilata i Balaguer) Date: Mon Dec 13 08:24:11 2004 Subject: [Numpy-discussion] Using new numeric instances in NumArray and RecArray. In-Reply-To: <1102942407.4631.9.camel@jaytmiller.comcast.net> References: <20041213113929.GR4752@orm-embar.terramar.selidor.net> <1102942407.4631.9.camel@jaytmiller.comcast.net> Message-ID: <20041213162320.GT4752@orm-embar.terramar.selidor.net> On Mon, Dec 13, 2004 at 07:53:27AM -0500, Todd Miller wrote: > On Mon, 2004-12-13 at 12:39 +0100, Ivan Vilata i Balaguer wrote: > [...] > > Is it possible to integrate new types seamlessly into NumArray and > > RecArray objects? > > I don't know. No one else has asked for it yet so you're in unfamiliar > territory. It does seem (to me anyway) like it would complicate an > already complex system. What are you trying to do? I'm trying to define new types that can be handled as normal types in numarray, but being different (distinguishable) from their equivalent types. For instance, an array of MyInt32 should work exactly as an Int32 array, but I would need to know it is a MyInt32 array and not an Int32 one. However, I you believe that the system is not prepared for that (or it would require too much work) I will try to identify it by some external means. Thank you anyway for your prompt answer. Best regards, Ivan -- Ivan Vilata i Balaguer >qo< http://www.carabos.com/ C?rabos Coop. V. V V Enjoy Data "" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From jmiller at stsci.edu Mon Dec 13 08:49:03 2004 From: jmiller at stsci.edu (Todd Miller) Date: Mon Dec 13 08:49:03 2004 Subject: [Numpy-discussion] Using new numeric instances in NumArray and RecArray. In-Reply-To: <20041213162320.GT4752@orm-embar.terramar.selidor.net> References: <20041213113929.GR4752@orm-embar.terramar.selidor.net> <1102942407.4631.9.camel@jaytmiller.comcast.net> <20041213162320.GT4752@orm-embar.terramar.selidor.net> Message-ID: <1102956503.18595.65.camel@halloween.stsci.edu> On Mon, 2004-12-13 at 11:23, Ivan Vilata i Balaguer wrote: > I'm trying to define new types that can be handled as normal types in > numarray, but being different (distinguishable) from their equivalent > types. For instance, an array of MyInt32 should work exactly as an > Int32 array, but I would need to know it is a MyInt32 array and not an > Int32 one. > > However, I you believe that the system is not prepared for that > (or it would require too much work) I will try to identify it by some > external means. Thank you anyway for your prompt answer. I think the system is simply not designed for that. For what you want, I think Int32 would need to be a class, not an instance, so that it could be subclassed while retaining a relatively simple C implementation. As you noted, the C infrastructure works using a type mapping table which maps type *instances* onto type numbers, so it won't work as it is now. Regards, Todd From haase at msg.ucsf.edu Mon Dec 13 09:10:01 2004 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Mon Dec 13 09:10:01 2004 Subject: [Numpy-discussion] Using new numeric instances in NumArray and RecArray. In-Reply-To: <1102956503.18595.65.camel@halloween.stsci.edu> References: <20041213113929.GR4752@orm-embar.terramar.selidor.net> <20041213162320.GT4752@orm-embar.terramar.selidor.net> <1102956503.18595.65.camel@halloween.stsci.edu> Message-ID: <200412130909.44454.haase@msg.ucsf.edu> Hi Ivan, I might be ignorant because if have never heard about the "same-but-distinguishable" concept (except from general oo-deriving of course). But I might actually have a similar situation. I have a 3d-image file-format which I wanted to handle mostly just by it's pixel-data content, that is: I return it as numarray. But to allow me to access the extra (header) information that comes with that file-format I "stick" a new attribute into the numarray instance. (this must sound very surprising to e.g. C++-people put is standard python as I understand) So here is some outline of my code: ''' class Mrc: # our "mrc"-file-format def __init__(self): open-file, make memmap for header+ data data.Mrc = self # this is the "sticking-in part" return data ''' This is very much paraphrased. I think __init__ doesn't have a return (so I use a seprarate function for that). And (after much debugging) I now use "weakref" for the "sticking-in part" to avoid circular references. Tell me if I guessed you case right (at least somewhat) ;-) Regards, Sebastian Haase On Monday 13 December 2004 08:48 am, Todd Miller wrote: > On Mon, 2004-12-13 at 11:23, Ivan Vilata i Balaguer wrote: > > I'm trying to define new types that can be handled as normal types in > > numarray, but being different (distinguishable) from their equivalent > > types. For instance, an array of MyInt32 should work exactly as an > > Int32 array, but I would need to know it is a MyInt32 array and not an > > Int32 one. > > > > However, I you believe that the system is not prepared for that > > (or it would require too much work) I will try to identify it by some > > external means. Thank you anyway for your prompt answer. > > I think the system is simply not designed for that. For what you want, > I think Int32 would need to be a class, not an instance, so that it > could be subclassed while retaining a relatively simple C > implementation. As you noted, the C infrastructure works using a type > mapping table which maps type *instances* onto type numbers, so it won't > work as it is now. > > Regards, > Todd > > > > ------------------------------------------------------- > SF email is sponsored by - The IT Product Guide > Read honest & candid reviews on hundreds of IT Products from real users. > Discover which products truly live up to the hype. Start reading now. > http://productguide.itmanagersjournal.com/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From ivilata at carabos.com Mon Dec 13 13:58:19 2004 From: ivilata at carabos.com (Ivan Vilata i Balaguer) Date: Mon Dec 13 13:58:19 2004 Subject: [Numpy-discussion] Using new numeric instances in NumArray and RecArray. In-Reply-To: <1102956503.18595.65.camel@halloween.stsci.edu> References: <20041213113929.GR4752@orm-embar.terramar.selidor.net> <1102942407.4631.9.camel@jaytmiller.comcast.net> <20041213162320.GT4752@orm-embar.terramar.selidor.net> <1102956503.18595.65.camel@halloween.stsci.edu> Message-ID: <20041213215740.GX4752@orm-embar.terramar.selidor.net> On Mon, Dec 13, 2004 at 11:48:23AM -0500, Todd Miller wrote: > [...] > I think the system is simply not designed for that. For what you want, > I think Int32 would need to be a class, not an instance, so that it > could be subclassed while retaining a relatively simple C > implementation. As you noted, the C infrastructure works using a type > mapping table which maps type *instances* onto type numbers, so it won't > work as it is now. Uh, what a pity. I thougth that creating new instances with IntegralType and FloatingType would be more useful. I will try with subclassing NumArray and RecArray as I was discussing with Colin. Thanks a lot for your help and useful information. Regards, Ivan -- Ivan Vilata i Balaguer >qo< http://www.carabos.com/ C?rabos Coop. V. V V Enjoy Data "" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From ivilata at carabos.com Mon Dec 13 14:21:06 2004 From: ivilata at carabos.com (Ivan Vilata i Balaguer) Date: Mon Dec 13 14:21:06 2004 Subject: [Numpy-discussion] Using new numeric instances in NumArray and RecArray. In-Reply-To: <41BDCF5D.5000609@sympatico.ca> References: <20041213113929.GR4752@orm-embar.terramar.selidor.net> <41BD9207.6060309@sympatico.ca> <20041213164000.GU4752@orm-embar.terramar.selidor.net> <41BDCF5D.5000609@sympatico.ca> Message-ID: <20041213222035.GY4752@orm-embar.terramar.selidor.net> On Mon, Dec 13, 2004 at 12:20:29PM -0500, Colin J. Williams wrote: > [...] > The complexity would depend partly on whether TimeStuff deals only with > data of the MyTime32 type. > > If all other data types (Int8, Float, Complex etc.) are barred, then the > needed overriding would probably be simpler. > [...] I have the feeling that MyInt32-only support should be enough for the NumArray subclass (it's a uniform array). However, the RecArray subclass should be able to handle all existing types in addition to MyInt32. > > I guess that another factor to consider is whether this is a one-off > project or something which will be in production for years to come. If > the former, it might be simpler to do the time related things outside of > numarray. On the other hand, if execution time is a significant factor, > then it could be worthwhile to modify the ufuncs - that is not a thing I > would like to tackle. Hehe, it will be production code with time-critical requirements. :) However, it now seems to me that subclassing NumArray and RecArray will be the right thing (not the easiest, though) to do. Thanks a lot Colin and Todd for your helpful comments. I'll try to do it and will tell you how it went. Bye! -- Ivan Vilata i Balaguer >qo< http://www.carabos.com/ C?rabos Coop. V. V V Enjoy Data "" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From ivilata at carabos.com Tue Dec 14 09:51:00 2004 From: ivilata at carabos.com (Ivan Vilata i Balaguer) Date: Tue Dec 14 09:51:00 2004 Subject: [Numpy-discussion] Re: Using new numeric instances in NumArray and RecArray. In-Reply-To: <20041213113929.GR4752@orm-embar.terramar.selidor.net> References: <20041213113929.GR4752@orm-embar.terramar.selidor.net> Message-ID: <20041214174949.GB20214@orm-embar.terramar.selidor.net> On Mon, Dec 13, 2004 at 12:39:29PM +0100, Ivan Vilata i Balaguer wrote: > > Hi all! I am trying to define new numeric types to use them in > NumArray and RecArray objects. > [...] Hello again. I have started to code a subclass of RecArray to map custom instance types (MyInt32) to standard Numarray instance types (Int32). The task seems plausible once numarray.typeDict, numarray.records.numfmt and numarray.records.revfmt are tailored to support the new types. Translating a format specification like formats='1MyInt32,(2,3)f8' to formats='1Int32,(2,3)f8' looks hard but it is a question of time. However, I have come across the fact that the constructors for NumArray and RecArray are seldom used directly. Instead, functions like numarray.array() and numarray.records.array() are used. These make heavy transformations on their arguments and then create new {Num,Rec}Array objects. This fact makes using classes different to {Num,Rec}Array very difficult, even more when using different signatures for their constructors. One solution is redefining the aforementioned functions to use custom classes. However, this looks cumbersome and not a simple task, but it seems to me the only way to keep the numarray interface. So I will try to implement those new types around Numarray instead of at the same level. Well, this was my report on my attempt to define new Numarray types! Thank you again for your suggestions. Best wishes, Ivan import disclaimer -- Ivan Vilata i Balaguer >qo< http://www.carabos.com/ C?rabos Coop. V. V V Enjoy Data "" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From luszczek at cs.utk.edu Tue Dec 14 09:53:03 2004 From: luszczek at cs.utk.edu (Piotr Luszczek) Date: Tue Dec 14 09:53:03 2004 Subject: [Numpy-discussion] started work on new version of LAPACK Message-ID: <41BF285D.2080809@cs.utk.edu> Hello all, on behalf of the LAPACK/ScaLAPACK team I'd like to inform you that we have just received funding to work on a new version of LAPACK and ScaLAPACK. Right now we are soliciting input from various groups that use the software. Numpy community is an important user and we would like to hear from you. I have subscribed to this list so I will receive any responses and pass them along to the rest the LAPACK/ScaLAPACK team. Here is a list to give you a brief idea of the issues we are trying to address: - programming language of new implementation (we will keep the old code as it is) - support for threading (OpenMP, pthreads, etc.) - 64-bit addressing - new/improved routines - IEEE 794 (floating-point) support Please let us know what you think. Peter (Piotr Luszczek) From rays at san.rr.com Wed Dec 15 20:13:01 2004 From: rays at san.rr.com (RJ) Date: Wed Dec 15 20:13:01 2004 Subject: [Numpy-discussion] copying ctypes arrays to numarray? Message-ID: <5.2.1.1.2.20041215195942.07e24590@pop-server.san.rr.com> I'm posting here to see if numarray has a method that could work like array.buffer_info(). numarray.info() returns a text output that can't be used like the array method is to memmove() between ctypes and Python arrays without parsing, apparently. ctypes thread with Thomas Heller below. His main question: "Maybe you should ask to make sure that there's no way to copy between objects implementing the buffer protocol with some Python function that I do not know about?" From bogus@does.not.exist.com Wed Dec 15 14:25:33 2004 From: bogus@does.not.exist.com () Date: Wed, 15 Dec 2004 20:25:33 +0100 Subject: [ctypes-users] copying/slicing ctypes arrays, (c_ulong *n)() In-Reply-To: <5.2.0.4.2.20041215100117.0dff97b0@blue-cove.com> (Ray Schumacher's message of "Wed, 15 Dec 2004 10:57:45 -0800") References: <5.2.1.1.2.20041214202542.05f02920@pop-server.san.rr.com> <5.2.1.1.2.20041214202542.05f02920@pop-server.san.rr.com> <5.2.0.4.2.20041215100117.0dff97b0@blue-cove.com> Message-ID: From florian.proff.schulze at gmx.net Thu Dec 16 01:15:02 2004 From: florian.proff.schulze at gmx.net (Florian Schulze) Date: Thu Dec 16 01:15:02 2004 Subject: [Numpy-discussion] Re: copying ctypes arrays to numarray? References: <5.2.1.1.2.20041215195942.07e24590@pop-server.san.rr.com> Message-ID: On Wed, 15 Dec 2004 20:16:07 -0800, RJ wrote: > I'm posting here to see if numarray has a method that could work like > array.buffer_info(). numarray.info() returns a text output that can't > be used like the array method is to memmove() between ctypes and Python > arrays without parsing, apparently. > > ctypes thread with Thomas Heller below. His main question: "Maybe you > should ask to make sure that there's no way to copy between > objects implementing the buffer protocol with some Python function that > I do not know about?" I just tried some things: >>> import ctypes >>> a = (ctypes.c_int * 5)() >>> a[0] = 1; a[1] = 2; a[2] = 3; a[3] = 4; a[4] = 5 >>> list(a) [1, 2, 3, 4, 5] >>> import numarray >>> buf = numarray.zeros(shape=20, type='i4') >>> buf array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) >>> buf[2:7] = a >>> buf array([0, 0, 1, 2, 3, 4, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) >>> temp = numarray.array(sequence=buffer(a), shape=5, type='i4') >>> temp array([1, 2, 3, 4, 5]) >>> temp._data >>> buffer(a) >>> a[2] = 10 >>> temp array([ 1, 2, 10, 4, 5]) The first block just creates the ctypes data. The second block uses slice assignment to copy the data from the ctypes array into the numarray.array. The third block uses the buffer interface to create a numarray.array which points to the same memory location as the ctypes array. The forth block illustrates that it's really the same memory. You have to benchmark which one of the two solutions is the better one for you. Regards, Florian Schulze From klimek at grc.nasa.gov Thu Dec 16 10:07:02 2004 From: klimek at grc.nasa.gov (Bob Klimek) Date: Thu Dec 16 10:07:02 2004 Subject: [Numpy-discussion] binary thinning In-Reply-To: References: <41B63AA8.5060205@grc.nasa.gov> Message-ID: <41C1CF1F.4000807@grc.nasa.gov> Peter Verveer wrote: > Hi Bob, > > Below I give my implementation, which does not produce the exact same > result as shown on the webpage you refer to, although the result seems > to be a proper skeleton. This may be because the order in which the > structures are applied matters (I think). Could you maybe test that by > permutating the order in which the structures are applied? I would be > interested if this is true and if one can get the same result as on > the webpage by finding the right sequence... Peter, I'm finally able to get back with some answers; unfortunately, for a number of reasons my time spent on this is quite fragmented. First, to answer your question about changing the order in which structures are applied. It turns out it does matter. I tried rotating the structures clockwise and counter-clockwise, pre-rotating the structure before doing the four rotations, and I permutating the order, and in some of those cases the results are different, although usually only slightly different - a couple pixels here, a few there. Second, no matter what I tried, I could not duplicate the picture in the webpage. Third, besides the test image (off the webpage) I also tested a skeleton function on few other images using several packages available to me, including an old Matrox MIL, ImageJ, and Matlab. Each one of them produced different results. I'm not sure that one is more correct than another, they're probably all correct. In general the nd_image skeleton produces results similar to Matlab. Peter, would you be interested in adding a few binary morphology functions to nd_image? So far I have working versions of borderkill, borderkeep, reconstruct, thinning, thickening, skeleton, skiz, and convex hull. Even though they were all produced with just what's there right now (erosion, dilation, and hit-or-miss) and a few numarray functions, it took a long time to figure out and could be helpful to the next guy. I'd be happy to send you what I got and/or post it. Regards, Bob From verveer at embl-heidelberg.de Thu Dec 16 10:43:03 2004 From: verveer at embl-heidelberg.de (Peter Verveer) Date: Thu Dec 16 10:43:03 2004 Subject: [Numpy-discussion] binary thinning In-Reply-To: <41C1CF1F.4000807@grc.nasa.gov> References: <41B63AA8.5060205@grc.nasa.gov> <41C1CF1F.4000807@grc.nasa.gov> Message-ID: <35A3C83E-4F92-11D9-8D51-000D932805AC@embl-heidelberg.de> Hi Bob, Thanks for using the nd_image code. It is helpful to get some user feedback. I think I did not tell you in my last email, but your last questions actually allowed me to find a few minor bugs and fix them... > First, to answer your question about changing the order in which > structures are applied. It turns out it does matter. I tried rotating > the structures clockwise and counter-clockwise, pre-rotating the > structure before doing the four rotations, and I permutating the > order, and in some of those cases the results are different, although > usually only slightly different - a couple pixels here, a few there. This seems reasonable to me, I would expect that the order matters. > Second, no matter what I tried, I could not duplicate the picture in > the webpage. > > Third, besides the test image (off the webpage) I also tested a > skeleton function on few other images using several packages available > to me, including an old Matrox MIL, ImageJ, and Matlab. Each one of > them produced different results. I'm not sure that one is more correct > than another, they're probably all correct. In general the nd_image > skeleton produces results similar to Matlab. It seems then that there is no accepted single solution for the skeleton, so I guess the code from nd_image is okay then. > Peter, would you be interested in adding a few binary morphology > functions to nd_image? So far I have working versions of borderkill, > borderkeep, reconstruct, thinning, thickening, skeleton, skiz, and > convex hull. Even though they were all produced with just what's there > right now (erosion, dilation, and hit-or-miss) and a few numarray > functions, it took a long time to figure out and could be helpful to > the next guy. I'd be happy to send you what I got and/or post it. One issue is that nd_image functions are supposed to be generic and work in any dimension. So I kept the binary morphology low-level on purpose, so far. But on the other hand, I don't see why we could not add some stuff even if it only works in 2D (which I assume is the case with your code.) Why don't you send me what you have and I have a look in the next weeks? Cheers, Peter From rays at san.rr.com Thu Dec 16 13:28:01 2004 From: rays at san.rr.com (rays) Date: Thu Dec 16 13:28:01 2004 Subject: [Numpy-discussion] copying ctypes arrays to numarray? In-Reply-To: <5.2.1.1.2.20041215195942.07e24590@pop-server.san.rr.com> Message-ID: <5.2.0.4.2.20041216132555.079aac28@pop-server.san.rr.com> I'm re-posting this here to see if anyone can shed light on numarray.info() behavior, and why assignment from a (ctypes.c_long * 2000)() to a numarray is so much slower than a memmove(). I was surprised that assignment wasn't faster and that numarray assignment was consistently ~.5% faster than Numeric. The N vs. n memmove() flip-flopped for fastest, with array.array always slower. After some discussion, I made a better direct comparison of ctypes and memmove, map etc., using all the methods suggested so far. I think it's fairly valid.(?) Run on a 2.4GHz WinXP, Py2.3, a number of times. Parsing numarray.info() was a royal pain, it writes directly to stdout! I.e.: >>> inf = src.info() class: shape: (32,) strides: (8,) byteoffset: 0 bytestride: 8 itemsize: 8 aligned: 1 contiguous: 1 data: byteorder: little byteswap: 0 type: Float64 >>> inf >>> type(inf) Since the ctype does support slicing, I was considering leaving the data in the device driver buffer (~1MB circular) and poking into it, but memmove is so much faster than slicing the ctype, I'm doing memmove()s to numarray. I presume that I should check for numarray.iscontiguous( ) or is_c_array( ) first to be safe... Results and code below. Thanks to all for the help, Ray >python test.py Array address 11443208 n address 17039424 N address 21102656 for loop 2027.0 us ea map 1781.0 us ea slice 1704.0 us ea assign N 1244.0 us ea assign n 1242.0 us ea memmove 4.3831 us ea memmove N 3.4773 us ea memmove n 3.4803 us ea _____________________________________________________ ## test.py import array import numarray import ctypes import time import string import StringIO import sys buf = (ctypes.c_long * 2000)() Array = array.array("l", [0]*10000) n = numarray.zeros((1000000), numarray.Int32) N = numarray.zeros((1000000), numarray.Int32) #!!!!!!!!!!!!!!! arrrrgggg! !!!!!!!!!!!!!!!! # n.info() writes directly to stdout! stdout = sys.stdout fileo = StringIO.StringIO() sys.stdout = fileo n.info() ninfo = fileo.getvalue( ) fileo.close() sys.stdout = stdout stdout = sys.stdout fileo = StringIO.StringIO() sys.stdout = fileo N.info() Ninfo = fileo.getvalue( ) fileo.close() sys.stdout = stdout #!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! print 'Array address', Array.buffer_info()[0] ninfo = string.split(ninfo) nAddress = int(ninfo[20], 16) print 'n address', nAddress Ninfo = string.split(Ninfo) NAddress = int(Ninfo[20], 16) print 'N address', NAddress t0 = time.clock() for loop in range(1000): for i in range(2000): n[loop+i] = buf[i] print 'for loop ', round((time.clock()-t0)*1000), 'us ea' t0 = time.clock() for loop in range(1000): n[loop:loop+2000] = map(None, buf) print 'map ', round((time.clock()-t0)*1000), 'us ea' t0 = time.clock() for loop in range(1000): n[loop:loop+2000] = buf[0:2000] print 'slice ', round((time.clock()-t0)*1000), 'us ea' t0 = time.clock() for loop in range(10000): N[loop:loop+2000] = buf print 'assign N', round((time.clock()-t0)*100), 'us ea' t0 = time.clock() for loop in range(10000): n[loop:loop+2000] = buf print 'assign n', round((time.clock()-t0)*100), 'us ea' t0 = time.clock() for loop in range(10000): ctypes.memmove(10+Array.buffer_info()[0], buf, 2000) print 'memmove ', round((time.clock()-t0)*1, 4), 'us ea' t0 = time.clock() for loop in range(10000): ctypes.memmove(10+NAddress, buf, 2000) print 'memmove N', round((time.clock()-t0)*1, 4), 'us ea' t0 = time.clock() for loop in range(10000): ctypes.memmove(10+nAddress, buf, 2000) print 'memmove n', round((time.clock()-t0)*1, 4), 'us ea' -------------- next part -------------- An HTML attachment was scrubbed... URL: From jmiller at stsci.edu Thu Dec 16 13:56:01 2004 From: jmiller at stsci.edu (Todd Miller) Date: Thu Dec 16 13:56:01 2004 Subject: [Numpy-discussion] copying ctypes arrays to numarray? In-Reply-To: <5.2.0.4.2.20041216132555.079aac28@pop-server.san.rr.com> References: <5.2.0.4.2.20041216132555.079aac28@pop-server.san.rr.com> Message-ID: <1103234122.3167.43.camel@jaytmiller.comcast.net> I agree that numarray's .info() would be better as a string, but I've been leery of changing it. Before we go to far with this, try: repr(src._data) to get a string like: "" Note: this technique depends on the buffer object used as the basis for the array. Here the repr of numarray's memory object is giving useful information. Other buffers, e.g. a string, might not. It looks to me like you forgot to import Numeric so I don't think there's any numarray:Numeric comparison going on... either that or I just don't understand what you're to compare. To be safe, check the .is_c_array() method. Regards, Todd On Thu, 2004-12-16 at 13:33 -0800, rays wrote: > I'm re-posting this here to see if anyone can shed light on > numarray.info() behavior, and why assignment from a (ctypes.c_long * > 2000)() to a numarray is so much slower than a memmove(). > > I was surprised that assignment wasn't faster and that numarray > assignment was consistently ~.5% faster than Numeric. > The N vs. n memmove() flip-flopped for fastest, with array.array > always slower. > > After some discussion, I made a better direct comparison of ctypes and > memmove, map etc., using all the methods suggested so far. I think > it's fairly valid.(?) Run on a 2.4GHz WinXP, Py2.3, a number of times. > > Parsing numarray.info() was a royal pain, it writes directly to > stdout! > I.e.: > >>> inf = src.info() > class: > shape: (32,) > strides: (8,) > byteoffset: 0 > bytestride: 8 > itemsize: 8 > aligned: 1 > contiguous: 1 > data: aliasing object > 00000000> > byteorder: little > byteswap: 0 > type: Float64 > >>> inf > >>> type(inf) > > > > Since the ctype does support slicing, I was considering leaving the > data in the device driver buffer (~1MB circular) and poking into it, > but memmove is so much faster than slicing the ctype, I'm doing > memmove()s to numarray. > I presume that I should check for numarray.iscontiguous( ) or > is_c_array( ) first to be safe... > > Results and code below. > > Thanks to all for the help, > Ray > > > > >python test.py > Array address 11443208 > n address 17039424 > N address 21102656 > for loop 2027.0 us ea > map 1781.0 us ea > slice 1704.0 us ea > assign N 1244.0 us ea > assign n 1242.0 us ea > memmove 4.3831 us ea > memmove N 3.4773 us ea > memmove n 3.4803 us ea > > _____________________________________________________ > > ## test.py > import array > import numarray > import ctypes > import time > import string > import StringIO > import sys > > buf = (ctypes.c_long * 2000)() > Array = array.array("l", [0]*10000) > n = numarray.zeros((1000000), numarray.Int32) > N = numarray.zeros((1000000), numarray.Int32) > > #!!!!!!!!!!!!!!! arrrrgggg! !!!!!!!!!!!!!!!! > # n.info() writes directly to stdout! > stdout = sys.stdout > fileo = StringIO.StringIO() > sys.stdout = fileo > n.info() > ninfo = fileo.getvalue( ) > fileo.close() > sys.stdout = stdout > > stdout = sys.stdout > fileo = StringIO.StringIO() > sys.stdout = fileo > N.info() > Ninfo = fileo.getvalue( ) > fileo.close() > sys.stdout = stdout > #!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! > > print 'Array address', Array.buffer_info()[0] > > ninfo = string.split(ninfo) > nAddress = int(ninfo[20], 16) > print 'n address', nAddress > Ninfo = string.split(Ninfo) > NAddress = int(Ninfo[20], 16) > print 'N address', NAddress > > t0 = time.clock() > for loop in range(1000): > for i in range(2000): > n[loop+i] = buf[i] > print 'for loop ', round((time.clock()-t0)*1000), 'us ea' > > t0 = time.clock() > for loop in range(1000): > n[loop:loop+2000] = map(None, buf) > print 'map ', round((time.clock()-t0)*1000), 'us ea' > > t0 = time.clock() > for loop in range(1000): > n[loop:loop+2000] = buf[0:2000] > print 'slice ', round((time.clock()-t0)*1000), 'us ea' > > t0 = time.clock() > for loop in range(10000): > N[loop:loop+2000] = buf > print 'assign N', round((time.clock()-t0)*100), 'us ea' > > t0 = time.clock() > for loop in range(10000): > n[loop:loop+2000] = buf > print 'assign n', round((time.clock()-t0)*100), 'us ea' > > t0 = time.clock() > for loop in range(10000): > ctypes.memmove(10+Array.buffer_info()[0], > buf, > 2000) > print 'memmove ', round((time.clock()-t0)*1, 4), 'us ea' > > t0 = time.clock() > for loop in range(10000): > ctypes.memmove(10+NAddress, > buf, > 2000) > print 'memmove N', round((time.clock()-t0)*1, 4), 'us ea' > > t0 = time.clock() > for loop in range(10000): > ctypes.memmove(10+nAddress, > buf, > 2000) > print 'memmove n', round((time.clock()-t0)*1, 4), 'us ea' > > > > > > > From rays at san.rr.com Thu Dec 16 17:05:29 2004 From: rays at san.rr.com (RJ) Date: Thu Dec 16 17:05:29 2004 Subject: [Numpy-discussion] copying ctypes arrays to numarray? In-Reply-To: <1103234122.3167.43.camel@jaytmiller.comcast.net> References: <5.2.0.4.2.20041216132555.079aac28@pop-server.san.rr.com> <5.2.0.4.2.20041216132555.079aac28@pop-server.san.rr.com> Message-ID: <5.2.1.1.2.20041216165316.00b1d4c0@pop-server.san.rr.com> Doh. Thanks Todd, At 04:55 PM 12/16/2004 -0500, Todd Miller wrote: > try: repr(src._data) ... >Other buffers, e.g. a string, might not. Which is fine for this app, so I will. >It looks to me like you forgot to import Numeric Yep. I screwed up the import statements - I'll re-do it tomorrow. In that light, it is interesting that the assign N were always longer than the n ones... >To be safe, check the .is_c_array() method. Will do. I am also a little concerned about the address changing in a long-running process (literally, weeks), so I might check that too. Also, why is assignment from a (ctypes.c_long * 2000)() to a numarray so much slower than a memmove()? Does the assign make a temp buffer? Ray -------------- next part -------------- An HTML attachment was scrubbed... URL: From konrad.hinsen at laposte.net Fri Dec 17 05:30:02 2004 From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net) Date: Fri Dec 17 05:30:02 2004 Subject: [Numpy-discussion] ufuncs on user-defined types in numarray Message-ID: I am working on making ScientificPython (http://dirac.cnrs-orleans.fr/ScientificPython/) compatible with numarray. One problem I stumbled across is that one feature of Numeric seems to be absent from numarray: the possibility to call ufuncs with arguments of non-numeric types, leading to a corresponding method call. As an illustration, the following example works fine with Numeric, but crashes with numarray: from Numeric import sqrt #from numarray import sqrt class DummyValue: def __init__(self, string): self.string = string def __str__(self): return self.string def sqrt(self): return DummyValue('sqrt(%s)' % self.string) x = DummyValue('x') print sqrt(x) Is this a bug or a feature? Is there another way to make ufuncs work with user-defined classes and types? Konrad. -- --------------------------------------------------------------------- Konrad Hinsen Laboratoire L?on Brillouin, CEA Saclay, 91191 Gif-sur-Yvette Cedex, France Tel.: +33-1 69 08 79 25 Fax: +33-1 69 08 82 61 E-Mail: hinsen at llb.saclay.cea.fr --------------------------------------------------------------------- From jmiller at stsci.edu Fri Dec 17 06:54:03 2004 From: jmiller at stsci.edu (Todd Miller) Date: Fri Dec 17 06:54:03 2004 Subject: [Numpy-discussion] copying ctypes arrays to numarray? In-Reply-To: <5.2.1.1.2.20041216165316.00b1d4c0@pop-server.san.rr.com> References: <5.2.0.4.2.20041216132555.079aac28@pop-server.san.rr.com> <5.2.0.4.2.20041216132555.079aac28@pop-server.san.rr.com> <5.2.1.1.2.20041216165316.00b1d4c0@pop-server.san.rr.com> Message-ID: <1103295187.31241.52.camel@halloween.stsci.edu> On Thu, 2004-12-16 at 20:04, RJ wrote: > Doh. > Thanks Todd, > > At 04:55 PM 12/16/2004 -0500, Todd Miller wrote: > > try: repr(src._data) > ... > > Other buffers, e.g. a string, might not. > > Which is fine for this app, so I will. > > > It looks to me like you forgot to import Numeric > > Yep. I screwed up the import statements - I'll re-do it tomorrow. In > that light, it is interesting that the assign N were always longer > than the n ones... > > > To be safe, check the .is_c_array() method. > > Will do. I am also a little concerned about the address changing in a > long-running process (literally, weeks), so I might check that too. As long as you let the NumArray allocate its own memory, it will allocate a numarray.memory object and those don't move. If you resize the array, that's a different story... > Also, why is assignment from a (ctypes.c_long * 2000)() to a numarray > so much slower than a memmove()? Does the assign make a temp buffer? For the assignment to numarray, I think the ctype is opaque and as a result Python uses the sequence protocol; this is a huge amount of work compared to a tight C loop moving bytes. I think every 4 bytes the ctype is doing a method call and building a Python long, numarray is doing a method call and extracting a Python long, then the Python long is destructed. I may be missing something, but the comparisons of for loop, map, and slice look like they're only doing 1000 moves but the array assigments are doing 10000, so I think there's another factor of 10 missing in that comparison. From jmiller at stsci.edu Fri Dec 17 07:10:12 2004 From: jmiller at stsci.edu (Todd Miller) Date: Fri Dec 17 07:10:12 2004 Subject: [Numpy-discussion] ufuncs on user-defined types in numarray In-Reply-To: References: Message-ID: <1103296191.31241.66.camel@halloween.stsci.edu> On Fri, 2004-12-17 at 08:31, konrad.hinsen at laposte.net wrote: > I am working on making ScientificPython > (http://dirac.cnrs-orleans.fr/ScientificPython/) compatible with > numarray. One problem I stumbled across is that one feature of Numeric > seems to be absent from numarray: the possibility to call ufuncs with > arguments of non-numeric types, leading to a corresponding method call. > > As an illustration, the following example works fine with Numeric, but > crashes with numarray: > > from Numeric import sqrt > #from numarray import sqrt > > class DummyValue: > > def __init__(self, string): > self.string = string > > def __str__(self): > return self.string > > def sqrt(self): > return DummyValue('sqrt(%s)' % self.string) > > x = DummyValue('x') > print sqrt(x) > > > Is this a bug or a feature? It's a missing feature. > Is there another way to make ufuncs work with user-defined classes and types? Not that I know of. How critical a feature is it? Do you have a work around? numarray ufunc handling is in flux as I'm adding numarray support to scipy. At some point, there's going to be a major ufunc consolidation, and perhaps I can add this additional feature then. Regards, Todd From konrad.hinsen at laposte.net Fri Dec 17 08:22:02 2004 From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net) Date: Fri Dec 17 08:22:02 2004 Subject: [Numpy-discussion] ufuncs on user-defined types in numarray In-Reply-To: <1103296191.31241.66.camel@halloween.stsci.edu> References: <1103296191.31241.66.camel@halloween.stsci.edu> Message-ID: On Dec 17, 2004, at 16:09, Todd Miller wrote: >> Is there another way to make ufuncs work with user-defined classes >> and types? > > Not that I know of. How critical a feature is it? Do you have a work > around? It is critical for some applications, though not for many. The most important application in ScientificPython is automatic derivatives. These are implemented as a number-like class whose instances store the values of a variable and of its derivatives. Such objects can be fed to any routine that does mathematical calculations, which is what makes it so powerful. Without this mechanism, all function applications would have to be written as method calls, which is less habitual, but more importantly creates code that doesn't work with plain numbers. > numarray ufunc handling is in flux as I'm adding numarray support to > scipy. At some point, there's going to be a major ufunc > consolidation, > and perhaps I can add this additional feature then. Sounds good. I'll postpone the ScientificPython update until then. Konrad. -- --------------------------------------------------------------------- Konrad Hinsen Laboratoire L?on Brillouin, CEA Saclay, 91191 Gif-sur-Yvette Cedex, France Tel.: +33-1 69 08 79 25 Fax: +33-1 69 08 82 61 E-Mail: hinsen at llb.saclay.cea.fr --------------------------------------------------------------------- From rays at san.rr.com Fri Dec 17 08:30:00 2004 From: rays at san.rr.com (RJ) Date: Fri Dec 17 08:30:00 2004 Subject: [Numpy-discussion] copying ctypes arrays to numarray? In-Reply-To: <1103295187.31241.52.camel@halloween.stsci.edu> References: <5.2.1.1.2.20041216165316.00b1d4c0@pop-server.san.rr.com> <5.2.0.4.2.20041216132555.079aac28@pop-server.san.rr.com> <5.2.0.4.2.20041216132555.079aac28@pop-server.san.rr.com> <5.2.1.1.2.20041216165316.00b1d4c0@pop-server.san.rr.com> Message-ID: <5.2.1.1.2.20041217082410.05e4bec0@pop-server.san.rr.com> At 09:53 AM 12/17/2004 -0500, Todd Miller wrote: >I may be missing something, but the comparisons of for loop, map, and >slice look like they're only doing 1000 moves but the array assigments >are doing 10000, so I think there's another factor of 10 missing in >that comparison. The multiplier in the print statement compensates to still give microseconds per move; I couldn't bear to wait for 10,000 slow iterations. I'll re-do the test shortly in any case. Ray Secret anti-spam filter-passing text. Include with reply: qwertyuiop -------------- next part -------------- An HTML attachment was scrubbed... URL: From jmiller at stsci.edu Sat Dec 18 05:53:01 2004 From: jmiller at stsci.edu (Todd Miller) Date: Sat Dec 18 05:53:01 2004 Subject: [Numpy-discussion] Re: simple numarray bug In-Reply-To: <41C395E3.5070903@caltech.edu> References: <41C395E3.5070903@caltech.edu> Message-ID: <1103377939.3258.14.camel@jaytmiller.comcast.net> On Fri, 2004-12-17 at 18:28 -0800, Andrew Straw wrote: > Hi Todd, > Hi Andrew, > (Nice meeting you at SciPy '04!) Yes, nice meeting you. > I think this is a simple bug which I just filed a bug report on SF for: > http://sourceforge.net/tracker/index.php?func=detail&aid=1087378&group_id=1369&atid=101369 > > This email is just a "heads up" that I created the bug report. Thanks for the bug report and heads up. Source Forge automatically gives me e-mail for numarray bug reports so "heads ups" are not necessary. However, I often fall behind real time response so if you make a report and nothing happens for a while gentle nagging is OK. I committed a "fix" for this to CVS... There is now a LinearAlgebraError subclass of Exception aliased as LinAlgError. Regards, Todd From gazzar at email.com Sat Dec 18 22:38:02 2004 From: gazzar at email.com (Gary Ruben) Date: Sat Dec 18 22:38:02 2004 Subject: [Numpy-discussion] ufuncs on user-defined types in numarray Message-ID: <20041219063737.35BB14BDAB@ws1-1.us4.outblaze.com> Thanks Konrad, I didn't realise this was possible. I should use it on my ErrorVal module . Last time I tried (quite a while ago), numarray's object array support wasn't up to allowing me to port ErrorVal from Numeric. I don't think it matters since I don't know of anyone, besides me, who has tried using my module :-( but I thought I'd mention it for Todd's benefit since I'm one who could benefit from implementing it in numarray. Gary R. ----- Original Message ----- From: "Todd Miller" To: konrad.hinsen at laposte.net Subject: Re: [Numpy-discussion] ufuncs on user-defined types in numarray Date: 17 Dec 2004 10:09:51 -0500 > > On Fri, 2004-12-17 at 08:31, konrad.hinsen at laposte.net wrote: > > I am working on making ScientificPython > > (http://dirac.cnrs-orleans.fr/ScientificPython/) compatible with > > numarray. One problem I stumbled across is that one feature of > > Numeric seems to be absent from numarray: the possibility to call > > ufuncs with arguments of non-numeric types, leading to a > > corresponding method call. > > > > As an illustration, the following example works fine with > > Numeric, but crashes with numarray: > > > > from Numeric import sqrt > > #from numarray import sqrt > > > > class DummyValue: > > > > def __init__(self, string): > > self.string = string > > > > def __str__(self): > > return self.string > > > > def sqrt(self): > > return DummyValue('sqrt(%s)' % self.string) > > > > x = DummyValue('x') > > print sqrt(x) > > > > > > Is this a bug or a feature? > > It's a missing feature. > > > Is there another way to make ufuncs work with user-defined classes and types? > > Not that I know of. How critical a feature is it? Do you have a work around? > > numarray ufunc handling is in flux as I'm adding numarray support to > scipy. At some point, there's going to be a major ufunc consolidation, > and perhaps I can add this additional feature then. > > Regards, > Todd > > > > ------------------------------------------------------- > SF email is sponsored by - The IT Product Guide > Read honest & candid reviews on hundreds of IT Products from real users. > Discover which products truly live up to the hype. Start reading now. > http://productguide.itmanagersjournal.com/ > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- ___________________________________________________________ Sign-up for Ads Free at Mail.com http://promo.mail.com/adsfreejump.htm From konrad.hinsen at laposte.net Sun Dec 19 00:52:02 2004 From: konrad.hinsen at laposte.net (konrad.hinsen at laposte.net) Date: Sun Dec 19 00:52:02 2004 Subject: [Numpy-discussion] ufuncs on user-defined types in numarray In-Reply-To: <20041219063737.35BB14BDAB@ws1-1.us4.outblaze.com> References: <20041219063737.35BB14BDAB@ws1-1.us4.outblaze.com> Message-ID: <2E538670-519B-11D9-909D-000A95AB5F10@laposte.net> On 19.12.2004, at 07:37, Gary Ruben wrote: > I didn't realise this was possible. I should use it on my ErrorVal > module . Last time I > tried (quite a Indeed! You could then even have arrays of ErrorVal objects with no extra effort, at least as long as ErrorVal doesn't implement the sequence protocol (because then all the Numeric functions will silently convert them to arrays). Your module looks interesting. What does it do internally, interval arithmetic? Konrad. From jbrandmeyer at earthlink.net Mon Dec 20 10:12:10 2004 From: jbrandmeyer at earthlink.net (Jonathan Brandmeyer) Date: Mon Dec 20 10:12:10 2004 Subject: [Numpy-discussion] Numeric 23.6 for Python 2.4 Message-ID: <2845281.1103566316940.JavaMail.root@waldorf.psp.pas.earthlink.net> Does the Numeric team intend to provide a build of Numeric 23.6 for Python 2.4? If not, I have managed to build a release using the free-as-in-beer version of MSVC 7.1, aka MS Visual C++ Toolkit 2003. I basically followed the directions here: http://www.vrplumber.com/programming/mstoolkit/index.html and altered setup.py to use Numeric's own math libraries rather than the atlas libs. The build has been tested only by running the supplied Test/test.py program after installing the package. For the moment, you can download the file from http://www4.ncsu.edu/~jdbrandm/Numeric-23.6.win32-py2.4.exe but I would prefer it to be hosted on the Sourceforge download page for the project. Its md5sum is 2df692d7b8136b68e3b511ed27cbbdd8 Thanks, -Jonathan Brandmeyer P.S. I apologize if the message's formatting is borked up; I'm sending this from a webmail interface. From jmiller at stsci.edu Mon Dec 20 11:34:07 2004 From: jmiller at stsci.edu (Todd Miller) Date: Mon Dec 20 11:34:07 2004 Subject: [Numpy-discussion] ufuncs on user-defined types in numarray In-Reply-To: <20041219063737.35BB14BDAB@ws1-1.us4.outblaze.com> References: <20041219063737.35BB14BDAB@ws1-1.us4.outblaze.com> Message-ID: <1103571033.3339.69.camel@jaytmiller.comcast.net> Hi Gary and Konrad, I filed a Numarray Enhancement Request on Source Forge for this feature. It's not clear if, when, or how it will be implemented but it won't be completely forgotten either. Regards, Todd On Sun, 2004-12-19 at 16:37 +1000, Gary Ruben wrote: > Thanks Konrad, > I didn't realise this was possible. I should use it on my ErrorVal module . Last time I tried (quite a while ago), numarray's object array support wasn't up to allowing me to port ErrorVal from Numeric. I don't think it matters since I don't know of anyone, besides me, who has tried using my module :-( but I thought I'd mention it for Todd's benefit since I'm one who could benefit from implementing it in numarray. > Gary R. > > ----- Original Message ----- > From: "Todd Miller" > To: konrad.hinsen at laposte.net > Subject: Re: [Numpy-discussion] ufuncs on user-defined types in numarray > Date: 17 Dec 2004 10:09:51 -0500 > > > > > On Fri, 2004-12-17 at 08:31, konrad.hinsen at laposte.net wrote: > > > I am working on making ScientificPython > > > (http://dirac.cnrs-orleans.fr/ScientificPython/) compatible with > > > numarray. One problem I stumbled across is that one feature of > > > Numeric seems to be absent from numarray: the possibility to call > > > ufuncs with arguments of non-numeric types, leading to a > > > corresponding method call. > > > > > > As an illustration, the following example works fine with > > > Numeric, but crashes with numarray: > > > > > > from Numeric import sqrt > > > #from numarray import sqrt > > > > > > class DummyValue: > > > > > > def __init__(self, string): > > > self.string = string > > > > > > def __str__(self): > > > return self.string > > > > > > def sqrt(self): > > > return DummyValue('sqrt(%s)' % self.string) > > > > > > x = DummyValue('x') > > > print sqrt(x) > > > > > > > > > Is this a bug or a feature? > > > > It's a missing feature. > > > > > Is there another way to make ufuncs work with user-defined classes and types? > > > > Not that I know of. How critical a feature is it? Do you have a work around? > > > > numarray ufunc handling is in flux as I'm adding numarray support to > > scipy. At some point, there's going to be a major ufunc consolidation, > > and perhaps I can add this additional feature then. > > > > Regards, > > Todd > > > > > > > > ------------------------------------------------------- > > SF email is sponsored by - The IT Product Guide > > Read honest & candid reviews on hundreds of IT Products from real users. > > Discover which products truly live up to the hype. Start reading now. > > http://productguide.itmanagersjournal.com/ > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From gazzar at email.com Mon Dec 20 17:13:22 2004 From: gazzar at email.com (Gary Ruben) Date: Mon Dec 20 17:13:22 2004 Subject: [Numpy-discussion] ufuncs on user-defined types in numarray Message-ID: <20041221011222.EE0076EEF6@ws1-5.us4.outblaze.com> First, thanks Todd for elevating this to 'requested feature' status. Konrad, I had no idea what interval arithmetic is, so I looked it up. I'm not sure if what I've done is considered interval arithmetic - I suspect it is, but that I've only implemented it partially. The code isn't that clever. It doesn't care about propagating rounding errors correctly at the error limits. It just implements a brute-force approach to calculating operations on the error limits of a data-point value along with the central value. This is already more accurate than what most people do, which is to linearise the error limits and treat them as infinitessimals. The central and outer values are grouped in an object and any operations are applied, without any consideration of rounding errors, to all three values for every operation. Gary ----- Original Message ----- > > I didn't realise this was possible. I should use it on my > > ErrorVal module . > > Last time I tried (quite a > > Indeed! You could then even have arrays of ErrorVal objects with no > extra effort, at least as long as ErrorVal doesn't implement the > sequence protocol (because then all the Numeric functions will > silently convert them to arrays). > > Your module looks interesting. What does it do internally, interval > arithmetic? > > Konrad. -- ___________________________________________________________ Sign-up for Ads Free at Mail.com http://promo.mail.com/adsfreejump.htm From faltet at carabos.com Tue Dec 21 05:08:01 2004 From: faltet at carabos.com (Francesc Altet) Date: Tue Dec 21 05:08:01 2004 Subject: [Numpy-discussion] Striding on NumArray objects Message-ID: <200412211406.57699.faltet@carabos.com> Hi, I'm a bit lost with the next example: In [28]: from numarray import * In [29]: a=arange(10) In [30]: a.iscontiguous() Out[30]: 1 In [31]: b=a[::2] In [32]: b.iscontiguous() Out[32]: 0 That seems to suggest that b shares the same data buffer than a. Indeed: In [36]: a._data Out[36]: In [37]: b._data Out[37]: At this point, I believe that _bytestride should be different on both arrays, but: In [33]: a._bytestride Out[33]: 4 In [34]: b._bytestride Out[34]: 4 while I expected to find b._bytestride equal to 8. Is that an error or a lack of understanding on my part? Cheers, -- Francesc Altet ? >qo< ? http://www.carabos.com/ C?rabos Coop. V. ? V ?V ? Enjoy Data ? ? ? ? ? ? ? ? ? ? "" From tim.hochberg at cox.net Tue Dec 21 06:18:06 2004 From: tim.hochberg at cox.net (Tim Hochberg) Date: Tue Dec 21 06:18:06 2004 Subject: [Numpy-discussion] Striding on NumArray objects In-Reply-To: <200412211406.57699.faltet@carabos.com> References: <200412211406.57699.faltet@carabos.com> Message-ID: <41C8307B.4080505@cox.net> Francesc Altet wrote: >Hi, > >I'm a bit lost with the next example: > >In [28]: from numarray import * >In [29]: a=arange(10) >In [30]: a.iscontiguous() >Out[30]: 1 >In [31]: b=a[::2] >In [32]: b.iscontiguous() >Out[32]: 0 > >That seems to suggest that b shares the same data buffer than a. Indeed: > >In [36]: a._data >Out[36]: >In [37]: b._data >Out[37]: > > OK so far. >At this point, I believe that _bytestride should be different on both >arrays, but: > >In [33]: a._bytestride >Out[33]: 4 >In [34]: b._bytestride >Out[34]: 4 > >while I expected to find b._bytestride equal to 8. > >Is that an error or a lack of understanding on my part? > > What you are looking for is _strides. Since, in general, arrays can be multidimensional, the stride(s) need to be specified as a sequence. _bytestride appears to be equivalent to itemsize(); that is, it tells how many bytes are required to represent one item of the array. Here's what strides looks like in your case: >>> a._strides (4,) >>> b._strides (8,) Hope that helps, -tim >Cheers, > > > From jmiller at stsci.edu Tue Dec 21 07:06:03 2004 From: jmiller at stsci.edu (Todd Miller) Date: Tue Dec 21 07:06:03 2004 Subject: [Numpy-discussion] Striding on NumArray objects In-Reply-To: <200412211406.57699.faltet@carabos.com> References: <200412211406.57699.faltet@carabos.com> Message-ID: <1103641503.30020.49.camel@halloween.stsci.edu> On Tue, 2004-12-21 at 08:06, Francesc Altet wrote: > Hi, > > I'm a bit lost with the next example: > > In [28]: from numarray import * > In [29]: a=arange(10) > In [30]: a.iscontiguous() > Out[30]: 1 > In [31]: b=a[::2] > In [32]: b.iscontiguous() > Out[32]: 0 > > That seems to suggest that b shares the same data buffer than a. Indeed: > > In [36]: a._data > Out[36]: > In [37]: b._data > Out[37]: > > At this point, I believe that _bytestride should be different on both > arrays, but: > > In [33]: a._bytestride > Out[33]: 4 > In [34]: b._bytestride > Out[34]: 4 > > while I expected to find b._bytestride equal to 8. > > Is that an error or a lack of understanding on my part? Hi Francesc, This is a difficult question for me, but I think the answer is that the use of _bytestride is very limited. _bytestride is used to compute the "natural" strides of an almost contiguous array, e.g. a field of a recarray. That is, given a bytestride and a shape, the strides of a field of a contiguous RecArray are implied. However, once we start slicing (say in more than one dimension), _strides contains more and more information and is no longer implied by just the shape and bytestride but also by the history of slicing. From that perspective, it's not clear what _bytestride can be relied upon for in general or that it needs to be (or can be) kept up to date during slicing. FWIW, looking into this uncovered a related bug in numarray.strings where I tried to use _bytestride to do a simple iteration over all the elements of an array... that doesn't work. Regards, Todd From faltet at carabos.com Tue Dec 21 07:29:03 2004 From: faltet at carabos.com (Francesc Altet) Date: Tue Dec 21 07:29:03 2004 Subject: [Numpy-discussion] Striding on NumArray objects In-Reply-To: <1103641503.30020.49.camel@halloween.stsci.edu> References: <200412211406.57699.faltet@carabos.com> <1103641503.30020.49.camel@halloween.stsci.edu> Message-ID: <200412211628.10968.faltet@carabos.com> Hi Todd, A Dimarts 21 Desembre 2004 16:05, vareu escriure: > This is a difficult question for me, but I think the answer is that the > use of _bytestride is very limited. _bytestride is used to compute the > "natural" strides of an almost contiguous array, e.g. a field of a > recarray. That is, given a bytestride and a shape, the strides of a > field of a contiguous RecArray are implied. Yes, I was trying to use _bytestride mainly in RecArray contexts. It just happens that I've used a NumArray object to check the _bytestride behaviour. > However, once we start slicing (say in more than one dimension), > _strides contains more and more information and is no longer implied by > just the shape and bytestride but also by the history of slicing. From > that perspective, it's not clear what _bytestride can be relied upon > for in general or that it needs to be (or can be) kept up to date during > slicing. Well, for the time being it seems that RecArray does not support multidimensionalty, so I can just use _bytestride as a shortcut of _strides[0] here. Although now that I think more about this, using itemsize() as Tim suggested, would be the best to compute strides in RecArrays; however this is supposing that fields in RecArrays are contiguous. This would be always the case? > FWIW, looking into this uncovered a related bug in numarray.strings > where I tried to use _bytestride to do a simple iteration over all the > elements of an array... that doesn't work. Good! Regards, -- Francesc Altet ? >qo< ? http://www.carabos.com/ C?rabos Coop. V. ? V ?V ? Enjoy Data ? ? ? ? ? ? ? ? ? ? "" From jmiller at stsci.edu Tue Dec 21 12:22:03 2004 From: jmiller at stsci.edu (Todd Miller) Date: Tue Dec 21 12:22:03 2004 Subject: [Numpy-discussion] Striding on NumArray objects Message-ID: <1103660500.30020.327.camel@halloween.stsci.edu> > > This is a difficult question for me, but I think the answer is that the > > use of _bytestride is very limited. _bytestride is used to compute the > > "natural" strides of an almost contiguous array, e.g. a field of a > > recarray. That is, given a bytestride and a shape, the strides of a > > field of a contiguous RecArray are implied. > > Yes, I was trying to use _bytestride mainly in RecArray contexts. It just > happens that I've used a NumArray object to check the _bytestride behaviour. > > > However, once we start slicing (say in more than one dimension), > > _strides contains more and more information and is no longer implied by > > just the shape and bytestride but also by the history of slicing. From > > that perspective, it's not clear what _bytestride can be relied upon > > for in general or that it needs to be (or can be) kept up to date during > > slicing. > > Well, for the time being it seems that RecArray does not support > multidimensionalty, so I can just use _bytestride as a shortcut of > _strides[0] here. I wouldn't. I had a veritable stroke of genius and talked to JC Hsu... He suggested doing concrete examples and also indicated that full multi-dimensional RecArray support is almost all (except for repr'ing) there. To be clear, in general, _bytestride != _itemsize. For a RecArray field (of a contiguous RecArray) where each row contains just one element, generally _strides[-1] == _bytestride != _itemsize: >>> r = rec.array(buffer=None, formats="i2,(3,4)f4,a3", shape=10) >>> r.field(0).info() class: shape: (10,) strides: (53,) byteoffset: 0 bytestride: 53 itemsize: 2 aligned: 0 contiguous: 0 data: byteorder: little byteswap: 0 type: Int16 For a RecArray field where each row contains more than one element, _strides[-1] == itemsize: >>> r.field(1).info() class: shape: (10, 3, 4) strides: (53, 16, 4) byteoffset: 2 bytestride: 53 itemsize: 4 aligned: 0 contiguous: 0 data: byteorder: little byteswap: 0 type: Float32 The situation gets more complicated still when someone starts slicing the field or RecArray: >>> x[::3].info() class: shape: (4, 3, 4) strides: (159, 16, 4) byteoffset: 2 bytestride: 53 itemsize: 4 aligned: 0 contiguous: 0 data: byteorder: little byteswap: 0 type: Float32 Multi-dimensional RecArrays add their own twist: >>> r = rec.array(buffer=None, formats="i2,(3,4)f4,a3",shape=(10,10,10)) >>> r.field(1).info() class: shape: (10, 10, 10, 3, 4) strides: (5300, 480, 48, 16, 4) byteoffset: 2 bytestride: 53 itemsize: 4 aligned: 0 contiguous: 0 data: byteorder: little byteswap: 0 type: Float32 HTH, Todd From faltet at carabos.com Wed Dec 22 04:42:04 2004 From: faltet at carabos.com (Francesc Altet) Date: Wed Dec 22 04:42:04 2004 Subject: [Numpy-discussion] Striding on NumArray objects In-Reply-To: <1103660500.30020.327.camel@halloween.stsci.edu> References: <1103660500.30020.327.camel@halloween.stsci.edu> Message-ID: <200412221341.08144.faltet@carabos.com> A Dimarts 21 Desembre 2004 21:21, Todd Miller va escriure: > I had a veritable stroke of genius and talked to JC Hsu... He suggested > doing concrete examples and also indicated that full multi-dimensional > RecArray support is almost all (except for repr'ing) there. Multidimensional RecArrays? Nice!. Is part of the code already in CVS? > To be clear, in general, _bytestride != _itemsize. For a RecArray > field (of a contiguous RecArray) where each row contains just one > element, generally _strides[-1] == _bytestride != _itemsize: Thanks for the examples. That has clarified *a lot* where to find the interesting data: it's clear now that what I need is _strides[0] (because I need to access elements in a column sequentially). And that even works for forthcoming multidimensional RecArrays which is good. Regards, -- Francesc Altet From juenglin at cs.pdx.edu Sun Dec 26 11:33:02 2004 From: juenglin at cs.pdx.edu (Ralf Juengling) Date: Sun Dec 26 11:33:02 2004 Subject: [Numpy-discussion] log bug? Message-ID: <1104089215.3979.6.camel@localhost.localdomain> Hi, I'm not sure this behavior is intended: >>> from numarray import * >>> log(array([0.0], type=Float64)) Warning: Encountered divide by zero(s) in log array([ inf]) >>> -inf would be a more useful result. ralf From jmiller at stsci.edu Sun Dec 26 14:16:05 2004 From: jmiller at stsci.edu (Todd Miller) Date: Sun Dec 26 14:16:05 2004 Subject: [Numpy-discussion] log bug? In-Reply-To: <1104089215.3979.6.camel@localhost.localdomain> References: <1104089215.3979.6.camel@localhost.localdomain> Message-ID: <1104099312.3332.1.camel@jaytmiller.comcast.net> On Sun, 2004-12-26 at 11:26 -0800, Ralf Juengling wrote: > Hi, > > I'm not sure this behavior is intended: > > >>> from numarray import * > >>> log(array([0.0], type=Float64)) > Warning: Encountered divide by zero(s) in log > array([ inf]) > >>> > > > -inf would be a more useful result. > > > ralf Hi Ralf, This is fixed in CVS now. Thanks, Todd From jdhunter at ace.bsd.uchicago.edu Fri Dec 31 05:46:05 2004 From: jdhunter at ace.bsd.uchicago.edu (John Hunter) Date: Fri Dec 31 05:46:05 2004 Subject: [Numpy-discussion] ANN: matplotlib-0.70 Message-ID: matplotlib is a 2D graphics package that produces plots from python scripts, the python shell, or embeds them in your favorite python GUI -- wx, gtk, tk, fltk currently supported with qt in the works. Unlike many python plotting alternatives is written in python, so it is easy to extend. matplotlib is used in the finance industry, web application servers, and many scientific and enginneering disciplines. With a large community of users and developers, matplotlib is approaching the goal of having a full featured, high quality, 2D plotting library for python. A lot of development has gone into matplotlib since the last major release, which I'll summarize here. For details, see the notes for the incremental releases at http://matplotlib.sf.net/whats_new.html. Major changes since matplotlib-0.60 - The alpha version of the users guide - http://matplotlib.sf.net/users_guide.pdf. There are still a number of sections to be completed, but it's a start! - The matlab namespace renamed pylab - if you are upgrading from a version older than 0.64, please remove site-packages/matplotlib before upgrading. See http://matplotlib.sourceforge.net/matlab_to_pylab.py - New plot types: contour plots (contour), polar charts (polar), horizontal bar charts (barh), pie charts (pie), sparse matrix visualization (spy and spy2). Eg, http://matplotlib.sf.net/screenshots.html#polar_demo - Full ipython http://ipython.scipy.org integration in the "pylab" mode for interactive control of matplotlib plots from the python shell. - A significantly improved interactive toolbar for panning, zooming, zoom to rect - see http://matplotlib.sf.net/tutorial.html#toolbar2. - New backends: FLTK, Cairo, GTKCairo - Text - rotated mathtext, mathtext for postscript, text bounding boxes - Colormaps - 14 colormaps built-in http://matplotlib.sf.net/screenshots.html#pcolor_demo - Images - performance optimizations for 4x faster large image handling, PIL support, interpolation and colorbar improvements, imread - Event handling for capturing mouse clicks, movements, keypresses, etc. - same pylab interface works across GUIs. See examples/keypress_demo.py, examples/picker_demo.py, examples/coords_demo.py - set and get matlab style property introspection - http://matplotlib.sf.net/examples/set_and_get.py - improved dates handling for dates and date string formatting from 0000-9999, eg http://matplotlib.sf.net/screenshots.html#finance_work - Be sure to check out the 120 examples at http://matplotlib.sf.net/examples Home page : http://matplotlib.sourceforge.net Downloads : http://sourceforge.net/projects/matplotlib Screenshots : http://matplotlib.sourceforge.net/screenshots.html Tutorial : http://matplotlib.sourceforge.net/tutorial.html Credits : http://matplotlib.sourceforge.net/credits.html John Hunter