From haase at msg.ucsf.edu Thu Dec 1 12:24:50 2005 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Thu Dec 1 12:24:50 2005 Subject: [Numpy-discussion] new scipyCore features in numarray - indexing... Message-ID: <200512011219.10076.haase@msg.ucsf.edu> (this was posted to SciPy on 2005-11-16 16:37 - maybe I got lost ;-) Todd, I'm just thinking of a nice feature that I think is now part of new scipyCore: Mixing index ranges in one axis with index lists in another. Example: I have index 4,7,9 that I'm intrested in: use a[ [4,7,9] ] If I want all section I obviously just say a[ : ] But what do I do in the 2d case where I want 4,7,9 in one axis and all in the other ? I understood that the new scipyCore allows a[:, [4,7,9]] whereas numarray gives an error !? Thanks, Sebastian Haase On Friday 04 November 2005 14:57, Todd Miller wrote: > Sebastian Haase wrote: > >Also I always need to thank Todd et al. for numarray which we are using > > for about 4 years now. > > I'm glad you found numarray useful. > > >I was following - I thought - all the postings here, but I don't remember > > when and what the reason was when a.type() changed to a.dtype (also > > there is a "dtypecode" somewhere !?). Any reference or explanation would > > be great. I have to say that the (old) parenthesis where always quite > > "annoying" ! ;-) > > > >Question: does the way allow assignments like "a.dtype = Float32". > >What does it do ? If not, is it raising an error (I had 2 different people > >yesterday who tried to assign to a.type here in our lab ...) > > > >Also is this now completely supported/tested and suggested for numarray ? > > (For the time numarray is still separate) > > I'm adding support for some of newcore's new interface features out of > desire to make it easier to migrate. Our intent is to make it possible > to write newcore code and run it on numarray now as newcore matures. > Not every newcore feature is going to be supported, but we'll make an > effort to support those which are easy to implement. Let me know is > there's some newcore idiom you want to use that numarray doesn't have yet. > > Regards, > Todd _______________________________________________ SciPy-user mailing list SciPy-user at scipy.net http://www.scipy.net/mailman/listinfo/scipy-user From perry at stsci.edu Thu Dec 1 12:35:45 2005 From: perry at stsci.edu (Perry Greenfield) Date: Thu Dec 1 12:35:45 2005 Subject: [Numpy-discussion] new scipyCore features in numarray - indexing... In-Reply-To: <200512011219.10076.haase@msg.ucsf.edu> References: <200512011219.10076.haase@msg.ucsf.edu> Message-ID: <7276d0988cd4efebba129819fb313251@stsci.edu> On Dec 1, 2005, at 3:19 PM, Sebastian Haase wrote: > (this was posted to SciPy on 2005-11-16 16:37 - maybe I got lost ;-) > Todd, > I'm just thinking of a nice feature that I think is now part of new > scipyCore: Mixing index ranges in one axis with index lists in > another. > Example: > I have index 4,7,9 that I'm intrested in: use a[ [4,7,9] ] > If I want all section I obviously just say a[ : ] > But what do I do in the 2d case where I want 4,7,9 in one axis and all > in the > other ? I understood that the new scipyCore allows a[:, [4,7,9]] > whereas numarray gives an error !? > > > Thanks, > Sebastian Haase If the question is whether we will be adding that feature to numarray, the answer is no. We don't have close to the resources to do that and work on porting our stuff to scipy_core at this time (anyone else that wants to is welcome). [As an aside, we considered adding that to numarray at the start, but decided against doing that because indexing seemed complex already; not that we are against the capability.] Perry From jmiller at stsci.edu Thu Dec 1 13:50:53 2005 From: jmiller at stsci.edu (Todd Miller) Date: Thu Dec 1 13:50:53 2005 Subject: [Numpy-discussion] new scipyCore features in numarray - indexing... In-Reply-To: <200512011219.10076.haase@msg.ucsf.edu> References: <200512011219.10076.haase@msg.ucsf.edu> Message-ID: <438F6DA9.3050003@stsci.edu> Sebastian Haase wrote: >(this was posted to SciPy on 2005-11-16 16:37 - maybe I got lost ;-) >Todd, >I'm just thinking of a nice feature that I think is now part of new > scipyCore: Mixing index ranges in one axis with index lists in another. >Example: > I have index 4,7,9 that I'm intrested in: use a[ [4,7,9] ] > If I want all section I obviously just say a[ : ] >But what do I do in the 2d case where I want 4,7,9 in one axis and all in the >other ? I understood that the new scipyCore allows a[:, [4,7,9]] >whereas numarray gives an error !? > > Yep, my impression is that the indexing in scipy newcore is an improvement over numarray which was itself a functional improvement over Numeric. I'm pretty sure fixing this in numarray is not a simple hack so, for now anyway, it won't get fixed. I wish I had better news... Regards, Todd >Thanks, >Sebastian Haase > >On Friday 04 November 2005 14:57, Todd Miller wrote: > > >>Sebastian Haase wrote: >> >> >>>Also I always need to thank Todd et al. for numarray which we are using >>>for about 4 years now. >>> >>> >>I'm glad you found numarray useful. >> >> >> >>>I was following - I thought - all the postings here, but I don't remember >>>when and what the reason was when a.type() changed to a.dtype (also >>>there is a "dtypecode" somewhere !?). Any reference or explanation would >>>be great. I have to say that the (old) parenthesis where always quite >>>"annoying" ! ;-) >>> >>>Question: does the way allow assignments like "a.dtype = Float32". >>>What does it do ? If not, is it raising an error (I had 2 different people >>>yesterday who tried to assign to a.type here in our lab ...) >>> >>>Also is this now completely supported/tested and suggested for numarray ? >>>(For the time numarray is still separate) >>> >>> >>I'm adding support for some of newcore's new interface features out of >>desire to make it easier to migrate. Our intent is to make it possible >>to write newcore code and run it on numarray now as newcore matures. >>Not every newcore feature is going to be supported, but we'll make an >>effort to support those which are easy to implement. Let me know is >>there's some newcore idiom you want to use that numarray doesn't have yet. >> >>Regards, >>Todd >> >> > >_______________________________________________ >SciPy-user mailing list >SciPy-user at scipy.net >http://www.scipy.net/mailman/listinfo/scipy-user > > >------------------------------------------------------- >This SF.net email is sponsored by: Splunk Inc. Do you grep through log files >for problems? Stop! Download the new AJAX search engine that makes >searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! >http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click >_______________________________________________ >Numpy-discussion mailing list >Numpy-discussion at lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > From oliphant at ee.byu.edu Thu Dec 1 17:12:09 2005 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu Dec 1 17:12:09 2005 Subject: [Numpy-discussion] Records in scipy core In-Reply-To: <438F6F10.3020503@stsci.edu> References: <438F5651.5060104@ee.byu.edu> <438F6F10.3020503@stsci.edu> Message-ID: <438F9EEE.904@ee.byu.edu> Christopher Hanley wrote: >Hi Travis, > >About a year ago (summer 2004) on the numpy distribution list there was >a lot of discussion of the records interface. I will dig through my >notes and put together a summary. > > Thanks for the pointers. I had forgotten about that discussion. I went back and re-read the thread. Here's a good link for others to re-read (the end of) this thread: http://news.gmane.org/find-root.php?message_id=%3cBD22BAC0.E9EB%25perry%40stsci.edu%3e I think some very good points were made. These points should be addressed from the context of scipy arrays which now support records in a very basic way. Because of this, we can support nested records of records --- but how is this to be presented to the user is still an open question (i.e. how do you build one...) I've finally been converted to believe that the notion of records is very important because it speaks of how to do the basic (typeless, mathless) array object that will go into Python correctly If we can get the general records type done right, then all the other types are examples of it. Thus, I would like to revive discussion of the record object for inclusion in scipy core. I pretty much agree with the semantics that Perry described in his final email (is this all implemented in numarray, yet?), except I would agree with Francesc Alted that a titles or labels concept should be allowed. I'm more enthusiastic about code than discussion, so I'm hoping for a short-lived discussion followed by actual code. I'm ready to do the implementation this week (I've already borrowed lots of great code from numarray which makes it easier), but feel free to chime in even if you read this later. In my mind, the discussion about the records array is primarily a discussion about the records data-type. The way I'm thinking, the scipy ndarray is a homogeneous collection of the same "thing." The big change in scipy core is that Numeric used to allow only certain data types, but now the ndarray can contain an arbitrary "void" data type. You can also add data-types to scipy core. These data-types are "almost" full members of the scipy data-type community. The "almost" is because the N*N casting matrix is not updated (this would require a re-design of how casting is considered). At some point, I'd like to fix this wart and make it so that data-types can be added at will -- I think if we get the record type right, I'll be able to figure out how to do this. We need to add a "record" data-type to scipy. Then, any array can be of "record" type, and there will be an additional "array scalar" that is what is returned when selecting a single element from the array. So, a record array would simply be an array of "records" plus some extra stuff for dealing with the mapping from field names to actual segments of the array element (we may decide that this mapping is general enough that all scipy arrays should have the capability of assigning names to sub-bytes of its main data-type and means of accessing those sub-bytes in which case the subclass is unnecessary). Let me explain further: Right now, the machinery is in place in scipy_core to get and set in any ndarray (regardless of its data-type) an arbitrary "field". A "field" in this context is defined as a sub-section of the basic element making up the array. Generically the sub-section is defined by an offset and a data-type or a tuple of a data type and a shape (to allow sub-arrays in a record). What I understand the user to want is the binding of a name to this generic sub-section descriptor. 1) Should we allow that for every scipy ndarray: complex data types have an obvious binding, would anybody want to name the first two bytes of their int32 array? I suggest holding off on this one until a records array is working.... 2) Supposing we don't go with number 1, we need to design a record data type that has this name-binding capability. The recarray class in scipy core SVN essentially just does this. Question: How important is backwards compatibility with old numarray specification. In particular, I would go with the .fields access described by Perry, and eliminate the .field() approach? Thanks for reading and any comments you can make. -Travis From perry at stsci.edu Fri Dec 2 05:31:03 2005 From: perry at stsci.edu (Perry Greenfield) Date: Fri Dec 2 05:31:03 2005 Subject: [Numpy-discussion] Records in scipy core In-Reply-To: <438F9EEE.904@ee.byu.edu> Message-ID: Travis Oliphant wrote: > > Thus, I would like to revive discussion of the record object for > inclusion in scipy core. I pretty much agree with the semantics that > Perry described in his final email (is this all implemented in numarray, > yet?), No, it was only talk to date, with plans to implment it, but other things have drawn our work up to now. > Question: How important is backwards compatibility with old numarray > specification. In particular, I would go with the .fields access > described by Perry, and eliminate the .field() approach? > For us, probably not critical since we have to do some rewriting anyway. (But it would be nice to retain for a while as deprecated). But what about field names that don't map well to attributes? I haven't had a chance to reread the past emails but I seem to recall this was a significant issue. That would imply that .field() would be needed for those cases anyway. Perry From strawman at astraw.com Fri Dec 2 08:22:07 2005 From: strawman at astraw.com (Andrew Straw) Date: Fri Dec 2 08:22:07 2005 Subject: [Numpy-discussion] Records in scipy core In-Reply-To: References: Message-ID: <439074A2.2090508@astraw.com> Perry Greenfield wrote: >Travis Oliphant wrote: > > >>Question: How important is backwards compatibility with old numarray >>specification. In particular, I would go with the .fields access >>described by Perry, and eliminate the .field() approach? >> >> >> >For us, probably not critical since we have to do some rewriting anyway. >(But it would be nice to retain for a while as deprecated). > >But what about field names that don't map well to attributes? >I haven't had a chance to reread the past emails but I seem to >recall this was a significant issue. That would imply that .field() >would be needed for those cases anyway. > > I haven't read the above thread extensively, but the issue of field names that don't map well to attributes is significant. For example, users of pytables often have columns with names that are not valid Python names. So, regardless of what solution is the most obvious, there should at least be a way to get as such field names. (pytables users are used to doing that.) Cheers! Andrew From cjw at sympatico.ca Fri Dec 2 08:52:03 2005 From: cjw at sympatico.ca (Colin J. Williams) Date: Fri Dec 2 08:52:03 2005 Subject: [Numpy-discussion] Records in scipy core In-Reply-To: <438F9EEE.904@ee.byu.edu> References: <438F5651.5060104@ee.byu.edu> <438F6F10.3020503@stsci.edu> <438F9EEE.904@ee.byu.edu> Message-ID: <43907B7A.7000905@sympatico.ca> Travis Oliphant wrote: > Christopher Hanley wrote: > >> Hi Travis, >> >> About a year ago (summer 2004) on the numpy distribution list there >> was a lot of discussion of the records interface. I will dig through >> my notes and put together a summary. >> >> > Thanks for the pointers. I had forgotten about that discussion. I > went back and re-read the thread. > > Here's a good link for others to re-read (the end of) this thread: > > http://news.gmane.org/find-root.php?message_id=%3cBD22BAC0.E9EB%25perry%40stsci.edu%3e > > > I think some very good points were made. These points should be > addressed from the context of scipy arrays which now support records > in a very basic way. Because of this, we can support nested records > of records --- but how is this to be presented to the user is still an > open question (i.e. how do you build one...) > > I've finally been converted to believe that the notion of records is > very important because it speaks of how to do the basic (typeless, > mathless) array object that will go into Python correctly If we can > get the general records type done right, then all the other types are > examples of it. > > Thus, I would like to revive discussion of the record object for > inclusion in scipy core. I pretty much agree with the semantics that > Perry described in his final email (is this all implemented in > numarray, yet?), except I would agree with Francesc Alted that a > titles or labels concept should be allowed. > I'm more enthusiastic about code than discussion, so I'm hoping for a > short-lived discussion followed by actual code. I'm ready to do the > implementation this week (I've already borrowed lots of great code > from numarray which makes it easier), but feel free to chime in even > if you read this later. > > In my mind, the discussion about the records array is primarily a > discussion about the records data-type. The way I'm thinking, the > scipy ndarray is a homogeneous collection of the same "thing." The > big change in scipy core is that Numeric used to allow only certain > data types, but now the ndarray can contain an arbitrary "void" data > type. You can also add data-types to scipy core. These data-types > are "almost" full members of the scipy data-type community. The > "almost" is because the N*N casting matrix is not updated (this would > require a re-design of how casting is considered). At some point, > I'd like to fix this wart and make it so that data-types can be added > at will -- I think if we get the record type right, I'll be able to > figure out how to do this. > > We need to add a "record" data-type to scipy. Then, any array can be > of "record" type, and there will be an additional "array scalar" that > is what is returned when selecting a single element from the array. > So, a record array would simply be an array of "records" plus some > extra stuff for dealing with the mapping from field names to actual > segments of the array element (we may decide that this mapping is > general enough that all scipy arrays should have the capability of > assigning names to sub-bytes of its main data-type and means of > accessing those sub-bytes in which case the subclass is unnecessary). > Let me explain further: Right now, the machinery is in place in > scipy_core to get and set in any ndarray (regardless of its data-type) > an arbitrary "field". A "field" in this context is defined as a > sub-section of the basic element making up the array. Generically > the sub-section is defined by an offset and a data-type or a tuple of > a data type and a shape (to allow sub-arrays in a record). What I > understand the user to want is the binding of a name to this generic > sub-section descriptor. > 1) Should we allow that for every scipy ndarray: complex data types > have an obvious binding, would anybody want to name the first two > bytes of their int32 array? I suggest holding off on this one until a > records array is working.... > > 2) Supposing we don't go with number 1, we need to design a record > data type that has this name-binding capability. > > The recarray class in scipy core SVN essentially just does this. > > Question: How important is backwards compatibility with old numarray > specification. In particular, I would go with the .fields access > described by Perry, and eliminate the .field() approach? > I feel that it is not particularly important. Having a good design is the thing to shoot for. > > Thanks for reading and any comments you can make. > > -Travis > I'm not clear as to what the current design objective is and so I'll try to recap and perhaps expand my pieces in the referenced discussion to set out the sort of arrangement I would like to see. We are moving towards having a multi-dimensional array which can hold objects of fixed size and type, the smallest being one byte (although the void would appear to be a collection of no size objects). Most of the need, and thus the focus, is on numeric objects, ranging in size from Int8 to Complex64. The Record is a fixed size object containing fields. Each field has a name, an optional title and data of a fixed type (perhaps including another record instance and maybe arrays of fixed size?). In the example below, AddressRecord and PersonRecord would be sub-classes of Record where the fields are named and, optionally, field titles given. The names would be consistent with Python naming whereas the title could be any Python string. The use of attributes raises the possibility that one could have nested records. For example, suppose one has an address record: addressRecord streetNumber streetName postalCode ... There could then be a personal record: personRecord ... officeAddress homeAddress ... One could address a component as rec.homeAddress.postalCode. Suppose one has a (n, n) array of persons then one could access the data in the following ways: persons[1] all records in the second row persons[:,1] all records in the second column persons[1, 1] return a specific person record persons[1, 1].homeAddress the home address record for a specific person persons[1, 1].homeAddress.postalCode the postal code for a specific person persons.homeAddress.postalCode an (n, n) array containing all postal codes persons.homeAddress.postalCode.title could be 'Zip Code' I see no need to have the attribute 'field' and would like to avoid the use of strings to identify a record component. This does require that fields be named as Python identifiers but is this restriction a killer? Colin W. From oliphant.travis at ieee.org Fri Dec 2 09:14:01 2005 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri Dec 2 09:14:01 2005 Subject: [Numpy-discussion] Records in scipy core In-Reply-To: <43907B7A.7000905@sympatico.ca> References: <438F5651.5060104@ee.byu.edu> <438F6F10.3020503@stsci.edu> <438F9EEE.904@ee.byu.edu> <43907B7A.7000905@sympatico.ca> Message-ID: <4390809F.7050004@ieee.org> > I'm not clear as to what the current design objective is and so I'll > try to recap and perhaps expand my pieces in the referenced discussion > to set out the sort of arrangement I would like to see. I have two objectives: 1) Make the core scipy array object flexible enough to support a very good records sub-class. In other works, I wonder if the core scipy array object could be made flexible enough to be used as a decent record array by itself, without adding much difficulty. In the process, I'm really trying to understand how the data-type of an array should be generally considered. An array object that has this generic perspective on data-type is what should go into Python, I believe. 2) Make a (more) useful records subclass of the ndarray object that is perhaps easier for the end-user to use. Involved with this, of course, is making functions that make it easy to create a records sub-class. > > We are moving towards having a multi-dimensional array which can hold > objects of fixed size and type, the smallest being one byte (although > the void would appear to be a collection of no size objects). Most of > the need, and thus the focus, is on numeric objects, ranging in size > from Int8 to Complex64. > > The Record is a fixed size object containing fields. Each field has a > name, an optional title and data of a fixed type (perhaps including > another record instance and maybe arrays of fixed size?). Right, so the record is really another kind of data-type. The concept of the multidimensional array does not need adjustment, but the concept of what constitutes a data-type may need some fixing up. > > In the example below, AddressRecord and PersonRecord would be > sub-classes of Record where the fields are named and, optionally, > field titles given. The names would be consistent with Python naming > whereas the title could be any Python string. I like the notion of titles and names. I think they are both useful. > > The use of attributes raises the possibility that one could have > nested records. For example, suppose one has an address record: Now, I'm in favor of attribute access. But, nested records is possible without attribute access (it's just extra typing). It's the underlying structure that provides the possibility for nested records (and that's what I'm trying to figure out how to support, generally). If I can support this generally in the basic ndarray object by augmenting the notion of data-type as appropriate, then making a subclass that has the nice syntatic sugar is easy. So, there are two issues really. 1) How to think about the data-type of a general ndarray object in order to support nested records in a straightforward way. 2) What should the syntatic sugar of a record array subclass be... I suppose a third is 3) How much of the syntatic sugar should be applied to all ndarray's? -Travis > I see no need to have the attribute 'field' and would like to avoid > the use of strings to identify a record component. This does require > that fields be named as Python identifiers but is this restriction a > killer? For a record array subclass that may be true. But, as was raised by others in the previous thread, there are problems of "name-space" collision with the methods and attributes of the array that would prevent certain names from being used (and any additions to the methods and attributes of the array would cause incompatibilities with some-people's records). But, at this point, I like the readability of the attribute access approach and could support it. -Travis From oliphant.travis at ieee.org Fri Dec 2 10:37:03 2005 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri Dec 2 10:37:03 2005 Subject: [Numpy-discussion] Records in scipy core In-Reply-To: References: Message-ID: <4390942F.6090302@ieee.org> Perry Greenfield wrote: > > >For us, probably not critical since we have to do some rewriting anyway. >(But it would be nice to retain for a while as deprecated). > > Easy enough to do by defining an actual record array (however, see below). I've been retaining backwards compatibility in other ways while not documenting it. For example, you can actually now pass in strings like 'Int32' for types. >But what about field names that don't map well to attributes? >I haven't had a chance to reread the past emails but I seem to >recall this was a significant issue. That would imply that .field() >would be needed for those cases anyway. > > What I'm referring to as the solution here is a slight modification to what Perry described. In other words, all arrays have the attribute .fields You can set this attribute to a dictionary which will automagically gives field names to any array (this dictionary has ordered lists of 'names', (optionally) 'titles', and "(data-descr, [offset])" lists which defines the mapping. If offset is not given, then the "next-available" offset is assumed. The data-descr is either 1) a data-type or 2) a tuple of (data-type, shape). The data-type is either a defined data-type or alias, or an object with a .fields attribute that provides the same dictionary and an .itemsize attribute that computes the total size of the data-type. You can get this attribute which returns a special fields object (written in Python initially like the flags attribute) that can look up field names like a dictionary, or with attribute access for names that are either 1) acceptable or 2) have a user-provided "python-name" associated with them. Thus, .fields['home address'] would always work but .fields.hmaddr would only work if the user had previously made the association hmaddr -> 'home address' for the data type of this array. Thus 'home address' would be a title but hmaddr would be the name. The records module would simply provide functions for making record arrays and a record data type. Driving my thinking is the concept that the notion of a record array is really a description of the data type of the array (not the array itself). Thus, all the fields information should really just be part of the data type itself. Now, I don't really want to create and register a new data type every time somebody has a new record layout. So, I've been re-thinking the notion of "registering a data-type". It seems to me that while it's O.K. to have a set of pre-defined data types. The notion of data-type ought to be flexible enough to allow the user to define one "on-the-fly". I'm thinking of ways to do this right now. Any suggestions are welcome. -Travis From golux at comcast.net Fri Dec 2 11:03:05 2005 From: golux at comcast.net (Stephen Waterbury) Date: Fri Dec 2 11:03:05 2005 Subject: [Numpy-discussion] Records in scipy core In-Reply-To: <4390942F.6090302@ieee.org> References: <4390942F.6090302@ieee.org> Message-ID: <439099D3.2050803@comcast.net> Travis Oliphant wrote: > So, I've been re-thinking the notion of "registering a data-type". It > seems to me that while it's O.K. to have a set of pre-defined data > types. The notion of data-type ought to be flexible enough to allow the > user to define one "on-the-fly". > I'm thinking of ways to do this right now. Any suggestions are welcome. I'm doing that in an application I'm developing. My objects have an attribute called '_schema' that is an instance of Zope InterfaceClass. An object (read "record" ;) is assigned a _schema when it is instantiated, and all information about its attributes (a.k.a. "fields") is contained in the _schema's Properties (my 'Property' subtypes the Zope interfaces 'Attribute' type, and has a host of (meta-)attributes like 'domain', 'range', 'id', 'name', etc. -- which could easily be extended to include things like 'title', but I use another mechanism for display characteristics, called 'DisplayMap', which can be used to specify the order in which you want the object's properties to appear in a grid, what you want their "display names" to be, etc. ... which are also customizable by the end-user. Let me know if this sounds interesting. Cheers, Steve From cjw at sympatico.ca Fri Dec 2 12:05:01 2005 From: cjw at sympatico.ca (Colin J. Williams) Date: Fri Dec 2 12:05:01 2005 Subject: [Numpy-discussion] Records in scipy core In-Reply-To: <4390942F.6090302@ieee.org> References: <4390942F.6090302@ieee.org> Message-ID: <4390A8C5.3070105@sympatico.ca> Travis Oliphant wrote: > Perry Greenfield wrote: > >> >> >> For us, probably not critical since we have to do some rewriting anyway. >> (But it would be nice to retain for a while as deprecated). >> >> > Easy enough to do by defining an actual record array (however, see > below). I've been retaining backwards compatibility in other ways > while not documenting it. For example, you can actually now pass in > strings like 'Int32' for types. > >> But what about field names that don't map well to attributes? >> I haven't had a chance to reread the past emails but I seem to >> recall this was a significant issue. That would imply that .field() >> would be needed for those cases anyway. >> >> > What I'm referring to as the solution here is a slight modification to > what Perry described. In other words, all arrays have the attribute > > .fields What I suggested in my posting was that there is no need and no benefit from the .fields attribute. The base class Record could be organized so that certain attributes which are used in arrays are not acceptable. For example, one would probably wish to avoid shape, size and the other attributes of the basic array but attributes associated with arrays with numeric types would probably not need to be barred. > > You can set this attribute to a dictionary which will automagically > gives field names to any array (this dictionary has ordered lists of > 'names', (optionally) 'titles', and "(data-descr, [offset])" lists > which defines the mapping. If offset is not given, then the > "next-available" offset is assumed. The data-descr is either 1) a > data-type or 2) a tuple of (data-type, shape). The data-type is > either a defined data-type or alias, or an object with a .fields > attribute that provides the same dictionary and an .itemsize attribute > that computes the total size of the data-type. > I wonder about the need for explicit dictionary operations. Can't this be handled through the class structure? > > You can get this attribute which returns a special fields object > (written in Python initially like the flags attribute) that can look > up field names like a dictionary, or with attribute access for names > that are either 1) acceptable or 2) have a user-provided "python-name" > associated with them. > Thus, > > .fields['home address'] > > would always work > > but > > .fields.hmaddr > > would only work if the user had previously made the association hmaddr > -> 'home address' for the data type of this array. Thus 'home > address' would be a title but hmaddr would be the name. > > The records module would simply provide functions for making record > arrays and a record data type. > Driving my thinking is the concept that the notion of a record array > is really a description of the data type of the array (not the array > itself). Thus, all the fields information should really just be part > of the data type itself. Now, I don't really want to create and > register a new data type every time somebody has a new record layout. > A record array is an array which has a record as its data element, in the same way that an integer array has an integer as its element. I don't understand the notion of registring a data type. Presumably an integer array has a pointer to the appropriate type of integer. Could the record array not have a pointer to the appropriate record type? > So, I've been re-thinking the notion of "registering a data-type". It > seems to me that while it's O.K. to have a set of pre-defined data > types. The notion of data-type ought to be flexible enough to allow > the user to define one "on-the-fly". > I'm thinking of ways to do this right now. Any suggestions are welcome. > The record types would be created "on-the-fly" as the class is instatiated. The array, through the dtype parameter would point to the record type. > > -Travis Colin W. From cjw at sympatico.ca Fri Dec 2 12:12:09 2005 From: cjw at sympatico.ca (Colin J. Williams) Date: Fri Dec 2 12:12:09 2005 Subject: [Numpy-discussion] Records in scipy core In-Reply-To: <439099D3.2050803@comcast.net> References: <4390942F.6090302@ieee.org> <439099D3.2050803@comcast.net> Message-ID: <4390AA7A.3050304@sympatico.ca> Stephen Waterbury wrote: > Travis Oliphant wrote: > >> So, I've been re-thinking the notion of "registering a data-type". >> It seems to me that while it's O.K. to have a set of pre-defined data >> types. The notion of data-type ought to be flexible enough to allow >> the user to define one "on-the-fly". >> I'm thinking of ways to do this right now. Any suggestions are welcome. > > > I'm doing that in an application I'm developing. > My objects have an attribute called '_schema' that is an instance > of Zope InterfaceClass. An object (read "record" ;) is assigned a > _schema > when it is instantiated, and all information about its attributes (a.k.a. > "fields") is contained in the _schema's Properties (my 'Property' > subtypes > the Zope interfaces 'Attribute' type, and has a host of (meta-)attributes > like 'domain', 'range', 'id', 'name', etc. -- which could easily be > extended to include things like 'title', but I use another mechanism > for display characteristics, called 'DisplayMap', which can be used to > specify the order in which you want the object's properties to appear > in a grid, what you want their "display names" to be, etc. ... which are > also customizable by the end-user. > > Let me know if this sounds interesting. > > Cheers, > Steve This is goes further than my suggestion. For arrays, it seems to me that an additional pointer to _schema is not needed as there is a pointer to the data type and the data type can contain the meta data Colin W. From golux at comcast.net Fri Dec 2 12:31:03 2005 From: golux at comcast.net (Stephen Waterbury) Date: Fri Dec 2 12:31:03 2005 Subject: [Numpy-discussion] Records in scipy core In-Reply-To: <4390AA7A.3050304@sympatico.ca> References: <4390942F.6090302@ieee.org> <439099D3.2050803@comcast.net> <4390AA7A.3050304@sympatico.ca> Message-ID: <4390AEE5.3050309@comcast.net> Colin J. Williams wrote: > Stephen Waterbury wrote: >> I'm doing that in an application I'm developing. >> My objects have an attribute called '_schema' that is an instance >> of Zope InterfaceClass. An object (read "record" ;) is assigned a >> _schema >> when it is instantiated, and all information about its attributes (a.k.a. >> "fields") is contained in the _schema's Properties (my 'Property' >> subtypes >> the Zope interfaces 'Attribute' type, and has a host of (meta-)attributes >> like 'domain', 'range', 'id', 'name', etc. -- which could easily be >> extended to include things like 'title', but I use another mechanism >> for display characteristics, called 'DisplayMap', which can be used to >> specify the order in which you want the object's properties to appear >> in a grid, what you want their "display names" to be, etc. ... which are >> also customizable by the end-user. > > This is goes further than my suggestion. For arrays, it seems to me > that an additional pointer to _schema is not needed as there is a > pointer to the data type and the data type can contain the meta data In effect, the _schema *is* the data type in my scenario. It contains all information necessary to describe the type of the object and has references to the types of all the object's attributes (which are called "fields" in record parlance, "Properties" in the world of ontologies, and "Attributes" in Zope Interfaces and UML terminology). Steve From golux at comcast.net Fri Dec 2 12:32:00 2005 From: golux at comcast.net (Stephen Waterbury) Date: Fri Dec 2 12:32:00 2005 Subject: [Numpy-discussion] Records in scipy core In-Reply-To: <4390AA7A.3050304@sympatico.ca> References: <4390942F.6090302@ieee.org> <439099D3.2050803@comcast.net> <4390AA7A.3050304@sympatico.ca> Message-ID: <4390AF0A.8080708@comcast.net> Colin J. Williams wrote: > Stephen Waterbury wrote: >> I'm doing that in an application I'm developing. >> My objects have an attribute called '_schema' that is an instance >> of Zope InterfaceClass. An object (read "record" ;) is assigned a >> _schema >> when it is instantiated, and all information about its attributes (a.k.a. >> "fields") is contained in the _schema's Properties (my 'Property' >> subtypes >> the Zope interfaces 'Attribute' type, and has a host of (meta-)attributes >> like 'domain', 'range', 'id', 'name', etc. -- which could easily be >> extended to include things like 'title', but I use another mechanism >> for display characteristics, called 'DisplayMap', which can be used to >> specify the order in which you want the object's properties to appear >> in a grid, what you want their "display names" to be, etc. ... which are >> also customizable by the end-user. > > This is goes further than my suggestion. For arrays, it seems to me > that an additional pointer to _schema is not needed as there is a > pointer to the data type and the data type can contain the meta data In effect, the _schema *is* the data type in my scenario. It contains all information necessary to describe the type of the object and references to the types of all the object's attributes (which are called "fields" in record parlance, "Properties" in the world of ontologies, and "Attributes" in Zope Interfaces and UML terminology). Steve From antonykummel at yahoo.com Sun Dec 4 05:30:02 2005 From: antonykummel at yahoo.com (Antony Kummel) Date: Sun Dec 4 05:30:02 2005 Subject: [Numpy-discussion] py2exe compliance Message-ID: <20051204132710.57542.qmail@web33903.mail.mud.yahoo.com> Hi! I have used a trick to make py2exe recognize all of numarray's modules, which I suggest be incorporated into the numarray CVS. It can be a real pain to solve this problem the first time you're py2exe-ing a script using numarray and find out that it simply doesn't work. I just added this to numarray's __init__.py: # this is needed for py2exe to be able to include numarray def _give_py2exe_hints(): # pyd's import numarray.libnumarray import numarray.libnumeric import numarray.memory import numarray._bytes import numarray._chararray import numarray._conv import numarray._converter import numarray._ndarray import numarray._numarray import numarray._objectarray import numarray._operator import numarray._sort import numarray._ufunc import numarray._ufuncBool import numarray._ufuncComplex32 import numarray._ufuncComplex64 import numarray._ufuncFloat32 import numarray._ufuncFloat64 import numarray._ufuncInt16 import numarray._ufuncInt32 import numarray._ufuncInt64 import numarray._ufuncInt8 import numarray._ufuncUInt16 import numarray._ufuncUInt32 import numarray._ufuncUInt8 # py's import numarray.arrayprint import numarray.array_persist import numarray.dotblas import numarray.generic import numarray.ieeespecial import numarray.memmap import numarray.memorytest import numarray.numarrayall import numarray.numarraycore import numarray.numarrayext import numarray.numeric import numarray.numerictypes import numarray.numinclude import numarray.numtest import numarray.objects import numarray.readonly import numarray.records import numarray.safethread import numarray.strings import numarray.teacup import numarray.testall import numarray.testdata import numarray.typeconv import numarray.ufunc import numarray._ufuncall # subpackages import numarray.convolve import numarray.convolve.lineshape import numarray.fft import numarray.image import numarray.linear_algebra import numarray.ma import numarray.matrix import numarray.mlab import numarray.nd_image import numarray.random_array --------------------------------- Yahoo! Personals Single? There's someone we'd like you to meet. Lots of someones, actually. Yahoo! Personals -------------- next part -------------- An HTML attachment was scrubbed... URL: From cjw at sympatico.ca Sun Dec 4 13:48:00 2005 From: cjw at sympatico.ca (Colin J. Williams) Date: Sun Dec 4 13:48:00 2005 Subject: [Numpy-discussion] Numeric3.0 - Currently available under scipy/base as version 0.6.1 (Windows) In-Reply-To: <43725D8A.9090307@ee.byu.edu> References: <437209BF.8090607@sympatico.ca> <43725D8A.9090307@ee.byu.edu> Message-ID: <439363D6.3010300@sympatico.ca> Travis Oliphant wrote: > Colin J. Williams wrote: > >> >> I have a package based on subclassing numarray, which is working >> satisfactorily, and am looking at the possibility of transferring the >> package to work under the revised Numeric. >> >> My feeling is that the transfer is probably feasible but that it >> would be premature to work on it at this time. > > > That's unfortunate. The more feedback we get early on about > subclassing, the better. The publication of Python style defs, together with doc strings describing the acceptable arguments for the various __new__ methods would help here. I don't find info on ndarray.__new__. Incidentally, ArrayType seems a better, more meaningful name for the class/type. >> >> One of the problems is the cluttered namespace, through the use of >> "from X import *". This is a style which is deprecated, see page 401 >> of Alex Martelli's /Python in a Nutshell/. > > > You will have to be more specific about what you think is wrong. What > namespace is "cluttered" exactly. Just because use is made of from X > import * in one module does not mean everything is "cluttered". SciPy > Core makes use of the __all__ variables to control what gets imported > and usually only specific functions are imported as necessary. > >> Another problem, at this stage, is that many doc strings are missing >> and that some which exist are a little cryptic. > > > I would submit there are more docstrings then Numeric had. Jump in > and help. The water is fine. > >> >> I would like to take another look when the next win32 binaries are >> available. > > > There has been much improvement since the last beta. I'm trying to > track down some remaining memory leaks before releasing new windows > binaries. The SVN code is always available for check out and it is > quite easy to build. We could always use more build testers to make > sure building is going as smoothly as we believe it is. > I now have 0.6.1, thanks. Now, some documentation on ndarray/ArrayType would be helpful, with the fact that __new__ alone is used instead of the usual __new__ + __init__. Among the parameters is buffer, it seems that this accepts buffers or strings, anything else? Iinformation on call backs (from ufuncs?) would help. For many cases, it would be better to create a Sub instance with Sub(aList) but, if the subclass must respond to callbacks then this may be inhibited. >> >> Some further thoughts on the present state of Numeric3.0 are >> available here . > > > > Most of your comments have more to do with differences between builtin > types and Python classes than anything about scipy. The type-class > chasm has shrunken of late, but there are still remaining issues. > These are Python issues that I believe are of little concern. > > I will comment on your issues that are not related to the above comment: > > > Use of package __init__.py to create namespace. > > If the epydoc and pydoc tools are not respecting the __init__.py code > then I would say they are broken. Using the __init__.py this way > frees the user from having to understand every little detail of the > package structure (which could also change as better organization is > obtained in the future). > > > Use of the from X import Y style > > Please give more support here. Just because one Python user advocates > against it is not sufficient argument. There is an argument to be > made for avoiding attribute lookups by importing the names directly in > this fashion. > > > *Methods vs functions* > > I agree that methods and functions are somewhat redundant. However, > the functions are still needed both to support lists and especially > for back-wards compatibility. Your example using take is odd (perhaps > it's a bug in an old release). I get consistent behavior. One > problem is you define a 1-d array in one case and a 2-d array in > another. Of course the answers will be different. > One difference is that the default axis argument is different. The > old functions have exactly the same default axis argument as they > always did, while the methods have a default of None for the axis > (which means treat the array as flat). > > > Lack of basic information in the doc strings > > Your examples are all class constructors. First of all, there is no > special __init__ method for the ndarray because __new__ takes care of > it. Second of all, the new method does need better documentation. > I'm not sure where to put it, though. The array_new function is > placed in the TypeObject of the array type. The __new__ attribute is > pasted on by PyTypeReady. I'm not sure how to give a docstring as well. > I suspect the proper thing to do is place the information in the > docstring for the Type Object. > > > -Travis > I was wrong in my earlier comment about take but there are some problems as, I hope, the script below illustrates. Colin W. # tTake.py ''' In those cases where there is a duplication, it is suggested that only the method is needed unless the function handles a wider range of data than a method. The function take does more than the method: >>> S.ndarray((1,2)).take([0]) array([[ 1.05176460e-305, 5.63390393e-311]]) >>> S.take([[1, 2]], [0]) array([[1, 2]]) >>> S.take.__doc__ >>> ''' import numarray.numarraycore as N import numarray.numerictypes as NT import scipy as S a= S.ndarray((2, ), dtype= S.int8, buffer= '12') print 'a:', `a` b= S.arange(12, dtype= S.int8).reshape(3, 4) print 'b:', `b` print 'b.take(((0, 0), (2, 2)), 1):', `b.take(((0, 0), (2, 2)), 1)` c= N.arange(12, dtype=NT.Int8, shape=(3, 4)) # this is a bit easier to use print 'c:', `c` print 'c.take(((0, 0), (2, 2)), 1):', `c.take(((0, 0), (2, 2)), 1)` # not quite the same as above print 'b-c:', `b-c` # this needs to raise an exception From vtrandal at yahoo.com Sun Dec 4 16:23:01 2005 From: vtrandal at yahoo.com (Vincent Randal) Date: Sun Dec 4 16:23:01 2005 Subject: [Numpy-discussion] Re: scipy cygwin compilation error References: <437F7D48.9070406@u.washington.edu> Message-ID: Jordan and others, I have the same problem installing scipy in cygwin. Has anyone found a solution? Specifically "python setup.py install" fails with messages like: "scipy/base/src/ufuncobject.c:475: undefined reference" Vincent Randal Longmont, CO From haase at msg.ucsf.edu Mon Dec 5 12:14:04 2005 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Mon Dec 5 12:14:04 2005 Subject: [Numpy-discussion] BUG in nd_image Message-ID: <200512051213.07316.haase@msg.ucsf.edu> Hi, When I call nd_image.rotate with reshape=False I always get "output shape not correct" >>> U.nd.rotate(d[0], 20, axes=(-1, -2), reshape=0, output=d[1], order=1, mode="constant", cval=0.0, prefilter=0) Traceback (most recent call last): File "", line 1, in ? File "/jws30/haase/PrLin/numarray/nd_image/interpolation.py", line 351, in rotate output, order, mode, cval, prefilter) File "/jws30/haase/PrLin/numarray/nd_image/interpolation.py", line 205, in affine_transform output_type) File "/jws30/haase/PrLin/numarray/nd_image/_ni_support.py", line 73, in _get_output raise RuntimeError, "output shape not correct" RuntimeError: output shape not correct I tracked the problem down to "inputShape != outputShape" one being a tuple the output shape being a list. (Pdb) p shape [128, 528] (Pdb) p output.shape (128, 528) (Pdb) p shape != output.shape 1 (Pdb) p shape , output.shape ([128, 528], (128, 528)) (Pdb) I'm using a CVS version around 1.3 ( /ni_interpolation.c/1.17/Fri Apr 22 20:35:27 2005//THEAD) but I took a look at the current CVS and it seems to still be a problem Looks like I'm the only one who doesn't want the reshape ;-) Thanks, Sebastian Haase From oliphant.travis at ieee.org Mon Dec 5 14:08:02 2005 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Dec 5 14:08:02 2005 Subject: [Numpy-discussion] Data type change completed Message-ID: <4394BA01.20501@ieee.org> I've committed the data-type change discussed at the end of last week to the SVN repository. Now the concept of a data type for an array has been replaced with a "data-descriptor". This data-descriptor is flexible enough to handle an arbitrary record specification with fields that include records and arrays or arrays of records. While nesting may not be the best data-layout for a new design, when memory-mapping an arbitrary fixed-record-length file, this capability allows you to handle even the most obsure record file. While the basic core tests pass for me, there may be lurking problems and so testing of the SVN trunk of scipy core will be appreciated. I've bumped up the version number because the C-API has changed (a few new functions and some functions becoming macros). I'd like to make a release of the new version by the end of the week (as soon as Chris Hanley at STSCI and I get records.py working better), so please test. Recently some intel c-compiler tests were failing on a 64-bit platform. It would be nice to figure out why that is happening as well, but I will probably not have time for that this week. Thanks, -Travis From verveer at embl-heidelberg.de Mon Dec 5 14:32:02 2005 From: verveer at embl-heidelberg.de (Peter Verveer) Date: Mon Dec 5 14:32:02 2005 Subject: [Numpy-discussion] BUG in nd_image In-Reply-To: <200512051213.07316.haase@msg.ucsf.edu> References: <200512051213.07316.haase@msg.ucsf.edu> Message-ID: <88B304C0-B5B3-461C-B8A7-FFD74F1BDAF6@embl-heidelberg.de> Works for me with the latest CVS version. On 5 Dec, 2005, at 21:13, Sebastian Haase wrote: > Hi, > When I call nd_image.rotate with reshape=False I always get > "output shape not correct" >>>> U.nd.rotate(d[0], 20, axes=(-1, -2), reshape=0, output=d[1], >>>> order=1, > mode="constant", cval=0.0, prefilter=0) > Traceback (most recent call last): > File "", line 1, in ? > File "/jws30/haase/PrLin/numarray/nd_image/interpolation.py", > line 351, in > rotate > output, order, mode, cval, prefilter) > File "/jws30/haase/PrLin/numarray/nd_image/interpolation.py", > line 205, in > affine_transform > output_type) > File "/jws30/haase/PrLin/numarray/nd_image/_ni_support.py", line > 73, in > _get_output > raise RuntimeError, "output shape not correct" > RuntimeError: output shape not correct > > I tracked the problem down to "inputShape != outputShape" one > being a tuple > the output shape being a list. > (Pdb) p shape > [128, 528] > (Pdb) p output.shape > (128, 528) > (Pdb) p shape != output.shape > 1 > (Pdb) p shape , output.shape > ([128, 528], (128, 528)) > (Pdb) > > I'm using a CVS version around 1.3 ( /ni_interpolation.c/1.17/Fri > Apr 22 > 20:35:27 2005//THEAD) > but I took a look at the current CVS and it seems to still be a > problem > > Looks like I'm the only one who doesn't want the reshape ;-) > > Thanks, > Sebastian Haase > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through > log files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD > SPLUNK! > http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From haase at msg.ucsf.edu Mon Dec 5 14:35:00 2005 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Mon Dec 5 14:35:00 2005 Subject: [Numpy-discussion] numarray: how to make sinc(x) function - "divide by zero" ! Message-ID: <200512051433.55884.haase@msg.ucsf.edu> Hi, The best I could come up with was: def sinc(r): na.Error.pushMode(all="ignore") a = na.where(r, na.divide(na.sin(r),r), 1) na.Error.popMode() return a but I still seem to get a warning... >>> F.sinc(0) Warning: Encountered invalid numeric result(s) in divide 1.0 Thanks, Sebastian Haase From haase at msg.ucsf.edu Mon Dec 5 16:43:03 2005 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Mon Dec 5 16:43:03 2005 Subject: [Numpy-discussion] performance of nd_image affine_transform In-Reply-To: <88B304C0-B5B3-461C-B8A7-FFD74F1BDAF6@embl-heidelberg.de> References: <200512051213.07316.haase@msg.ucsf.edu> <88B304C0-B5B3-461C-B8A7-FFD74F1BDAF6@embl-heidelberg.de> Message-ID: <200512051642.40135.haase@msg.ucsf.edu> Hi, Thanks Peter for checking on the problem I reported in my last posting... Now I'm looking into using nd_image.affine_transform inplace of a fortran routine that I have been using to do this. a) I need to run this on Windows - where I don't have Fortran b) My Fortran routine does only do linear interpolation and I like the idea of experimenting with splines. A and B would of course be good reasons to use nd_image, BUT c) for a 512x512 float32 image my fortran takes about 14ms nd.affine_transform with given output array, prefilter=0 and order=1 takes about 132ms ! With prefilter=1 it takes 138ms; with prefilter=1 and order=3 it takes 279ms !! (order=2,prefilter=1 takes 226ms ; order=3,prefilter=0 222ms) All these are averaged over 10 runs on Linux (P4 2.8GHz) Why is nd_image 10x slower ? (spline order 1 does the same as linear (non-spline) interpolation, right ?) I would call this many (100, 1000 ?) times inside a simplex algorithm which takes already many seconds to complete using the Fortran routine... Thanks, Sebastian Haase On Monday 05 December 2005 14:31, Peter Verveer wrote: > Works for me with the latest CVS version. > > On 5 Dec, 2005, at 21:13, Sebastian Haase wrote: > > Hi, > > When I call nd_image.rotate with reshape=False I always get > > "output shape not correct" > > > >>>> U.nd.rotate(d[0], 20, axes=(-1, -2), reshape=0, output=d[1], > >>>> order=1, > > > > mode="constant", cval=0.0, prefilter=0) > > Traceback (most recent call last): > > File "", line 1, in ? > > File "/jws30/haase/PrLin/numarray/nd_image/interpolation.py", > > line 351, in > > rotate > > output, order, mode, cval, prefilter) > > File "/jws30/haase/PrLin/numarray/nd_image/interpolation.py", > > line 205, in > > affine_transform > > output_type) > > File "/jws30/haase/PrLin/numarray/nd_image/_ni_support.py", line > > 73, in > > _get_output > > raise RuntimeError, "output shape not correct" > > RuntimeError: output shape not correct > > > > I tracked the problem down to "inputShape != outputShape" one > > being a tuple > > the output shape being a list. > > (Pdb) p shape > > [128, 528] > > (Pdb) p output.shape > > (128, 528) > > (Pdb) p shape != output.shape > > 1 > > (Pdb) p shape , output.shape > > ([128, 528], (128, 528)) > > (Pdb) > > > > I'm using a CVS version around 1.3 ( /ni_interpolation.c/1.17/Fri > > Apr 22 > > 20:35:27 2005//THEAD) > > but I took a look at the current CVS and it seems to still be a > > problem > > > > Looks like I'm the only one who doesn't want the reshape ;-) > > > > Thanks, > > Sebastian Haase > > > > > > ------------------------------------------------------- > > This SF.net email is sponsored by: Splunk Inc. Do you grep through > > log files > > for problems? Stop! Download the new AJAX search engine that makes > > searching your log files as easy as surfing the web. DOWNLOAD > > SPLUNK! > > http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log > files for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From cjw at sympatico.ca Mon Dec 5 18:51:03 2005 From: cjw at sympatico.ca (Colin J. Williams) Date: Mon Dec 5 18:51:03 2005 Subject: [Numpy-discussion] Data type change completed In-Reply-To: <4394BA01.20501@ieee.org> References: <4394BA01.20501@ieee.org> Message-ID: <4394FC7B.6040005@sympatico.ca> Travis Oliphant wrote: > > I've committed the data-type change discussed at the end of last week > to the SVN repository. Now the concept of a data type for an array > has been replaced with a "data-descriptor". This data-descriptor is > flexible enough to handle an arbitrary record specification with > fields that include records and arrays or arrays of records. While > nesting may not be the best data-layout for a new design, when > memory-mapping an arbitrary fixed-record-length file, this capability > allows you to handle even the most obsure record file. > Does this mean that the dtype parameter is changed? obscure?? > While the basic core tests pass for me, there may be lurking problems > and so testing of the SVN trunk of scipy core will be appreciated. > I've bumped up the version number because the C-API has changed (a few > new functions and some functions becoming macros). > I'd like to make a release of the new version by the end of the week > (as soon as Chris Hanley at STSCI and I get records.py working > better), so please test. > > Recently some intel c-compiler tests were failing on a 64-bit > platform. It would be nice to figure out why that is happening as > well, but I will probably not have time for that this week. > > Thanks, > > -Travis From oliphant.travis at ieee.org Mon Dec 5 21:44:02 2005 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Dec 5 21:44:02 2005 Subject: ***[Possible UCE]*** Re: [Numpy-discussion] Data type change completed In-Reply-To: <4394FC7B.6040005@sympatico.ca> References: <4394BA01.20501@ieee.org> <4394FC7B.6040005@sympatico.ca> Message-ID: <43952502.1050609@ieee.org> Colin J. Williams wrote: > Travis Oliphant wrote: > >> >> I've committed the data-type change discussed at the end of last week >> to the SVN repository. Now the concept of a data type for an array >> has been replaced with a "data-descriptor". This data-descriptor is >> flexible enough to handle an arbitrary record specification with >> fields that include records and arrays or arrays of records. While >> nesting may not be the best data-layout for a new design, when >> memory-mapping an arbitrary fixed-record-length file, this capability >> allows you to handle even the most obsure record file. >> > Does this mean that the dtype parameter is changed? obscure?? No, it's not changed. The dtype parameter is still used and it is still called the same thing. It's just that what constitutes a data-type has changed significantly. For example now tuples and dictionaries can be used to describe a data-type. These definitions are recursive so that whenever data-type is used it means anything that can be interpreted as a data-type. And I really mean data-descriptor, but data-type is in such common usage that I still use it. Tuple: ======== (fixed-size-data-type, shape) (generic-size-data-type, itemsize) (base-type-data-type, new-type-data-type) Examples: dtype=(int32, (5,5)) --- a 5x5 array of int32 is the description of this item. dtype=(str, 10) --- a length-10 string dtype=(int16, {'real':(int8,0),'imag':(int8,4)} --- a descriptor that acts like an int16 array mathematically (in ufuncs) but has real and imag fields. Dictionary (defaults to a dtypechar == 'V') ========== format1: {"names": list-of-field-names, "formats": list of data-types, "offsets" : list of start-of-the-field "titles" : extra field names } format2 (and how it's stored internally) {key1 : (data-type1, offset1 [, title1]), key2 : (data-type2, offset2 [, title2]), ... keyn : (data-typen, offsetn [, titlen]) } Other objects not already covered: ===================== ???? Right now, it just passes the tp_dict of the typeobject to the dictionary-conversion routine. I'm open for ideas here and will probably have better ideas once the actual record data-type (not data-descriptor but actual subclass of the scipy.void data type) looks like. All of these can be used as the dtype parameter wherever it is taken (of course you can't always do something useful with every data-descriptor). When an ndarray has an associated type descriptor with fields (that's where the field information is stored), then those fields can be accessed using string or unicode keys to the getitem call. Thus, you can do something like this: >>> a = ones((4,3), dtype=(int16, {'real':(int8, 0), 'imag':(int8, 1)})) >>> a['imag'] = 2 >>> a['real'] = 1 >>> a.tostring() '\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02' Note that there are now three distinct but interacting Python objects: 1) the N-dimensional array of a fixed itemsize. 2) a Python object representing one element of the array. 3) the data-descriptor object describing the data-type of the array. These three things were always there under the covers (the PyArray_Descr* has been there since Numeric), and the standard Python types were always filling in for number 2. Now we are just being more explicit about it. Now, all three things are present and accounted for. I'm really quite happy with the resulting infrastructure. I think it will allow some really neat possibilities. I'm thinking the record array subclass will allow attribute-based look-up and register a nice record type for the actual "element" in of the record array. -Travis From cjw at sympatico.ca Tue Dec 6 07:05:06 2005 From: cjw at sympatico.ca (Colin J. Williams) Date: Tue Dec 6 07:05:06 2005 Subject: [Numpy-discussion] Data type change completed In-Reply-To: <43952502.1050609@ieee.org> References: <4394BA01.20501@ieee.org> <4394FC7B.6040005@sympatico.ca> <43952502.1050609@ieee.org> Message-ID: <4395A857.1040601@sympatico.ca> Travis Oliphant wrote: > Colin J. Williams wrote: > >> Travis Oliphant wrote: >> >>> >>> I've committed the data-type change discussed at the end of last >>> week to the SVN repository. Now the concept of a data type for an >>> array has been replaced with a "data-descriptor". This >>> data-descriptor is flexible enough to handle an arbitrary record >>> specification with fields that include records and arrays or arrays >>> of records. While nesting may not be the best data-layout for a new >>> design, when memory-mapping an arbitrary fixed-record-length file, >>> this capability allows you to handle even the most obsure record file. >>> >> Does this mean that the dtype parameter is changed? obscure?? > > > No, it's not changed. The dtype parameter is still used and it is > still called the same thing. It's just that what constitutes a > data-type has changed significantly. > > For example now tuples and dictionaries can be used to describe a > data-type. These definitions are recursive so that whenever data-type > is used it means anything that can be interpreted as a data-type. And > I really mean data-descriptor, but data-type is in such common usage > that I still use it. This would appear to be a good step forward but with all of the immutable types (int8, FloatType, TupleType, etc.) the data is stored in the ArrayType instance (array_data?) whereas, with a dictionary, it would appear to be necessary to store the items outside the array. Is that desirable? Even the tuple can have its content modified, as the example below shows: >>> a= [] >>> b= (a, [2, 3]) >>> b[0] [] >>> b[0]=99 Traceback (most recent call last): File "", line 1, in ? TypeError: object does not support item assignment <<< GOOD >>> b[1][0] 2 >>> b[1][0]=99 >>> b ([], [99, 3]) <<< HERE WE CHANGE THE VALUE OF THE TUPLE >>> > > Tuple: > ======== > (fixed-size-data-type, shape) > (generic-size-data-type, itemsize) > (base-type-data-type, new-type-data-type) > > Examples: > > dtype=(int32, (5,5)) --- a 5x5 array of int32 is the description of > this item. > dtype=(str, 10) --- a length-10 string So dtype now contains both the data type of each element and the shape of the array? This seems a significant change from numarray or Numeric. > dtype=(int16, {'real':(int8,0),'imag':(int8,4)} --- a descriptor that > acts > > like an int16 array mathematically > > (in ufuncs) but has real and imag > > > fields. > > This adds complexity, is there a compensating benefit? Do all of the complex operations apply? > > Dictionary (defaults to a dtypechar == 'V') > ========== Why no clean things up by dropping typechar? These seemed to be one of the warts in numarray, only carried forward for compatibility reasons. Could the compatibility objectives of the project not be achieved, outside the ArrayType object, with a wrapper of some sort? > format1: > > {"names": list-of-field-names, > "formats": list of data-types, > > > "offsets" : list of start-of-the-field > "titles" : extra field names > } > Couldn't the use of records avoid the cumbersome use of keys? > format2 (and how it's stored internally) > > {key1 : (data-type1, offset1 [, title1]), > key2 : (data-type2, offset2 [, title2]), > ... > keyn : (data-typen, offsetn [, titlen]) > } > This is cleaner, but couldn't this inormation be contained within the Record instance? > > Other objects not already covered: > ===================== > ???? > Right now, it just passes the tp_dict of the typeobject to the > dictionary-conversion routine. > I'm open for ideas here and will probably have better ideas once the > actual record data-type (not data-descriptor but actual subclass of > the scipy.void data type) looks like. > > All of these can be used as the dtype parameter wherever it is taken > (of course you can't > always do something useful with every data-descriptor). > When an ndarray has an associated type descriptor with fields (that's > where the field information is > stored), then those fields can be accessed using string or unicode > keys to the getitem call. > I've used ArrayType in place of ndarray (or maybe it should have been ndbigarray?) above as it appear to be more descriptive and fits with the Python convention on class naming. > Thus, you can do something like this: > > >>> a = ones((4,3), dtype=(int16, {'real':(int8, 0), 'imag':(int8, 1)})) > >>> a['imag'] = 2 > >>> a['real'] = 1 > >>> a.tostring() > '\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02' > > Or, one could have something like: class SmallComplex(Record): ..''' This class typically has no instances in user code. ''' ..real= (int8, ) ..imag= (int8) ..def __init__(self): .... ..def __new__(self): .... >>> a = ones((4,3), dtype= SmallComplex) >>> a.imag = 2 >>> a.real = 1 >>> a.tostring() '\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02' > > Note that there are now three distinct but interacting Python objects: > > 1) the N-dimensional array of a fixed itemsize. > 2) a Python object representing one element of the array. > 3) the data-descriptor object describing the data-type of the array. This looks cleaner. Perehaps 2) and 3) could be phrased a little differently: 2) a Python object which is one element of the array. 3) the data-descriptor object describing the data-type of the array element. > > These three things were always there under the covers (the > PyArray_Descr* has been there since Numeric), and the standard Python > types were always filling in for number 2. Now we are just being more > explicit about it. > > Now, all three things are present and accounted for. I'm really quite > happy with the resulting infrastructure. I think it will allow some > really neat possibilities. > > I'm thinking the record array subclass will allow attribute-based > look-up and register a nice record type for the actual "element" in of > the record array. > This is good but the major structure is the array which can have elements of various types such as ComplexType, NoneType, int8 or a variety of user defined immutable records. Colin W. PS My Record sketch above needs a lot more thinking through > > -Travis > > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log > files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From christos.siopis at ulb.ac.be Tue Dec 6 09:06:24 2005 From: christos.siopis at ulb.ac.be (Christos Siopis) Date: Tue Dec 6 09:06:24 2005 Subject: [Numpy-discussion] Indices of min/max element of an array? In-Reply-To: <4395A857.1040601@sympatico.ca> References: <4394BA01.20501@ieee.org> <4394FC7B.6040005@sympatico.ca> <43952502.1050609@ieee.org> <4395A857.1040601@sympatico.ca> Message-ID: <20051206170617.GC5410@ulb.ac.be> Hi all, I was wondering if i'm missing a numpy/numarray function that would return the indices of the minimum/maximum element of an array. Argmin/max can only do this one axis at a time. Alternatively, one can find the index for the flattened array and then play with modulo arithmetic, which gets quickly annoying for > 2 dimensions. Since this is a rather frequent operation, i would think there's already a function for it?... Thanks, Christos From perry at stsci.edu Tue Dec 6 09:16:16 2005 From: perry at stsci.edu (Perry Greenfield) Date: Tue Dec 6 09:16:16 2005 Subject: [Numpy-discussion] Indices of min/max element of an array? In-Reply-To: <20051206170617.GC5410@ulb.ac.be> References: <4394BA01.20501@ieee.org> <4394FC7B.6040005@sympatico.ca> <43952502.1050609@ieee.org> <4395A857.1040601@sympatico.ca> <20051206170617.GC5410@ulb.ac.be> Message-ID: Why not: ind = where(arr == arr.max()) and use the first element of the index array(s) Perry Greenfield On Dec 6, 2005, at 12:06 PM, Christos Siopis wrote: > Hi all, > > I was wondering if i'm missing a numpy/numarray function that would > return the > indices of the minimum/maximum element of an array. Argmin/max can > only do this > one axis at a time. Alternatively, one can find the index for the > flattened array > and then play with modulo arithmetic, which gets quickly annoying for > > 2 > dimensions. Since this is a rather frequent operation, i would think > there's > already a function for it?... > > Thanks, > Christos > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log > files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From haase at msg.ucsf.edu Tue Dec 6 09:25:13 2005 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue Dec 6 09:25:13 2005 Subject: [Numpy-discussion] performance of nd_image affine_transform In-Reply-To: References: <200512051213.07316.haase@msg.ucsf.edu> <200512051642.40135.haase@msg.ucsf.edu> Message-ID: <200512060924.19745.haase@msg.ucsf.edu> (Peter, I hope you meant to reply to the mailing list...) Hi Peter, Does that mean that the performance hit happens "at the lowest level", i.e. there is lot's of extra stuff in the in the innermost c loop that iterates over the output array ? Or is there just so much setup and cleanup code that makes the overall function slow ? How could I profile C extensions anyway - pointers/hints are appreciated ! Thanks for your nd_image (In case that wasn't clear - I like it a lot !) Sebastian Haase On Tuesday 06 December 2005 01:32, you wrote: > Hi Sebastian, > > The interpolation routines are not really optimized very well. The > underlying code is quite generic for spline order, data type, and > arbritatry array dimension and data layout. Your fortran code is > probably much more optimized. It would be nice to have more optimized > code in nd_image, but I do not really have the resources to do that. > > > Hi, > > Thanks Peter for checking on the problem I reported in my last > > posting... > > > > Now I'm looking into using nd_image.affine_transform inplace of a > > fortran > > routine that I have been using to do this. > > a) I need to run this on Windows - where I don't have Fortran > > b) My Fortran routine does only do linear interpolation and I like > > the idea of > > experimenting with splines. > > > > A and B would of course be good reasons to use nd_image, > > BUT c) > > for a 512x512 float32 image my fortran takes about 14ms > > nd.affine_transform with given output array, prefilter=0 and order=1 > > takes about 132ms ! > > With prefilter=1 it takes 138ms; with prefilter=1 and order=3 it > > takes > > 279ms !! (order=2,prefilter=1 takes 226ms ; order=3,prefilter=0 > > 222ms) > > All these are averaged over 10 runs on Linux (P4 2.8GHz) > > > > Why is nd_image 10x slower ? (spline order 1 does the same as linear > > (non-spline) interpolation, right ?) I would call this many (100, > > 1000 ?) > > times inside a simplex algorithm which takes already many seconds > > to complete > > using the Fortran routine... > > > > Thanks, > > Sebastian Haase > > > > On Monday 05 December 2005 14:31, Peter Verveer wrote: > >> Works for me with the latest CVS version. > >> > >> On 5 Dec, 2005, at 21:13, Sebastian Haase wrote: > >>> Hi, > >>> When I call nd_image.rotate with reshape=False I always get > >>> "output shape not correct" > >>> > >>>>>> U.nd.rotate(d[0], 20, axes=(-1, -2), reshape=0, output=d[1], > >>>>>> order=1, > >>> > >>> mode="constant", cval=0.0, prefilter=0) > >>> Traceback (most recent call last): > >>> File "", line 1, in ? > >>> File "/jws30/haase/PrLin/numarray/nd_image/interpolation.py", > >>> line 351, in > >>> rotate > >>> output, order, mode, cval, prefilter) > >>> File "/jws30/haase/PrLin/numarray/nd_image/interpolation.py", > >>> line 205, in > >>> affine_transform > >>> output_type) > >>> File "/jws30/haase/PrLin/numarray/nd_image/_ni_support.py", line > >>> 73, in > >>> _get_output > >>> raise RuntimeError, "output shape not correct" > >>> RuntimeError: output shape not correct > >>> > >>> I tracked the problem down to "inputShape != outputShape" one > >>> being a tuple > >>> the output shape being a list. > >>> (Pdb) p shape > >>> [128, 528] > >>> (Pdb) p output.shape > >>> (128, 528) > >>> (Pdb) p shape != output.shape > >>> 1 > >>> (Pdb) p shape , output.shape > >>> ([128, 528], (128, 528)) > >>> (Pdb) > >>> > >>> I'm using a CVS version around 1.3 ( /ni_interpolation.c/1.17/Fri > >>> Apr 22 > >>> 20:35:27 2005//THEAD) > >>> but I took a look at the current CVS and it seems to still be a > >>> problem > >>> > >>> Looks like I'm the only one who doesn't want the reshape ;-) > >>> > >>> Thanks, > >>> Sebastian Haase > >>> > >>> > >>> ------------------------------------------------------- > >>> This SF.net email is sponsored by: Splunk Inc. Do you grep through > >>> log files > >>> for problems? Stop! Download the new AJAX search engine that makes > >>> searching your log files as easy as surfing the web. DOWNLOAD > >>> SPLUNK! > >>> http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > >>> _______________________________________________ > >>> Numpy-discussion mailing list > >>> Numpy-discussion at lists.sourceforge.net > >>> https://lists.sourceforge.net/lists/listinfo/numpy-discussion > >> > >> ------------------------------------------------------- > >> This SF.net email is sponsored by: Splunk Inc. Do you grep through > >> log > >> files for problems? Stop! Download the new AJAX search engine > >> that makes > >> searching your log files as easy as surfing the web. DOWNLOAD > >> SPLUNK! > >> http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > >> _______________________________________________ > >> Numpy-discussion mailing list > >> Numpy-discussion at lists.sourceforge.net > >> https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > > ------------------------------------------------------- > > This SF.net email is sponsored by: Splunk Inc. Do you grep through > > log files > > for problems? Stop! Download the new AJAX search engine that makes > > searching your log files as easy as surfing the web. DOWNLOAD > > SPLUNK! > > http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From verveer at embl-heidelberg.de Tue Dec 6 09:37:13 2005 From: verveer at embl-heidelberg.de (Peter Verveer) Date: Tue Dec 6 09:37:13 2005 Subject: [Numpy-discussion] performance of nd_image affine_transform In-Reply-To: <200512060924.19745.haase@msg.ucsf.edu> References: <200512051213.07316.haase@msg.ucsf.edu> <200512051642.40135.haase@msg.ucsf.edu> <200512060924.19745.haase@msg.ucsf.edu> Message-ID: <8D594C13-1DF2-4507-9A2A-6C8602B01F75@embl-heidelberg.de> On Dec 6, 2005, at 18:24, Sebastian Haase wrote: > (Peter, I hope you meant to reply to the mailing list...) Yes, I did :-) > Hi Peter, > Does that mean that the performance hit happens "at the lowest > level", i.e. > there is lot's of extra stuff in the in the innermost c loop that > iterates > over the output array ? Yes, I would say so. > Or is there just so much setup and cleanup code that > makes the overall function slow ? Some of that may be a problem too, for instance, the function does malloc and free some temporary data, so that would add if you apply the function repeatedly. > How could I profile C extensions anyway - pointers/hints are > appreciated ! Not sure how you would go about that. I guess optimizing would require a rewrite for the specific cases that you like to optimize. > > Thanks for your nd_image (In case that wasn't clear - I like it a > lot !) Glad to hear that! > > Sebastian Haase > > > > On Tuesday 06 December 2005 01:32, you wrote: >> Hi Sebastian, >> >> The interpolation routines are not really optimized very well. The >> underlying code is quite generic for spline order, data type, and >> arbritatry array dimension and data layout. Your fortran code is >> probably much more optimized. It would be nice to have more optimized >> code in nd_image, but I do not really have the resources to do that. >> >>> Hi, >>> Thanks Peter for checking on the problem I reported in my last >>> posting... >>> >>> Now I'm looking into using nd_image.affine_transform inplace of a >>> fortran >>> routine that I have been using to do this. >>> a) I need to run this on Windows - where I don't have Fortran >>> b) My Fortran routine does only do linear interpolation and I like >>> the idea of >>> experimenting with splines. >>> >>> A and B would of course be good reasons to use nd_image, >>> BUT c) >>> for a 512x512 float32 image my fortran takes about 14ms >>> nd.affine_transform with given output array, prefilter=0 and order=1 >>> takes about 132ms ! >>> With prefilter=1 it takes 138ms; with prefilter=1 and order=3 it >>> takes >>> 279ms !! (order=2,prefilter=1 takes 226ms ; order=3,prefilter=0 >>> 222ms) >>> All these are averaged over 10 runs on Linux (P4 2.8GHz) >>> >>> Why is nd_image 10x slower ? (spline order 1 does the same as linear >>> (non-spline) interpolation, right ?) I would call this many (100, >>> 1000 ?) >>> times inside a simplex algorithm which takes already many seconds >>> to complete >>> using the Fortran routine... >>> >>> Thanks, >>> Sebastian Haase >>> >>> On Monday 05 December 2005 14:31, Peter Verveer wrote: >>>> Works for me with the latest CVS version. >>>> >>>> On 5 Dec, 2005, at 21:13, Sebastian Haase wrote: >>>>> Hi, >>>>> When I call nd_image.rotate with reshape=False I always get >>>>> "output shape not correct" >>>>> >>>>>>>> U.nd.rotate(d[0], 20, axes=(-1, -2), reshape=0, output=d[1], >>>>>>>> order=1, >>>>> >>>>> mode="constant", cval=0.0, prefilter=0) >>>>> Traceback (most recent call last): >>>>> File "", line 1, in ? >>>>> File "/jws30/haase/PrLin/numarray/nd_image/interpolation.py", >>>>> line 351, in >>>>> rotate >>>>> output, order, mode, cval, prefilter) >>>>> File "/jws30/haase/PrLin/numarray/nd_image/interpolation.py", >>>>> line 205, in >>>>> affine_transform >>>>> output_type) >>>>> File "/jws30/haase/PrLin/numarray/nd_image/_ni_support.py", line >>>>> 73, in >>>>> _get_output >>>>> raise RuntimeError, "output shape not correct" >>>>> RuntimeError: output shape not correct >>>>> >>>>> I tracked the problem down to "inputShape != outputShape" one >>>>> being a tuple >>>>> the output shape being a list. >>>>> (Pdb) p shape >>>>> [128, 528] >>>>> (Pdb) p output.shape >>>>> (128, 528) >>>>> (Pdb) p shape != output.shape >>>>> 1 >>>>> (Pdb) p shape , output.shape >>>>> ([128, 528], (128, 528)) >>>>> (Pdb) >>>>> >>>>> I'm using a CVS version around 1.3 ( /ni_interpolation.c/1.17/Fri >>>>> Apr 22 >>>>> 20:35:27 2005//THEAD) >>>>> but I took a look at the current CVS and it seems to still be a >>>>> problem >>>>> >>>>> Looks like I'm the only one who doesn't want the reshape ;-) >>>>> >>>>> Thanks, >>>>> Sebastian Haase >>>>> >>>>> >>>>> ------------------------------------------------------- >>>>> This SF.net email is sponsored by: Splunk Inc. Do you grep through >>>>> log files >>>>> for problems? Stop! Download the new AJAX search engine that >>>>> makes >>>>> searching your log files as easy as surfing the web. DOWNLOAD >>>>> SPLUNK! >>>>> http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click >>>>> _______________________________________________ >>>>> Numpy-discussion mailing list >>>>> Numpy-discussion at lists.sourceforge.net >>>>> https://lists.sourceforge.net/lists/listinfo/numpy-discussion >>>> >>>> ------------------------------------------------------- >>>> This SF.net email is sponsored by: Splunk Inc. Do you grep through >>>> log >>>> files for problems? Stop! Download the new AJAX search engine >>>> that makes >>>> searching your log files as easy as surfing the web. DOWNLOAD >>>> SPLUNK! >>>> http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click >>>> _______________________________________________ >>>> Numpy-discussion mailing list >>>> Numpy-discussion at lists.sourceforge.net >>>> https://lists.sourceforge.net/lists/listinfo/numpy-discussion >>> >>> ------------------------------------------------------- >>> This SF.net email is sponsored by: Splunk Inc. Do you grep through >>> log files >>> for problems? Stop! Download the new AJAX search engine that makes >>> searching your log files as easy as surfing the web. DOWNLOAD >>> SPLUNK! >>> http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click >>> _______________________________________________ >>> Numpy-discussion mailing list >>> Numpy-discussion at lists.sourceforge.net >>> https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through > log files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD > SPLUNK! > http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From jmiller at stsci.edu Tue Dec 6 11:11:13 2005 From: jmiller at stsci.edu (Todd Miller) Date: Tue Dec 6 11:11:13 2005 Subject: [Numpy-discussion] ANN: numarray-1.5.0 Message-ID: <4395E1D8.2090709@stsci.edu> Numarray is an array processing package designed to efficiently manipulate large multi-dimensional arrays. Numarray is modelled after Numeric and features c-code generated from python template scripts, the capacity to operate directly on arrays in files, arrays of heterogeneous records, string arrays, and in-place operation on memory mapped files. ========================================================================= Release Notes for numarray-1.5.0 I. ENHANCEMENTS 1. Implementation of scipy newcore array interface and __array_struct__ support (supplying and consuming) for numarray. This should facilitate better interoperability with scipy newcore and Numeric. This protocol is documented here: http://numeric.scipy.org/array_interface.html but also in numarray's source code in Include/numarray/arrayif.h. Thanks to everyone who gave feedback on 1.4.1 and particularly to Francesc Altet for his diligent testing (and doctest) of Numeric<->numarray data interchange. II. BUGS FIXED / CLOSED 1365121 Definition of PyObject* operator conflicts with C++ 1350954 nummacro.h:27: error: expected type-specifier before ';' tok 1346470 Please document more of what byteswap and byteswapped do 1364215 Cannot combine array and masked array (e.g. via divide) 1363723 my_array = +my_other_array uncovers a bug 1364811 Infinite loop converting empty CharArrays to lists 1364815 Strange behaviour when creating empty Int32 arrays 1340983 Fix __get_array_data__ 1346480 byteswapped doesn't match doc string 1346425 Please document if copy "fixes" byte order 1346426 Please document what array does with numarray arrays See http://sourceforge.net/tracker/?atid=450446&group_id=1369&func=browse for more details. III. CAUTIONS This release should be binary compatible with numarray-1.3.x. New extensions compiled against 1.5.0 may not work with prior versions of numarray. WHERE ----------- Numarray-1.5.0 windows executable installers, source code, and manual is here: http://sourceforge.net/project/showfiles.php?group_id=1369 Numarray is hosted by Source Forge in the same project which hosts Numeric: http://sourceforge.net/projects/numpy/ The web page for Numarray information is at: http://www.stsci.edu/resources/software_hardware/numarray Trackers for Numarray Bugs, Feature Requests, Support, and Patches are at the Source Forge project for NumPy at: http://sourceforge.net/tracker/?group_id=1369 REQUIREMENTS ------------------------------ numarray-1.5.0 requires Python 2.2.3 or greater. AUTHORS, LICENSE ------------------------------ Numarray was written by Perry Greenfield, Rick White, Todd Miller, JC Hsu, Paul Barrett, Phil Hodge at the Space Telescope Science Institute. We'd like to acknowledge the assitance of Francesc Alted, Paul Dubois, Sebastian Haase, Chuck Harris, Tim Hochberg, Nadav Horesh, Edward C. Jones, Eric Jones, Jochen Kuepper, Travis Oliphant, Pearu Peterson, Peter Verveer, Colin Williams, Rory Yorke, and everyone else who has contributed with comments and feedback. Numarray is made available under a BSD-style License. See LICENSE.txt in the source distribution for details. -- Todd Miller jmiller at stsci.edu From christos.siopis at ulb.ac.be Tue Dec 6 12:29:04 2005 From: christos.siopis at ulb.ac.be (Christos Siopis) Date: Tue Dec 6 12:29:04 2005 Subject: [Numpy-discussion] Indices of min/max element of an array? In-Reply-To: References: <4394BA01.20501@ieee.org> <4394FC7B.6040005@sympatico.ca> <43952502.1050609@ieee.org> <4395A857.1040601@sympatico.ca> <20051206170617.GC5410@ulb.ac.be> Message-ID: <20051206202905.GO5410@ulb.ac.be> On Tue, Dec 06, 2005 at 12:17:14PM -0500, Perry Greenfield wrote: > Why not: > > ind = where(arr == arr.max()) > > and use the first element of the index array(s) Thanks, that worked nice! I was stuck with the old Numeric definition of where(). Could it perhaps be made more obvious in numarray's where() documentation that it can also be called with one parameter only? (I.e., put this information in the section header, not mid-way in the description). I'm looking at the April 20, 2005 documentation release. Would it be safe to assume that the numarray definition of where() will be the one propagated into scipy.core? As a side note, i am wondering if there is a semantic asymmetry in using min() and max() to signify the min/max element in the entire array while argmin() and argmax() signify the min/max element along each axis. At the same time, and as far as i can tell, there is no min()/max() method to provide the min/max element along each axis, and there is no method to do the equivalent of "argmin/max_all()", as implemented by where(arr == arr.min/max()). Apologies if this has been discussed before. Thank you, Christos From oliphant at ee.byu.edu Tue Dec 6 12:33:00 2005 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue Dec 6 12:33:00 2005 Subject: [Numpy-discussion] Indices of min/max element of an array? In-Reply-To: <20051206202905.GO5410@ulb.ac.be> References: <4394BA01.20501@ieee.org> <4394FC7B.6040005@sympatico.ca> <43952502.1050609@ieee.org> <4395A857.1040601@sympatico.ca> <20051206170617.GC5410@ulb.ac.be> <20051206202905.GO5410@ulb.ac.be> Message-ID: <4395F526.6030902@ee.byu.edu> Christos Siopis wrote: >On Tue, Dec 06, 2005 at 12:17:14PM -0500, Perry Greenfield wrote: > > > >>Why not: >> >>ind = where(arr == arr.max()) >> >>and use the first element of the index array(s) >> >> > > >Would it be safe to assume that the numarray definition of where() will be the one >propagated into scipy.core? > > Yes, it is already there. -Travis From aisaac at american.edu Tue Dec 6 15:34:01 2005 From: aisaac at american.edu (Alan G Isaac) Date: Tue Dec 6 15:34:01 2005 Subject: [Numpy-discussion] Indices of min/max element of an array? In-Reply-To: <20051206202905.GO5410@ulb.ac.be> References: <4394BA01.20501@ieee.org> <4394FC7B.6040005@sympatico.ca> <43952502.1050609@ieee.org> <4395A857.1040601@sympatico.ca> <20051206170617.GC5410@ulb.ac.be> <20051206202905.GO5410@ulb.ac.be> Message-ID: On Tue, 6 Dec 2005, Christos Siopis apparently wrote: > As a side note, i am wondering if there is a semantic > asymmetry in using min() and max() to signify the min/max > element in the entire array while argmin() and argmax() > signify the min/max element along each axis. SciPy arrays function as "expected" in this sense: >>> import scipy >>> x=scipy.array([[1,2],[3,4]]) >>> x.max() 4 >>> x.argmax() 3 Note that, as I understand, argmax gives the index from x.flat Also, the scipy ufuncs max and argmax have the symmetry you seek, if I understand correctly. hth, Alan Isaac From arnd.baecker at web.de Wed Dec 7 02:08:01 2005 From: arnd.baecker at web.de (Arnd Baecker) Date: Wed Dec 7 02:08:01 2005 Subject: [Numpy-discussion] ACML and dotblas Message-ID: Hi, has anyone figured out how to use dotblas with the AMD Core Math Library (ACML, http://developer.amd.com/acml.aspx)? Darren Dale asked this a month ago here http://aspn.activestate.com/ASPN/Mail/Message/numpy-discussion/2886285 but I could not find any answer. Many thanx, Arnd From dd55 at cornell.edu Wed Dec 7 03:31:06 2005 From: dd55 at cornell.edu (Darren Dale) Date: Wed Dec 7 03:31:06 2005 Subject: [Numpy-discussion] ACML and dotblas In-Reply-To: References: Message-ID: <200512070630.06109.dd55@cornell.edu> On Wednesday 07 December 2005 5:07 am, Arnd Baecker wrote: > Hi, > > has anyone figured out how to use dotblas > with the AMD Core Math Library (ACML, http://developer.amd.com/acml.aspx)? > Darren Dale asked this a month ago here > http://aspn.activestate.com/ASPN/Mail/Message/numpy-discussion/2886285 > but I could not find any answer. I don't have any progress to report, sorry. Darren From service at lntl.ebay.com Fri Dec 9 11:58:07 2005 From: service at lntl.ebay.com (eBay) Date: Fri Dec 9 11:58:07 2005 Subject: [Numpy-discussion] Important eBay! Account Information. Message-ID: An HTML attachment was scrubbed... URL: From christos.siopis at ulb.ac.be Sat Dec 10 10:32:12 2005 From: christos.siopis at ulb.ac.be (Christos Siopis) Date: Sat Dec 10 10:32:12 2005 Subject: [Numpy-discussion] Indices of min/max element of an array? In-Reply-To: References: <4394BA01.20501@ieee.org> <4394FC7B.6040005@sympatico.ca> <43952502.1050609@ieee.org> <4395A857.1040601@sympatico.ca> <20051206170617.GC5410@ulb.ac.be> Message-ID: <20051210183306.GD8898@ulb.ac.be> On Tue, Dec 06, 2005 at 06:37:11PM -0500, Alan G Isaac wrote: > On Tue, 6 Dec 2005, Christos Siopis apparently wrote: > > > As a side note, i am wondering if there is a semantic > > asymmetry in using min() and max() to signify the min/max > > element in the entire array while argmin() and argmax() > > signify the min/max element along each axis. > > SciPy arrays function as "expected" in this sense: > >>> import scipy > >>> x=scipy.array([[1,2],[3,4]]) > >>> x.max() > 4 > >>> x.argmax() > 3 > > Note that, as I understand, argmax gives the index from x.flat Thanks for the note. I do not have SciPy installed to test this, and i am not sure which version of SciPy you are using. I believe in the (remote?) past, SciPy was using Numeric as a core, but using the latest Numeric available on Gentoo AMD64 i obtain different results: Python 2.4.2 (#1, Nov 19 2005, 12:30:12) [GCC 3.4.4 (Gentoo 3.4.4-r1, ssp-3.4.4-1.0, pie-8.7.8)] on linux2 >>> import Numeric >>> Numeric.__version__ '23.7' >>> x=Numeric.array([[1,2],[3,4]]) >>> x.max() Traceback (most recent call last): File "", line 1, in ? AttributeError: max >>> x.argmax() Traceback (most recent call last): File "", line 1, in ? AttributeError: argmax i.e., these methods are not supported for Numeric arrays. The max() function does not exist either: >>> Numeric.max(x) Traceback (most recent call last): File "", line 1, in ? AttributeError: 'module' object has no attribute 'max' (i think one would have to use MA.Maximum() to find the global array maximum back at those days...). And: >>> Numeric.argmax(x) array([1, 1]) i.e., it does not flatten the array. --- Now, using numarray: >>> import numarray >>> numarray.__version__ '1.3.1' >>> x=numarray.array([[1,2],[3,4]]) >>> x.max() 4 >>> x.argmax() array([1, 1]) i.e., these methods for the array object now do exist, but the behavior of argmax() is not the same as in your SciPy. My conjencture is that your SciPy uses some "intermediate" version between the old Numeric and the current scipy.core which, as i understand from what Travis said, supports the above numarray behavior. > Also, the scipy ufuncs max and argmax have the symmetry you > seek, if I understand correctly. With so many versions of things floating around, i think it's hard to tell what has what any more... One more reason to look forward to the outcome of Travis' work, and hope that things (or at least the API) will stabilize... Thanks, Christos From cjw at sympatico.ca Sun Dec 11 06:53:01 2005 From: cjw at sympatico.ca (Colin J. Williams) Date: Sun Dec 11 06:53:01 2005 Subject: [Numpy-discussion] Re: Scipy Core (Was Numeric 3.0) In-Reply-To: <439B297E.3030808@ieee.org> References: <439AEA74.3090407@sympatico.ca> <439B297E.3030808@ieee.org> Message-ID: <439C3D1A.1010502@sympatico.ca> Travis, This is intended to restore the context of your response. TO is Travis Oliphant, CW is Colin Williams TO: The dtype parameter is still used and it is still called the same thing. It's just that what constitutes a data-type has changed significantly. For example now tuples and dictionaries can be used to describe a data-type. These definitions are recursive so that whenever data-type is used it means anything that can be interpreted as a data-type. And I really mean data-descriptor, but data-type is in such common usage that I still use it. CW: This would appear to be a good step forward but with all of the immutable types (int8, FloatType, TupleType, etc.) the data is stored in the ArrayType instance (array_data?) whereas, with a dictionary, it would appear to be necessary to store the items outside the array. Is that desirable? TO: I don't even understand what you are questioning here. Perhaps you misunderstood me. Items are always stored in the array. Even records "items" are stored in the array. The descriptor just allows you to describe what kind of data and how it is stored in the array. Response: Yes, I had assumed that int8 and FloatType are equally data-descriptors, i.e. objects which describe the elements of a numeric array. Wrongly, I assumed that TupleType or DictType are also a data descriptors. Mea culpa. On another subject, it would help, for those of us who are not afficionados of C, if you provided the equivalent Python def statement for those function implemented in C, especially the ArrayType's __new__ method. perhaps in the docstrings. Colin W. Travis Oliphant wrote: > Colin J. Williams wrote: > >> >> This would appear to be a good step forward but with all of the >> immutable types (int8, FloatType, TupleType, etc.) the data is stored >> in the ArrayType instance (array_data?) whereas, with a dictionary, >> it would appear to be necessary to store the items outside the >> array. Is that desirable? > > > I don't even understand what you are questioning here. Perhaps you > misunderstood me. Items are always stored in the array. Even records > "items" are stored in the array. The descriptor just allows you to > describe what kind of data and how it is stored in the array. > >> >> Even the tuple can have its content modified, as the example below >> shows: >> > I don't understand what you mean to show by the > tuple-content-modifying example. > >>> dtype=(int32, (5,5)) --- a 5x5 array of int32 is the description >>> of this item. >>> dtype=(str, 10) --- a length-10 string >> >> >> >> So dtype now contains both the data type of each element and the >> shape of the array? This seems a significant change from numarray or >> Numeric. > > > No, no. Standard usage is the same. In normal usage you would not > create an array this way. You could, of course, but it's not the > documented procedure. The reason for this descriptor is to allow > you to have a field-element that itself is an array of items. > >> >> >>> dtype=(int16, {'real':(int8,0),'imag':(int8,4)} --- a descriptor >>> that acts >>> >>> like an int16 array mathematically >>> >>> (in ufuncs) but has real and imag >>> >>> >>> fields. >>> >>> >> This adds complexity, is there a compensating benefit? Do all of the >> complex operations apply? > > > I'm only showing what is possible and that the notion of data-type > descriptor is complete. > >> >> Why not clean things up by dropping typechar? These seemed to be one >> of the warts in numarray, only carried forward for >> compatibility reasons. Could the compatibility objectives of the >> project not be achieved, outside the ArrayType object, with a wrapper >> of some sort? > > > Too hard to do at this point. Too much code uses the characters. I > also don't mind the characters so much (the struct module and the > Python array module use them). > >> >> Couldn't the use of records avoid the cumbersome use of keys? > > > Yes. But, this is how you specify fields generally. > >> >>> format2 (and how it's stored internally) >>> >>> {key1 : (data-type1, offset1 [, title1]), >>> key2 : (data-type2, offset2 [, title2]), >>> ... >>> keyn : (data-typen, offsetn [, titlen]) >>> } >>> >> This is cleaner, but couldn't this inormation be contained within the >> Record instance? > > > > Yes, but I am describing a general concept of data-type description > not just something applicable to records. > >>> Thus, you can do something like this: >>> >>> >>> a = ones((4,3), dtype=(int16, {'real':(int8, 0), 'imag':(int8, >>> 1)})) >>> >>> a['imag'] = 2 >>> >>> a['real'] = 1 >>> >>> a.tostring() >>> '\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02' >>> >>> >> Or, one could have something like: >> class SmallComplex(Record): >> ..''' This class typically has no instances in user code. ''' >> ..real= (int8, ) >> ..imag= (int8) >> ..def __init__(self): >> .... >> ..def __new__(self): >> .... >> >> >>> a = ones((4,3), dtype= SmallComplex) >> >>> a.imag = 2 >> >>> a.real = 1 >> >>> a.tostring() >> '\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02' >> > > > Yes, something like this should be possible, though I have not fleshed > out the user-interface at all. I've just been working on the basic > structure that would support such things. Do: > > class small_complex(record): > dtypedescr = {'r':(int8,0),'i':(int8,1)} > > a = ndrecarray((4,3), formats=small_complex) > a.r = 1 > a.i = 2 > a.tostring() > > and your example would work (with current SVN...) > > The ndrecarray subclass allows the attribute-to-field conversion, > regular arrays do not. > > -Travis > > > > > > > > > > -Travis > > From dfg at 456.com Sun Dec 11 10:14:04 2005 From: dfg at 456.com (ghkkuk) Date: Sun Dec 11 10:14:04 2005 Subject: [Numpy-discussion] =?GB2312?B?obZbwMrPzMa9XSDW0Ln6xvPStdW9wtTNu86n0dDM1rvhIKG3?= Message-ID: An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Sun Dec 11 12:54:04 2005 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sun Dec 11 12:54:04 2005 Subject: [Numpy-discussion] Re: [SciPy-user] transition problems In-Reply-To: <439C4AAB.70308@hoc.net> References: <439C4AAB.70308@hoc.net> Message-ID: <439C91DB.1060505@ieee.org> Christian Kristukat wrote: >Hi, >I noticed that using the new core together with older packages that are based on >numeric, like the plotting module from wxPython, matplotlib, grace_np.py, etc. >is problematic since numeric seems to be unable to recognize new core arrays as >equivalent to the numeric ones. > If you get a recent version (>=24.0) of Numeric then it shouldn't be a problem. One of the purposes of the array interface is to ease the transition, but only recent versions of Numeric use it properly. If there are problems with recent versions of Numeric and the array interface then please post them. Older versions of Numeric would need to use tostring and fromstring (better than tolist...) >In some cases it is sufficient to add a >.tolist() to array parameters but that is not really satisfactory. >Will a final version of new core have solved these incompatibilities? > > Unfortunately, I don't know how to solve incompatibilities with older versions of Numeric. All third-party libraries should either start using only the array interface or switch to scipy_core. I've noticed there is some confusion still circulating on the net that the new arrayobject in scipy_core is going to get into Python at some point, and some people are holding off making any changes until that happens. I am probably the source of that confusion since at one time I did have that goal. However, early on (in March) after discussions with Paul, Perry, and Guido we decided that trying to force an arrayobject into Python would place us on a release cycle that would be constraining. So, don't wait to try out scipy_core because there is nothing going into Python any time soon that is even as capable as Numeric. There are plans for getting a very simple arrayobject into Python (largely to enshrine the array interface), but we really need help moving that forward. With the recent changes to scipy_core, I think we have a very good design for a generic array object with associated data-descriptor that could go into Python. For python itself, I would only say that descriptors should only be written for Object, integer, float, and complex-float arrays. A PEP has been started and sits at http://svn.scipy.org/svn/PEP waiting for more people to help push it forward. Best, -Travis From oliphant.travis at ieee.org Sun Dec 11 20:38:01 2005 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sun Dec 11 20:38:01 2005 Subject: [Numpy-discussion] Re: [SciPy-dev] New release on Monday In-Reply-To: <767A9640-49E7-42B8-BE28-6B4001D93F38@ftw.at> References: <439BE8F4.3070302@ieee.org> <767A9640-49E7-42B8-BE28-6B4001D93F38@ftw.at> Message-ID: <439CFE79.6050303@ieee.org> Ed Schofield wrote: >On 11/12/2005, at 8:53 AM, Travis Oliphant wrote: > > >Is there any chance of including type checking for this release for >unsafe in-place operations? Several people supported the idea in the >thread [In-place operators and casting] a few weeks ago. In summary, >the existing behaviour would still be achievable with an explicit cast: > >>> my_int_array += my_float_array.astype(int) > > Actually, this would not do the same thing because it would force the entire float array into memory as an integer array and then add it. The current behavior creates only bufsize memories for this kind of copying. So, for large-array performance this approach would be worse, which is a big reason I'm not really supportive of a switch. I'm also hesitant to change this because it is the default behavior of numarray, so I'd like to receive more feedback from members of that community who are coming over to scipy_core before doing something different. I think, however, Numeric raises an error in this circumstance. So, I would advise changing this behavior in the current release, but I don't see this issue as closed. While I would never support changing the data-type of the array when using an inplace operator, I could see the logic in raising an error when the type cannot be cast safely. -Travis From oliphant.travis at ieee.org Mon Dec 12 12:49:06 2005 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Dec 12 12:49:06 2005 Subject: [Numpy-discussion] Thoughts on making it easier for numarray users to transition to scipy_core Message-ID: <439DE22B.9070400@ieee.org> A few days ago I played with nd_image and was able to make it compile for scipy_core. In the process, I had some ideas for making the transition to scipy_core easier for numarray users: 1) It would be nice if there was someway to document on the Python level any needed changes. In the process, we might find things that need to be added to scipy_core. I would like to have some kind of program for automatically making most of those needed changes before a 1.0 release of scipy_core. 2) On the C-API side, my experience with nd_image showed me that quite a few of the numarray C-API calls can be written as macros while some will need to be additional functions. I think we could easily write a numcompatmodule.c and associated numcompat.h file so that users needing the numarray C-API could include the numcompat.h file and then the import_libnumarray() command would load the numcompatmodule.c with it's compatibility functions. In this way, the transition for C-API users could be as easy as changing the include file to numcompat.h? -Travis From verveer at embl-heidelberg.de Mon Dec 12 13:36:02 2005 From: verveer at embl-heidelberg.de (Peter Verveer) Date: Mon Dec 12 13:36:02 2005 Subject: [Numpy-discussion] Thoughts on making it easier for numarray users to transition to scipy_core In-Reply-To: <439DE22B.9070400@ieee.org> References: <439DE22B.9070400@ieee.org> Message-ID: <79153027-85E8-4DB3-A69F-2B1398CB2043@embl-heidelberg.de> Hi Travis, I am curious about your experience with moving nd_image to scipy_core. I have not had the time to look at scipy_core, so I have not tried to do it myself, although it was on my very-long-term list of things to do. Did it require a lot of changes and did you get it to pass the unit tests? Cheers, Peter > A few days ago I played with nd_image and was able to make it > compile for scipy_core. In the process, I had some ideas for > making the transition to scipy_core easier for numarray users: > > 1) It would be nice if there was someway to document on the Python > level any needed changes. In the process, we might find things > that need to be added to scipy_core. I would like to have some > kind of program for automatically making most of those needed > changes before a 1.0 release of scipy_core. > > 2) On the C-API side, my experience with nd_image showed me that > quite a few of the numarray C-API calls can be written as macros > while some will need to be additional functions. I think we could > easily write a numcompatmodule.c and associated numcompat.h file so > that users needing the numarray C-API could include the numcompat.h > file and then the import_libnumarray() command would load the > numcompatmodule.c with it's compatibility functions. > > In this way, the transition for C-API users could be as easy as > changing the include file to numcompat.h? > -Travis > > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through > log files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD > SPLUNK! > http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From oliphant.travis at ieee.org Mon Dec 12 15:36:04 2005 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Dec 12 15:36:04 2005 Subject: [SciPy-dev] [Numpy-discussion] Thoughts on making it easier for numarray users to transition to scipy_core In-Reply-To: <367c3c211512e3bac5ff4aabe127721d@stsci.edu> References: <439DE22B.9070400@ieee.org> <367c3c211512e3bac5ff4aabe127721d@stsci.edu> Message-ID: <439DFE67.2020600@ieee.org> >That certainly would be nice. We are starting to look migrating some >Python code. It may be a little while >(a couple months?) before we can start tackling migrating some of the C >extension code so we won't be exercising that right away (but maybe >someone else can). > > I think I will have something basic in-place by then. We can add needed API calls as extensions get ported. On a related note, where should the Packages (like nd_image) from numarray go? Should they all go into scipy core, full scipy, or be separately downloadable as scipy_core addons? -Travis From verveer at embl-heidelberg.de Tue Dec 13 00:27:02 2005 From: verveer at embl-heidelberg.de (Peter Verveer) Date: Tue Dec 13 00:27:02 2005 Subject: [SciPy-dev] [Numpy-discussion] Thoughts on making it easier for numarray users to transition to scipy_core In-Reply-To: <439DFE67.2020600@ieee.org> References: <439DE22B.9070400@ieee.org> <367c3c211512e3bac5ff4aabe127721d@stsci.edu> <439DFE67.2020600@ieee.org> Message-ID: On Dec 12, 2005, at 23:49, Travis Oliphant wrote: > >> That certainly would be nice. We are starting to look migrating >> some Python code. It may be a little while >> (a couple months?) before we can start tackling migrating some of >> the C extension code so we won't be exercising that right away >> (but maybe someone else can). >> > I think I will have something basic in-place by then. We can add > needed API calls as extensions get ported. > > On a related note, where should the Packages (like nd_image) from > numarray go? Should they all go into scipy core, full scipy, or be > separately downloadable as scipy_core addons? Regarding nd_image, I would guess that it should be an addon, or go at some point in scipy. However, even though I am the author, I am most likely not going to support it outside numarray. I do realize that at some point numarray may be retired, but I have not decided yet if I will be using/supporting nd_image beyond that point. It would be nice to have a common code-base with the numarray version so that all can profit from any bug-fixes that I am still doing in the numarray version. peter From edcjones at comcast.net Tue Dec 13 20:28:13 2005 From: edcjones at comcast.net (Edward C. Jones) Date: Tue Dec 13 20:28:13 2005 Subject: [Numpy-discussion] numarray bug: random_array/dtest.py fails on 64 bit machine Message-ID: <439F9F3D.1080508@comcast.net> I have an Athlon 64 chip. I use Debian unstable but I compile all Python related programs myself. They go into /usr/local. I have Python 2.4.2 and numarray 1.5.0. I got the following error from testall.test(): File "/usr/local/lib/python2.4/site-packages/numarray/random_array/dtest.py", line 57, in numarray.random_array.dtest Failed example: x = multivariate_normal(num.array([10,20]), num.array(([1,2],[2,4]))); x Expected: array([ 9.95170432, 19.90340867]) Got: array([ 9.95170432, 19.90340866]) From verveer at embl-heidelberg.de Wed Dec 14 01:29:04 2005 From: verveer at embl-heidelberg.de (Peter Verveer) Date: Wed Dec 14 01:29:04 2005 Subject: [SciPy-dev] [Numpy-discussion] Thoughts on making it easier for numarray users to transition to scipy_core In-Reply-To: <439F4DBE.5040601@ieee.org> References: <439DE22B.9070400@ieee.org> <367c3c211512e3bac5ff4aabe127721d@stsci.edu> <439DFE67.2020600@ieee.org> <439F4DBE.5040601@ieee.org> Message-ID: On 13 Dec, 2005, at 23:39, Travis Oliphant wrote: > >> However, even though I am the author, I am most likely not going >> to support it outside numarray. I do realize that at some point >> numarray may be retired, but I have not decided yet if I will be >> using/supporting nd_image beyond that point. It would be nice to >> have a common code-base with the numarray version so that all can >> profit from any bug-fixes that I am still doing in the numarray >> version. > > > So, do I take this to mean that you are not enthusiastic about a > merger between the Numeric and numarray communities? I would love > to hear why you feel this way. > From my perspective, it just makes unification that much harder to > have important third-party authors make these kinds of statements, > so I'm curious as to why you don't want to convert to a common code- > base. Because others may have the same feelings, it would help all > of us if we could address them. Don't worry about offending me or > others in the scipy world. We have pretty thick skins by this point. I think I should clarify what I intent to do with nd_image in the future. It affects current nd_image users and the numarray package also, so I am also sending this to the numpy list. First of all, I should stress that I am quite happy about a possible merger. It is quite likely that I will be using python for numerical work in the future, and a merger of numarray and Numeric into a package of equal or better quality will only make my life easier. Currently I am the only software developer within a scientific laboratory, but software development is not my main focus. I used python with numarray for virtual all of my numerical work, and nd_image was sort of a side effect of that. Right now I do not have the time to make a switch to a new package, nor can I really further expand nd_image. It would be nice to try moving to scipy_core now, which would help the community too I guess, but I just cannot affort it at the moment, so I am sticking with numarray for the time being. My situation will most likely change next year, since our group will move and expand a lot, and I will hopefully be able to have other people working on software development. I guess around that time we will see scipy_core mature, so that would be a good time to switch. I understand the numarray people take a wait-and-see approach, but intent to switch when scipy_core matures and fulfills their requirements. I guess I am following their example. So next year I will might be in the position of having more resources and I then will start thinking about developing some open-source scientific software. An imaging package like nd_image would be one of them. However, in contrast to the current design, I would like to move to a package that is fully written in C, with a thin interface layer to numarray/scipy_core for use with python. That would allow me to use the routines in other environments than python,. Sofar I have been very comfortable to make my package available under the license that is used by numarray. The same would be true for a scipy_core/scipy version. However, for a general C package that could have a much broader audience in the same way, I feel different. I would most likely release such a package using the GPL. In addition, that would allow me to use other GPL licensed software such as GSL with it. However, that would mean that a python package based on it would also be released with that license. I am not complete sure what the implications for distribution with a package like scipy would be, this may not be possible. But it would be freely available as a separate module which we would maintain ourself. If it pans out this way, it means that nd_image will not be further developed at some point. I am currently doing bug-fixes in the numarray version, and I certainly would not mind if there would be a common code-base allowing it to work with scipy_core. But if I am going to developer package as described above, I will to focus on that and recommend nd_image users to switch at some point. > > Even if your statement is made from the perspective of someone who > is moving away from Python altogether, I would still very-much > appreciate hearing your reasons. Now, perhaps, your just going to > experience a career-change altogether in which case, best wishes, > and thanks for all you've contributed. I will certainly stick with Python :-) > > Regardless of whether or not you have time to respond, I do > appreciate all your efforts. The nd_image package is very nice as > I've told you before. Thank you! Thanks! nd_image seems to find some users, and I would like to continue to provide something like nd_image. It might be a while though until nd_image or a replacement will start moving again and further developing. Meanwhile I am committed to keep nd_image working with numarray and doing minor bug-fixes. If we can at some point find a solution for a single code-base which would work with scipy_core, I am happy to expand this commitment to scipy_core. I cannot promise indefinite support, but as long as I am in the field and as long as I do not release a 'successor' package, I will try... Cheers, Peter From Chris.Barker at noaa.gov Wed Dec 14 10:26:13 2005 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed Dec 14 10:26:13 2005 Subject: [SciPy-dev] [Numpy-discussion] Thoughts on making it easier for =?iso-8859-1?q?numarray=09users_to_transition_to?= scipy_core In-Reply-To: References: <439DE22B.9070400@ieee.org> <439F4DBE.5040601@ieee.org> Message-ID: <200512141025.05985.Chris.Barker@noaa.gov> On Wednesday 14 December 2005 1:28 am, Peter Verveer wrote: > Sofar I have been very comfortable to make my package available under > the license that is used by numarray. The same would be true for a > scipy_core/scipy version. However, for a general C package that could > have a much broader audience in the same way, I feel different. I > would most likely release such a package using the GPL. In addition, > that would allow me to use other GPL licensed software such as GSL > with it. You could release it under two licenses, GPL for a C package that uses other GPL software, and the SciPy license for the python extension. The only thing that would kill that plan is if you use GPL code in the Python extension. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From aisaac at american.edu Wed Dec 14 12:09:06 2005 From: aisaac at american.edu (Alan G Isaac) Date: Wed Dec 14 12:09:06 2005 Subject: [Numpy-discussion] =?ISO-8859-1?Q?Re[2]:=20[SciPy-dev]=20[Numpy-discussion]=20Thoughts=20on?==?ISO-8859-1?Q?=20making=20it=20easier=20for=20numarray=09users=20to=20tra?==?ISO-8859-1?Q?nsition=20to=20scipy_core?= In-Reply-To: <200512141025.05985.Chris.Barker@noaa.gov> References: <439DE22B.9070400@ieee.org> <439F4DBE.5040601@ieee.org><200512141025.05985.Chris.Barker@noaa.gov> Message-ID: > On Wednesday 14 December 2005 1:28 am, Peter Verveer wrote: > Sofar I have been very comfortable to make my package > available under the license that is used by numarray. The > same would be true for a scipy_core/scipy version. > However, for a general C package that could have a much > broader audience in the same way, I feel different. > I would most likely release such a package using the GPL. > In addition, that would allow me to use other GPL > licensed software such as GSL with it. "Use with" means many things. You write it, you choose the license, of course. But as an academic user/observer, rather than a substantial contributor, I have noticed that if your goal is to see your code used and improved, it is not clear that the GPL is the best way to pursue that goal. See e.g. the history of Matplotlib and MayaVi. (Or Python itself, for that matter, for a larger example.) Each case is individual, of course. If you need extant GPL code to be an *integral* part of your code, that makes the GPL natural. fwiw, Alan From greg.ewing at canterbury.ac.nz Wed Dec 14 15:38:04 2005 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed Dec 14 15:38:04 2005 Subject: [Numpy-discussion] Re[2]: [SciPy-dev] [Numpy-discussion] Thoughts on making it easier for numarray users to transition to scipy core In-Reply-To: References: <439DE22B.9070400@ieee.org> <439F4DBE.5040601@ieee.org> <200512141025.05985.Chris.Barker@noaa.gov> Message-ID: <43A0ACB0.2050905@canterbury.ac.nz> On Wednesday 14 December 2005 1:28 am, Peter Verveer wrote: > I would most likely release such a package using the GPL. > In addition, that would allow me to use other GPL > licensed software such as GSL with it. If you combine your package with something GPL and release the result, then of course you would have to release that combination under the GPL. But that doesn't prevent you from releasing your own package separately under a different licence if you want. The question you need to ask yourself is whether you really want to impose all the restrictions of the GPL on users of your package. If you have no desire to be that restrictive, then use a more liberal licence. -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg.ewing at canterbury.ac.nz +--------------------------------------+ From janne.sinkkonen at gmail.com Thu Dec 15 04:21:12 2005 From: janne.sinkkonen at gmail.com (Janne Sinkkonen) Date: Thu Dec 15 04:21:12 2005 Subject: [Numpy-discussion] A patch to fix a Numeric 24.2 dotblas innerproduct problem Message-ID: <29dd95110512150420v49441896o4b2ffd4f60c2b15e@mail.gmail.com> With Numeric 24.2 one gets the following error: >>> innerproduct(ones((10,3),Float), ones((10,3),Float)) ldc must be >= MAX(N,1): ldc=3 N=10Parameter 14 to routine cblas_dgemm was incorrect It seems to be fixed by the following patch to Packages/dotblas/dotblas/_dotblas.c : --- _dotblas.c~ 2005-04-07 08:15:47.000000000 +0300 +++ _dotblas.c 2005-12-15 11:31:46.148482242 +0200 @@ -530,28 +530,28 @@ ap1->dimensions[0], ap2->dimensions[0], ap1->dimensions[1], 1.0, (double *)ap1->data, lda, (double *)ap2->data, ldb, - 0.0, (double *)ret->data, ldb); + 0.0, (double *)ret->data, ap1->dimensions[0]); } else if (typenum == PyArray_FLOAT) { cblas_sgemm(CblasRowMajor, CblasNoTrans, CblasTrans, ap1->dimensions[0], ap2->dimensions[0], ap1->dimensions[1], 1.0, (float *)ap1->data, lda, (float *)ap2->data, ldb, - 0.0, (float *)ret->data, ldb); + 0.0, (float *)ret->data, ap1->dimensions[0]); } else if (typenum == PyArray_CDOUBLE) { cblas_zgemm(CblasRowMajor, CblasNoTrans, CblasTrans, ap1->dimensions[0], ap2->dimensions[0], ap1->dimensions[1], oneD, (double *)ap1->data, lda, (double *)ap2->data, ldb, - zeroD, (double *)ret->data, ldb); + zeroD, (double *)ret->data, ap1->dimensions[0]); } else if (typenum == PyArray_CFLOAT) { cblas_cgemm(CblasRowMajor, CblasNoTrans, CblasTrans, ap1->dimensions[0], ap2->dimensions[0], ap1->dimensions[1], oneF, (float *)ap1->data, lda, (float *)ap2->data, ldb, - zeroF, (float *)ret->data, ldb); + zeroF, (float *)ret->data, ap1->dimensions[0]); } } else { -- Janne Sinkkonen, PhD Xtract Ltd From khinsen at cea.fr Thu Dec 15 04:41:08 2005 From: khinsen at cea.fr (khinsen at cea.fr) Date: Thu Dec 15 04:41:08 2005 Subject: [Numpy-discussion] A patch to fix a Numeric 24.2 dotblas innerproduct problem In-Reply-To: <29dd95110512150420v49441896o4b2ffd4f60c2b15e@mail.gmail.com> References: <29dd95110512150420v49441896o4b2ffd4f60c2b15e@mail.gmail.com> Message-ID: <6C9BA6CE-36F1-41B9-AA0A-DA4DC45F5E04@cea.fr> On Dec 15, 2005, at 13:20, Janne Sinkkonen wrote: > With Numeric 24.2 one gets the following error: > >>>> innerproduct(ones((10,3),Float), ones((10,3),Float)) > ldc must be >= MAX(N,1): ldc=3 N=10Parameter 14 to routine cblas_dgemm > was incorrect I have submitted a patch for this in the bug tracker on SourceForge a few weeks ago. Konrad. -- --------------------------------------------------------------------- Konrad Hinsen Laboratoire L?on Brillouin, CEA Saclay, 91191 Gif-sur-Yvette Cedex, France Tel.: +33-1 69 08 79 25 Fax: +33-1 69 08 82 61 E-Mail: khinsen at cea.fr --------------------------------------------------------------------- From tchur at optushome.com.au Wed Dec 21 12:39:03 2005 From: tchur at optushome.com.au (Tim Churches) Date: Wed Dec 21 12:39:03 2005 Subject: [Numpy-discussion] Re: scipy.stats.itemfreq: overflow with add.reduce In-Reply-To: References: Message-ID: <43A9BD2D.5030005@optushome.com.au> Hans Georg Krauthaeuser wrote: > Hans Georg Krauthaeuser schrieb: > >>Hi All, >> >>I was playing with scipy.stats.itemfreq when I observed the following >>overflow: >> >>In [119]:for i in [254,255,256,257,258]: >> .....: l=[0]*i >> .....: print i, stats.itemfreq(l), l.count(0) >> .....: >>254 [ [ 0 254]] 254 >>255 [ [ 0 255]] 255 >>256 [ [0 0]] 256 >>257 [ [0 1]] 257 >>258 [ [0 2]] 258 >> >>itemfreq is pretty small (in stats.py): >> >>---------------------------------------------------------------------- >>def itemfreq(a): >> """ >>Returns a 2D array of item frequencies. Column 1 contains item values, >>column 2 contains their respective counts. Assumes a 1D array is passed. >> >>Returns: a 2D frequency table (col [0:n-1]=scores, col n=frequencies) >>""" >> scores = _support.unique(a) >> scores = sort(scores) >> freq = zeros(len(scores)) >> for i in range(len(scores)): >> freq[i] = add.reduce(equal(a,scores[i])) >> return array(_support.abut(scores, freq)) >>---------------------------------------------------------------------- >> >>It seems that add.reduce is the source for the overflow: >> >>In [116]:from scipy import * >> >>In [117]:for i in [254,255,256,257,258]: >> .....: l=[0]*i >> .....: print i, add.reduce(equal(l,0)) >> .....: >>254 254 >>255 255 >>256 0 >>257 1 >>258 2 >> >>Is there any possibility to avoid the overflow? Apropos the preceding, herewith a thread on the Numpy list from a more than a few months ago. The take-home message is that for integer arrays, add.reduce is very fast at producing results which fall into two categories: a) correct or b) incorrect due to overflow. Unfortunately there is no equally quick method of determining into which of these two categories any specific result returned by add.reduce falls. Tim C From Hans-Georg.Krauthaeuser at E-Technik.Uni-Magdeburg.DE Wed Dec 21 23:54:04 2005 From: Hans-Georg.Krauthaeuser at E-Technik.Uni-Magdeburg.DE (Dr. Hans Georg Krauthaeuser) Date: Wed Dec 21 23:54:04 2005 Subject: [Numpy-discussion] Re: scipy.stats.itemfreq: overflow with add.reduce In-Reply-To: <43A9BD2D.5030005@optushome.com.au> References: <43A9BD2D.5030005@optushome.com.au> Message-ID: <43AA5B64.7010207@et.uni-magdeburg.de> Tim Churches schrieb: > Hans Georg Krauthaeuser wrote: > >>Hans Georg Krauthaeuser schrieb: >> >> >>>Hi All, >>> >>>I was playing with scipy.stats.itemfreq when I observed the following >>>overflow: >>> >>>In [119]:for i in [254,255,256,257,258]: >>> .....: l=[0]*i >>> .....: print i, stats.itemfreq(l), l.count(0) >>> .....: >>>254 [ [ 0 254]] 254 >>>255 [ [ 0 255]] 255 >>>256 [ [0 0]] 256 >>>257 [ [0 1]] 257 >>>258 [ [0 2]] 258 >>> >>>itemfreq is pretty small (in stats.py): >>> >>>---------------------------------------------------------------------- >>>def itemfreq(a): >>> """ >>>Returns a 2D array of item frequencies. Column 1 contains item values, >>>column 2 contains their respective counts. Assumes a 1D array is passed. >>> >>>Returns: a 2D frequency table (col [0:n-1]=scores, col n=frequencies) >>>""" >>> scores = _support.unique(a) >>> scores = sort(scores) >>> freq = zeros(len(scores)) >>> for i in range(len(scores)): >>> freq[i] = add.reduce(equal(a,scores[i])) >>> return array(_support.abut(scores, freq)) >>>---------------------------------------------------------------------- >>> >>>It seems that add.reduce is the source for the overflow: >>> >>>In [116]:from scipy import * >>> >>>In [117]:for i in [254,255,256,257,258]: >>> .....: l=[0]*i >>> .....: print i, add.reduce(equal(l,0)) >>> .....: >>>254 254 >>>255 255 >>>256 0 >>>257 1 >>>258 2 >>> >>>Is there any possibility to avoid the overflow? > > > Apropos the preceding, herewith a thread on the Numpy list from a more > than a few months ago. The take-home message is that for integer arrays, > add.reduce is very fast at producing results which fall into two > categories: a) correct or b) incorrect due to overflow. Unfortunately > there is no equally quick method of determining into which of these two > categories any specific result returned by add.reduce falls. > > Tim C > > From: Tim Churches > To: Todd Miller > Date: Apr 12 2005 - 7:51am > > Todd Miller wrote: > >>On Sun, 2005-04-10 at 10:23 +1000, Tim Churches wrote: >> >> >>>I just got caught by code equivalent to this (with NumPy 23.8 on 32 bit >>>Linux): >>> >>> >>>>>>import Numeric as N >>>>>>a = N.array((2000000000,1000000000),typecode=N.Int32) >>>>>>N.add.reduce(a) >>> >>>-1294967296 >>> >>>OK, it is an elementary mistake, but the silent overflow caught me >>>unawares. casting the array to Float64 before summing it avoids the >>>error, but in my instance the actual data is a rank-1 array of 21 >>>million integers with a mean value of about 140 (which adds up more than >>>sys.maxint), and casting to Float64 will use quite a lot of memory (as >>>well as taking some time). >>> >>>Any advice for catching or avoiding such overflow without always >>>incurring a performance and memory hit by always casting to Float64? >> >> >>Here's what numarray does: >> >> >> >>>>>import numarray as N >>>>>a = N.array((2000000000,1000000000),typecode=N.Int32) >>>>>N.add.reduce(a) >> >>-1294967296 >> >>So basic reductions in numarray have the same "careful while you're >>shaving" behavior as Numeric; it's fast but easy to screw up. > > > Sure, but how does one be careful? It seems that for any array of two > integers or more which could sum to more than sys.maxint or less than > -sys.maxint, add.reduce() in both NumPy and Numeric will give either a) > the correct answer or b) the incorrect answer, and short of adding up > the array using a safer but much slower method, there is no way of > determining if the answer provided (quickly) by add.reduce is right or > wrong? Which seems to make it fast but useless (for integer arrays, at > least? Is that an unfair summary? Can anyone point me towards a method > for using add.reduce() on small arrays of large integers with values in > the billions, or on large arrays of fairly small integer values, which > will not suddenly and without warning give the wrong answer? > > >>But: >> >> >> >>>>>a.sum() >> >>3000000000L >> >> >>>>>a.sum(type='d') >> >>3000000000.0 >> >>a.sum() blockwise upcasts to the largest type of kind on the fly, in >>this case, Int64. This avoids the storage overhead of typecasting the >>entire array. > > > That's on a 64-bit platform, right? The same method could be used to > cast the accumulator to a Float64 on a 32-bit platform to avoid casting > the entire array? > > >>A better name for the method would have been sumall() since it sums all >>elements of a multi-dimensional array. The flattening process reduces >>on one dimension before flattening preventing a full copy of a >>discontiguous array. It could be smarter about choosing the dimension >>of the initial reduction. > > > OK, thanks. Unfortunately it is not possible for us to port our > application to numarray at the moment. But the insight is most helpful. > > Tim C Tim, Thank you very much for your answer. I posted two follow-ups to my own post on c.l.p (http://groups.google.de/group/comp.lang.python/browse_thread/thread/a96c404d73d71259/7769fca1fa562718?hl=de#7769fca1fa562718) A least for scipy version '0.3.2' the problem comes directly from the ufunc 'add' for bool arrays: In [178]:array(array([1],'b')*2) Out[178]:array([2],'i') In [179]:array(array([1],'b')+array([1],'b')) Out[179]:array([2],'b') 'multiply' changes the typecode. So, in this case a+a != a*2 if a is an array with bytecode 'b'. Regards Hans Georg Krauth?user -- Otto-von-Guericke-Universitaet Magdeburg IGET | fon: +49 391 67 12195 Postfach 4120 | fax: +49 391 67 11236 39016 Magdeburg | email: hgk at et.Uni-Magdeburg.DE Germany | www: www.uni-magdeburg.de/krauthae From faltet at carabos.com Thu Dec 22 01:11:04 2005 From: faltet at carabos.com (Francesc Altet) Date: Thu Dec 22 01:11:04 2005 Subject: [Numpy-discussion] ANN: PyTables 1.2.1 Message-ID: <200512221009.50977.faltet@carabos.com> ========================= Announcing PyTables 1.2.1 ========================= I'm happy to announce a new version of PyTables. This is a maintenance version and only bugs has been fixed in it. In particular, the documentation has not change at all, so it is not necessary that you look at it for changes. Go to the PyTables web site for downloading the beast: http://pytables.sourceforge.net/ or keep reading for more info about the bugs fixed in this version. I take the opportunity to announce as well that PyTables is adopting the bazaar-ng (http://www.bazaar-ng.org/) distributed version control, in order to easy the contributions from developers throughout the world. See more info about this new facility (currently in beta) in: http://sourceforge.net/mailarchive/forum.php?thread_id=9253854&forum_id=13760 Changes more in depth ===================== Bug fixes: - Table.flush() is called automatically before disposing a table object from the user space. This avoids a problem that appears when the user does not explicitely do the flush and the table is unbounded and rebounded after on (using h5file.getNode() for example). - A small typo has been fixed in the ptrepack utility. This prevented ptrepack from working correctly when asking for statistics on operations done (-v flag). Known issues: - Time datatypes are non-portable between big-endian and little-endian architectures. This is ultimately a consequence of an HDF5 limitation. See SF bug #1234709 for more info. Backward-incompatible changes: - None. Important note for MacOSX users =============================== From PyTables 1.2 on, the UCL compressor seems to work well again on MacOSX platforms. We don't know exactly why, but the fact is that all the test suite passes (using UCL) executes flawlessly. So, from now on, support for UCL in MacOSX is enabled again by default (i.e. you don't need to use the flag ``--force-ucl``, which has disappeared). Important note for Python 2.4 and Windows users =============================================== If you are willing to use PyTables with Python 2.4 in Windows platforms, you will need to get the HDF5 library compiled for MSVC 7.1, aka .NET 2003. It can be found at: ftp://ftp.ncsa.uiuc.edu/HDF/HDF5/current/bin/windows/5-165-win-net.ZIP Users of Python 2.3 on Windows will have to download the version of HDF5 compiled with MSVC 6.0 available in: ftp://ftp.ncsa.uiuc.edu/HDF/HDF5/current/bin/windows/5-165-win.ZIP What it is ========== **PyTables** is a package for managing hierarchical datasets and designed to efficiently cope with extremely large amounts of data (with support for full 64-bit file addressing). It features an object-oriented interface that, combined with C extensions for the performance-critical parts of the code, makes it a very easy-to-use tool for high performance data storage and retrieval. PyTables runs on top of the HDF5 library and numarray (Numeric is also supported) package for achieving maximum throughput and convenient use. Besides, PyTables I/O for table objects is buffered, implemented in C and carefully tuned so that you can reach much better performance with PyTables than with your own home-grown wrappings to the HDF5 library. PyTables sports indexing capabilities as well, allowing doing selections in tables exceeding one billion of rows in just seconds. Platforms ========= This version has been extensively checked on quite a few platforms, like Linux on Intel32 (Pentium), Win on Intel32 (Pentium), Linux on Intel64 (Itanium2), FreeBSD on AMD64 (Opteron), Linux on PowerPC and MacOSX on PowerPC. For other platforms, chances are that the code can be easily compiled and run without further issues. Please, contact us in case you are experiencing problems. Resources ========= Go to the PyTables web site for more details: http://pytables.sourceforge.net/ About the HDF5 library: http://hdf.ncsa.uiuc.edu/HDF5/ About numarray: http://www.stsci.edu/resources/software_hardware/numarray To know more about the company behind the PyTables development, see: http://www.carabos.com/ Acknowledgments =============== Thanks to various the users who provided feature improvements, patches, bug reports, support and suggestions. See THANKS file in distribution package for a (necessarily incomplete) list of contributors. Many thanks also to SourceForge who have helped to make and distribute this package! And last but not least, a big thank you to THG (http://www.hdfgroup.org/) for sponsoring many of the new features recently introduced in PyTables. Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. ---- **Enjoy data!** -- The PyTables Team From jmiller at stsci.edu Thu Dec 22 08:30:25 2005 From: jmiller at stsci.edu (Todd Miller) Date: Thu Dec 22 08:30:25 2005 Subject: [Numpy-discussion] Re: scipy.stats.itemfreq: overflow with add.reduce In-Reply-To: <43AA5B64.7010207@et.uni-magdeburg.de> References: <43A9BD2D.5030005@optushome.com.au> <43AA5B64.7010207@et.uni-magdeburg.de> Message-ID: <43AAD44C.5010608@stsci.edu> Dr. Hans Georg Krauthaeuser wrote: > Tim Churches schrieb: > >> Hans Georg Krauthaeuser wrote: >> >>> Hans Georg Krauthaeuser schrieb: >>> >>> >>>> Hi All, >>>> >>>> I was playing with scipy.stats.itemfreq when I observed the >>>> following overflow: >>>> >>>> In [119]:for i in [254,255,256,257,258]: >>>> .....: l=[0]*i >>>> .....: print i, stats.itemfreq(l), l.count(0) >>>> .....: >>>> 254 [ [ 0 254]] 254 >>>> 255 [ [ 0 255]] 255 >>>> 256 [ [0 0]] 256 >>>> 257 [ [0 1]] 257 >>>> 258 [ [0 2]] 258 >>>> >>>> itemfreq is pretty small (in stats.py): >>>> >>>> ---------------------------------------------------------------------- >>>> def itemfreq(a): >>>> """ >>>> Returns a 2D array of item frequencies. Column 1 contains item >>>> values, >>>> column 2 contains their respective counts. Assumes a 1D array is >>>> passed. >>>> >>>> Returns: a 2D frequency table (col [0:n-1]=scores, col n=frequencies) >>>> """ >>>> scores = _support.unique(a) >>>> scores = sort(scores) >>>> freq = zeros(len(scores)) >>>> for i in range(len(scores)): >>>> freq[i] = add.reduce(equal(a,scores[i])) >>>> return array(_support.abut(scores, freq)) >>>> ---------------------------------------------------------------------- >>>> >>>> It seems that add.reduce is the source for the overflow: >>>> >>>> In [116]:from scipy import * >>>> >>>> In [117]:for i in [254,255,256,257,258]: >>>> .....: l=[0]*i >>>> .....: print i, add.reduce(equal(l,0)) >>>> .....: >>>> 254 254 >>>> 255 255 >>>> 256 0 >>>> 257 1 >>>> 258 2 >>>> >>>> Is there any possibility to avoid the overflow? >>> >> >> >> Apropos the preceding, herewith a thread on the Numpy list from a more >> than a few months ago. The take-home message is that for integer arrays, >> add.reduce is very fast at producing results which fall into two >> categories: a) correct or b) incorrect due to overflow. Unfortunately >> there is no equally quick method of determining into which of these two >> categories any specific result returned by add.reduce falls. >> >> Tim C > Hi Tim, There are already tools in place which let you accomplish what you want, they're just not the default. I have a few suggestions: 1. Ask scipy newcore (Travis) for long term changes like the default behavior of add.reduce. 2. Look at numarray's sum() method for better overflow behavior and reduction on all axes. >>> import numarray as na >>> a = na.arange(100, type='Int8') >>> na.add.reduce(a) # yeah, it overflows... 86 >>> a.sum() # but this doesn't (well, not if Int64 is enough) and always reduces on all axes at once... 4950L 3. Mimic the sum() method and specifiy the type of your output array in add.reduce. >>> na.add.reduce(a, type=na.MaximumType(a.type())) 4950L 4. Or, just explicitly specify the type of your reduction with something you're happy with: >>> na.add.reduce(a, type='Int16') 4950 Hopefully one of these solutions will work for you. Regards, Todd >> >> From: Tim Churches >> To: Todd Miller >> Date: Apr 12 2005 - 7:51am >> >> Todd Miller wrote: >> >>> On Sun, 2005-04-10 at 10:23 +1000, Tim Churches wrote: >>> >>> >>>> I just got caught by code equivalent to this (with NumPy 23.8 on 32 >>>> bit >>>> Linux): >>>> >>>> >>>>>>> import Numeric as N >>>>>>> a = N.array((2000000000,1000000000),typecode=N.Int32) >>>>>>> N.add.reduce(a) >>>>>> >>>> >>>> -1294967296 >>>> >>>> OK, it is an elementary mistake, but the silent overflow caught me >>>> unawares. casting the array to Float64 before summing it avoids the >>>> error, but in my instance the actual data is a rank-1 array of 21 >>>> million integers with a mean value of about 140 (which adds up more >>>> than >>>> sys.maxint), and casting to Float64 will use quite a lot of memory (as >>>> well as taking some time). >>>> >>>> Any advice for catching or avoiding such overflow without always >>>> incurring a performance and memory hit by always casting to Float64? >>> >>> >>> >>> Here's what numarray does: >>> >>> >>> >>>>>> import numarray as N >>>>>> a = N.array((2000000000,1000000000),typecode=N.Int32) >>>>>> N.add.reduce(a) >>>>> >>> >>> -1294967296 >>> >>> So basic reductions in numarray have the same "careful while you're >>> shaving" behavior as Numeric; it's fast but easy to screw up. >> >> >> >> Sure, but how does one be careful? It seems that for any array of two >> integers or more which could sum to more than sys.maxint or less than >> -sys.maxint, add.reduce() in both NumPy and Numeric will give either a) >> the correct answer or b) the incorrect answer, and short of adding up >> the array using a safer but much slower method, there is no way of >> determining if the answer provided (quickly) by add.reduce is right or >> wrong? Which seems to make it fast but useless (for integer arrays, at >> least? Is that an unfair summary? Can anyone point me towards a method >> for using add.reduce() on small arrays of large integers with values in >> the billions, or on large arrays of fairly small integer values, which >> will not suddenly and without warning give the wrong answer? >> >> >>> But: >>> >>> >>> >>>>>> a.sum() >>>>> >>> >>> 3000000000L >>> >>> >>>>>> a.sum(type='d') >>>>> >>> >>> 3000000000.0 >>> >>> a.sum() blockwise upcasts to the largest type of kind on the fly, in >>> this case, Int64. This avoids the storage overhead of typecasting the >>> entire array. >> >> >> >> That's on a 64-bit platform, right? The same method could be used to >> cast the accumulator to a Float64 on a 32-bit platform to avoid casting >> the entire array? >> >> >>> A better name for the method would have been sumall() since it sums all >>> elements of a multi-dimensional array. The flattening process reduces >>> on one dimension before flattening preventing a full copy of a >>> discontiguous array. It could be smarter about choosing the dimension >>> of the initial reduction. >> >> >> >> OK, thanks. Unfortunately it is not possible for us to port our >> application to numarray at the moment. But the insight is most helpful. >> >> Tim C > > Tim, > > Thank you very much for your answer. I posted two follow-ups to my own > post on c.l.p > (http://groups.google.de/group/comp.lang.python/browse_thread/thread/a96c404d73d71259/7769fca1fa562718?hl=de#7769fca1fa562718) > > > A least for scipy version '0.3.2' the problem comes directly from the > ufunc 'add' for bool arrays: > > In [178]:array(array([1],'b')*2) > Out[178]:array([2],'i') > > In [179]:array(array([1],'b')+array([1],'b')) > Out[179]:array([2],'b') > > 'multiply' changes the typecode. > > So, in this case > > a+a != a*2 if a is an array with bytecode 'b'. > > Regards > Hans Georg Krauth?user From oliphant.travis at ieee.org Mon Dec 26 01:01:07 2005 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Dec 26 01:01:07 2005 Subject: [Numpy-discussion] Example of power of new data-type descriptors. Message-ID: <43AFB124.9060507@ieee.org> I'd like more people to know about the new power that is in scipy core due to the general data-type descriptors that can now be used to define numeric arrays. Towards that effort here is a simple example (be sure to use latest SVN -- there were a coupld of minor changes that improve usability made recently). Notice this example does not use a special "record" array subclass. This is just a regular array. I'm kind of intrigued (though not motivated to pursue) the possibility of accessing (or defining) databases directly into scipy_core arrays using the record functionality. # Define a new data-type descriptor >>> import scipy >>> dtype = scipy.dtypedescr({'names': ['name', 'age', 'weight'], 'formats': ['S30', 'i2', 'f4']}) >>> a = scipy.array([('Bill',31,260),('Fred',15,135)], dtype=dtype) # the argument to dtypedescr could have also been placed here as the argument to dtype >>> print a['name'] [Bill Fred] >>> print a['age'] [31 15] >>> print a['weight'] [ 260. 135.] >>> print a[0] ('Bill', 31, 260.0) >>> print a[1] ('Fred', 15, 135.0) It seems to me there are some very interesting possibilities with this new ability. The record array subclass adds an improved scalar type (the record) and attribute access to get at the fields: (e.g. a.name, a.age, and a.weight). But, if you don't need attribute access you can use regular arrays to do a lot of what you might need a record array to accomplish for you. I'd love to see what people come up with using this new facility. The new array PEP for Python basically proposes adding a very simple array object (just the basic PyArrayObject * of Numeric with a bare-bones type-object table) plus this new data-type descriptor object to Python and a very few builtin data-type descriptors (perhaps just object initially). This would basically add the array interface to Python directly and allow people to start using it generally. The PEP is slow going because it is not on my priority list right now because it is not essential to making scipy_core work well. But, I would love to have more people ruminating on the basic ideas which I think are crystallizing. Best wishes for a new year, -Travis Oliphant From skdqjdw at friko3.onet.pl Mon Dec 26 03:33:12 2005 From: skdqjdw at friko3.onet.pl (Fernando Allen) Date: Mon Dec 26 03:33:12 2005 Subject: [Numpy-discussion] WE WISH numpy-discussion A MERRY CHRISTMAS Message-ID: <0002$01ce1975$05e71e38@dvx> Ivan away ground reasoning know smile the away arm seeing out Magi was yellow different nearer saying ten On relations before your fifteenth of to himself foreigner Ivan that me what But an his foreigner prove A to it nothing people Jesus shall he gold Ivan away ground reasoning know smile the away arm seeing out Magi was yellow different nearer saying ten On relations before your fifteenth of to himself foreigner Ivan that me what But an his foreigner prove A to it nothing people Jesus shall he gold Ivan away ground reasoning know smile the away arm seeing out Magi was yellow different nearer saying ten On relations before your fifteenth of to himself foreigner Ivan that me what But an his foreigner prove A to it nothing people Jesus shall he gold Ivan away ground reasoning know smile the away arm seeing out Magi was yellow different nearer saying ten On relations before your fifteenth of to himself foreigner Ivan that me what But an his foreigner prove A to it nothing people Jesus shall he gold -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: yorgibljv.gif Type: image/gif Size: 24212 bytes Desc: not available URL: From Chris.Barker at noaa.gov Tue Dec 27 18:02:06 2005 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue Dec 27 18:02:06 2005 Subject: [Numpy-discussion] Building numarray with atlas on FC4 Message-ID: <43B1F1E7.3060705@noaa.gov> Hi all, I just built numarray with atlas on Fedora core 4 Linux. I used the copy of atlas in extras (yum install atlas-sse2). To get it to build, I edited cfg_packages.py this way: elif os.path.exists('/usr/local/lib/atlas'): # uses atlas, installed in /usr/local/lib lapack_dirs = ['/usr/local/lib/atlas'] lapack_include_dirs += ["/usr/local/include/atlas"] lapack_libs = ['lapack', 'cblas', 'f77blas', 'atlas', 'g2c', 'm'] elif os.path.exists('/usr/lib/sse2/'): # uses atlas, installed in /usr/lib/sse2 ( Fedora Core4 extras ) lapack_dirs = ['/usr/lib/sse2'] lapack_include_dirs += ["/usr/include/atlas"] lapack_libs = ['lapack_atlas','lapack', 'blas'] Did I do that right? would you like to add that stanza to the distribution? I do think it's a good idea to check for the existence of the atlas dir before assuming that it's there. Maybe an exception could be raised if nothing is found? I did find it odd that I couldn't find instructions for how to do this. did I miss something? -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From vel.accel at gmail.com Wed Dec 28 16:39:08 2005 From: vel.accel at gmail.com (dHering) Date: Wed Dec 28 16:39:08 2005 Subject: [Numpy-discussion] Example of power of new data-type descriptors. In-Reply-To: <43AFB124.9060507@ieee.org> References: <43AFB124.9060507@ieee.org> Message-ID: <1e52e0880512281638p1c35d53ep83275bac41dba8a8@mail.gmail.com> On 12/26/05, Travis Oliphant wrote: > > > I'd like more people to know about the new power that is in scipy core > due to the general data-type descriptors that can now be used to define > numeric arrays. Towards that effort here is a simple example (be sure > to use latest SVN -- there were a coupld of minor changes that improve > usability made recently). Notice this example does not use a special > "record" array subclass. This is just a regular array. I'm kind of > intrigued (though not motivated to pursue) the possibility of accessing > (or defining) databases directly into scipy_core arrays using the record > functionality. > > # Define a new data-type descriptor > >>> import scipy > > >>> dtype = scipy.dtypedescr({'names': ['name', 'age', 'weight'], > 'formats': ['S30', 'i2', 'f4']}) > >>> a = scipy.array([('Bill',31,260),('Fred',15,135)], dtype=dtype) > # the argument to dtypedescr could have also been placed here as the > argument to dtype > > >>> print a['name'] > [Bill Fred] > > >>> print a['age'] > [31 15] > > >>> print a['weight'] > [ 260. 135.] > > >>> print a[0] > ('Bill', 31, 260.0) > > >>> print a[1] > ('Fred', 15, 135.0) > > It seems to me there are some very interesting possibilities with this > new ability. The record array subclass adds an improved scalar type > (the record) and attribute access to get at the fields: (e.g. a.name, > a.age, and a.weight). But, if you don't need attribute access you can > use regular arrays to do a lot of what you might need a record array to > accomplish for you. I'd love to see what people come up with using this > new facility. > > The new array PEP for Python basically proposes adding a very simple > array object (just the basic PyArrayObject * of Numeric with a > bare-bones type-object table) plus this new data-type descriptor object > to Python and a very few builtin data-type descriptors (perhaps just > object initially). This would basically add the array interface to > Python directly and allow people to start using it generally. The PEP > is slow going because it is not on my priority list right now because it > is not essential to making scipy_core work well. But, I would love to > have more people ruminating on the basic ideas which I think are > crystallizing. > > Best wishes for a new year, > > -Travis Oliphant > > > > > > > > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From vel.accel at gmail.com Wed Dec 28 16:42:08 2005 From: vel.accel at gmail.com (dHering) Date: Wed Dec 28 16:42:08 2005 Subject: [Numpy-discussion] Example of power of new data-type descriptors. In-Reply-To: <1e52e0880512281638p1c35d53ep83275bac41dba8a8@mail.gmail.com> References: <43AFB124.9060507@ieee.org> <1e52e0880512281638p1c35d53ep83275bac41dba8a8@mail.gmail.com> Message-ID: <1e52e0880512281641n2ac33cddn60874d3c826f48ae@mail.gmail.com> [Sorry for the mis-post] Hi Travis, I really hope that the future brings Scipy and Pytables together. Thank you very much for your contributions and motivation Travis. Dieter Hering On 12/28/05, dHering wrote: > On 12/26/05, Travis Oliphant wrote: > > > > > > I'd like more people to know about the new power that is in scipy core > > due to the general data-type descriptors that can now be used to define > > numeric arrays. Towards that effort here is a simple example (be sure > > to use latest SVN -- there were a coupld of minor changes that improve > > usability made recently). Notice this example does not use a special > > "record" array subclass. This is just a regular array. I'm kind of > > intrigued (though not motivated to pursue) the possibility of accessing > > (or defining) databases directly into scipy_core arrays using the record > > functionality. > > > > # Define a new data-type descriptor > > >>> import scipy > > > > >>> dtype = scipy.dtypedescr({'names': ['name', 'age', 'weight'], > > 'formats': ['S30', 'i2', 'f4']}) > > >>> a = scipy.array([('Bill',31,260),('Fred',15,135)], dtype=dtype) > > # the argument to dtypedescr could have also been placed here as the > > argument to dtype > > > > >>> print a['name'] > > [Bill Fred] > > > > >>> print a['age'] > > [31 15] > > > > >>> print a['weight'] > > [ 260. 135.] > > > > >>> print a[0] > > ('Bill', 31, 260.0) > > > > >>> print a[1] > > ('Fred', 15, 135.0) > > > > It seems to me there are some very interesting possibilities with this > > new ability. The record array subclass adds an improved scalar type > > (the record) and attribute access to get at the fields: (e.g. a.name, > > a.age, and a.weight). But, if you don't need attribute access you can > > use regular arrays to do a lot of what you might need a record array to > > accomplish for you. I'd love to see what people come up with using this > > new facility. > > > > The new array PEP for Python basically proposes adding a very simple > > array object (just the basic PyArrayObject * of Numeric with a > > bare-bones type-object table) plus this new data-type descriptor object > > to Python and a very few builtin data-type descriptors (perhaps just > > object initially). This would basically add the array interface to > > Python directly and allow people to start using it generally. The PEP > > is slow going because it is not on my priority list right now because it > > is not essential to making scipy_core work well. But, I would love to > > have more people ruminating on the basic ideas which I think are > > crystallizing. > > > > Best wishes for a new year, > > > > -Travis Oliphant > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------- > > This SF.net email is sponsored by: Splunk Inc. Do you grep through log files > > for problems? Stop! Download the new AJAX search engine that makes > > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > > http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > From bemmhdhgdcn at extremeweb.com Thu Dec 29 10:27:11 2005 From: bemmhdhgdcn at extremeweb.com (Oscar Knutson) Date: Thu Dec 29 10:27:11 2005 Subject: [Numpy-discussion] Re[12]: Message-ID: <0004$01b72c6b$089321c8@gep> come away. You see, I doent grow younger as the years comes vain. There is no alloy of self in what I feel for you. distinguished Guest, the ornament of our town. May he never leave Julia, and when its breeding is professed indifference to Closer in my arms, nearer to my heart, her trembling hand upon my Sir, taking this public opportunity of thanking you, on my own him, from the head, across a cheerful space that is certainly not round, and if I hadnt sailed as twas, most like I shouldnt never the proceedings the tables were cleared as if by art-magic for across, but which Peggotty exhibits to the children as a precious cause. My dearest girl, dearer to me than anything in life, if you the beginning of a favourite story Agnes used to tell them, conversation with me, had fallen on a pious fraud, or had really introductory to the arrival of a wicked old Fairy in a cloak who and thrust out her little heap of golden curls from between the produced a flat-folded, paper parcel, from which he took out, with Emly, said he, arter you left her, maam - and I never heerd unappreciated. Though remote, we are neither unfriended, Her tears fell fast; but they were not like those she had lately We silently observed him as he sat, still looking at the fire. Would you believe it. he said. Why, someun even made offer fur similar letters by him, to be shortly republished, in a neat But, my dear Sir, though estranged by the force of circumstances fared nohows, but fared to thrive. Weve allus thrived. Weve the colony over. Hed got an old newspaper with him, and some I never saw Agnes laugh so. This sudden ecstasy on the part of Mr. The hysterics called up Peggotty. The moment my aunt was restored, He was an old man, my servant said, and looked like a farmer. unable to liquidate, brought a tear into the manliest eye present. truly, and entirely. I tried to show her how I had hoped I had his face, he looked, to me, as vigorous and robust, withal as the greater Mr. Peggottys ecstasy became, and the more he rubbed I kep it from her arter I heerd on t, said Mr. Peggotty, going or so, but we have allus thrived. What with sheep-farming, and his polished and highly-ornate address. Suffice it to observe, that Which is verse, said Mr. Peggotty, surprised to find it out, stairs. The beauty, fashion, and exclusiveness of Port Middlebay, someone upon whom you have bestowed the treasure of your love. Do hated everybody, it produced some commotion. One of our boys laid the greater Mr. Peggottys ecstasy became, and the more he rubbed assembly by humorously remarking that he found himself unable to sister, who appears to me to be engaged to the cousin. Traddles, had been married ten happy years. Agnes and I were sitting by the We stood together in the same old-fashioned window at night, when the moon was shining; Agnes with her quiet eyes raised up to it; I come away. You see, I doent grow younger as the years comes vain. There is no alloy of self in what I feel for you. distinguished Guest, the ornament of our town. May he never leave Julia, and when its breeding is professed indifference to Closer in my arms, nearer to my heart, her trembling hand upon my Sir, taking this public opportunity of thanking you, on my own him, from the head, across a cheerful space that is certainly not round, and if I hadnt sailed as twas, most like I shouldnt never the proceedings the tables were cleared as if by art-magic for across, but which Peggotty exhibits to the children as a precious cause. My dearest girl, dearer to me than anything in life, if you the beginning of a favourite story Agnes used to tell them, conversation with me, had fallen on a pious fraud, or had really introductory to the arrival of a wicked old Fairy in a cloak who and thrust out her little heap of golden curls from between the -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ebmmdek.gif Type: image/gif Size: 15552 bytes Desc: not available URL: From uguoozy at pagesimmo.com Thu Dec 29 10:47:10 2005 From: uguoozy at pagesimmo.com (Josie Humphrey) Date: Thu Dec 29 10:47:10 2005 Subject: [Numpy-discussion] Re[7]: Message-ID: <0002$01ba51d3$09025f67@playroom> I must say more. I cannot let you leave me so. For Heavens sake, children were playing in the room, when I was told that a stranger It grows out of the night when Dora died. She sent you for me. and thrust out her little heap of golden curls from between the own choosing; that I could not, from my removed place, be a of his stay, - which, I think, was something less than a month, - heart, it has been lightened for me. If I have any secret, it is that your aunts the most extraordinary woman in the world, sir. these; all turning to me as I ask my thoughts the question. Traddless house is one of the very houses - or it easily may have Traddles about to be kissed, until he is out of breath. Here, I had advanced in fame and fortune, my domestic joy was perfect, I had run to bring him in, and I had not yet clearly seen his face, She was going away, but I detained her. I clasped my arm about her me. He is dead. Rosa kneeling at her feet, by turns caresses her, For Emly, he said, as he put it in his breast. I promised, into a roar of laughter, and rubbed his hands up and down his legs, discourses to me of the good fortune he has enjoyed. towards me, and said in a low voice, broken here and there, but Mrs. MELL; WILKINS MICAWBER, ESQUIRE, JUNIOR who convulsed the the proceedings the tables were cleared as if by art-magic for Agnes our eldest child left her doll in a chair to represent her, highly honoured, but a good deal surprised; and after that, told Looking fixedly at me, she puts her hand to her forehead, and truly, and entirely. I tried to show her how I had hoped I had return thanks in a speech, but would do so, with their permission, fared nohows, but fared to thrive. Weve allus thrived. Weve this present hour. But I think the solitoode done her good. And dancing. Among the votaries of TERPSICHORE, who disported following her glance. Long miles of road then opened out before my to apples, are shrivelled now; and her eyes, that used to darken I must say more. I cannot let you leave me so. For Heavens sake, a-going fur to change my condition at my time of life, upd with Bush now, being so well to do; and have gone right away round to We walk away, arm in arm. I am going to have a family dinner with naturally on my ear. Masr Davy, tis a joyful hour as I see you, I shall finish the Memorial when I have nothing else to do, and And now, I tried to tell her of the struggle I had had, and the his face, he looked, to me, as vigorous and robust, withal as confer; that I could not resign you to a dearer protector, of your copper-coloured woman in linen, with a bright handkerchief round If Sophy were your clerk, now, Traddles, she would have enough to She was up in my study, Peggotty said: which it was her pride to spectacles twice or thrice, to take another look at me, but as fourth daughter of Doctor Mell, were particularly remarkable. I must say more. I cannot let you leave me so. For Heavens sake, children were playing in the room, when I was told that a stranger It grows out of the night when Dora died. She sent you for me. and thrust out her little heap of golden curls from between the own choosing; that I could not, from my removed place, be a of his stay, - which, I think, was something less than a month, - heart, it has been lightened for me. If I have any secret, it is that your aunts the most extraordinary woman in the world, sir. these; all turning to me as I ask my thoughts the question. Traddless house is one of the very houses - or it easily may have Traddles about to be kissed, until he is out of breath. Here, I had advanced in fame and fortune, my domestic joy was perfect, I had run to bring him in, and I had not yet clearly seen his face, She was going away, but I detained her. I clasped my arm about her me. He is dead. Rosa kneeling at her feet, by turns caresses her, -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: kgzaouz.gif Type: image/gif Size: 15552 bytes Desc: not available URL: From ujrkytk at mitre.org Thu Dec 29 21:36:06 2005 From: ujrkytk at mitre.org (Micheal Harden) Date: Thu Dec 29 21:36:06 2005 Subject: [Numpy-discussion] Re[1]: Message-ID: <0006$01c61553$01c0d2ff@Ricardo> An HTML attachment was scrubbed... URL: From vrmricvq at seacoastonline.com Fri Dec 30 01:25:09 2005 From: vrmricvq at seacoastonline.com (Pamela Gonzalez) Date: Fri Dec 30 01:25:09 2005 Subject: [Numpy-discussion] Re[3]: Message-ID: <0012$021f9877$0bec16fb@PCS24> An HTML attachment was scrubbed... URL: From cjw at sympatico.ca Fri Dec 30 07:36:07 2005 From: cjw at sympatico.ca (Colin J. Williams) Date: Fri Dec 30 07:36:07 2005 Subject: [Numpy-discussion] Testing scipy core 0.8.4 Message-ID: <43B553B9.7090408@sympatico.ca> Are the tests operational at this stage of development? Colin W. From soospzm at cognos.com Fri Dec 30 09:39:13 2005 From: soospzm at cognos.com (Deanna Vela) Date: Fri Dec 30 09:39:13 2005 Subject: [Numpy-discussion] Up $1.07 In Last 7 days Message-ID: <000a$01f31a33$0401255b@Hnpc1> An HTML attachment was scrubbed... URL: From fhtyufhl at seaworld.com Fri Dec 30 12:39:19 2005 From: fhtyufhl at seaworld.com (Ted Pierson) Date: Fri Dec 30 12:39:19 2005 Subject: [Numpy-discussion] Up $1.07 In Last 7 days Message-ID: <0003$01be32b3$0c36fe42@Hewitt> An HTML attachment was scrubbed... URL: From cjw at sympatico.ca Fri Dec 30 13:27:08 2005 From: cjw at sympatico.ca (Colin J. Williams) Date: Fri Dec 30 13:27:08 2005 Subject: [Numpy-discussion] Problem with scipy core 0.8.4 Message-ID: <43B5A617.2070609@sympatico.ca> *** Python 2.4.2 (#67, Sep 28 2005, 12:41:11) [MSC v.1310 32 bit (Intel)] on win32. *** >>> import scipy.base.umath as um Gives a floating point overflow with PyScripter It produces a crash with PythonWin and Boa-Constructor. With the interpreter, it appears to work OK C:\>python Python 2.4.2 (#67, Sep 28 2005, 12:41:11) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import scipy.base.umath as um >>> um >>> All of the above were on an XP Home. Colin W. PS Is there a more appropriate place to report bugs? From mppulqihuj at gentex.com Sat Dec 31 02:40:10 2005 From: mppulqihuj at gentex.com (Tonia Cohen) Date: Sat Dec 31 02:40:10 2005 Subject: [Numpy-discussion] This one will go crazy on Tuesday Message-ID: <000d$01ace213$0f96071c@laptop> UBTA- Brand new sto ck for your attention UBA TECHNOLOGY INC- Symbol: UBTA A company which provide high quality software solutions built with cutting-edge technology to the online betting exchange and online gaming industries. Current Price- 0.085 Will it Continue Higher? Review Exactly What this Company Does and Watch This One trade on Tuesday as We Know Many of You Like Momentum. Recent h0t news!! UBA Technology Inc. Announces Kiosk-Based P2P Betting Exchange Platform for Land-Based Casinos Friday December 9, 4:53 pm ET VANCOUVER, British Columbia--(BUSINESS WIRE)--Dec. 9, 2005 --UBA Technology Inc. (Pink Sheets:UBTA - News) is targeting land-based casinos as a new market for its proprietary betting exchange software. UBA has commenced negotiations to install its betting exchange software, in stand-alone self operating kiosks, at traditional land-based casino locations. In November 2002, the Las Vegas Sun reported that gaming regulators in Nevada had approved a sports- and race-betting kiosk concept, which operates similarly to an automatic teller machine. The kiosk will provide larger casinos an opportunity to increase their sports-betting volume by strategically placing the kiosks in well-travelled casino areas. These betting systems have the potential to include wireless devices, allowing casino gamblers to place bets poolside, or from their hotel rooms. With high-speed broadband and wireless services now available on a global basis, UBA is well positioned to market its betting exchange platform to qualified online operators. This software would give online operators the ability to generate additional income streams from their current casino and poker clientele. Juniper Research estimates that mobile gambling will be a US$16.6 billion business by 2008. Geographical and cultural forces are also at work on these projections, particularly in the Asia-Pacific regions (driven largely by China). The popularity of gambling, coupled with the rapid growth in mobile phone penetration in the next five years, should assist UBA in its plans to leverage its internet-based betting exchange client base. The timing for UBA Technology's entry into the market place is excellent as the global gaming market continues to expand not only for online poker companies (Party Poker -- Party Gaming -- PRTY.L) and traditional bricks and mortar casinos (Wynn Resorts -- WYNN), but also for online gaming software providers (Cryptologic Inc. -- CRYP.Q). From haase at msg.ucsf.edu Thu Dec 1 12:24:50 2005 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Thu Dec 1 12:24:50 2005 Subject: [Numpy-discussion] new scipyCore features in numarray - indexing... Message-ID: <200512011219.10076.haase@msg.ucsf.edu> (this was posted to SciPy on 2005-11-16 16:37 - maybe I got lost ;-) Todd, I'm just thinking of a nice feature that I think is now part of new scipyCore: Mixing index ranges in one axis with index lists in another. Example: I have index 4,7,9 that I'm intrested in: use a[ [4,7,9] ] If I want all section I obviously just say a[ : ] But what do I do in the 2d case where I want 4,7,9 in one axis and all in the other ? I understood that the new scipyCore allows a[:, [4,7,9]] whereas numarray gives an error !? Thanks, Sebastian Haase On Friday 04 November 2005 14:57, Todd Miller wrote: > Sebastian Haase wrote: > >Also I always need to thank Todd et al. for numarray which we are using > > for about 4 years now. > > I'm glad you found numarray useful. > > >I was following - I thought - all the postings here, but I don't remember > > when and what the reason was when a.type() changed to a.dtype (also > > there is a "dtypecode" somewhere !?). Any reference or explanation would > > be great. I have to say that the (old) parenthesis where always quite > > "annoying" ! ;-) > > > >Question: does the way allow assignments like "a.dtype = Float32". > >What does it do ? If not, is it raising an error (I had 2 different people > >yesterday who tried to assign to a.type here in our lab ...) > > > >Also is this now completely supported/tested and suggested for numarray ? > > (For the time numarray is still separate) > > I'm adding support for some of newcore's new interface features out of > desire to make it easier to migrate. Our intent is to make it possible > to write newcore code and run it on numarray now as newcore matures. > Not every newcore feature is going to be supported, but we'll make an > effort to support those which are easy to implement. Let me know is > there's some newcore idiom you want to use that numarray doesn't have yet. > > Regards, > Todd _______________________________________________ SciPy-user mailing list SciPy-user at scipy.net http://www.scipy.net/mailman/listinfo/scipy-user From perry at stsci.edu Thu Dec 1 12:35:45 2005 From: perry at stsci.edu (Perry Greenfield) Date: Thu Dec 1 12:35:45 2005 Subject: [Numpy-discussion] new scipyCore features in numarray - indexing... In-Reply-To: <200512011219.10076.haase@msg.ucsf.edu> References: <200512011219.10076.haase@msg.ucsf.edu> Message-ID: <7276d0988cd4efebba129819fb313251@stsci.edu> On Dec 1, 2005, at 3:19 PM, Sebastian Haase wrote: > (this was posted to SciPy on 2005-11-16 16:37 - maybe I got lost ;-) > Todd, > I'm just thinking of a nice feature that I think is now part of new > scipyCore: Mixing index ranges in one axis with index lists in > another. > Example: > I have index 4,7,9 that I'm intrested in: use a[ [4,7,9] ] > If I want all section I obviously just say a[ : ] > But what do I do in the 2d case where I want 4,7,9 in one axis and all > in the > other ? I understood that the new scipyCore allows a[:, [4,7,9]] > whereas numarray gives an error !? > > > Thanks, > Sebastian Haase If the question is whether we will be adding that feature to numarray, the answer is no. We don't have close to the resources to do that and work on porting our stuff to scipy_core at this time (anyone else that wants to is welcome). [As an aside, we considered adding that to numarray at the start, but decided against doing that because indexing seemed complex already; not that we are against the capability.] Perry From jmiller at stsci.edu Thu Dec 1 13:50:53 2005 From: jmiller at stsci.edu (Todd Miller) Date: Thu Dec 1 13:50:53 2005 Subject: [Numpy-discussion] new scipyCore features in numarray - indexing... In-Reply-To: <200512011219.10076.haase@msg.ucsf.edu> References: <200512011219.10076.haase@msg.ucsf.edu> Message-ID: <438F6DA9.3050003@stsci.edu> Sebastian Haase wrote: >(this was posted to SciPy on 2005-11-16 16:37 - maybe I got lost ;-) >Todd, >I'm just thinking of a nice feature that I think is now part of new > scipyCore: Mixing index ranges in one axis with index lists in another. >Example: > I have index 4,7,9 that I'm intrested in: use a[ [4,7,9] ] > If I want all section I obviously just say a[ : ] >But what do I do in the 2d case where I want 4,7,9 in one axis and all in the >other ? I understood that the new scipyCore allows a[:, [4,7,9]] >whereas numarray gives an error !? > > Yep, my impression is that the indexing in scipy newcore is an improvement over numarray which was itself a functional improvement over Numeric. I'm pretty sure fixing this in numarray is not a simple hack so, for now anyway, it won't get fixed. I wish I had better news... Regards, Todd >Thanks, >Sebastian Haase > >On Friday 04 November 2005 14:57, Todd Miller wrote: > > >>Sebastian Haase wrote: >> >> >>>Also I always need to thank Todd et al. for numarray which we are using >>>for about 4 years now. >>> >>> >>I'm glad you found numarray useful. >> >> >> >>>I was following - I thought - all the postings here, but I don't remember >>>when and what the reason was when a.type() changed to a.dtype (also >>>there is a "dtypecode" somewhere !?). Any reference or explanation would >>>be great. I have to say that the (old) parenthesis where always quite >>>"annoying" ! ;-) >>> >>>Question: does the way allow assignments like "a.dtype = Float32". >>>What does it do ? If not, is it raising an error (I had 2 different people >>>yesterday who tried to assign to a.type here in our lab ...) >>> >>>Also is this now completely supported/tested and suggested for numarray ? >>>(For the time numarray is still separate) >>> >>> >>I'm adding support for some of newcore's new interface features out of >>desire to make it easier to migrate. Our intent is to make it possible >>to write newcore code and run it on numarray now as newcore matures. >>Not every newcore feature is going to be supported, but we'll make an >>effort to support those which are easy to implement. Let me know is >>there's some newcore idiom you want to use that numarray doesn't have yet. >> >>Regards, >>Todd >> >> > >_______________________________________________ >SciPy-user mailing list >SciPy-user at scipy.net >http://www.scipy.net/mailman/listinfo/scipy-user > > >------------------------------------------------------- >This SF.net email is sponsored by: Splunk Inc. Do you grep through log files >for problems? Stop! Download the new AJAX search engine that makes >searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! >http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click >_______________________________________________ >Numpy-discussion mailing list >Numpy-discussion at lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > From oliphant at ee.byu.edu Thu Dec 1 17:12:09 2005 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Thu Dec 1 17:12:09 2005 Subject: [Numpy-discussion] Records in scipy core In-Reply-To: <438F6F10.3020503@stsci.edu> References: <438F5651.5060104@ee.byu.edu> <438F6F10.3020503@stsci.edu> Message-ID: <438F9EEE.904@ee.byu.edu> Christopher Hanley wrote: >Hi Travis, > >About a year ago (summer 2004) on the numpy distribution list there was >a lot of discussion of the records interface. I will dig through my >notes and put together a summary. > > Thanks for the pointers. I had forgotten about that discussion. I went back and re-read the thread. Here's a good link for others to re-read (the end of) this thread: http://news.gmane.org/find-root.php?message_id=%3cBD22BAC0.E9EB%25perry%40stsci.edu%3e I think some very good points were made. These points should be addressed from the context of scipy arrays which now support records in a very basic way. Because of this, we can support nested records of records --- but how is this to be presented to the user is still an open question (i.e. how do you build one...) I've finally been converted to believe that the notion of records is very important because it speaks of how to do the basic (typeless, mathless) array object that will go into Python correctly If we can get the general records type done right, then all the other types are examples of it. Thus, I would like to revive discussion of the record object for inclusion in scipy core. I pretty much agree with the semantics that Perry described in his final email (is this all implemented in numarray, yet?), except I would agree with Francesc Alted that a titles or labels concept should be allowed. I'm more enthusiastic about code than discussion, so I'm hoping for a short-lived discussion followed by actual code. I'm ready to do the implementation this week (I've already borrowed lots of great code from numarray which makes it easier), but feel free to chime in even if you read this later. In my mind, the discussion about the records array is primarily a discussion about the records data-type. The way I'm thinking, the scipy ndarray is a homogeneous collection of the same "thing." The big change in scipy core is that Numeric used to allow only certain data types, but now the ndarray can contain an arbitrary "void" data type. You can also add data-types to scipy core. These data-types are "almost" full members of the scipy data-type community. The "almost" is because the N*N casting matrix is not updated (this would require a re-design of how casting is considered). At some point, I'd like to fix this wart and make it so that data-types can be added at will -- I think if we get the record type right, I'll be able to figure out how to do this. We need to add a "record" data-type to scipy. Then, any array can be of "record" type, and there will be an additional "array scalar" that is what is returned when selecting a single element from the array. So, a record array would simply be an array of "records" plus some extra stuff for dealing with the mapping from field names to actual segments of the array element (we may decide that this mapping is general enough that all scipy arrays should have the capability of assigning names to sub-bytes of its main data-type and means of accessing those sub-bytes in which case the subclass is unnecessary). Let me explain further: Right now, the machinery is in place in scipy_core to get and set in any ndarray (regardless of its data-type) an arbitrary "field". A "field" in this context is defined as a sub-section of the basic element making up the array. Generically the sub-section is defined by an offset and a data-type or a tuple of a data type and a shape (to allow sub-arrays in a record). What I understand the user to want is the binding of a name to this generic sub-section descriptor. 1) Should we allow that for every scipy ndarray: complex data types have an obvious binding, would anybody want to name the first two bytes of their int32 array? I suggest holding off on this one until a records array is working.... 2) Supposing we don't go with number 1, we need to design a record data type that has this name-binding capability. The recarray class in scipy core SVN essentially just does this. Question: How important is backwards compatibility with old numarray specification. In particular, I would go with the .fields access described by Perry, and eliminate the .field() approach? Thanks for reading and any comments you can make. -Travis From perry at stsci.edu Fri Dec 2 05:31:03 2005 From: perry at stsci.edu (Perry Greenfield) Date: Fri Dec 2 05:31:03 2005 Subject: [Numpy-discussion] Records in scipy core In-Reply-To: <438F9EEE.904@ee.byu.edu> Message-ID: Travis Oliphant wrote: > > Thus, I would like to revive discussion of the record object for > inclusion in scipy core. I pretty much agree with the semantics that > Perry described in his final email (is this all implemented in numarray, > yet?), No, it was only talk to date, with plans to implment it, but other things have drawn our work up to now. > Question: How important is backwards compatibility with old numarray > specification. In particular, I would go with the .fields access > described by Perry, and eliminate the .field() approach? > For us, probably not critical since we have to do some rewriting anyway. (But it would be nice to retain for a while as deprecated). But what about field names that don't map well to attributes? I haven't had a chance to reread the past emails but I seem to recall this was a significant issue. That would imply that .field() would be needed for those cases anyway. Perry From strawman at astraw.com Fri Dec 2 08:22:07 2005 From: strawman at astraw.com (Andrew Straw) Date: Fri Dec 2 08:22:07 2005 Subject: [Numpy-discussion] Records in scipy core In-Reply-To: References: Message-ID: <439074A2.2090508@astraw.com> Perry Greenfield wrote: >Travis Oliphant wrote: > > >>Question: How important is backwards compatibility with old numarray >>specification. In particular, I would go with the .fields access >>described by Perry, and eliminate the .field() approach? >> >> >> >For us, probably not critical since we have to do some rewriting anyway. >(But it would be nice to retain for a while as deprecated). > >But what about field names that don't map well to attributes? >I haven't had a chance to reread the past emails but I seem to >recall this was a significant issue. That would imply that .field() >would be needed for those cases anyway. > > I haven't read the above thread extensively, but the issue of field names that don't map well to attributes is significant. For example, users of pytables often have columns with names that are not valid Python names. So, regardless of what solution is the most obvious, there should at least be a way to get as such field names. (pytables users are used to doing that.) Cheers! Andrew From cjw at sympatico.ca Fri Dec 2 08:52:03 2005 From: cjw at sympatico.ca (Colin J. Williams) Date: Fri Dec 2 08:52:03 2005 Subject: [Numpy-discussion] Records in scipy core In-Reply-To: <438F9EEE.904@ee.byu.edu> References: <438F5651.5060104@ee.byu.edu> <438F6F10.3020503@stsci.edu> <438F9EEE.904@ee.byu.edu> Message-ID: <43907B7A.7000905@sympatico.ca> Travis Oliphant wrote: > Christopher Hanley wrote: > >> Hi Travis, >> >> About a year ago (summer 2004) on the numpy distribution list there >> was a lot of discussion of the records interface. I will dig through >> my notes and put together a summary. >> >> > Thanks for the pointers. I had forgotten about that discussion. I > went back and re-read the thread. > > Here's a good link for others to re-read (the end of) this thread: > > http://news.gmane.org/find-root.php?message_id=%3cBD22BAC0.E9EB%25perry%40stsci.edu%3e > > > I think some very good points were made. These points should be > addressed from the context of scipy arrays which now support records > in a very basic way. Because of this, we can support nested records > of records --- but how is this to be presented to the user is still an > open question (i.e. how do you build one...) > > I've finally been converted to believe that the notion of records is > very important because it speaks of how to do the basic (typeless, > mathless) array object that will go into Python correctly If we can > get the general records type done right, then all the other types are > examples of it. > > Thus, I would like to revive discussion of the record object for > inclusion in scipy core. I pretty much agree with the semantics that > Perry described in his final email (is this all implemented in > numarray, yet?), except I would agree with Francesc Alted that a > titles or labels concept should be allowed. > I'm more enthusiastic about code than discussion, so I'm hoping for a > short-lived discussion followed by actual code. I'm ready to do the > implementation this week (I've already borrowed lots of great code > from numarray which makes it easier), but feel free to chime in even > if you read this later. > > In my mind, the discussion about the records array is primarily a > discussion about the records data-type. The way I'm thinking, the > scipy ndarray is a homogeneous collection of the same "thing." The > big change in scipy core is that Numeric used to allow only certain > data types, but now the ndarray can contain an arbitrary "void" data > type. You can also add data-types to scipy core. These data-types > are "almost" full members of the scipy data-type community. The > "almost" is because the N*N casting matrix is not updated (this would > require a re-design of how casting is considered). At some point, > I'd like to fix this wart and make it so that data-types can be added > at will -- I think if we get the record type right, I'll be able to > figure out how to do this. > > We need to add a "record" data-type to scipy. Then, any array can be > of "record" type, and there will be an additional "array scalar" that > is what is returned when selecting a single element from the array. > So, a record array would simply be an array of "records" plus some > extra stuff for dealing with the mapping from field names to actual > segments of the array element (we may decide that this mapping is > general enough that all scipy arrays should have the capability of > assigning names to sub-bytes of its main data-type and means of > accessing those sub-bytes in which case the subclass is unnecessary). > Let me explain further: Right now, the machinery is in place in > scipy_core to get and set in any ndarray (regardless of its data-type) > an arbitrary "field". A "field" in this context is defined as a > sub-section of the basic element making up the array. Generically > the sub-section is defined by an offset and a data-type or a tuple of > a data type and a shape (to allow sub-arrays in a record). What I > understand the user to want is the binding of a name to this generic > sub-section descriptor. > 1) Should we allow that for every scipy ndarray: complex data types > have an obvious binding, would anybody want to name the first two > bytes of their int32 array? I suggest holding off on this one until a > records array is working.... > > 2) Supposing we don't go with number 1, we need to design a record > data type that has this name-binding capability. > > The recarray class in scipy core SVN essentially just does this. > > Question: How important is backwards compatibility with old numarray > specification. In particular, I would go with the .fields access > described by Perry, and eliminate the .field() approach? > I feel that it is not particularly important. Having a good design is the thing to shoot for. > > Thanks for reading and any comments you can make. > > -Travis > I'm not clear as to what the current design objective is and so I'll try to recap and perhaps expand my pieces in the referenced discussion to set out the sort of arrangement I would like to see. We are moving towards having a multi-dimensional array which can hold objects of fixed size and type, the smallest being one byte (although the void would appear to be a collection of no size objects). Most of the need, and thus the focus, is on numeric objects, ranging in size from Int8 to Complex64. The Record is a fixed size object containing fields. Each field has a name, an optional title and data of a fixed type (perhaps including another record instance and maybe arrays of fixed size?). In the example below, AddressRecord and PersonRecord would be sub-classes of Record where the fields are named and, optionally, field titles given. The names would be consistent with Python naming whereas the title could be any Python string. The use of attributes raises the possibility that one could have nested records. For example, suppose one has an address record: addressRecord streetNumber streetName postalCode ... There could then be a personal record: personRecord ... officeAddress homeAddress ... One could address a component as rec.homeAddress.postalCode. Suppose one has a (n, n) array of persons then one could access the data in the following ways: persons[1] all records in the second row persons[:,1] all records in the second column persons[1, 1] return a specific person record persons[1, 1].homeAddress the home address record for a specific person persons[1, 1].homeAddress.postalCode the postal code for a specific person persons.homeAddress.postalCode an (n, n) array containing all postal codes persons.homeAddress.postalCode.title could be 'Zip Code' I see no need to have the attribute 'field' and would like to avoid the use of strings to identify a record component. This does require that fields be named as Python identifiers but is this restriction a killer? Colin W. From oliphant.travis at ieee.org Fri Dec 2 09:14:01 2005 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri Dec 2 09:14:01 2005 Subject: [Numpy-discussion] Records in scipy core In-Reply-To: <43907B7A.7000905@sympatico.ca> References: <438F5651.5060104@ee.byu.edu> <438F6F10.3020503@stsci.edu> <438F9EEE.904@ee.byu.edu> <43907B7A.7000905@sympatico.ca> Message-ID: <4390809F.7050004@ieee.org> > I'm not clear as to what the current design objective is and so I'll > try to recap and perhaps expand my pieces in the referenced discussion > to set out the sort of arrangement I would like to see. I have two objectives: 1) Make the core scipy array object flexible enough to support a very good records sub-class. In other works, I wonder if the core scipy array object could be made flexible enough to be used as a decent record array by itself, without adding much difficulty. In the process, I'm really trying to understand how the data-type of an array should be generally considered. An array object that has this generic perspective on data-type is what should go into Python, I believe. 2) Make a (more) useful records subclass of the ndarray object that is perhaps easier for the end-user to use. Involved with this, of course, is making functions that make it easy to create a records sub-class. > > We are moving towards having a multi-dimensional array which can hold > objects of fixed size and type, the smallest being one byte (although > the void would appear to be a collection of no size objects). Most of > the need, and thus the focus, is on numeric objects, ranging in size > from Int8 to Complex64. > > The Record is a fixed size object containing fields. Each field has a > name, an optional title and data of a fixed type (perhaps including > another record instance and maybe arrays of fixed size?). Right, so the record is really another kind of data-type. The concept of the multidimensional array does not need adjustment, but the concept of what constitutes a data-type may need some fixing up. > > In the example below, AddressRecord and PersonRecord would be > sub-classes of Record where the fields are named and, optionally, > field titles given. The names would be consistent with Python naming > whereas the title could be any Python string. I like the notion of titles and names. I think they are both useful. > > The use of attributes raises the possibility that one could have > nested records. For example, suppose one has an address record: Now, I'm in favor of attribute access. But, nested records is possible without attribute access (it's just extra typing). It's the underlying structure that provides the possibility for nested records (and that's what I'm trying to figure out how to support, generally). If I can support this generally in the basic ndarray object by augmenting the notion of data-type as appropriate, then making a subclass that has the nice syntatic sugar is easy. So, there are two issues really. 1) How to think about the data-type of a general ndarray object in order to support nested records in a straightforward way. 2) What should the syntatic sugar of a record array subclass be... I suppose a third is 3) How much of the syntatic sugar should be applied to all ndarray's? -Travis > I see no need to have the attribute 'field' and would like to avoid > the use of strings to identify a record component. This does require > that fields be named as Python identifiers but is this restriction a > killer? For a record array subclass that may be true. But, as was raised by others in the previous thread, there are problems of "name-space" collision with the methods and attributes of the array that would prevent certain names from being used (and any additions to the methods and attributes of the array would cause incompatibilities with some-people's records). But, at this point, I like the readability of the attribute access approach and could support it. -Travis From oliphant.travis at ieee.org Fri Dec 2 10:37:03 2005 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri Dec 2 10:37:03 2005 Subject: [Numpy-discussion] Records in scipy core In-Reply-To: References: Message-ID: <4390942F.6090302@ieee.org> Perry Greenfield wrote: > > >For us, probably not critical since we have to do some rewriting anyway. >(But it would be nice to retain for a while as deprecated). > > Easy enough to do by defining an actual record array (however, see below). I've been retaining backwards compatibility in other ways while not documenting it. For example, you can actually now pass in strings like 'Int32' for types. >But what about field names that don't map well to attributes? >I haven't had a chance to reread the past emails but I seem to >recall this was a significant issue. That would imply that .field() >would be needed for those cases anyway. > > What I'm referring to as the solution here is a slight modification to what Perry described. In other words, all arrays have the attribute .fields You can set this attribute to a dictionary which will automagically gives field names to any array (this dictionary has ordered lists of 'names', (optionally) 'titles', and "(data-descr, [offset])" lists which defines the mapping. If offset is not given, then the "next-available" offset is assumed. The data-descr is either 1) a data-type or 2) a tuple of (data-type, shape). The data-type is either a defined data-type or alias, or an object with a .fields attribute that provides the same dictionary and an .itemsize attribute that computes the total size of the data-type. You can get this attribute which returns a special fields object (written in Python initially like the flags attribute) that can look up field names like a dictionary, or with attribute access for names that are either 1) acceptable or 2) have a user-provided "python-name" associated with them. Thus, .fields['home address'] would always work but .fields.hmaddr would only work if the user had previously made the association hmaddr -> 'home address' for the data type of this array. Thus 'home address' would be a title but hmaddr would be the name. The records module would simply provide functions for making record arrays and a record data type. Driving my thinking is the concept that the notion of a record array is really a description of the data type of the array (not the array itself). Thus, all the fields information should really just be part of the data type itself. Now, I don't really want to create and register a new data type every time somebody has a new record layout. So, I've been re-thinking the notion of "registering a data-type". It seems to me that while it's O.K. to have a set of pre-defined data types. The notion of data-type ought to be flexible enough to allow the user to define one "on-the-fly". I'm thinking of ways to do this right now. Any suggestions are welcome. -Travis From golux at comcast.net Fri Dec 2 11:03:05 2005 From: golux at comcast.net (Stephen Waterbury) Date: Fri Dec 2 11:03:05 2005 Subject: [Numpy-discussion] Records in scipy core In-Reply-To: <4390942F.6090302@ieee.org> References: <4390942F.6090302@ieee.org> Message-ID: <439099D3.2050803@comcast.net> Travis Oliphant wrote: > So, I've been re-thinking the notion of "registering a data-type". It > seems to me that while it's O.K. to have a set of pre-defined data > types. The notion of data-type ought to be flexible enough to allow the > user to define one "on-the-fly". > I'm thinking of ways to do this right now. Any suggestions are welcome. I'm doing that in an application I'm developing. My objects have an attribute called '_schema' that is an instance of Zope InterfaceClass. An object (read "record" ;) is assigned a _schema when it is instantiated, and all information about its attributes (a.k.a. "fields") is contained in the _schema's Properties (my 'Property' subtypes the Zope interfaces 'Attribute' type, and has a host of (meta-)attributes like 'domain', 'range', 'id', 'name', etc. -- which could easily be extended to include things like 'title', but I use another mechanism for display characteristics, called 'DisplayMap', which can be used to specify the order in which you want the object's properties to appear in a grid, what you want their "display names" to be, etc. ... which are also customizable by the end-user. Let me know if this sounds interesting. Cheers, Steve From cjw at sympatico.ca Fri Dec 2 12:05:01 2005 From: cjw at sympatico.ca (Colin J. Williams) Date: Fri Dec 2 12:05:01 2005 Subject: [Numpy-discussion] Records in scipy core In-Reply-To: <4390942F.6090302@ieee.org> References: <4390942F.6090302@ieee.org> Message-ID: <4390A8C5.3070105@sympatico.ca> Travis Oliphant wrote: > Perry Greenfield wrote: > >> >> >> For us, probably not critical since we have to do some rewriting anyway. >> (But it would be nice to retain for a while as deprecated). >> >> > Easy enough to do by defining an actual record array (however, see > below). I've been retaining backwards compatibility in other ways > while not documenting it. For example, you can actually now pass in > strings like 'Int32' for types. > >> But what about field names that don't map well to attributes? >> I haven't had a chance to reread the past emails but I seem to >> recall this was a significant issue. That would imply that .field() >> would be needed for those cases anyway. >> >> > What I'm referring to as the solution here is a slight modification to > what Perry described. In other words, all arrays have the attribute > > .fields What I suggested in my posting was that there is no need and no benefit from the .fields attribute. The base class Record could be organized so that certain attributes which are used in arrays are not acceptable. For example, one would probably wish to avoid shape, size and the other attributes of the basic array but attributes associated with arrays with numeric types would probably not need to be barred. > > You can set this attribute to a dictionary which will automagically > gives field names to any array (this dictionary has ordered lists of > 'names', (optionally) 'titles', and "(data-descr, [offset])" lists > which defines the mapping. If offset is not given, then the > "next-available" offset is assumed. The data-descr is either 1) a > data-type or 2) a tuple of (data-type, shape). The data-type is > either a defined data-type or alias, or an object with a .fields > attribute that provides the same dictionary and an .itemsize attribute > that computes the total size of the data-type. > I wonder about the need for explicit dictionary operations. Can't this be handled through the class structure? > > You can get this attribute which returns a special fields object > (written in Python initially like the flags attribute) that can look > up field names like a dictionary, or with attribute access for names > that are either 1) acceptable or 2) have a user-provided "python-name" > associated with them. > Thus, > > .fields['home address'] > > would always work > > but > > .fields.hmaddr > > would only work if the user had previously made the association hmaddr > -> 'home address' for the data type of this array. Thus 'home > address' would be a title but hmaddr would be the name. > > The records module would simply provide functions for making record > arrays and a record data type. > Driving my thinking is the concept that the notion of a record array > is really a description of the data type of the array (not the array > itself). Thus, all the fields information should really just be part > of the data type itself. Now, I don't really want to create and > register a new data type every time somebody has a new record layout. > A record array is an array which has a record as its data element, in the same way that an integer array has an integer as its element. I don't understand the notion of registring a data type. Presumably an integer array has a pointer to the appropriate type of integer. Could the record array not have a pointer to the appropriate record type? > So, I've been re-thinking the notion of "registering a data-type". It > seems to me that while it's O.K. to have a set of pre-defined data > types. The notion of data-type ought to be flexible enough to allow > the user to define one "on-the-fly". > I'm thinking of ways to do this right now. Any suggestions are welcome. > The record types would be created "on-the-fly" as the class is instatiated. The array, through the dtype parameter would point to the record type. > > -Travis Colin W. From cjw at sympatico.ca Fri Dec 2 12:12:09 2005 From: cjw at sympatico.ca (Colin J. Williams) Date: Fri Dec 2 12:12:09 2005 Subject: [Numpy-discussion] Records in scipy core In-Reply-To: <439099D3.2050803@comcast.net> References: <4390942F.6090302@ieee.org> <439099D3.2050803@comcast.net> Message-ID: <4390AA7A.3050304@sympatico.ca> Stephen Waterbury wrote: > Travis Oliphant wrote: > >> So, I've been re-thinking the notion of "registering a data-type". >> It seems to me that while it's O.K. to have a set of pre-defined data >> types. The notion of data-type ought to be flexible enough to allow >> the user to define one "on-the-fly". >> I'm thinking of ways to do this right now. Any suggestions are welcome. > > > I'm doing that in an application I'm developing. > My objects have an attribute called '_schema' that is an instance > of Zope InterfaceClass. An object (read "record" ;) is assigned a > _schema > when it is instantiated, and all information about its attributes (a.k.a. > "fields") is contained in the _schema's Properties (my 'Property' > subtypes > the Zope interfaces 'Attribute' type, and has a host of (meta-)attributes > like 'domain', 'range', 'id', 'name', etc. -- which could easily be > extended to include things like 'title', but I use another mechanism > for display characteristics, called 'DisplayMap', which can be used to > specify the order in which you want the object's properties to appear > in a grid, what you want their "display names" to be, etc. ... which are > also customizable by the end-user. > > Let me know if this sounds interesting. > > Cheers, > Steve This is goes further than my suggestion. For arrays, it seems to me that an additional pointer to _schema is not needed as there is a pointer to the data type and the data type can contain the meta data Colin W. From golux at comcast.net Fri Dec 2 12:31:03 2005 From: golux at comcast.net (Stephen Waterbury) Date: Fri Dec 2 12:31:03 2005 Subject: [Numpy-discussion] Records in scipy core In-Reply-To: <4390AA7A.3050304@sympatico.ca> References: <4390942F.6090302@ieee.org> <439099D3.2050803@comcast.net> <4390AA7A.3050304@sympatico.ca> Message-ID: <4390AEE5.3050309@comcast.net> Colin J. Williams wrote: > Stephen Waterbury wrote: >> I'm doing that in an application I'm developing. >> My objects have an attribute called '_schema' that is an instance >> of Zope InterfaceClass. An object (read "record" ;) is assigned a >> _schema >> when it is instantiated, and all information about its attributes (a.k.a. >> "fields") is contained in the _schema's Properties (my 'Property' >> subtypes >> the Zope interfaces 'Attribute' type, and has a host of (meta-)attributes >> like 'domain', 'range', 'id', 'name', etc. -- which could easily be >> extended to include things like 'title', but I use another mechanism >> for display characteristics, called 'DisplayMap', which can be used to >> specify the order in which you want the object's properties to appear >> in a grid, what you want their "display names" to be, etc. ... which are >> also customizable by the end-user. > > This is goes further than my suggestion. For arrays, it seems to me > that an additional pointer to _schema is not needed as there is a > pointer to the data type and the data type can contain the meta data In effect, the _schema *is* the data type in my scenario. It contains all information necessary to describe the type of the object and has references to the types of all the object's attributes (which are called "fields" in record parlance, "Properties" in the world of ontologies, and "Attributes" in Zope Interfaces and UML terminology). Steve From golux at comcast.net Fri Dec 2 12:32:00 2005 From: golux at comcast.net (Stephen Waterbury) Date: Fri Dec 2 12:32:00 2005 Subject: [Numpy-discussion] Records in scipy core In-Reply-To: <4390AA7A.3050304@sympatico.ca> References: <4390942F.6090302@ieee.org> <439099D3.2050803@comcast.net> <4390AA7A.3050304@sympatico.ca> Message-ID: <4390AF0A.8080708@comcast.net> Colin J. Williams wrote: > Stephen Waterbury wrote: >> I'm doing that in an application I'm developing. >> My objects have an attribute called '_schema' that is an instance >> of Zope InterfaceClass. An object (read "record" ;) is assigned a >> _schema >> when it is instantiated, and all information about its attributes (a.k.a. >> "fields") is contained in the _schema's Properties (my 'Property' >> subtypes >> the Zope interfaces 'Attribute' type, and has a host of (meta-)attributes >> like 'domain', 'range', 'id', 'name', etc. -- which could easily be >> extended to include things like 'title', but I use another mechanism >> for display characteristics, called 'DisplayMap', which can be used to >> specify the order in which you want the object's properties to appear >> in a grid, what you want their "display names" to be, etc. ... which are >> also customizable by the end-user. > > This is goes further than my suggestion. For arrays, it seems to me > that an additional pointer to _schema is not needed as there is a > pointer to the data type and the data type can contain the meta data In effect, the _schema *is* the data type in my scenario. It contains all information necessary to describe the type of the object and references to the types of all the object's attributes (which are called "fields" in record parlance, "Properties" in the world of ontologies, and "Attributes" in Zope Interfaces and UML terminology). Steve From antonykummel at yahoo.com Sun Dec 4 05:30:02 2005 From: antonykummel at yahoo.com (Antony Kummel) Date: Sun Dec 4 05:30:02 2005 Subject: [Numpy-discussion] py2exe compliance Message-ID: <20051204132710.57542.qmail@web33903.mail.mud.yahoo.com> Hi! I have used a trick to make py2exe recognize all of numarray's modules, which I suggest be incorporated into the numarray CVS. It can be a real pain to solve this problem the first time you're py2exe-ing a script using numarray and find out that it simply doesn't work. I just added this to numarray's __init__.py: # this is needed for py2exe to be able to include numarray def _give_py2exe_hints(): # pyd's import numarray.libnumarray import numarray.libnumeric import numarray.memory import numarray._bytes import numarray._chararray import numarray._conv import numarray._converter import numarray._ndarray import numarray._numarray import numarray._objectarray import numarray._operator import numarray._sort import numarray._ufunc import numarray._ufuncBool import numarray._ufuncComplex32 import numarray._ufuncComplex64 import numarray._ufuncFloat32 import numarray._ufuncFloat64 import numarray._ufuncInt16 import numarray._ufuncInt32 import numarray._ufuncInt64 import numarray._ufuncInt8 import numarray._ufuncUInt16 import numarray._ufuncUInt32 import numarray._ufuncUInt8 # py's import numarray.arrayprint import numarray.array_persist import numarray.dotblas import numarray.generic import numarray.ieeespecial import numarray.memmap import numarray.memorytest import numarray.numarrayall import numarray.numarraycore import numarray.numarrayext import numarray.numeric import numarray.numerictypes import numarray.numinclude import numarray.numtest import numarray.objects import numarray.readonly import numarray.records import numarray.safethread import numarray.strings import numarray.teacup import numarray.testall import numarray.testdata import numarray.typeconv import numarray.ufunc import numarray._ufuncall # subpackages import numarray.convolve import numarray.convolve.lineshape import numarray.fft import numarray.image import numarray.linear_algebra import numarray.ma import numarray.matrix import numarray.mlab import numarray.nd_image import numarray.random_array --------------------------------- Yahoo! Personals Single? There's someone we'd like you to meet. Lots of someones, actually. Yahoo! Personals -------------- next part -------------- An HTML attachment was scrubbed... URL: From cjw at sympatico.ca Sun Dec 4 13:48:00 2005 From: cjw at sympatico.ca (Colin J. Williams) Date: Sun Dec 4 13:48:00 2005 Subject: [Numpy-discussion] Numeric3.0 - Currently available under scipy/base as version 0.6.1 (Windows) In-Reply-To: <43725D8A.9090307@ee.byu.edu> References: <437209BF.8090607@sympatico.ca> <43725D8A.9090307@ee.byu.edu> Message-ID: <439363D6.3010300@sympatico.ca> Travis Oliphant wrote: > Colin J. Williams wrote: > >> >> I have a package based on subclassing numarray, which is working >> satisfactorily, and am looking at the possibility of transferring the >> package to work under the revised Numeric. >> >> My feeling is that the transfer is probably feasible but that it >> would be premature to work on it at this time. > > > That's unfortunate. The more feedback we get early on about > subclassing, the better. The publication of Python style defs, together with doc strings describing the acceptable arguments for the various __new__ methods would help here. I don't find info on ndarray.__new__. Incidentally, ArrayType seems a better, more meaningful name for the class/type. >> >> One of the problems is the cluttered namespace, through the use of >> "from X import *". This is a style which is deprecated, see page 401 >> of Alex Martelli's /Python in a Nutshell/. > > > You will have to be more specific about what you think is wrong. What > namespace is "cluttered" exactly. Just because use is made of from X > import * in one module does not mean everything is "cluttered". SciPy > Core makes use of the __all__ variables to control what gets imported > and usually only specific functions are imported as necessary. > >> Another problem, at this stage, is that many doc strings are missing >> and that some which exist are a little cryptic. > > > I would submit there are more docstrings then Numeric had. Jump in > and help. The water is fine. > >> >> I would like to take another look when the next win32 binaries are >> available. > > > There has been much improvement since the last beta. I'm trying to > track down some remaining memory leaks before releasing new windows > binaries. The SVN code is always available for check out and it is > quite easy to build. We could always use more build testers to make > sure building is going as smoothly as we believe it is. > I now have 0.6.1, thanks. Now, some documentation on ndarray/ArrayType would be helpful, with the fact that __new__ alone is used instead of the usual __new__ + __init__. Among the parameters is buffer, it seems that this accepts buffers or strings, anything else? Iinformation on call backs (from ufuncs?) would help. For many cases, it would be better to create a Sub instance with Sub(aList) but, if the subclass must respond to callbacks then this may be inhibited. >> >> Some further thoughts on the present state of Numeric3.0 are >> available here . > > > > Most of your comments have more to do with differences between builtin > types and Python classes than anything about scipy. The type-class > chasm has shrunken of late, but there are still remaining issues. > These are Python issues that I believe are of little concern. > > I will comment on your issues that are not related to the above comment: > > > Use of package __init__.py to create namespace. > > If the epydoc and pydoc tools are not respecting the __init__.py code > then I would say they are broken. Using the __init__.py this way > frees the user from having to understand every little detail of the > package structure (which could also change as better organization is > obtained in the future). > > > Use of the from X import Y style > > Please give more support here. Just because one Python user advocates > against it is not sufficient argument. There is an argument to be > made for avoiding attribute lookups by importing the names directly in > this fashion. > > > *Methods vs functions* > > I agree that methods and functions are somewhat redundant. However, > the functions are still needed both to support lists and especially > for back-wards compatibility. Your example using take is odd (perhaps > it's a bug in an old release). I get consistent behavior. One > problem is you define a 1-d array in one case and a 2-d array in > another. Of course the answers will be different. > One difference is that the default axis argument is different. The > old functions have exactly the same default axis argument as they > always did, while the methods have a default of None for the axis > (which means treat the array as flat). > > > Lack of basic information in the doc strings > > Your examples are all class constructors. First of all, there is no > special __init__ method for the ndarray because __new__ takes care of > it. Second of all, the new method does need better documentation. > I'm not sure where to put it, though. The array_new function is > placed in the TypeObject of the array type. The __new__ attribute is > pasted on by PyTypeReady. I'm not sure how to give a docstring as well. > I suspect the proper thing to do is place the information in the > docstring for the Type Object. > > > -Travis > I was wrong in my earlier comment about take but there are some problems as, I hope, the script below illustrates. Colin W. # tTake.py ''' In those cases where there is a duplication, it is suggested that only the method is needed unless the function handles a wider range of data than a method. The function take does more than the method: >>> S.ndarray((1,2)).take([0]) array([[ 1.05176460e-305, 5.63390393e-311]]) >>> S.take([[1, 2]], [0]) array([[1, 2]]) >>> S.take.__doc__ >>> ''' import numarray.numarraycore as N import numarray.numerictypes as NT import scipy as S a= S.ndarray((2, ), dtype= S.int8, buffer= '12') print 'a:', `a` b= S.arange(12, dtype= S.int8).reshape(3, 4) print 'b:', `b` print 'b.take(((0, 0), (2, 2)), 1):', `b.take(((0, 0), (2, 2)), 1)` c= N.arange(12, dtype=NT.Int8, shape=(3, 4)) # this is a bit easier to use print 'c:', `c` print 'c.take(((0, 0), (2, 2)), 1):', `c.take(((0, 0), (2, 2)), 1)` # not quite the same as above print 'b-c:', `b-c` # this needs to raise an exception From vtrandal at yahoo.com Sun Dec 4 16:23:01 2005 From: vtrandal at yahoo.com (Vincent Randal) Date: Sun Dec 4 16:23:01 2005 Subject: [Numpy-discussion] Re: scipy cygwin compilation error References: <437F7D48.9070406@u.washington.edu> Message-ID: Jordan and others, I have the same problem installing scipy in cygwin. Has anyone found a solution? Specifically "python setup.py install" fails with messages like: "scipy/base/src/ufuncobject.c:475: undefined reference" Vincent Randal Longmont, CO From haase at msg.ucsf.edu Mon Dec 5 12:14:04 2005 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Mon Dec 5 12:14:04 2005 Subject: [Numpy-discussion] BUG in nd_image Message-ID: <200512051213.07316.haase@msg.ucsf.edu> Hi, When I call nd_image.rotate with reshape=False I always get "output shape not correct" >>> U.nd.rotate(d[0], 20, axes=(-1, -2), reshape=0, output=d[1], order=1, mode="constant", cval=0.0, prefilter=0) Traceback (most recent call last): File "", line 1, in ? File "/jws30/haase/PrLin/numarray/nd_image/interpolation.py", line 351, in rotate output, order, mode, cval, prefilter) File "/jws30/haase/PrLin/numarray/nd_image/interpolation.py", line 205, in affine_transform output_type) File "/jws30/haase/PrLin/numarray/nd_image/_ni_support.py", line 73, in _get_output raise RuntimeError, "output shape not correct" RuntimeError: output shape not correct I tracked the problem down to "inputShape != outputShape" one being a tuple the output shape being a list. (Pdb) p shape [128, 528] (Pdb) p output.shape (128, 528) (Pdb) p shape != output.shape 1 (Pdb) p shape , output.shape ([128, 528], (128, 528)) (Pdb) I'm using a CVS version around 1.3 ( /ni_interpolation.c/1.17/Fri Apr 22 20:35:27 2005//THEAD) but I took a look at the current CVS and it seems to still be a problem Looks like I'm the only one who doesn't want the reshape ;-) Thanks, Sebastian Haase From oliphant.travis at ieee.org Mon Dec 5 14:08:02 2005 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Dec 5 14:08:02 2005 Subject: [Numpy-discussion] Data type change completed Message-ID: <4394BA01.20501@ieee.org> I've committed the data-type change discussed at the end of last week to the SVN repository. Now the concept of a data type for an array has been replaced with a "data-descriptor". This data-descriptor is flexible enough to handle an arbitrary record specification with fields that include records and arrays or arrays of records. While nesting may not be the best data-layout for a new design, when memory-mapping an arbitrary fixed-record-length file, this capability allows you to handle even the most obsure record file. While the basic core tests pass for me, there may be lurking problems and so testing of the SVN trunk of scipy core will be appreciated. I've bumped up the version number because the C-API has changed (a few new functions and some functions becoming macros). I'd like to make a release of the new version by the end of the week (as soon as Chris Hanley at STSCI and I get records.py working better), so please test. Recently some intel c-compiler tests were failing on a 64-bit platform. It would be nice to figure out why that is happening as well, but I will probably not have time for that this week. Thanks, -Travis From verveer at embl-heidelberg.de Mon Dec 5 14:32:02 2005 From: verveer at embl-heidelberg.de (Peter Verveer) Date: Mon Dec 5 14:32:02 2005 Subject: [Numpy-discussion] BUG in nd_image In-Reply-To: <200512051213.07316.haase@msg.ucsf.edu> References: <200512051213.07316.haase@msg.ucsf.edu> Message-ID: <88B304C0-B5B3-461C-B8A7-FFD74F1BDAF6@embl-heidelberg.de> Works for me with the latest CVS version. On 5 Dec, 2005, at 21:13, Sebastian Haase wrote: > Hi, > When I call nd_image.rotate with reshape=False I always get > "output shape not correct" >>>> U.nd.rotate(d[0], 20, axes=(-1, -2), reshape=0, output=d[1], >>>> order=1, > mode="constant", cval=0.0, prefilter=0) > Traceback (most recent call last): > File "", line 1, in ? > File "/jws30/haase/PrLin/numarray/nd_image/interpolation.py", > line 351, in > rotate > output, order, mode, cval, prefilter) > File "/jws30/haase/PrLin/numarray/nd_image/interpolation.py", > line 205, in > affine_transform > output_type) > File "/jws30/haase/PrLin/numarray/nd_image/_ni_support.py", line > 73, in > _get_output > raise RuntimeError, "output shape not correct" > RuntimeError: output shape not correct > > I tracked the problem down to "inputShape != outputShape" one > being a tuple > the output shape being a list. > (Pdb) p shape > [128, 528] > (Pdb) p output.shape > (128, 528) > (Pdb) p shape != output.shape > 1 > (Pdb) p shape , output.shape > ([128, 528], (128, 528)) > (Pdb) > > I'm using a CVS version around 1.3 ( /ni_interpolation.c/1.17/Fri > Apr 22 > 20:35:27 2005//THEAD) > but I took a look at the current CVS and it seems to still be a > problem > > Looks like I'm the only one who doesn't want the reshape ;-) > > Thanks, > Sebastian Haase > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through > log files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD > SPLUNK! > http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From haase at msg.ucsf.edu Mon Dec 5 14:35:00 2005 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Mon Dec 5 14:35:00 2005 Subject: [Numpy-discussion] numarray: how to make sinc(x) function - "divide by zero" ! Message-ID: <200512051433.55884.haase@msg.ucsf.edu> Hi, The best I could come up with was: def sinc(r): na.Error.pushMode(all="ignore") a = na.where(r, na.divide(na.sin(r),r), 1) na.Error.popMode() return a but I still seem to get a warning... >>> F.sinc(0) Warning: Encountered invalid numeric result(s) in divide 1.0 Thanks, Sebastian Haase From haase at msg.ucsf.edu Mon Dec 5 16:43:03 2005 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Mon Dec 5 16:43:03 2005 Subject: [Numpy-discussion] performance of nd_image affine_transform In-Reply-To: <88B304C0-B5B3-461C-B8A7-FFD74F1BDAF6@embl-heidelberg.de> References: <200512051213.07316.haase@msg.ucsf.edu> <88B304C0-B5B3-461C-B8A7-FFD74F1BDAF6@embl-heidelberg.de> Message-ID: <200512051642.40135.haase@msg.ucsf.edu> Hi, Thanks Peter for checking on the problem I reported in my last posting... Now I'm looking into using nd_image.affine_transform inplace of a fortran routine that I have been using to do this. a) I need to run this on Windows - where I don't have Fortran b) My Fortran routine does only do linear interpolation and I like the idea of experimenting with splines. A and B would of course be good reasons to use nd_image, BUT c) for a 512x512 float32 image my fortran takes about 14ms nd.affine_transform with given output array, prefilter=0 and order=1 takes about 132ms ! With prefilter=1 it takes 138ms; with prefilter=1 and order=3 it takes 279ms !! (order=2,prefilter=1 takes 226ms ; order=3,prefilter=0 222ms) All these are averaged over 10 runs on Linux (P4 2.8GHz) Why is nd_image 10x slower ? (spline order 1 does the same as linear (non-spline) interpolation, right ?) I would call this many (100, 1000 ?) times inside a simplex algorithm which takes already many seconds to complete using the Fortran routine... Thanks, Sebastian Haase On Monday 05 December 2005 14:31, Peter Verveer wrote: > Works for me with the latest CVS version. > > On 5 Dec, 2005, at 21:13, Sebastian Haase wrote: > > Hi, > > When I call nd_image.rotate with reshape=False I always get > > "output shape not correct" > > > >>>> U.nd.rotate(d[0], 20, axes=(-1, -2), reshape=0, output=d[1], > >>>> order=1, > > > > mode="constant", cval=0.0, prefilter=0) > > Traceback (most recent call last): > > File "", line 1, in ? > > File "/jws30/haase/PrLin/numarray/nd_image/interpolation.py", > > line 351, in > > rotate > > output, order, mode, cval, prefilter) > > File "/jws30/haase/PrLin/numarray/nd_image/interpolation.py", > > line 205, in > > affine_transform > > output_type) > > File "/jws30/haase/PrLin/numarray/nd_image/_ni_support.py", line > > 73, in > > _get_output > > raise RuntimeError, "output shape not correct" > > RuntimeError: output shape not correct > > > > I tracked the problem down to "inputShape != outputShape" one > > being a tuple > > the output shape being a list. > > (Pdb) p shape > > [128, 528] > > (Pdb) p output.shape > > (128, 528) > > (Pdb) p shape != output.shape > > 1 > > (Pdb) p shape , output.shape > > ([128, 528], (128, 528)) > > (Pdb) > > > > I'm using a CVS version around 1.3 ( /ni_interpolation.c/1.17/Fri > > Apr 22 > > 20:35:27 2005//THEAD) > > but I took a look at the current CVS and it seems to still be a > > problem > > > > Looks like I'm the only one who doesn't want the reshape ;-) > > > > Thanks, > > Sebastian Haase > > > > > > ------------------------------------------------------- > > This SF.net email is sponsored by: Splunk Inc. Do you grep through > > log files > > for problems? Stop! Download the new AJAX search engine that makes > > searching your log files as easy as surfing the web. DOWNLOAD > > SPLUNK! > > http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log > files for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From cjw at sympatico.ca Mon Dec 5 18:51:03 2005 From: cjw at sympatico.ca (Colin J. Williams) Date: Mon Dec 5 18:51:03 2005 Subject: [Numpy-discussion] Data type change completed In-Reply-To: <4394BA01.20501@ieee.org> References: <4394BA01.20501@ieee.org> Message-ID: <4394FC7B.6040005@sympatico.ca> Travis Oliphant wrote: > > I've committed the data-type change discussed at the end of last week > to the SVN repository. Now the concept of a data type for an array > has been replaced with a "data-descriptor". This data-descriptor is > flexible enough to handle an arbitrary record specification with > fields that include records and arrays or arrays of records. While > nesting may not be the best data-layout for a new design, when > memory-mapping an arbitrary fixed-record-length file, this capability > allows you to handle even the most obsure record file. > Does this mean that the dtype parameter is changed? obscure?? > While the basic core tests pass for me, there may be lurking problems > and so testing of the SVN trunk of scipy core will be appreciated. > I've bumped up the version number because the C-API has changed (a few > new functions and some functions becoming macros). > I'd like to make a release of the new version by the end of the week > (as soon as Chris Hanley at STSCI and I get records.py working > better), so please test. > > Recently some intel c-compiler tests were failing on a 64-bit > platform. It would be nice to figure out why that is happening as > well, but I will probably not have time for that this week. > > Thanks, > > -Travis From oliphant.travis at ieee.org Mon Dec 5 21:44:02 2005 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Dec 5 21:44:02 2005 Subject: ***[Possible UCE]*** Re: [Numpy-discussion] Data type change completed In-Reply-To: <4394FC7B.6040005@sympatico.ca> References: <4394BA01.20501@ieee.org> <4394FC7B.6040005@sympatico.ca> Message-ID: <43952502.1050609@ieee.org> Colin J. Williams wrote: > Travis Oliphant wrote: > >> >> I've committed the data-type change discussed at the end of last week >> to the SVN repository. Now the concept of a data type for an array >> has been replaced with a "data-descriptor". This data-descriptor is >> flexible enough to handle an arbitrary record specification with >> fields that include records and arrays or arrays of records. While >> nesting may not be the best data-layout for a new design, when >> memory-mapping an arbitrary fixed-record-length file, this capability >> allows you to handle even the most obsure record file. >> > Does this mean that the dtype parameter is changed? obscure?? No, it's not changed. The dtype parameter is still used and it is still called the same thing. It's just that what constitutes a data-type has changed significantly. For example now tuples and dictionaries can be used to describe a data-type. These definitions are recursive so that whenever data-type is used it means anything that can be interpreted as a data-type. And I really mean data-descriptor, but data-type is in such common usage that I still use it. Tuple: ======== (fixed-size-data-type, shape) (generic-size-data-type, itemsize) (base-type-data-type, new-type-data-type) Examples: dtype=(int32, (5,5)) --- a 5x5 array of int32 is the description of this item. dtype=(str, 10) --- a length-10 string dtype=(int16, {'real':(int8,0),'imag':(int8,4)} --- a descriptor that acts like an int16 array mathematically (in ufuncs) but has real and imag fields. Dictionary (defaults to a dtypechar == 'V') ========== format1: {"names": list-of-field-names, "formats": list of data-types, "offsets" : list of start-of-the-field "titles" : extra field names } format2 (and how it's stored internally) {key1 : (data-type1, offset1 [, title1]), key2 : (data-type2, offset2 [, title2]), ... keyn : (data-typen, offsetn [, titlen]) } Other objects not already covered: ===================== ???? Right now, it just passes the tp_dict of the typeobject to the dictionary-conversion routine. I'm open for ideas here and will probably have better ideas once the actual record data-type (not data-descriptor but actual subclass of the scipy.void data type) looks like. All of these can be used as the dtype parameter wherever it is taken (of course you can't always do something useful with every data-descriptor). When an ndarray has an associated type descriptor with fields (that's where the field information is stored), then those fields can be accessed using string or unicode keys to the getitem call. Thus, you can do something like this: >>> a = ones((4,3), dtype=(int16, {'real':(int8, 0), 'imag':(int8, 1)})) >>> a['imag'] = 2 >>> a['real'] = 1 >>> a.tostring() '\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02' Note that there are now three distinct but interacting Python objects: 1) the N-dimensional array of a fixed itemsize. 2) a Python object representing one element of the array. 3) the data-descriptor object describing the data-type of the array. These three things were always there under the covers (the PyArray_Descr* has been there since Numeric), and the standard Python types were always filling in for number 2. Now we are just being more explicit about it. Now, all three things are present and accounted for. I'm really quite happy with the resulting infrastructure. I think it will allow some really neat possibilities. I'm thinking the record array subclass will allow attribute-based look-up and register a nice record type for the actual "element" in of the record array. -Travis From cjw at sympatico.ca Tue Dec 6 07:05:06 2005 From: cjw at sympatico.ca (Colin J. Williams) Date: Tue Dec 6 07:05:06 2005 Subject: [Numpy-discussion] Data type change completed In-Reply-To: <43952502.1050609@ieee.org> References: <4394BA01.20501@ieee.org> <4394FC7B.6040005@sympatico.ca> <43952502.1050609@ieee.org> Message-ID: <4395A857.1040601@sympatico.ca> Travis Oliphant wrote: > Colin J. Williams wrote: > >> Travis Oliphant wrote: >> >>> >>> I've committed the data-type change discussed at the end of last >>> week to the SVN repository. Now the concept of a data type for an >>> array has been replaced with a "data-descriptor". This >>> data-descriptor is flexible enough to handle an arbitrary record >>> specification with fields that include records and arrays or arrays >>> of records. While nesting may not be the best data-layout for a new >>> design, when memory-mapping an arbitrary fixed-record-length file, >>> this capability allows you to handle even the most obsure record file. >>> >> Does this mean that the dtype parameter is changed? obscure?? > > > No, it's not changed. The dtype parameter is still used and it is > still called the same thing. It's just that what constitutes a > data-type has changed significantly. > > For example now tuples and dictionaries can be used to describe a > data-type. These definitions are recursive so that whenever data-type > is used it means anything that can be interpreted as a data-type. And > I really mean data-descriptor, but data-type is in such common usage > that I still use it. This would appear to be a good step forward but with all of the immutable types (int8, FloatType, TupleType, etc.) the data is stored in the ArrayType instance (array_data?) whereas, with a dictionary, it would appear to be necessary to store the items outside the array. Is that desirable? Even the tuple can have its content modified, as the example below shows: >>> a= [] >>> b= (a, [2, 3]) >>> b[0] [] >>> b[0]=99 Traceback (most recent call last): File "", line 1, in ? TypeError: object does not support item assignment <<< GOOD >>> b[1][0] 2 >>> b[1][0]=99 >>> b ([], [99, 3]) <<< HERE WE CHANGE THE VALUE OF THE TUPLE >>> > > Tuple: > ======== > (fixed-size-data-type, shape) > (generic-size-data-type, itemsize) > (base-type-data-type, new-type-data-type) > > Examples: > > dtype=(int32, (5,5)) --- a 5x5 array of int32 is the description of > this item. > dtype=(str, 10) --- a length-10 string So dtype now contains both the data type of each element and the shape of the array? This seems a significant change from numarray or Numeric. > dtype=(int16, {'real':(int8,0),'imag':(int8,4)} --- a descriptor that > acts > > like an int16 array mathematically > > (in ufuncs) but has real and imag > > > fields. > > This adds complexity, is there a compensating benefit? Do all of the complex operations apply? > > Dictionary (defaults to a dtypechar == 'V') > ========== Why no clean things up by dropping typechar? These seemed to be one of the warts in numarray, only carried forward for compatibility reasons. Could the compatibility objectives of the project not be achieved, outside the ArrayType object, with a wrapper of some sort? > format1: > > {"names": list-of-field-names, > "formats": list of data-types, > > > "offsets" : list of start-of-the-field > "titles" : extra field names > } > Couldn't the use of records avoid the cumbersome use of keys? > format2 (and how it's stored internally) > > {key1 : (data-type1, offset1 [, title1]), > key2 : (data-type2, offset2 [, title2]), > ... > keyn : (data-typen, offsetn [, titlen]) > } > This is cleaner, but couldn't this inormation be contained within the Record instance? > > Other objects not already covered: > ===================== > ???? > Right now, it just passes the tp_dict of the typeobject to the > dictionary-conversion routine. > I'm open for ideas here and will probably have better ideas once the > actual record data-type (not data-descriptor but actual subclass of > the scipy.void data type) looks like. > > All of these can be used as the dtype parameter wherever it is taken > (of course you can't > always do something useful with every data-descriptor). > When an ndarray has an associated type descriptor with fields (that's > where the field information is > stored), then those fields can be accessed using string or unicode > keys to the getitem call. > I've used ArrayType in place of ndarray (or maybe it should have been ndbigarray?) above as it appear to be more descriptive and fits with the Python convention on class naming. > Thus, you can do something like this: > > >>> a = ones((4,3), dtype=(int16, {'real':(int8, 0), 'imag':(int8, 1)})) > >>> a['imag'] = 2 > >>> a['real'] = 1 > >>> a.tostring() > '\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02' > > Or, one could have something like: class SmallComplex(Record): ..''' This class typically has no instances in user code. ''' ..real= (int8, ) ..imag= (int8) ..def __init__(self): .... ..def __new__(self): .... >>> a = ones((4,3), dtype= SmallComplex) >>> a.imag = 2 >>> a.real = 1 >>> a.tostring() '\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02' > > Note that there are now three distinct but interacting Python objects: > > 1) the N-dimensional array of a fixed itemsize. > 2) a Python object representing one element of the array. > 3) the data-descriptor object describing the data-type of the array. This looks cleaner. Perehaps 2) and 3) could be phrased a little differently: 2) a Python object which is one element of the array. 3) the data-descriptor object describing the data-type of the array element. > > These three things were always there under the covers (the > PyArray_Descr* has been there since Numeric), and the standard Python > types were always filling in for number 2. Now we are just being more > explicit about it. > > Now, all three things are present and accounted for. I'm really quite > happy with the resulting infrastructure. I think it will allow some > really neat possibilities. > > I'm thinking the record array subclass will allow attribute-based > look-up and register a nice record type for the actual "element" in of > the record array. > This is good but the major structure is the array which can have elements of various types such as ComplexType, NoneType, int8 or a variety of user defined immutable records. Colin W. PS My Record sketch above needs a lot more thinking through > > -Travis > > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log > files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From christos.siopis at ulb.ac.be Tue Dec 6 09:06:24 2005 From: christos.siopis at ulb.ac.be (Christos Siopis) Date: Tue Dec 6 09:06:24 2005 Subject: [Numpy-discussion] Indices of min/max element of an array? In-Reply-To: <4395A857.1040601@sympatico.ca> References: <4394BA01.20501@ieee.org> <4394FC7B.6040005@sympatico.ca> <43952502.1050609@ieee.org> <4395A857.1040601@sympatico.ca> Message-ID: <20051206170617.GC5410@ulb.ac.be> Hi all, I was wondering if i'm missing a numpy/numarray function that would return the indices of the minimum/maximum element of an array. Argmin/max can only do this one axis at a time. Alternatively, one can find the index for the flattened array and then play with modulo arithmetic, which gets quickly annoying for > 2 dimensions. Since this is a rather frequent operation, i would think there's already a function for it?... Thanks, Christos From perry at stsci.edu Tue Dec 6 09:16:16 2005 From: perry at stsci.edu (Perry Greenfield) Date: Tue Dec 6 09:16:16 2005 Subject: [Numpy-discussion] Indices of min/max element of an array? In-Reply-To: <20051206170617.GC5410@ulb.ac.be> References: <4394BA01.20501@ieee.org> <4394FC7B.6040005@sympatico.ca> <43952502.1050609@ieee.org> <4395A857.1040601@sympatico.ca> <20051206170617.GC5410@ulb.ac.be> Message-ID: Why not: ind = where(arr == arr.max()) and use the first element of the index array(s) Perry Greenfield On Dec 6, 2005, at 12:06 PM, Christos Siopis wrote: > Hi all, > > I was wondering if i'm missing a numpy/numarray function that would > return the > indices of the minimum/maximum element of an array. Argmin/max can > only do this > one axis at a time. Alternatively, one can find the index for the > flattened array > and then play with modulo arithmetic, which gets quickly annoying for > > 2 > dimensions. Since this is a rather frequent operation, i would think > there's > already a function for it?... > > Thanks, > Christos > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log > files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From haase at msg.ucsf.edu Tue Dec 6 09:25:13 2005 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue Dec 6 09:25:13 2005 Subject: [Numpy-discussion] performance of nd_image affine_transform In-Reply-To: References: <200512051213.07316.haase@msg.ucsf.edu> <200512051642.40135.haase@msg.ucsf.edu> Message-ID: <200512060924.19745.haase@msg.ucsf.edu> (Peter, I hope you meant to reply to the mailing list...) Hi Peter, Does that mean that the performance hit happens "at the lowest level", i.e. there is lot's of extra stuff in the in the innermost c loop that iterates over the output array ? Or is there just so much setup and cleanup code that makes the overall function slow ? How could I profile C extensions anyway - pointers/hints are appreciated ! Thanks for your nd_image (In case that wasn't clear - I like it a lot !) Sebastian Haase On Tuesday 06 December 2005 01:32, you wrote: > Hi Sebastian, > > The interpolation routines are not really optimized very well. The > underlying code is quite generic for spline order, data type, and > arbritatry array dimension and data layout. Your fortran code is > probably much more optimized. It would be nice to have more optimized > code in nd_image, but I do not really have the resources to do that. > > > Hi, > > Thanks Peter for checking on the problem I reported in my last > > posting... > > > > Now I'm looking into using nd_image.affine_transform inplace of a > > fortran > > routine that I have been using to do this. > > a) I need to run this on Windows - where I don't have Fortran > > b) My Fortran routine does only do linear interpolation and I like > > the idea of > > experimenting with splines. > > > > A and B would of course be good reasons to use nd_image, > > BUT c) > > for a 512x512 float32 image my fortran takes about 14ms > > nd.affine_transform with given output array, prefilter=0 and order=1 > > takes about 132ms ! > > With prefilter=1 it takes 138ms; with prefilter=1 and order=3 it > > takes > > 279ms !! (order=2,prefilter=1 takes 226ms ; order=3,prefilter=0 > > 222ms) > > All these are averaged over 10 runs on Linux (P4 2.8GHz) > > > > Why is nd_image 10x slower ? (spline order 1 does the same as linear > > (non-spline) interpolation, right ?) I would call this many (100, > > 1000 ?) > > times inside a simplex algorithm which takes already many seconds > > to complete > > using the Fortran routine... > > > > Thanks, > > Sebastian Haase > > > > On Monday 05 December 2005 14:31, Peter Verveer wrote: > >> Works for me with the latest CVS version. > >> > >> On 5 Dec, 2005, at 21:13, Sebastian Haase wrote: > >>> Hi, > >>> When I call nd_image.rotate with reshape=False I always get > >>> "output shape not correct" > >>> > >>>>>> U.nd.rotate(d[0], 20, axes=(-1, -2), reshape=0, output=d[1], > >>>>>> order=1, > >>> > >>> mode="constant", cval=0.0, prefilter=0) > >>> Traceback (most recent call last): > >>> File "", line 1, in ? > >>> File "/jws30/haase/PrLin/numarray/nd_image/interpolation.py", > >>> line 351, in > >>> rotate > >>> output, order, mode, cval, prefilter) > >>> File "/jws30/haase/PrLin/numarray/nd_image/interpolation.py", > >>> line 205, in > >>> affine_transform > >>> output_type) > >>> File "/jws30/haase/PrLin/numarray/nd_image/_ni_support.py", line > >>> 73, in > >>> _get_output > >>> raise RuntimeError, "output shape not correct" > >>> RuntimeError: output shape not correct > >>> > >>> I tracked the problem down to "inputShape != outputShape" one > >>> being a tuple > >>> the output shape being a list. > >>> (Pdb) p shape > >>> [128, 528] > >>> (Pdb) p output.shape > >>> (128, 528) > >>> (Pdb) p shape != output.shape > >>> 1 > >>> (Pdb) p shape , output.shape > >>> ([128, 528], (128, 528)) > >>> (Pdb) > >>> > >>> I'm using a CVS version around 1.3 ( /ni_interpolation.c/1.17/Fri > >>> Apr 22 > >>> 20:35:27 2005//THEAD) > >>> but I took a look at the current CVS and it seems to still be a > >>> problem > >>> > >>> Looks like I'm the only one who doesn't want the reshape ;-) > >>> > >>> Thanks, > >>> Sebastian Haase > >>> > >>> > >>> ------------------------------------------------------- > >>> This SF.net email is sponsored by: Splunk Inc. Do you grep through > >>> log files > >>> for problems? Stop! Download the new AJAX search engine that makes > >>> searching your log files as easy as surfing the web. DOWNLOAD > >>> SPLUNK! > >>> http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > >>> _______________________________________________ > >>> Numpy-discussion mailing list > >>> Numpy-discussion at lists.sourceforge.net > >>> https://lists.sourceforge.net/lists/listinfo/numpy-discussion > >> > >> ------------------------------------------------------- > >> This SF.net email is sponsored by: Splunk Inc. Do you grep through > >> log > >> files for problems? Stop! Download the new AJAX search engine > >> that makes > >> searching your log files as easy as surfing the web. DOWNLOAD > >> SPLUNK! > >> http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > >> _______________________________________________ > >> Numpy-discussion mailing list > >> Numpy-discussion at lists.sourceforge.net > >> https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > > ------------------------------------------------------- > > This SF.net email is sponsored by: Splunk Inc. Do you grep through > > log files > > for problems? Stop! Download the new AJAX search engine that makes > > searching your log files as easy as surfing the web. DOWNLOAD > > SPLUNK! > > http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From verveer at embl-heidelberg.de Tue Dec 6 09:37:13 2005 From: verveer at embl-heidelberg.de (Peter Verveer) Date: Tue Dec 6 09:37:13 2005 Subject: [Numpy-discussion] performance of nd_image affine_transform In-Reply-To: <200512060924.19745.haase@msg.ucsf.edu> References: <200512051213.07316.haase@msg.ucsf.edu> <200512051642.40135.haase@msg.ucsf.edu> <200512060924.19745.haase@msg.ucsf.edu> Message-ID: <8D594C13-1DF2-4507-9A2A-6C8602B01F75@embl-heidelberg.de> On Dec 6, 2005, at 18:24, Sebastian Haase wrote: > (Peter, I hope you meant to reply to the mailing list...) Yes, I did :-) > Hi Peter, > Does that mean that the performance hit happens "at the lowest > level", i.e. > there is lot's of extra stuff in the in the innermost c loop that > iterates > over the output array ? Yes, I would say so. > Or is there just so much setup and cleanup code that > makes the overall function slow ? Some of that may be a problem too, for instance, the function does malloc and free some temporary data, so that would add if you apply the function repeatedly. > How could I profile C extensions anyway - pointers/hints are > appreciated ! Not sure how you would go about that. I guess optimizing would require a rewrite for the specific cases that you like to optimize. > > Thanks for your nd_image (In case that wasn't clear - I like it a > lot !) Glad to hear that! > > Sebastian Haase > > > > On Tuesday 06 December 2005 01:32, you wrote: >> Hi Sebastian, >> >> The interpolation routines are not really optimized very well. The >> underlying code is quite generic for spline order, data type, and >> arbritatry array dimension and data layout. Your fortran code is >> probably much more optimized. It would be nice to have more optimized >> code in nd_image, but I do not really have the resources to do that. >> >>> Hi, >>> Thanks Peter for checking on the problem I reported in my last >>> posting... >>> >>> Now I'm looking into using nd_image.affine_transform inplace of a >>> fortran >>> routine that I have been using to do this. >>> a) I need to run this on Windows - where I don't have Fortran >>> b) My Fortran routine does only do linear interpolation and I like >>> the idea of >>> experimenting with splines. >>> >>> A and B would of course be good reasons to use nd_image, >>> BUT c) >>> for a 512x512 float32 image my fortran takes about 14ms >>> nd.affine_transform with given output array, prefilter=0 and order=1 >>> takes about 132ms ! >>> With prefilter=1 it takes 138ms; with prefilter=1 and order=3 it >>> takes >>> 279ms !! (order=2,prefilter=1 takes 226ms ; order=3,prefilter=0 >>> 222ms) >>> All these are averaged over 10 runs on Linux (P4 2.8GHz) >>> >>> Why is nd_image 10x slower ? (spline order 1 does the same as linear >>> (non-spline) interpolation, right ?) I would call this many (100, >>> 1000 ?) >>> times inside a simplex algorithm which takes already many seconds >>> to complete >>> using the Fortran routine... >>> >>> Thanks, >>> Sebastian Haase >>> >>> On Monday 05 December 2005 14:31, Peter Verveer wrote: >>>> Works for me with the latest CVS version. >>>> >>>> On 5 Dec, 2005, at 21:13, Sebastian Haase wrote: >>>>> Hi, >>>>> When I call nd_image.rotate with reshape=False I always get >>>>> "output shape not correct" >>>>> >>>>>>>> U.nd.rotate(d[0], 20, axes=(-1, -2), reshape=0, output=d[1], >>>>>>>> order=1, >>>>> >>>>> mode="constant", cval=0.0, prefilter=0) >>>>> Traceback (most recent call last): >>>>> File "", line 1, in ? >>>>> File "/jws30/haase/PrLin/numarray/nd_image/interpolation.py", >>>>> line 351, in >>>>> rotate >>>>> output, order, mode, cval, prefilter) >>>>> File "/jws30/haase/PrLin/numarray/nd_image/interpolation.py", >>>>> line 205, in >>>>> affine_transform >>>>> output_type) >>>>> File "/jws30/haase/PrLin/numarray/nd_image/_ni_support.py", line >>>>> 73, in >>>>> _get_output >>>>> raise RuntimeError, "output shape not correct" >>>>> RuntimeError: output shape not correct >>>>> >>>>> I tracked the problem down to "inputShape != outputShape" one >>>>> being a tuple >>>>> the output shape being a list. >>>>> (Pdb) p shape >>>>> [128, 528] >>>>> (Pdb) p output.shape >>>>> (128, 528) >>>>> (Pdb) p shape != output.shape >>>>> 1 >>>>> (Pdb) p shape , output.shape >>>>> ([128, 528], (128, 528)) >>>>> (Pdb) >>>>> >>>>> I'm using a CVS version around 1.3 ( /ni_interpolation.c/1.17/Fri >>>>> Apr 22 >>>>> 20:35:27 2005//THEAD) >>>>> but I took a look at the current CVS and it seems to still be a >>>>> problem >>>>> >>>>> Looks like I'm the only one who doesn't want the reshape ;-) >>>>> >>>>> Thanks, >>>>> Sebastian Haase >>>>> >>>>> >>>>> ------------------------------------------------------- >>>>> This SF.net email is sponsored by: Splunk Inc. Do you grep through >>>>> log files >>>>> for problems? Stop! Download the new AJAX search engine that >>>>> makes >>>>> searching your log files as easy as surfing the web. DOWNLOAD >>>>> SPLUNK! >>>>> http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click >>>>> _______________________________________________ >>>>> Numpy-discussion mailing list >>>>> Numpy-discussion at lists.sourceforge.net >>>>> https://lists.sourceforge.net/lists/listinfo/numpy-discussion >>>> >>>> ------------------------------------------------------- >>>> This SF.net email is sponsored by: Splunk Inc. Do you grep through >>>> log >>>> files for problems? Stop! Download the new AJAX search engine >>>> that makes >>>> searching your log files as easy as surfing the web. DOWNLOAD >>>> SPLUNK! >>>> http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click >>>> _______________________________________________ >>>> Numpy-discussion mailing list >>>> Numpy-discussion at lists.sourceforge.net >>>> https://lists.sourceforge.net/lists/listinfo/numpy-discussion >>> >>> ------------------------------------------------------- >>> This SF.net email is sponsored by: Splunk Inc. Do you grep through >>> log files >>> for problems? Stop! Download the new AJAX search engine that makes >>> searching your log files as easy as surfing the web. DOWNLOAD >>> SPLUNK! >>> http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click >>> _______________________________________________ >>> Numpy-discussion mailing list >>> Numpy-discussion at lists.sourceforge.net >>> https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through > log files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD > SPLUNK! > http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From jmiller at stsci.edu Tue Dec 6 11:11:13 2005 From: jmiller at stsci.edu (Todd Miller) Date: Tue Dec 6 11:11:13 2005 Subject: [Numpy-discussion] ANN: numarray-1.5.0 Message-ID: <4395E1D8.2090709@stsci.edu> Numarray is an array processing package designed to efficiently manipulate large multi-dimensional arrays. Numarray is modelled after Numeric and features c-code generated from python template scripts, the capacity to operate directly on arrays in files, arrays of heterogeneous records, string arrays, and in-place operation on memory mapped files. ========================================================================= Release Notes for numarray-1.5.0 I. ENHANCEMENTS 1. Implementation of scipy newcore array interface and __array_struct__ support (supplying and consuming) for numarray. This should facilitate better interoperability with scipy newcore and Numeric. This protocol is documented here: http://numeric.scipy.org/array_interface.html but also in numarray's source code in Include/numarray/arrayif.h. Thanks to everyone who gave feedback on 1.4.1 and particularly to Francesc Altet for his diligent testing (and doctest) of Numeric<->numarray data interchange. II. BUGS FIXED / CLOSED 1365121 Definition of PyObject* operator conflicts with C++ 1350954 nummacro.h:27: error: expected type-specifier before ';' tok 1346470 Please document more of what byteswap and byteswapped do 1364215 Cannot combine array and masked array (e.g. via divide) 1363723 my_array = +my_other_array uncovers a bug 1364811 Infinite loop converting empty CharArrays to lists 1364815 Strange behaviour when creating empty Int32 arrays 1340983 Fix __get_array_data__ 1346480 byteswapped doesn't match doc string 1346425 Please document if copy "fixes" byte order 1346426 Please document what array does with numarray arrays See http://sourceforge.net/tracker/?atid=450446&group_id=1369&func=browse for more details. III. CAUTIONS This release should be binary compatible with numarray-1.3.x. New extensions compiled against 1.5.0 may not work with prior versions of numarray. WHERE ----------- Numarray-1.5.0 windows executable installers, source code, and manual is here: http://sourceforge.net/project/showfiles.php?group_id=1369 Numarray is hosted by Source Forge in the same project which hosts Numeric: http://sourceforge.net/projects/numpy/ The web page for Numarray information is at: http://www.stsci.edu/resources/software_hardware/numarray Trackers for Numarray Bugs, Feature Requests, Support, and Patches are at the Source Forge project for NumPy at: http://sourceforge.net/tracker/?group_id=1369 REQUIREMENTS ------------------------------ numarray-1.5.0 requires Python 2.2.3 or greater. AUTHORS, LICENSE ------------------------------ Numarray was written by Perry Greenfield, Rick White, Todd Miller, JC Hsu, Paul Barrett, Phil Hodge at the Space Telescope Science Institute. We'd like to acknowledge the assitance of Francesc Alted, Paul Dubois, Sebastian Haase, Chuck Harris, Tim Hochberg, Nadav Horesh, Edward C. Jones, Eric Jones, Jochen Kuepper, Travis Oliphant, Pearu Peterson, Peter Verveer, Colin Williams, Rory Yorke, and everyone else who has contributed with comments and feedback. Numarray is made available under a BSD-style License. See LICENSE.txt in the source distribution for details. -- Todd Miller jmiller at stsci.edu From christos.siopis at ulb.ac.be Tue Dec 6 12:29:04 2005 From: christos.siopis at ulb.ac.be (Christos Siopis) Date: Tue Dec 6 12:29:04 2005 Subject: [Numpy-discussion] Indices of min/max element of an array? In-Reply-To: References: <4394BA01.20501@ieee.org> <4394FC7B.6040005@sympatico.ca> <43952502.1050609@ieee.org> <4395A857.1040601@sympatico.ca> <20051206170617.GC5410@ulb.ac.be> Message-ID: <20051206202905.GO5410@ulb.ac.be> On Tue, Dec 06, 2005 at 12:17:14PM -0500, Perry Greenfield wrote: > Why not: > > ind = where(arr == arr.max()) > > and use the first element of the index array(s) Thanks, that worked nice! I was stuck with the old Numeric definition of where(). Could it perhaps be made more obvious in numarray's where() documentation that it can also be called with one parameter only? (I.e., put this information in the section header, not mid-way in the description). I'm looking at the April 20, 2005 documentation release. Would it be safe to assume that the numarray definition of where() will be the one propagated into scipy.core? As a side note, i am wondering if there is a semantic asymmetry in using min() and max() to signify the min/max element in the entire array while argmin() and argmax() signify the min/max element along each axis. At the same time, and as far as i can tell, there is no min()/max() method to provide the min/max element along each axis, and there is no method to do the equivalent of "argmin/max_all()", as implemented by where(arr == arr.min/max()). Apologies if this has been discussed before. Thank you, Christos From oliphant at ee.byu.edu Tue Dec 6 12:33:00 2005 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Tue Dec 6 12:33:00 2005 Subject: [Numpy-discussion] Indices of min/max element of an array? In-Reply-To: <20051206202905.GO5410@ulb.ac.be> References: <4394BA01.20501@ieee.org> <4394FC7B.6040005@sympatico.ca> <43952502.1050609@ieee.org> <4395A857.1040601@sympatico.ca> <20051206170617.GC5410@ulb.ac.be> <20051206202905.GO5410@ulb.ac.be> Message-ID: <4395F526.6030902@ee.byu.edu> Christos Siopis wrote: >On Tue, Dec 06, 2005 at 12:17:14PM -0500, Perry Greenfield wrote: > > > >>Why not: >> >>ind = where(arr == arr.max()) >> >>and use the first element of the index array(s) >> >> > > >Would it be safe to assume that the numarray definition of where() will be the one >propagated into scipy.core? > > Yes, it is already there. -Travis From aisaac at american.edu Tue Dec 6 15:34:01 2005 From: aisaac at american.edu (Alan G Isaac) Date: Tue Dec 6 15:34:01 2005 Subject: [Numpy-discussion] Indices of min/max element of an array? In-Reply-To: <20051206202905.GO5410@ulb.ac.be> References: <4394BA01.20501@ieee.org> <4394FC7B.6040005@sympatico.ca> <43952502.1050609@ieee.org> <4395A857.1040601@sympatico.ca> <20051206170617.GC5410@ulb.ac.be> <20051206202905.GO5410@ulb.ac.be> Message-ID: On Tue, 6 Dec 2005, Christos Siopis apparently wrote: > As a side note, i am wondering if there is a semantic > asymmetry in using min() and max() to signify the min/max > element in the entire array while argmin() and argmax() > signify the min/max element along each axis. SciPy arrays function as "expected" in this sense: >>> import scipy >>> x=scipy.array([[1,2],[3,4]]) >>> x.max() 4 >>> x.argmax() 3 Note that, as I understand, argmax gives the index from x.flat Also, the scipy ufuncs max and argmax have the symmetry you seek, if I understand correctly. hth, Alan Isaac From arnd.baecker at web.de Wed Dec 7 02:08:01 2005 From: arnd.baecker at web.de (Arnd Baecker) Date: Wed Dec 7 02:08:01 2005 Subject: [Numpy-discussion] ACML and dotblas Message-ID: Hi, has anyone figured out how to use dotblas with the AMD Core Math Library (ACML, http://developer.amd.com/acml.aspx)? Darren Dale asked this a month ago here http://aspn.activestate.com/ASPN/Mail/Message/numpy-discussion/2886285 but I could not find any answer. Many thanx, Arnd From dd55 at cornell.edu Wed Dec 7 03:31:06 2005 From: dd55 at cornell.edu (Darren Dale) Date: Wed Dec 7 03:31:06 2005 Subject: [Numpy-discussion] ACML and dotblas In-Reply-To: References: Message-ID: <200512070630.06109.dd55@cornell.edu> On Wednesday 07 December 2005 5:07 am, Arnd Baecker wrote: > Hi, > > has anyone figured out how to use dotblas > with the AMD Core Math Library (ACML, http://developer.amd.com/acml.aspx)? > Darren Dale asked this a month ago here > http://aspn.activestate.com/ASPN/Mail/Message/numpy-discussion/2886285 > but I could not find any answer. I don't have any progress to report, sorry. Darren From service at lntl.ebay.com Fri Dec 9 11:58:07 2005 From: service at lntl.ebay.com (eBay) Date: Fri Dec 9 11:58:07 2005 Subject: [Numpy-discussion] Important eBay! Account Information. Message-ID: An HTML attachment was scrubbed... URL: From christos.siopis at ulb.ac.be Sat Dec 10 10:32:12 2005 From: christos.siopis at ulb.ac.be (Christos Siopis) Date: Sat Dec 10 10:32:12 2005 Subject: [Numpy-discussion] Indices of min/max element of an array? In-Reply-To: References: <4394BA01.20501@ieee.org> <4394FC7B.6040005@sympatico.ca> <43952502.1050609@ieee.org> <4395A857.1040601@sympatico.ca> <20051206170617.GC5410@ulb.ac.be> Message-ID: <20051210183306.GD8898@ulb.ac.be> On Tue, Dec 06, 2005 at 06:37:11PM -0500, Alan G Isaac wrote: > On Tue, 6 Dec 2005, Christos Siopis apparently wrote: > > > As a side note, i am wondering if there is a semantic > > asymmetry in using min() and max() to signify the min/max > > element in the entire array while argmin() and argmax() > > signify the min/max element along each axis. > > SciPy arrays function as "expected" in this sense: > >>> import scipy > >>> x=scipy.array([[1,2],[3,4]]) > >>> x.max() > 4 > >>> x.argmax() > 3 > > Note that, as I understand, argmax gives the index from x.flat Thanks for the note. I do not have SciPy installed to test this, and i am not sure which version of SciPy you are using. I believe in the (remote?) past, SciPy was using Numeric as a core, but using the latest Numeric available on Gentoo AMD64 i obtain different results: Python 2.4.2 (#1, Nov 19 2005, 12:30:12) [GCC 3.4.4 (Gentoo 3.4.4-r1, ssp-3.4.4-1.0, pie-8.7.8)] on linux2 >>> import Numeric >>> Numeric.__version__ '23.7' >>> x=Numeric.array([[1,2],[3,4]]) >>> x.max() Traceback (most recent call last): File "", line 1, in ? AttributeError: max >>> x.argmax() Traceback (most recent call last): File "", line 1, in ? AttributeError: argmax i.e., these methods are not supported for Numeric arrays. The max() function does not exist either: >>> Numeric.max(x) Traceback (most recent call last): File "", line 1, in ? AttributeError: 'module' object has no attribute 'max' (i think one would have to use MA.Maximum() to find the global array maximum back at those days...). And: >>> Numeric.argmax(x) array([1, 1]) i.e., it does not flatten the array. --- Now, using numarray: >>> import numarray >>> numarray.__version__ '1.3.1' >>> x=numarray.array([[1,2],[3,4]]) >>> x.max() 4 >>> x.argmax() array([1, 1]) i.e., these methods for the array object now do exist, but the behavior of argmax() is not the same as in your SciPy. My conjencture is that your SciPy uses some "intermediate" version between the old Numeric and the current scipy.core which, as i understand from what Travis said, supports the above numarray behavior. > Also, the scipy ufuncs max and argmax have the symmetry you > seek, if I understand correctly. With so many versions of things floating around, i think it's hard to tell what has what any more... One more reason to look forward to the outcome of Travis' work, and hope that things (or at least the API) will stabilize... Thanks, Christos From cjw at sympatico.ca Sun Dec 11 06:53:01 2005 From: cjw at sympatico.ca (Colin J. Williams) Date: Sun Dec 11 06:53:01 2005 Subject: [Numpy-discussion] Re: Scipy Core (Was Numeric 3.0) In-Reply-To: <439B297E.3030808@ieee.org> References: <439AEA74.3090407@sympatico.ca> <439B297E.3030808@ieee.org> Message-ID: <439C3D1A.1010502@sympatico.ca> Travis, This is intended to restore the context of your response. TO is Travis Oliphant, CW is Colin Williams TO: The dtype parameter is still used and it is still called the same thing. It's just that what constitutes a data-type has changed significantly. For example now tuples and dictionaries can be used to describe a data-type. These definitions are recursive so that whenever data-type is used it means anything that can be interpreted as a data-type. And I really mean data-descriptor, but data-type is in such common usage that I still use it. CW: This would appear to be a good step forward but with all of the immutable types (int8, FloatType, TupleType, etc.) the data is stored in the ArrayType instance (array_data?) whereas, with a dictionary, it would appear to be necessary to store the items outside the array. Is that desirable? TO: I don't even understand what you are questioning here. Perhaps you misunderstood me. Items are always stored in the array. Even records "items" are stored in the array. The descriptor just allows you to describe what kind of data and how it is stored in the array. Response: Yes, I had assumed that int8 and FloatType are equally data-descriptors, i.e. objects which describe the elements of a numeric array. Wrongly, I assumed that TupleType or DictType are also a data descriptors. Mea culpa. On another subject, it would help, for those of us who are not afficionados of C, if you provided the equivalent Python def statement for those function implemented in C, especially the ArrayType's __new__ method. perhaps in the docstrings. Colin W. Travis Oliphant wrote: > Colin J. Williams wrote: > >> >> This would appear to be a good step forward but with all of the >> immutable types (int8, FloatType, TupleType, etc.) the data is stored >> in the ArrayType instance (array_data?) whereas, with a dictionary, >> it would appear to be necessary to store the items outside the >> array. Is that desirable? > > > I don't even understand what you are questioning here. Perhaps you > misunderstood me. Items are always stored in the array. Even records > "items" are stored in the array. The descriptor just allows you to > describe what kind of data and how it is stored in the array. > >> >> Even the tuple can have its content modified, as the example below >> shows: >> > I don't understand what you mean to show by the > tuple-content-modifying example. > >>> dtype=(int32, (5,5)) --- a 5x5 array of int32 is the description >>> of this item. >>> dtype=(str, 10) --- a length-10 string >> >> >> >> So dtype now contains both the data type of each element and the >> shape of the array? This seems a significant change from numarray or >> Numeric. > > > No, no. Standard usage is the same. In normal usage you would not > create an array this way. You could, of course, but it's not the > documented procedure. The reason for this descriptor is to allow > you to have a field-element that itself is an array of items. > >> >> >>> dtype=(int16, {'real':(int8,0),'imag':(int8,4)} --- a descriptor >>> that acts >>> >>> like an int16 array mathematically >>> >>> (in ufuncs) but has real and imag >>> >>> >>> fields. >>> >>> >> This adds complexity, is there a compensating benefit? Do all of the >> complex operations apply? > > > I'm only showing what is possible and that the notion of data-type > descriptor is complete. > >> >> Why not clean things up by dropping typechar? These seemed to be one >> of the warts in numarray, only carried forward for >> compatibility reasons. Could the compatibility objectives of the >> project not be achieved, outside the ArrayType object, with a wrapper >> of some sort? > > > Too hard to do at this point. Too much code uses the characters. I > also don't mind the characters so much (the struct module and the > Python array module use them). > >> >> Couldn't the use of records avoid the cumbersome use of keys? > > > Yes. But, this is how you specify fields generally. > >> >>> format2 (and how it's stored internally) >>> >>> {key1 : (data-type1, offset1 [, title1]), >>> key2 : (data-type2, offset2 [, title2]), >>> ... >>> keyn : (data-typen, offsetn [, titlen]) >>> } >>> >> This is cleaner, but couldn't this inormation be contained within the >> Record instance? > > > > Yes, but I am describing a general concept of data-type description > not just something applicable to records. > >>> Thus, you can do something like this: >>> >>> >>> a = ones((4,3), dtype=(int16, {'real':(int8, 0), 'imag':(int8, >>> 1)})) >>> >>> a['imag'] = 2 >>> >>> a['real'] = 1 >>> >>> a.tostring() >>> '\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02' >>> >>> >> Or, one could have something like: >> class SmallComplex(Record): >> ..''' This class typically has no instances in user code. ''' >> ..real= (int8, ) >> ..imag= (int8) >> ..def __init__(self): >> .... >> ..def __new__(self): >> .... >> >> >>> a = ones((4,3), dtype= SmallComplex) >> >>> a.imag = 2 >> >>> a.real = 1 >> >>> a.tostring() >> '\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02\x01\x02' >> > > > Yes, something like this should be possible, though I have not fleshed > out the user-interface at all. I've just been working on the basic > structure that would support such things. Do: > > class small_complex(record): > dtypedescr = {'r':(int8,0),'i':(int8,1)} > > a = ndrecarray((4,3), formats=small_complex) > a.r = 1 > a.i = 2 > a.tostring() > > and your example would work (with current SVN...) > > The ndrecarray subclass allows the attribute-to-field conversion, > regular arrays do not. > > -Travis > > > > > > > > > > -Travis > > From dfg at 456.com Sun Dec 11 10:14:04 2005 From: dfg at 456.com (ghkkuk) Date: Sun Dec 11 10:14:04 2005 Subject: [Numpy-discussion] =?GB2312?B?obZbwMrPzMa9XSDW0Ln6xvPStdW9wtTNu86n0dDM1rvhIKG3?= Message-ID: An HTML attachment was scrubbed... URL: From oliphant.travis at ieee.org Sun Dec 11 12:54:04 2005 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sun Dec 11 12:54:04 2005 Subject: [Numpy-discussion] Re: [SciPy-user] transition problems In-Reply-To: <439C4AAB.70308@hoc.net> References: <439C4AAB.70308@hoc.net> Message-ID: <439C91DB.1060505@ieee.org> Christian Kristukat wrote: >Hi, >I noticed that using the new core together with older packages that are based on >numeric, like the plotting module from wxPython, matplotlib, grace_np.py, etc. >is problematic since numeric seems to be unable to recognize new core arrays as >equivalent to the numeric ones. > If you get a recent version (>=24.0) of Numeric then it shouldn't be a problem. One of the purposes of the array interface is to ease the transition, but only recent versions of Numeric use it properly. If there are problems with recent versions of Numeric and the array interface then please post them. Older versions of Numeric would need to use tostring and fromstring (better than tolist...) >In some cases it is sufficient to add a >.tolist() to array parameters but that is not really satisfactory. >Will a final version of new core have solved these incompatibilities? > > Unfortunately, I don't know how to solve incompatibilities with older versions of Numeric. All third-party libraries should either start using only the array interface or switch to scipy_core. I've noticed there is some confusion still circulating on the net that the new arrayobject in scipy_core is going to get into Python at some point, and some people are holding off making any changes until that happens. I am probably the source of that confusion since at one time I did have that goal. However, early on (in March) after discussions with Paul, Perry, and Guido we decided that trying to force an arrayobject into Python would place us on a release cycle that would be constraining. So, don't wait to try out scipy_core because there is nothing going into Python any time soon that is even as capable as Numeric. There are plans for getting a very simple arrayobject into Python (largely to enshrine the array interface), but we really need help moving that forward. With the recent changes to scipy_core, I think we have a very good design for a generic array object with associated data-descriptor that could go into Python. For python itself, I would only say that descriptors should only be written for Object, integer, float, and complex-float arrays. A PEP has been started and sits at http://svn.scipy.org/svn/PEP waiting for more people to help push it forward. Best, -Travis From oliphant.travis at ieee.org Sun Dec 11 20:38:01 2005 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Sun Dec 11 20:38:01 2005 Subject: [Numpy-discussion] Re: [SciPy-dev] New release on Monday In-Reply-To: <767A9640-49E7-42B8-BE28-6B4001D93F38@ftw.at> References: <439BE8F4.3070302@ieee.org> <767A9640-49E7-42B8-BE28-6B4001D93F38@ftw.at> Message-ID: <439CFE79.6050303@ieee.org> Ed Schofield wrote: >On 11/12/2005, at 8:53 AM, Travis Oliphant wrote: > > >Is there any chance of including type checking for this release for >unsafe in-place operations? Several people supported the idea in the >thread [In-place operators and casting] a few weeks ago. In summary, >the existing behaviour would still be achievable with an explicit cast: > >>> my_int_array += my_float_array.astype(int) > > Actually, this would not do the same thing because it would force the entire float array into memory as an integer array and then add it. The current behavior creates only bufsize memories for this kind of copying. So, for large-array performance this approach would be worse, which is a big reason I'm not really supportive of a switch. I'm also hesitant to change this because it is the default behavior of numarray, so I'd like to receive more feedback from members of that community who are coming over to scipy_core before doing something different. I think, however, Numeric raises an error in this circumstance. So, I would advise changing this behavior in the current release, but I don't see this issue as closed. While I would never support changing the data-type of the array when using an inplace operator, I could see the logic in raising an error when the type cannot be cast safely. -Travis From oliphant.travis at ieee.org Mon Dec 12 12:49:06 2005 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Dec 12 12:49:06 2005 Subject: [Numpy-discussion] Thoughts on making it easier for numarray users to transition to scipy_core Message-ID: <439DE22B.9070400@ieee.org> A few days ago I played with nd_image and was able to make it compile for scipy_core. In the process, I had some ideas for making the transition to scipy_core easier for numarray users: 1) It would be nice if there was someway to document on the Python level any needed changes. In the process, we might find things that need to be added to scipy_core. I would like to have some kind of program for automatically making most of those needed changes before a 1.0 release of scipy_core. 2) On the C-API side, my experience with nd_image showed me that quite a few of the numarray C-API calls can be written as macros while some will need to be additional functions. I think we could easily write a numcompatmodule.c and associated numcompat.h file so that users needing the numarray C-API could include the numcompat.h file and then the import_libnumarray() command would load the numcompatmodule.c with it's compatibility functions. In this way, the transition for C-API users could be as easy as changing the include file to numcompat.h? -Travis From verveer at embl-heidelberg.de Mon Dec 12 13:36:02 2005 From: verveer at embl-heidelberg.de (Peter Verveer) Date: Mon Dec 12 13:36:02 2005 Subject: [Numpy-discussion] Thoughts on making it easier for numarray users to transition to scipy_core In-Reply-To: <439DE22B.9070400@ieee.org> References: <439DE22B.9070400@ieee.org> Message-ID: <79153027-85E8-4DB3-A69F-2B1398CB2043@embl-heidelberg.de> Hi Travis, I am curious about your experience with moving nd_image to scipy_core. I have not had the time to look at scipy_core, so I have not tried to do it myself, although it was on my very-long-term list of things to do. Did it require a lot of changes and did you get it to pass the unit tests? Cheers, Peter > A few days ago I played with nd_image and was able to make it > compile for scipy_core. In the process, I had some ideas for > making the transition to scipy_core easier for numarray users: > > 1) It would be nice if there was someway to document on the Python > level any needed changes. In the process, we might find things > that need to be added to scipy_core. I would like to have some > kind of program for automatically making most of those needed > changes before a 1.0 release of scipy_core. > > 2) On the C-API side, my experience with nd_image showed me that > quite a few of the numarray C-API calls can be written as macros > while some will need to be additional functions. I think we could > easily write a numcompatmodule.c and associated numcompat.h file so > that users needing the numarray C-API could include the numcompat.h > file and then the import_libnumarray() command would load the > numcompatmodule.c with it's compatibility functions. > > In this way, the transition for C-API users could be as easy as > changing the include file to numcompat.h? > -Travis > > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through > log files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD > SPLUNK! > http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion From oliphant.travis at ieee.org Mon Dec 12 15:36:04 2005 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Dec 12 15:36:04 2005 Subject: [SciPy-dev] [Numpy-discussion] Thoughts on making it easier for numarray users to transition to scipy_core In-Reply-To: <367c3c211512e3bac5ff4aabe127721d@stsci.edu> References: <439DE22B.9070400@ieee.org> <367c3c211512e3bac5ff4aabe127721d@stsci.edu> Message-ID: <439DFE67.2020600@ieee.org> >That certainly would be nice. We are starting to look migrating some >Python code. It may be a little while >(a couple months?) before we can start tackling migrating some of the C >extension code so we won't be exercising that right away (but maybe >someone else can). > > I think I will have something basic in-place by then. We can add needed API calls as extensions get ported. On a related note, where should the Packages (like nd_image) from numarray go? Should they all go into scipy core, full scipy, or be separately downloadable as scipy_core addons? -Travis From verveer at embl-heidelberg.de Tue Dec 13 00:27:02 2005 From: verveer at embl-heidelberg.de (Peter Verveer) Date: Tue Dec 13 00:27:02 2005 Subject: [SciPy-dev] [Numpy-discussion] Thoughts on making it easier for numarray users to transition to scipy_core In-Reply-To: <439DFE67.2020600@ieee.org> References: <439DE22B.9070400@ieee.org> <367c3c211512e3bac5ff4aabe127721d@stsci.edu> <439DFE67.2020600@ieee.org> Message-ID: On Dec 12, 2005, at 23:49, Travis Oliphant wrote: > >> That certainly would be nice. We are starting to look migrating >> some Python code. It may be a little while >> (a couple months?) before we can start tackling migrating some of >> the C extension code so we won't be exercising that right away >> (but maybe someone else can). >> > I think I will have something basic in-place by then. We can add > needed API calls as extensions get ported. > > On a related note, where should the Packages (like nd_image) from > numarray go? Should they all go into scipy core, full scipy, or be > separately downloadable as scipy_core addons? Regarding nd_image, I would guess that it should be an addon, or go at some point in scipy. However, even though I am the author, I am most likely not going to support it outside numarray. I do realize that at some point numarray may be retired, but I have not decided yet if I will be using/supporting nd_image beyond that point. It would be nice to have a common code-base with the numarray version so that all can profit from any bug-fixes that I am still doing in the numarray version. peter From edcjones at comcast.net Tue Dec 13 20:28:13 2005 From: edcjones at comcast.net (Edward C. Jones) Date: Tue Dec 13 20:28:13 2005 Subject: [Numpy-discussion] numarray bug: random_array/dtest.py fails on 64 bit machine Message-ID: <439F9F3D.1080508@comcast.net> I have an Athlon 64 chip. I use Debian unstable but I compile all Python related programs myself. They go into /usr/local. I have Python 2.4.2 and numarray 1.5.0. I got the following error from testall.test(): File "/usr/local/lib/python2.4/site-packages/numarray/random_array/dtest.py", line 57, in numarray.random_array.dtest Failed example: x = multivariate_normal(num.array([10,20]), num.array(([1,2],[2,4]))); x Expected: array([ 9.95170432, 19.90340867]) Got: array([ 9.95170432, 19.90340866]) From verveer at embl-heidelberg.de Wed Dec 14 01:29:04 2005 From: verveer at embl-heidelberg.de (Peter Verveer) Date: Wed Dec 14 01:29:04 2005 Subject: [SciPy-dev] [Numpy-discussion] Thoughts on making it easier for numarray users to transition to scipy_core In-Reply-To: <439F4DBE.5040601@ieee.org> References: <439DE22B.9070400@ieee.org> <367c3c211512e3bac5ff4aabe127721d@stsci.edu> <439DFE67.2020600@ieee.org> <439F4DBE.5040601@ieee.org> Message-ID: On 13 Dec, 2005, at 23:39, Travis Oliphant wrote: > >> However, even though I am the author, I am most likely not going >> to support it outside numarray. I do realize that at some point >> numarray may be retired, but I have not decided yet if I will be >> using/supporting nd_image beyond that point. It would be nice to >> have a common code-base with the numarray version so that all can >> profit from any bug-fixes that I am still doing in the numarray >> version. > > > So, do I take this to mean that you are not enthusiastic about a > merger between the Numeric and numarray communities? I would love > to hear why you feel this way. > From my perspective, it just makes unification that much harder to > have important third-party authors make these kinds of statements, > so I'm curious as to why you don't want to convert to a common code- > base. Because others may have the same feelings, it would help all > of us if we could address them. Don't worry about offending me or > others in the scipy world. We have pretty thick skins by this point. I think I should clarify what I intent to do with nd_image in the future. It affects current nd_image users and the numarray package also, so I am also sending this to the numpy list. First of all, I should stress that I am quite happy about a possible merger. It is quite likely that I will be using python for numerical work in the future, and a merger of numarray and Numeric into a package of equal or better quality will only make my life easier. Currently I am the only software developer within a scientific laboratory, but software development is not my main focus. I used python with numarray for virtual all of my numerical work, and nd_image was sort of a side effect of that. Right now I do not have the time to make a switch to a new package, nor can I really further expand nd_image. It would be nice to try moving to scipy_core now, which would help the community too I guess, but I just cannot affort it at the moment, so I am sticking with numarray for the time being. My situation will most likely change next year, since our group will move and expand a lot, and I will hopefully be able to have other people working on software development. I guess around that time we will see scipy_core mature, so that would be a good time to switch. I understand the numarray people take a wait-and-see approach, but intent to switch when scipy_core matures and fulfills their requirements. I guess I am following their example. So next year I will might be in the position of having more resources and I then will start thinking about developing some open-source scientific software. An imaging package like nd_image would be one of them. However, in contrast to the current design, I would like to move to a package that is fully written in C, with a thin interface layer to numarray/scipy_core for use with python. That would allow me to use the routines in other environments than python,. Sofar I have been very comfortable to make my package available under the license that is used by numarray. The same would be true for a scipy_core/scipy version. However, for a general C package that could have a much broader audience in the same way, I feel different. I would most likely release such a package using the GPL. In addition, that would allow me to use other GPL licensed software such as GSL with it. However, that would mean that a python package based on it would also be released with that license. I am not complete sure what the implications for distribution with a package like scipy would be, this may not be possible. But it would be freely available as a separate module which we would maintain ourself. If it pans out this way, it means that nd_image will not be further developed at some point. I am currently doing bug-fixes in the numarray version, and I certainly would not mind if there would be a common code-base allowing it to work with scipy_core. But if I am going to developer package as described above, I will to focus on that and recommend nd_image users to switch at some point. > > Even if your statement is made from the perspective of someone who > is moving away from Python altogether, I would still very-much > appreciate hearing your reasons. Now, perhaps, your just going to > experience a career-change altogether in which case, best wishes, > and thanks for all you've contributed. I will certainly stick with Python :-) > > Regardless of whether or not you have time to respond, I do > appreciate all your efforts. The nd_image package is very nice as > I've told you before. Thank you! Thanks! nd_image seems to find some users, and I would like to continue to provide something like nd_image. It might be a while though until nd_image or a replacement will start moving again and further developing. Meanwhile I am committed to keep nd_image working with numarray and doing minor bug-fixes. If we can at some point find a solution for a single code-base which would work with scipy_core, I am happy to expand this commitment to scipy_core. I cannot promise indefinite support, but as long as I am in the field and as long as I do not release a 'successor' package, I will try... Cheers, Peter From Chris.Barker at noaa.gov Wed Dec 14 10:26:13 2005 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed Dec 14 10:26:13 2005 Subject: [SciPy-dev] [Numpy-discussion] Thoughts on making it easier for =?iso-8859-1?q?numarray=09users_to_transition_to?= scipy_core In-Reply-To: References: <439DE22B.9070400@ieee.org> <439F4DBE.5040601@ieee.org> Message-ID: <200512141025.05985.Chris.Barker@noaa.gov> On Wednesday 14 December 2005 1:28 am, Peter Verveer wrote: > Sofar I have been very comfortable to make my package available under > the license that is used by numarray. The same would be true for a > scipy_core/scipy version. However, for a general C package that could > have a much broader audience in the same way, I feel different. I > would most likely release such a package using the GPL. In addition, > that would allow me to use other GPL licensed software such as GSL > with it. You could release it under two licenses, GPL for a C package that uses other GPL software, and the SciPy license for the python extension. The only thing that would kill that plan is if you use GPL code in the Python extension. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From aisaac at american.edu Wed Dec 14 12:09:06 2005 From: aisaac at american.edu (Alan G Isaac) Date: Wed Dec 14 12:09:06 2005 Subject: [Numpy-discussion] =?ISO-8859-1?Q?Re[2]:=20[SciPy-dev]=20[Numpy-discussion]=20Thoughts=20on?==?ISO-8859-1?Q?=20making=20it=20easier=20for=20numarray=09users=20to=20tra?==?ISO-8859-1?Q?nsition=20to=20scipy_core?= In-Reply-To: <200512141025.05985.Chris.Barker@noaa.gov> References: <439DE22B.9070400@ieee.org> <439F4DBE.5040601@ieee.org><200512141025.05985.Chris.Barker@noaa.gov> Message-ID: > On Wednesday 14 December 2005 1:28 am, Peter Verveer wrote: > Sofar I have been very comfortable to make my package > available under the license that is used by numarray. The > same would be true for a scipy_core/scipy version. > However, for a general C package that could have a much > broader audience in the same way, I feel different. > I would most likely release such a package using the GPL. > In addition, that would allow me to use other GPL > licensed software such as GSL with it. "Use with" means many things. You write it, you choose the license, of course. But as an academic user/observer, rather than a substantial contributor, I have noticed that if your goal is to see your code used and improved, it is not clear that the GPL is the best way to pursue that goal. See e.g. the history of Matplotlib and MayaVi. (Or Python itself, for that matter, for a larger example.) Each case is individual, of course. If you need extant GPL code to be an *integral* part of your code, that makes the GPL natural. fwiw, Alan From greg.ewing at canterbury.ac.nz Wed Dec 14 15:38:04 2005 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed Dec 14 15:38:04 2005 Subject: [Numpy-discussion] Re[2]: [SciPy-dev] [Numpy-discussion] Thoughts on making it easier for numarray users to transition to scipy core In-Reply-To: References: <439DE22B.9070400@ieee.org> <439F4DBE.5040601@ieee.org> <200512141025.05985.Chris.Barker@noaa.gov> Message-ID: <43A0ACB0.2050905@canterbury.ac.nz> On Wednesday 14 December 2005 1:28 am, Peter Verveer wrote: > I would most likely release such a package using the GPL. > In addition, that would allow me to use other GPL > licensed software such as GSL with it. If you combine your package with something GPL and release the result, then of course you would have to release that combination under the GPL. But that doesn't prevent you from releasing your own package separately under a different licence if you want. The question you need to ask yourself is whether you really want to impose all the restrictions of the GPL on users of your package. If you have no desire to be that restrictive, then use a more liberal licence. -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg.ewing at canterbury.ac.nz +--------------------------------------+ From janne.sinkkonen at gmail.com Thu Dec 15 04:21:12 2005 From: janne.sinkkonen at gmail.com (Janne Sinkkonen) Date: Thu Dec 15 04:21:12 2005 Subject: [Numpy-discussion] A patch to fix a Numeric 24.2 dotblas innerproduct problem Message-ID: <29dd95110512150420v49441896o4b2ffd4f60c2b15e@mail.gmail.com> With Numeric 24.2 one gets the following error: >>> innerproduct(ones((10,3),Float), ones((10,3),Float)) ldc must be >= MAX(N,1): ldc=3 N=10Parameter 14 to routine cblas_dgemm was incorrect It seems to be fixed by the following patch to Packages/dotblas/dotblas/_dotblas.c : --- _dotblas.c~ 2005-04-07 08:15:47.000000000 +0300 +++ _dotblas.c 2005-12-15 11:31:46.148482242 +0200 @@ -530,28 +530,28 @@ ap1->dimensions[0], ap2->dimensions[0], ap1->dimensions[1], 1.0, (double *)ap1->data, lda, (double *)ap2->data, ldb, - 0.0, (double *)ret->data, ldb); + 0.0, (double *)ret->data, ap1->dimensions[0]); } else if (typenum == PyArray_FLOAT) { cblas_sgemm(CblasRowMajor, CblasNoTrans, CblasTrans, ap1->dimensions[0], ap2->dimensions[0], ap1->dimensions[1], 1.0, (float *)ap1->data, lda, (float *)ap2->data, ldb, - 0.0, (float *)ret->data, ldb); + 0.0, (float *)ret->data, ap1->dimensions[0]); } else if (typenum == PyArray_CDOUBLE) { cblas_zgemm(CblasRowMajor, CblasNoTrans, CblasTrans, ap1->dimensions[0], ap2->dimensions[0], ap1->dimensions[1], oneD, (double *)ap1->data, lda, (double *)ap2->data, ldb, - zeroD, (double *)ret->data, ldb); + zeroD, (double *)ret->data, ap1->dimensions[0]); } else if (typenum == PyArray_CFLOAT) { cblas_cgemm(CblasRowMajor, CblasNoTrans, CblasTrans, ap1->dimensions[0], ap2->dimensions[0], ap1->dimensions[1], oneF, (float *)ap1->data, lda, (float *)ap2->data, ldb, - zeroF, (float *)ret->data, ldb); + zeroF, (float *)ret->data, ap1->dimensions[0]); } } else { -- Janne Sinkkonen, PhD Xtract Ltd From khinsen at cea.fr Thu Dec 15 04:41:08 2005 From: khinsen at cea.fr (khinsen at cea.fr) Date: Thu Dec 15 04:41:08 2005 Subject: [Numpy-discussion] A patch to fix a Numeric 24.2 dotblas innerproduct problem In-Reply-To: <29dd95110512150420v49441896o4b2ffd4f60c2b15e@mail.gmail.com> References: <29dd95110512150420v49441896o4b2ffd4f60c2b15e@mail.gmail.com> Message-ID: <6C9BA6CE-36F1-41B9-AA0A-DA4DC45F5E04@cea.fr> On Dec 15, 2005, at 13:20, Janne Sinkkonen wrote: > With Numeric 24.2 one gets the following error: > >>>> innerproduct(ones((10,3),Float), ones((10,3),Float)) > ldc must be >= MAX(N,1): ldc=3 N=10Parameter 14 to routine cblas_dgemm > was incorrect I have submitted a patch for this in the bug tracker on SourceForge a few weeks ago. Konrad. -- --------------------------------------------------------------------- Konrad Hinsen Laboratoire L?on Brillouin, CEA Saclay, 91191 Gif-sur-Yvette Cedex, France Tel.: +33-1 69 08 79 25 Fax: +33-1 69 08 82 61 E-Mail: khinsen at cea.fr --------------------------------------------------------------------- From tchur at optushome.com.au Wed Dec 21 12:39:03 2005 From: tchur at optushome.com.au (Tim Churches) Date: Wed Dec 21 12:39:03 2005 Subject: [Numpy-discussion] Re: scipy.stats.itemfreq: overflow with add.reduce In-Reply-To: References: Message-ID: <43A9BD2D.5030005@optushome.com.au> Hans Georg Krauthaeuser wrote: > Hans Georg Krauthaeuser schrieb: > >>Hi All, >> >>I was playing with scipy.stats.itemfreq when I observed the following >>overflow: >> >>In [119]:for i in [254,255,256,257,258]: >> .....: l=[0]*i >> .....: print i, stats.itemfreq(l), l.count(0) >> .....: >>254 [ [ 0 254]] 254 >>255 [ [ 0 255]] 255 >>256 [ [0 0]] 256 >>257 [ [0 1]] 257 >>258 [ [0 2]] 258 >> >>itemfreq is pretty small (in stats.py): >> >>---------------------------------------------------------------------- >>def itemfreq(a): >> """ >>Returns a 2D array of item frequencies. Column 1 contains item values, >>column 2 contains their respective counts. Assumes a 1D array is passed. >> >>Returns: a 2D frequency table (col [0:n-1]=scores, col n=frequencies) >>""" >> scores = _support.unique(a) >> scores = sort(scores) >> freq = zeros(len(scores)) >> for i in range(len(scores)): >> freq[i] = add.reduce(equal(a,scores[i])) >> return array(_support.abut(scores, freq)) >>---------------------------------------------------------------------- >> >>It seems that add.reduce is the source for the overflow: >> >>In [116]:from scipy import * >> >>In [117]:for i in [254,255,256,257,258]: >> .....: l=[0]*i >> .....: print i, add.reduce(equal(l,0)) >> .....: >>254 254 >>255 255 >>256 0 >>257 1 >>258 2 >> >>Is there any possibility to avoid the overflow? Apropos the preceding, herewith a thread on the Numpy list from a more than a few months ago. The take-home message is that for integer arrays, add.reduce is very fast at producing results which fall into two categories: a) correct or b) incorrect due to overflow. Unfortunately there is no equally quick method of determining into which of these two categories any specific result returned by add.reduce falls. Tim C From Hans-Georg.Krauthaeuser at E-Technik.Uni-Magdeburg.DE Wed Dec 21 23:54:04 2005 From: Hans-Georg.Krauthaeuser at E-Technik.Uni-Magdeburg.DE (Dr. Hans Georg Krauthaeuser) Date: Wed Dec 21 23:54:04 2005 Subject: [Numpy-discussion] Re: scipy.stats.itemfreq: overflow with add.reduce In-Reply-To: <43A9BD2D.5030005@optushome.com.au> References: <43A9BD2D.5030005@optushome.com.au> Message-ID: <43AA5B64.7010207@et.uni-magdeburg.de> Tim Churches schrieb: > Hans Georg Krauthaeuser wrote: > >>Hans Georg Krauthaeuser schrieb: >> >> >>>Hi All, >>> >>>I was playing with scipy.stats.itemfreq when I observed the following >>>overflow: >>> >>>In [119]:for i in [254,255,256,257,258]: >>> .....: l=[0]*i >>> .....: print i, stats.itemfreq(l), l.count(0) >>> .....: >>>254 [ [ 0 254]] 254 >>>255 [ [ 0 255]] 255 >>>256 [ [0 0]] 256 >>>257 [ [0 1]] 257 >>>258 [ [0 2]] 258 >>> >>>itemfreq is pretty small (in stats.py): >>> >>>---------------------------------------------------------------------- >>>def itemfreq(a): >>> """ >>>Returns a 2D array of item frequencies. Column 1 contains item values, >>>column 2 contains their respective counts. Assumes a 1D array is passed. >>> >>>Returns: a 2D frequency table (col [0:n-1]=scores, col n=frequencies) >>>""" >>> scores = _support.unique(a) >>> scores = sort(scores) >>> freq = zeros(len(scores)) >>> for i in range(len(scores)): >>> freq[i] = add.reduce(equal(a,scores[i])) >>> return array(_support.abut(scores, freq)) >>>---------------------------------------------------------------------- >>> >>>It seems that add.reduce is the source for the overflow: >>> >>>In [116]:from scipy import * >>> >>>In [117]:for i in [254,255,256,257,258]: >>> .....: l=[0]*i >>> .....: print i, add.reduce(equal(l,0)) >>> .....: >>>254 254 >>>255 255 >>>256 0 >>>257 1 >>>258 2 >>> >>>Is there any possibility to avoid the overflow? > > > Apropos the preceding, herewith a thread on the Numpy list from a more > than a few months ago. The take-home message is that for integer arrays, > add.reduce is very fast at producing results which fall into two > categories: a) correct or b) incorrect due to overflow. Unfortunately > there is no equally quick method of determining into which of these two > categories any specific result returned by add.reduce falls. > > Tim C > > From: Tim Churches > To: Todd Miller > Date: Apr 12 2005 - 7:51am > > Todd Miller wrote: > >>On Sun, 2005-04-10 at 10:23 +1000, Tim Churches wrote: >> >> >>>I just got caught by code equivalent to this (with NumPy 23.8 on 32 bit >>>Linux): >>> >>> >>>>>>import Numeric as N >>>>>>a = N.array((2000000000,1000000000),typecode=N.Int32) >>>>>>N.add.reduce(a) >>> >>>-1294967296 >>> >>>OK, it is an elementary mistake, but the silent overflow caught me >>>unawares. casting the array to Float64 before summing it avoids the >>>error, but in my instance the actual data is a rank-1 array of 21 >>>million integers with a mean value of about 140 (which adds up more than >>>sys.maxint), and casting to Float64 will use quite a lot of memory (as >>>well as taking some time). >>> >>>Any advice for catching or avoiding such overflow without always >>>incurring a performance and memory hit by always casting to Float64? >> >> >>Here's what numarray does: >> >> >> >>>>>import numarray as N >>>>>a = N.array((2000000000,1000000000),typecode=N.Int32) >>>>>N.add.reduce(a) >> >>-1294967296 >> >>So basic reductions in numarray have the same "careful while you're >>shaving" behavior as Numeric; it's fast but easy to screw up. > > > Sure, but how does one be careful? It seems that for any array of two > integers or more which could sum to more than sys.maxint or less than > -sys.maxint, add.reduce() in both NumPy and Numeric will give either a) > the correct answer or b) the incorrect answer, and short of adding up > the array using a safer but much slower method, there is no way of > determining if the answer provided (quickly) by add.reduce is right or > wrong? Which seems to make it fast but useless (for integer arrays, at > least? Is that an unfair summary? Can anyone point me towards a method > for using add.reduce() on small arrays of large integers with values in > the billions, or on large arrays of fairly small integer values, which > will not suddenly and without warning give the wrong answer? > > >>But: >> >> >> >>>>>a.sum() >> >>3000000000L >> >> >>>>>a.sum(type='d') >> >>3000000000.0 >> >>a.sum() blockwise upcasts to the largest type of kind on the fly, in >>this case, Int64. This avoids the storage overhead of typecasting the >>entire array. > > > That's on a 64-bit platform, right? The same method could be used to > cast the accumulator to a Float64 on a 32-bit platform to avoid casting > the entire array? > > >>A better name for the method would have been sumall() since it sums all >>elements of a multi-dimensional array. The flattening process reduces >>on one dimension before flattening preventing a full copy of a >>discontiguous array. It could be smarter about choosing the dimension >>of the initial reduction. > > > OK, thanks. Unfortunately it is not possible for us to port our > application to numarray at the moment. But the insight is most helpful. > > Tim C Tim, Thank you very much for your answer. I posted two follow-ups to my own post on c.l.p (http://groups.google.de/group/comp.lang.python/browse_thread/thread/a96c404d73d71259/7769fca1fa562718?hl=de#7769fca1fa562718) A least for scipy version '0.3.2' the problem comes directly from the ufunc 'add' for bool arrays: In [178]:array(array([1],'b')*2) Out[178]:array([2],'i') In [179]:array(array([1],'b')+array([1],'b')) Out[179]:array([2],'b') 'multiply' changes the typecode. So, in this case a+a != a*2 if a is an array with bytecode 'b'. Regards Hans Georg Krauth?user -- Otto-von-Guericke-Universitaet Magdeburg IGET | fon: +49 391 67 12195 Postfach 4120 | fax: +49 391 67 11236 39016 Magdeburg | email: hgk at et.Uni-Magdeburg.DE Germany | www: www.uni-magdeburg.de/krauthae From faltet at carabos.com Thu Dec 22 01:11:04 2005 From: faltet at carabos.com (Francesc Altet) Date: Thu Dec 22 01:11:04 2005 Subject: [Numpy-discussion] ANN: PyTables 1.2.1 Message-ID: <200512221009.50977.faltet@carabos.com> ========================= Announcing PyTables 1.2.1 ========================= I'm happy to announce a new version of PyTables. This is a maintenance version and only bugs has been fixed in it. In particular, the documentation has not change at all, so it is not necessary that you look at it for changes. Go to the PyTables web site for downloading the beast: http://pytables.sourceforge.net/ or keep reading for more info about the bugs fixed in this version. I take the opportunity to announce as well that PyTables is adopting the bazaar-ng (http://www.bazaar-ng.org/) distributed version control, in order to easy the contributions from developers throughout the world. See more info about this new facility (currently in beta) in: http://sourceforge.net/mailarchive/forum.php?thread_id=9253854&forum_id=13760 Changes more in depth ===================== Bug fixes: - Table.flush() is called automatically before disposing a table object from the user space. This avoids a problem that appears when the user does not explicitely do the flush and the table is unbounded and rebounded after on (using h5file.getNode() for example). - A small typo has been fixed in the ptrepack utility. This prevented ptrepack from working correctly when asking for statistics on operations done (-v flag). Known issues: - Time datatypes are non-portable between big-endian and little-endian architectures. This is ultimately a consequence of an HDF5 limitation. See SF bug #1234709 for more info. Backward-incompatible changes: - None. Important note for MacOSX users =============================== From PyTables 1.2 on, the UCL compressor seems to work well again on MacOSX platforms. We don't know exactly why, but the fact is that all the test suite passes (using UCL) executes flawlessly. So, from now on, support for UCL in MacOSX is enabled again by default (i.e. you don't need to use the flag ``--force-ucl``, which has disappeared). Important note for Python 2.4 and Windows users =============================================== If you are willing to use PyTables with Python 2.4 in Windows platforms, you will need to get the HDF5 library compiled for MSVC 7.1, aka .NET 2003. It can be found at: ftp://ftp.ncsa.uiuc.edu/HDF/HDF5/current/bin/windows/5-165-win-net.ZIP Users of Python 2.3 on Windows will have to download the version of HDF5 compiled with MSVC 6.0 available in: ftp://ftp.ncsa.uiuc.edu/HDF/HDF5/current/bin/windows/5-165-win.ZIP What it is ========== **PyTables** is a package for managing hierarchical datasets and designed to efficiently cope with extremely large amounts of data (with support for full 64-bit file addressing). It features an object-oriented interface that, combined with C extensions for the performance-critical parts of the code, makes it a very easy-to-use tool for high performance data storage and retrieval. PyTables runs on top of the HDF5 library and numarray (Numeric is also supported) package for achieving maximum throughput and convenient use. Besides, PyTables I/O for table objects is buffered, implemented in C and carefully tuned so that you can reach much better performance with PyTables than with your own home-grown wrappings to the HDF5 library. PyTables sports indexing capabilities as well, allowing doing selections in tables exceeding one billion of rows in just seconds. Platforms ========= This version has been extensively checked on quite a few platforms, like Linux on Intel32 (Pentium), Win on Intel32 (Pentium), Linux on Intel64 (Itanium2), FreeBSD on AMD64 (Opteron), Linux on PowerPC and MacOSX on PowerPC. For other platforms, chances are that the code can be easily compiled and run without further issues. Please, contact us in case you are experiencing problems. Resources ========= Go to the PyTables web site for more details: http://pytables.sourceforge.net/ About the HDF5 library: http://hdf.ncsa.uiuc.edu/HDF5/ About numarray: http://www.stsci.edu/resources/software_hardware/numarray To know more about the company behind the PyTables development, see: http://www.carabos.com/ Acknowledgments =============== Thanks to various the users who provided feature improvements, patches, bug reports, support and suggestions. See THANKS file in distribution package for a (necessarily incomplete) list of contributors. Many thanks also to SourceForge who have helped to make and distribute this package! And last but not least, a big thank you to THG (http://www.hdfgroup.org/) for sponsoring many of the new features recently introduced in PyTables. Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. ---- **Enjoy data!** -- The PyTables Team From jmiller at stsci.edu Thu Dec 22 08:30:25 2005 From: jmiller at stsci.edu (Todd Miller) Date: Thu Dec 22 08:30:25 2005 Subject: [Numpy-discussion] Re: scipy.stats.itemfreq: overflow with add.reduce In-Reply-To: <43AA5B64.7010207@et.uni-magdeburg.de> References: <43A9BD2D.5030005@optushome.com.au> <43AA5B64.7010207@et.uni-magdeburg.de> Message-ID: <43AAD44C.5010608@stsci.edu> Dr. Hans Georg Krauthaeuser wrote: > Tim Churches schrieb: > >> Hans Georg Krauthaeuser wrote: >> >>> Hans Georg Krauthaeuser schrieb: >>> >>> >>>> Hi All, >>>> >>>> I was playing with scipy.stats.itemfreq when I observed the >>>> following overflow: >>>> >>>> In [119]:for i in [254,255,256,257,258]: >>>> .....: l=[0]*i >>>> .....: print i, stats.itemfreq(l), l.count(0) >>>> .....: >>>> 254 [ [ 0 254]] 254 >>>> 255 [ [ 0 255]] 255 >>>> 256 [ [0 0]] 256 >>>> 257 [ [0 1]] 257 >>>> 258 [ [0 2]] 258 >>>> >>>> itemfreq is pretty small (in stats.py): >>>> >>>> ---------------------------------------------------------------------- >>>> def itemfreq(a): >>>> """ >>>> Returns a 2D array of item frequencies. Column 1 contains item >>>> values, >>>> column 2 contains their respective counts. Assumes a 1D array is >>>> passed. >>>> >>>> Returns: a 2D frequency table (col [0:n-1]=scores, col n=frequencies) >>>> """ >>>> scores = _support.unique(a) >>>> scores = sort(scores) >>>> freq = zeros(len(scores)) >>>> for i in range(len(scores)): >>>> freq[i] = add.reduce(equal(a,scores[i])) >>>> return array(_support.abut(scores, freq)) >>>> ---------------------------------------------------------------------- >>>> >>>> It seems that add.reduce is the source for the overflow: >>>> >>>> In [116]:from scipy import * >>>> >>>> In [117]:for i in [254,255,256,257,258]: >>>> .....: l=[0]*i >>>> .....: print i, add.reduce(equal(l,0)) >>>> .....: >>>> 254 254 >>>> 255 255 >>>> 256 0 >>>> 257 1 >>>> 258 2 >>>> >>>> Is there any possibility to avoid the overflow? >>> >> >> >> Apropos the preceding, herewith a thread on the Numpy list from a more >> than a few months ago. The take-home message is that for integer arrays, >> add.reduce is very fast at producing results which fall into two >> categories: a) correct or b) incorrect due to overflow. Unfortunately >> there is no equally quick method of determining into which of these two >> categories any specific result returned by add.reduce falls. >> >> Tim C > Hi Tim, There are already tools in place which let you accomplish what you want, they're just not the default. I have a few suggestions: 1. Ask scipy newcore (Travis) for long term changes like the default behavior of add.reduce. 2. Look at numarray's sum() method for better overflow behavior and reduction on all axes. >>> import numarray as na >>> a = na.arange(100, type='Int8') >>> na.add.reduce(a) # yeah, it overflows... 86 >>> a.sum() # but this doesn't (well, not if Int64 is enough) and always reduces on all axes at once... 4950L 3. Mimic the sum() method and specifiy the type of your output array in add.reduce. >>> na.add.reduce(a, type=na.MaximumType(a.type())) 4950L 4. Or, just explicitly specify the type of your reduction with something you're happy with: >>> na.add.reduce(a, type='Int16') 4950 Hopefully one of these solutions will work for you. Regards, Todd >> >> From: Tim Churches >> To: Todd Miller >> Date: Apr 12 2005 - 7:51am >> >> Todd Miller wrote: >> >>> On Sun, 2005-04-10 at 10:23 +1000, Tim Churches wrote: >>> >>> >>>> I just got caught by code equivalent to this (with NumPy 23.8 on 32 >>>> bit >>>> Linux): >>>> >>>> >>>>>>> import Numeric as N >>>>>>> a = N.array((2000000000,1000000000),typecode=N.Int32) >>>>>>> N.add.reduce(a) >>>>>> >>>> >>>> -1294967296 >>>> >>>> OK, it is an elementary mistake, but the silent overflow caught me >>>> unawares. casting the array to Float64 before summing it avoids the >>>> error, but in my instance the actual data is a rank-1 array of 21 >>>> million integers with a mean value of about 140 (which adds up more >>>> than >>>> sys.maxint), and casting to Float64 will use quite a lot of memory (as >>>> well as taking some time). >>>> >>>> Any advice for catching or avoiding such overflow without always >>>> incurring a performance and memory hit by always casting to Float64? >>> >>> >>> >>> Here's what numarray does: >>> >>> >>> >>>>>> import numarray as N >>>>>> a = N.array((2000000000,1000000000),typecode=N.Int32) >>>>>> N.add.reduce(a) >>>>> >>> >>> -1294967296 >>> >>> So basic reductions in numarray have the same "careful while you're >>> shaving" behavior as Numeric; it's fast but easy to screw up. >> >> >> >> Sure, but how does one be careful? It seems that for any array of two >> integers or more which could sum to more than sys.maxint or less than >> -sys.maxint, add.reduce() in both NumPy and Numeric will give either a) >> the correct answer or b) the incorrect answer, and short of adding up >> the array using a safer but much slower method, there is no way of >> determining if the answer provided (quickly) by add.reduce is right or >> wrong? Which seems to make it fast but useless (for integer arrays, at >> least? Is that an unfair summary? Can anyone point me towards a method >> for using add.reduce() on small arrays of large integers with values in >> the billions, or on large arrays of fairly small integer values, which >> will not suddenly and without warning give the wrong answer? >> >> >>> But: >>> >>> >>> >>>>>> a.sum() >>>>> >>> >>> 3000000000L >>> >>> >>>>>> a.sum(type='d') >>>>> >>> >>> 3000000000.0 >>> >>> a.sum() blockwise upcasts to the largest type of kind on the fly, in >>> this case, Int64. This avoids the storage overhead of typecasting the >>> entire array. >> >> >> >> That's on a 64-bit platform, right? The same method could be used to >> cast the accumulator to a Float64 on a 32-bit platform to avoid casting >> the entire array? >> >> >>> A better name for the method would have been sumall() since it sums all >>> elements of a multi-dimensional array. The flattening process reduces >>> on one dimension before flattening preventing a full copy of a >>> discontiguous array. It could be smarter about choosing the dimension >>> of the initial reduction. >> >> >> >> OK, thanks. Unfortunately it is not possible for us to port our >> application to numarray at the moment. But the insight is most helpful. >> >> Tim C > > Tim, > > Thank you very much for your answer. I posted two follow-ups to my own > post on c.l.p > (http://groups.google.de/group/comp.lang.python/browse_thread/thread/a96c404d73d71259/7769fca1fa562718?hl=de#7769fca1fa562718) > > > A least for scipy version '0.3.2' the problem comes directly from the > ufunc 'add' for bool arrays: > > In [178]:array(array([1],'b')*2) > Out[178]:array([2],'i') > > In [179]:array(array([1],'b')+array([1],'b')) > Out[179]:array([2],'b') > > 'multiply' changes the typecode. > > So, in this case > > a+a != a*2 if a is an array with bytecode 'b'. > > Regards > Hans Georg Krauth?user From oliphant.travis at ieee.org Mon Dec 26 01:01:07 2005 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon Dec 26 01:01:07 2005 Subject: [Numpy-discussion] Example of power of new data-type descriptors. Message-ID: <43AFB124.9060507@ieee.org> I'd like more people to know about the new power that is in scipy core due to the general data-type descriptors that can now be used to define numeric arrays. Towards that effort here is a simple example (be sure to use latest SVN -- there were a coupld of minor changes that improve usability made recently). Notice this example does not use a special "record" array subclass. This is just a regular array. I'm kind of intrigued (though not motivated to pursue) the possibility of accessing (or defining) databases directly into scipy_core arrays using the record functionality. # Define a new data-type descriptor >>> import scipy >>> dtype = scipy.dtypedescr({'names': ['name', 'age', 'weight'], 'formats': ['S30', 'i2', 'f4']}) >>> a = scipy.array([('Bill',31,260),('Fred',15,135)], dtype=dtype) # the argument to dtypedescr could have also been placed here as the argument to dtype >>> print a['name'] [Bill Fred] >>> print a['age'] [31 15] >>> print a['weight'] [ 260. 135.] >>> print a[0] ('Bill', 31, 260.0) >>> print a[1] ('Fred', 15, 135.0) It seems to me there are some very interesting possibilities with this new ability. The record array subclass adds an improved scalar type (the record) and attribute access to get at the fields: (e.g. a.name, a.age, and a.weight). But, if you don't need attribute access you can use regular arrays to do a lot of what you might need a record array to accomplish for you. I'd love to see what people come up with using this new facility. The new array PEP for Python basically proposes adding a very simple array object (just the basic PyArrayObject * of Numeric with a bare-bones type-object table) plus this new data-type descriptor object to Python and a very few builtin data-type descriptors (perhaps just object initially). This would basically add the array interface to Python directly and allow people to start using it generally. The PEP is slow going because it is not on my priority list right now because it is not essential to making scipy_core work well. But, I would love to have more people ruminating on the basic ideas which I think are crystallizing. Best wishes for a new year, -Travis Oliphant From skdqjdw at friko3.onet.pl Mon Dec 26 03:33:12 2005 From: skdqjdw at friko3.onet.pl (Fernando Allen) Date: Mon Dec 26 03:33:12 2005 Subject: [Numpy-discussion] WE WISH numpy-discussion A MERRY CHRISTMAS Message-ID: <0002$01ce1975$05e71e38@dvx> Ivan away ground reasoning know smile the away arm seeing out Magi was yellow different nearer saying ten On relations before your fifteenth of to himself foreigner Ivan that me what But an his foreigner prove A to it nothing people Jesus shall he gold Ivan away ground reasoning know smile the away arm seeing out Magi was yellow different nearer saying ten On relations before your fifteenth of to himself foreigner Ivan that me what But an his foreigner prove A to it nothing people Jesus shall he gold Ivan away ground reasoning know smile the away arm seeing out Magi was yellow different nearer saying ten On relations before your fifteenth of to himself foreigner Ivan that me what But an his foreigner prove A to it nothing people Jesus shall he gold Ivan away ground reasoning know smile the away arm seeing out Magi was yellow different nearer saying ten On relations before your fifteenth of to himself foreigner Ivan that me what But an his foreigner prove A to it nothing people Jesus shall he gold -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: yorgibljv.gif Type: image/gif Size: 24212 bytes Desc: not available URL: From Chris.Barker at noaa.gov Tue Dec 27 18:02:06 2005 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue Dec 27 18:02:06 2005 Subject: [Numpy-discussion] Building numarray with atlas on FC4 Message-ID: <43B1F1E7.3060705@noaa.gov> Hi all, I just built numarray with atlas on Fedora core 4 Linux. I used the copy of atlas in extras (yum install atlas-sse2). To get it to build, I edited cfg_packages.py this way: elif os.path.exists('/usr/local/lib/atlas'): # uses atlas, installed in /usr/local/lib lapack_dirs = ['/usr/local/lib/atlas'] lapack_include_dirs += ["/usr/local/include/atlas"] lapack_libs = ['lapack', 'cblas', 'f77blas', 'atlas', 'g2c', 'm'] elif os.path.exists('/usr/lib/sse2/'): # uses atlas, installed in /usr/lib/sse2 ( Fedora Core4 extras ) lapack_dirs = ['/usr/lib/sse2'] lapack_include_dirs += ["/usr/include/atlas"] lapack_libs = ['lapack_atlas','lapack', 'blas'] Did I do that right? would you like to add that stanza to the distribution? I do think it's a good idea to check for the existence of the atlas dir before assuming that it's there. Maybe an exception could be raised if nothing is found? I did find it odd that I couldn't find instructions for how to do this. did I miss something? -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From vel.accel at gmail.com Wed Dec 28 16:39:08 2005 From: vel.accel at gmail.com (dHering) Date: Wed Dec 28 16:39:08 2005 Subject: [Numpy-discussion] Example of power of new data-type descriptors. In-Reply-To: <43AFB124.9060507@ieee.org> References: <43AFB124.9060507@ieee.org> Message-ID: <1e52e0880512281638p1c35d53ep83275bac41dba8a8@mail.gmail.com> On 12/26/05, Travis Oliphant wrote: > > > I'd like more people to know about the new power that is in scipy core > due to the general data-type descriptors that can now be used to define > numeric arrays. Towards that effort here is a simple example (be sure > to use latest SVN -- there were a coupld of minor changes that improve > usability made recently). Notice this example does not use a special > "record" array subclass. This is just a regular array. I'm kind of > intrigued (though not motivated to pursue) the possibility of accessing > (or defining) databases directly into scipy_core arrays using the record > functionality. > > # Define a new data-type descriptor > >>> import scipy > > >>> dtype = scipy.dtypedescr({'names': ['name', 'age', 'weight'], > 'formats': ['S30', 'i2', 'f4']}) > >>> a = scipy.array([('Bill',31,260),('Fred',15,135)], dtype=dtype) > # the argument to dtypedescr could have also been placed here as the > argument to dtype > > >>> print a['name'] > [Bill Fred] > > >>> print a['age'] > [31 15] > > >>> print a['weight'] > [ 260. 135.] > > >>> print a[0] > ('Bill', 31, 260.0) > > >>> print a[1] > ('Fred', 15, 135.0) > > It seems to me there are some very interesting possibilities with this > new ability. The record array subclass adds an improved scalar type > (the record) and attribute access to get at the fields: (e.g. a.name, > a.age, and a.weight). But, if you don't need attribute access you can > use regular arrays to do a lot of what you might need a record array to > accomplish for you. I'd love to see what people come up with using this > new facility. > > The new array PEP for Python basically proposes adding a very simple > array object (just the basic PyArrayObject * of Numeric with a > bare-bones type-object table) plus this new data-type descriptor object > to Python and a very few builtin data-type descriptors (perhaps just > object initially). This would basically add the array interface to > Python directly and allow people to start using it generally. The PEP > is slow going because it is not on my priority list right now because it > is not essential to making scipy_core work well. But, I would love to > have more people ruminating on the basic ideas which I think are > crystallizing. > > Best wishes for a new year, > > -Travis Oliphant > > > > > > > > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From vel.accel at gmail.com Wed Dec 28 16:42:08 2005 From: vel.accel at gmail.com (dHering) Date: Wed Dec 28 16:42:08 2005 Subject: [Numpy-discussion] Example of power of new data-type descriptors. In-Reply-To: <1e52e0880512281638p1c35d53ep83275bac41dba8a8@mail.gmail.com> References: <43AFB124.9060507@ieee.org> <1e52e0880512281638p1c35d53ep83275bac41dba8a8@mail.gmail.com> Message-ID: <1e52e0880512281641n2ac33cddn60874d3c826f48ae@mail.gmail.com> [Sorry for the mis-post] Hi Travis, I really hope that the future brings Scipy and Pytables together. Thank you very much for your contributions and motivation Travis. Dieter Hering On 12/28/05, dHering wrote: > On 12/26/05, Travis Oliphant wrote: > > > > > > I'd like more people to know about the new power that is in scipy core > > due to the general data-type descriptors that can now be used to define > > numeric arrays. Towards that effort here is a simple example (be sure > > to use latest SVN -- there were a coupld of minor changes that improve > > usability made recently). Notice this example does not use a special > > "record" array subclass. This is just a regular array. I'm kind of > > intrigued (though not motivated to pursue) the possibility of accessing > > (or defining) databases directly into scipy_core arrays using the record > > functionality. > > > > # Define a new data-type descriptor > > >>> import scipy > > > > >>> dtype = scipy.dtypedescr({'names': ['name', 'age', 'weight'], > > 'formats': ['S30', 'i2', 'f4']}) > > >>> a = scipy.array([('Bill',31,260),('Fred',15,135)], dtype=dtype) > > # the argument to dtypedescr could have also been placed here as the > > argument to dtype > > > > >>> print a['name'] > > [Bill Fred] > > > > >>> print a['age'] > > [31 15] > > > > >>> print a['weight'] > > [ 260. 135.] > > > > >>> print a[0] > > ('Bill', 31, 260.0) > > > > >>> print a[1] > > ('Fred', 15, 135.0) > > > > It seems to me there are some very interesting possibilities with this > > new ability. The record array subclass adds an improved scalar type > > (the record) and attribute access to get at the fields: (e.g. a.name, > > a.age, and a.weight). But, if you don't need attribute access you can > > use regular arrays to do a lot of what you might need a record array to > > accomplish for you. I'd love to see what people come up with using this > > new facility. > > > > The new array PEP for Python basically proposes adding a very simple > > array object (just the basic PyArrayObject * of Numeric with a > > bare-bones type-object table) plus this new data-type descriptor object > > to Python and a very few builtin data-type descriptors (perhaps just > > object initially). This would basically add the array interface to > > Python directly and allow people to start using it generally. The PEP > > is slow going because it is not on my priority list right now because it > > is not essential to making scipy_core work well. But, I would love to > > have more people ruminating on the basic ideas which I think are > > crystallizing. > > > > Best wishes for a new year, > > > > -Travis Oliphant > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------- > > This SF.net email is sponsored by: Splunk Inc. Do you grep through log files > > for problems? Stop! Download the new AJAX search engine that makes > > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > > http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > From bemmhdhgdcn at extremeweb.com Thu Dec 29 10:27:11 2005 From: bemmhdhgdcn at extremeweb.com (Oscar Knutson) Date: Thu Dec 29 10:27:11 2005 Subject: [Numpy-discussion] Re[12]: Message-ID: <0004$01b72c6b$089321c8@gep> come away. You see, I doent grow younger as the years comes vain. There is no alloy of self in what I feel for you. distinguished Guest, the ornament of our town. May he never leave Julia, and when its breeding is professed indifference to Closer in my arms, nearer to my heart, her trembling hand upon my Sir, taking this public opportunity of thanking you, on my own him, from the head, across a cheerful space that is certainly not round, and if I hadnt sailed as twas, most like I shouldnt never the proceedings the tables were cleared as if by art-magic for across, but which Peggotty exhibits to the children as a precious cause. My dearest girl, dearer to me than anything in life, if you the beginning of a favourite story Agnes used to tell them, conversation with me, had fallen on a pious fraud, or had really introductory to the arrival of a wicked old Fairy in a cloak who and thrust out her little heap of golden curls from between the produced a flat-folded, paper parcel, from which he took out, with Emly, said he, arter you left her, maam - and I never heerd unappreciated. Though remote, we are neither unfriended, Her tears fell fast; but they were not like those she had lately We silently observed him as he sat, still looking at the fire. Would you believe it. he said. Why, someun even made offer fur similar letters by him, to be shortly republished, in a neat But, my dear Sir, though estranged by the force of circumstances fared nohows, but fared to thrive. Weve allus thrived. Weve the colony over. Hed got an old newspaper with him, and some I never saw Agnes laugh so. This sudden ecstasy on the part of Mr. The hysterics called up Peggotty. The moment my aunt was restored, He was an old man, my servant said, and looked like a farmer. unable to liquidate, brought a tear into the manliest eye present. truly, and entirely. I tried to show her how I had hoped I had his face, he looked, to me, as vigorous and robust, withal as the greater Mr. Peggottys ecstasy became, and the more he rubbed I kep it from her arter I heerd on t, said Mr. Peggotty, going or so, but we have allus thrived. What with sheep-farming, and his polished and highly-ornate address. Suffice it to observe, that Which is verse, said Mr. Peggotty, surprised to find it out, stairs. The beauty, fashion, and exclusiveness of Port Middlebay, someone upon whom you have bestowed the treasure of your love. Do hated everybody, it produced some commotion. One of our boys laid the greater Mr. Peggottys ecstasy became, and the more he rubbed assembly by humorously remarking that he found himself unable to sister, who appears to me to be engaged to the cousin. Traddles, had been married ten happy years. Agnes and I were sitting by the We stood together in the same old-fashioned window at night, when the moon was shining; Agnes with her quiet eyes raised up to it; I come away. You see, I doent grow younger as the years comes vain. There is no alloy of self in what I feel for you. distinguished Guest, the ornament of our town. May he never leave Julia, and when its breeding is professed indifference to Closer in my arms, nearer to my heart, her trembling hand upon my Sir, taking this public opportunity of thanking you, on my own him, from the head, across a cheerful space that is certainly not round, and if I hadnt sailed as twas, most like I shouldnt never the proceedings the tables were cleared as if by art-magic for across, but which Peggotty exhibits to the children as a precious cause. My dearest girl, dearer to me than anything in life, if you the beginning of a favourite story Agnes used to tell them, conversation with me, had fallen on a pious fraud, or had really introductory to the arrival of a wicked old Fairy in a cloak who and thrust out her little heap of golden curls from between the -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ebmmdek.gif Type: image/gif Size: 15552 bytes Desc: not available URL: From uguoozy at pagesimmo.com Thu Dec 29 10:47:10 2005 From: uguoozy at pagesimmo.com (Josie Humphrey) Date: Thu Dec 29 10:47:10 2005 Subject: [Numpy-discussion] Re[7]: Message-ID: <0002$01ba51d3$09025f67@playroom> I must say more. I cannot let you leave me so. For Heavens sake, children were playing in the room, when I was told that a stranger It grows out of the night when Dora died. She sent you for me. and thrust out her little heap of golden curls from between the own choosing; that I could not, from my removed place, be a of his stay, - which, I think, was something less than a month, - heart, it has been lightened for me. If I have any secret, it is that your aunts the most extraordinary woman in the world, sir. these; all turning to me as I ask my thoughts the question. Traddless house is one of the very houses - or it easily may have Traddles about to be kissed, until he is out of breath. Here, I had advanced in fame and fortune, my domestic joy was perfect, I had run to bring him in, and I had not yet clearly seen his face, She was going away, but I detained her. I clasped my arm about her me. He is dead. Rosa kneeling at her feet, by turns caresses her, For Emly, he said, as he put it in his breast. I promised, into a roar of laughter, and rubbed his hands up and down his legs, discourses to me of the good fortune he has enjoyed. towards me, and said in a low voice, broken here and there, but Mrs. MELL; WILKINS MICAWBER, ESQUIRE, JUNIOR who convulsed the the proceedings the tables were cleared as if by art-magic for Agnes our eldest child left her doll in a chair to represent her, highly honoured, but a good deal surprised; and after that, told Looking fixedly at me, she puts her hand to her forehead, and truly, and entirely. I tried to show her how I had hoped I had return thanks in a speech, but would do so, with their permission, fared nohows, but fared to thrive. Weve allus thrived. Weve this present hour. But I think the solitoode done her good. And dancing. Among the votaries of TERPSICHORE, who disported following her glance. Long miles of road then opened out before my to apples, are shrivelled now; and her eyes, that used to darken I must say more. I cannot let you leave me so. For Heavens sake, a-going fur to change my condition at my time of life, upd with Bush now, being so well to do; and have gone right away round to We walk away, arm in arm. I am going to have a family dinner with naturally on my ear. Masr Davy, tis a joyful hour as I see you, I shall finish the Memorial when I have nothing else to do, and And now, I tried to tell her of the struggle I had had, and the his face, he looked, to me, as vigorous and robust, withal as confer; that I could not resign you to a dearer protector, of your copper-coloured woman in linen, with a bright handkerchief round If Sophy were your clerk, now, Traddles, she would have enough to She was up in my study, Peggotty said: which it was her pride to spectacles twice or thrice, to take another look at me, but as fourth daughter of Doctor Mell, were particularly remarkable. I must say more. I cannot let you leave me so. For Heavens sake, children were playing in the room, when I was told that a stranger It grows out of the night when Dora died. She sent you for me. and thrust out her little heap of golden curls from between the own choosing; that I could not, from my removed place, be a of his stay, - which, I think, was something less than a month, - heart, it has been lightened for me. If I have any secret, it is that your aunts the most extraordinary woman in the world, sir. these; all turning to me as I ask my thoughts the question. Traddless house is one of the very houses - or it easily may have Traddles about to be kissed, until he is out of breath. Here, I had advanced in fame and fortune, my domestic joy was perfect, I had run to bring him in, and I had not yet clearly seen his face, She was going away, but I detained her. I clasped my arm about her me. He is dead. Rosa kneeling at her feet, by turns caresses her, -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: kgzaouz.gif Type: image/gif Size: 15552 bytes Desc: not available URL: From ujrkytk at mitre.org Thu Dec 29 21:36:06 2005 From: ujrkytk at mitre.org (Micheal Harden) Date: Thu Dec 29 21:36:06 2005 Subject: [Numpy-discussion] Re[1]: Message-ID: <0006$01c61553$01c0d2ff@Ricardo> An HTML attachment was scrubbed... URL: From vrmricvq at seacoastonline.com Fri Dec 30 01:25:09 2005 From: vrmricvq at seacoastonline.com (Pamela Gonzalez) Date: Fri Dec 30 01:25:09 2005 Subject: [Numpy-discussion] Re[3]: Message-ID: <0012$021f9877$0bec16fb@PCS24> An HTML attachment was scrubbed... URL: From cjw at sympatico.ca Fri Dec 30 07:36:07 2005 From: cjw at sympatico.ca (Colin J. Williams) Date: Fri Dec 30 07:36:07 2005 Subject: [Numpy-discussion] Testing scipy core 0.8.4 Message-ID: <43B553B9.7090408@sympatico.ca> Are the tests operational at this stage of development? Colin W. From soospzm at cognos.com Fri Dec 30 09:39:13 2005 From: soospzm at cognos.com (Deanna Vela) Date: Fri Dec 30 09:39:13 2005 Subject: [Numpy-discussion] Up $1.07 In Last 7 days Message-ID: <000a$01f31a33$0401255b@Hnpc1> An HTML attachment was scrubbed... URL: From fhtyufhl at seaworld.com Fri Dec 30 12:39:19 2005 From: fhtyufhl at seaworld.com (Ted Pierson) Date: Fri Dec 30 12:39:19 2005 Subject: [Numpy-discussion] Up $1.07 In Last 7 days Message-ID: <0003$01be32b3$0c36fe42@Hewitt> An HTML attachment was scrubbed... URL: From cjw at sympatico.ca Fri Dec 30 13:27:08 2005 From: cjw at sympatico.ca (Colin J. Williams) Date: Fri Dec 30 13:27:08 2005 Subject: [Numpy-discussion] Problem with scipy core 0.8.4 Message-ID: <43B5A617.2070609@sympatico.ca> *** Python 2.4.2 (#67, Sep 28 2005, 12:41:11) [MSC v.1310 32 bit (Intel)] on win32. *** >>> import scipy.base.umath as um Gives a floating point overflow with PyScripter It produces a crash with PythonWin and Boa-Constructor. With the interpreter, it appears to work OK C:\>python Python 2.4.2 (#67, Sep 28 2005, 12:41:11) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import scipy.base.umath as um >>> um >>> All of the above were on an XP Home. Colin W. PS Is there a more appropriate place to report bugs? From mppulqihuj at gentex.com Sat Dec 31 02:40:10 2005 From: mppulqihuj at gentex.com (Tonia Cohen) Date: Sat Dec 31 02:40:10 2005 Subject: [Numpy-discussion] This one will go crazy on Tuesday Message-ID: <000d$01ace213$0f96071c@laptop> UBTA- Brand new sto ck for your attention UBA TECHNOLOGY INC- Symbol: UBTA A company which provide high quality software solutions built with cutting-edge technology to the online betting exchange and online gaming industries. Current Price- 0.085 Will it Continue Higher? Review Exactly What this Company Does and Watch This One trade on Tuesday as We Know Many of You Like Momentum. Recent h0t news!! UBA Technology Inc. Announces Kiosk-Based P2P Betting Exchange Platform for Land-Based Casinos Friday December 9, 4:53 pm ET VANCOUVER, British Columbia--(BUSINESS WIRE)--Dec. 9, 2005 --UBA Technology Inc. (Pink Sheets:UBTA - News) is targeting land-based casinos as a new market for its proprietary betting exchange software. UBA has commenced negotiations to install its betting exchange software, in stand-alone self operating kiosks, at traditional land-based casino locations. In November 2002, the Las Vegas Sun reported that gaming regulators in Nevada had approved a sports- and race-betting kiosk concept, which operates similarly to an automatic teller machine. The kiosk will provide larger casinos an opportunity to increase their sports-betting volume by strategically placing the kiosks in well-travelled casino areas. These betting systems have the potential to include wireless devices, allowing casino gamblers to place bets poolside, or from their hotel rooms. With high-speed broadband and wireless services now available on a global basis, UBA is well positioned to market its betting exchange platform to qualified online operators. This software would give online operators the ability to generate additional income streams from their current casino and poker clientele. Juniper Research estimates that mobile gambling will be a US$16.6 billion business by 2008. Geographical and cultural forces are also at work on these projections, particularly in the Asia-Pacific regions (driven largely by China). The popularity of gambling, coupled with the rapid growth in mobile phone penetration in the next five years, should assist UBA in its plans to leverage its internet-based betting exchange client base. The timing for UBA Technology's entry into the market place is excellent as the global gaming market continues to expand not only for online poker companies (Party Poker -- Party Gaming -- PRTY.L) and traditional bricks and mortar casinos (Wynn Resorts -- WYNN), but also for online gaming software providers (Cryptologic Inc. -- CRYP.Q).