From eric at scipy.org Wed Jan 2 18:00:36 2002 From: eric at scipy.org (eric) Date: Wed, 2 Jan 2002 18:00:36 -0500 Subject: [SciPy-user] pre-release peek of compiler... and I need help with a name Message-ID: <039701c193e1$4a384190$777ba8c0@ericlaptop> Hey folks, Happy new year to all. In celebration, I had planned on releasing a new version of the scipy.compiler module, but as a stand alone package since it is useful outside the scientific community. So, the module was to be released as 'compiler'. Unfortunately, at the 11th hour, I just noticed that name is already taken in 2.2. Doh! So, I need a new name. The module has three main functions (or modules). inline() -- allows you to inline C/C++ code in Python. blitz() -- translates python Numeric expressions into Blitz++ (C++) expressions for fast execution. ext_tools -- Module useful for building Python extension modules. You can look here at the temporary site (until I get the real name...) http://www.scipy.org/site_content/compiler/ Please let me know if you have a suitable name. Travis V. has the front runner: cwrap Putting that along side cow.py is a little much though... so we're still looking. Also, there are still tons of typos, etc. in the docs(Trav suggests running a find/replace on your/you're would help. :| ) Let me know how the tests fair if anyone gets a chance to try it, and we'll hopefully have things shaped up with a new name for release by the end of the week. eric From staelin at exch.hpl.hp.com Thu Jan 3 02:26:37 2002 From: staelin at exch.hpl.hp.com (Staelin, Carl) Date: Wed, 2 Jan 2002 23:26:37 -0800 Subject: [SciPy-user] pre-release peek of compiler... and I need help with a name Message-ID: <40700B4C02ABD5119F00009027876644014B83FB@hplex1.hpl.hp.com> Why not one of: py2c ccompiler c Cheers, Carl _________________________________________________ [(hp)] Carl Staelin Senior Research Scientist Hewlett-Packard Laboratories Technion City Haifa, 32000 ISRAEL +972(4)823-1237x221 +972(4)822-0407 fax staelin at hpl.hp.com _______http://www.hpl.hp.com/personal/Carl_Staelin_______ > -----Original Message----- > From: eric [mailto:eric at scipy.org] > Sent: Thursday, January 03, 2002 1:01 AM > To: scipy-user at scipy.org; scipy-dev at scipy.org > Subject: [SciPy-user] pre-release peek of compiler... and I need help > with a name > > > Hey folks, > > Happy new year to all. In celebration, I had planned on > releasing a new > version of the scipy.compiler module, but as a stand alone > package since it > is useful outside the scientific community. So, the module was to be > released as 'compiler'. Unfortunately, at the 11th hour, I > just noticed > that name is already taken in 2.2. Doh! > > So, I need a new name. The module has three main functions > (or modules). > > inline() -- allows you to inline C/C++ code in Python. > blitz() -- translates python Numeric expressions into Blitz++ (C++) > expressions for fast execution. > ext_tools -- Module useful for building Python extension modules. > > You can look here at the temporary site (until I get the real name...) > > http://www.scipy.org/site_content/compiler/ > > Please let me know if you have a suitable name. Travis V. > has the front > runner: > > cwrap > > Putting that along side cow.py is a little much though... so > we're still > looking. > > Also, there are still tons of typos, etc. in the docs(Trav > suggests running > a find/replace on your/you're would help. :| ) Let me know > how the tests > fair if anyone gets a chance to try it, and we'll hopefully > have things > shaped up with a new name for release by the end of the week. > > eric > > > > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.net > http://www.scipy.net/mailman/listinfo/scipy-user > From eric at scipy.org Thu Jan 3 11:32:53 2002 From: eric at scipy.org (eric) Date: Thu, 3 Jan 2002 11:32:53 -0500 Subject: [SciPy-user] pre-release peek of compiler... and I need help with a name References: <40700B4C02ABD5119F00009027876644014B83FB@hplex1.hpl.hp.com> Message-ID: <041e01c19474$4af4f810$777ba8c0@ericlaptop> Thanks Carl and others, All would work. c would be short and sweet, but to many people use c as a variable. I think py2c is already taken by another project. Thanks for all the suggestions everyone. In the end, it came down to 'bicep' and 'weave', and 'weave' won. Why? It weaves snippets of C/C++ into Python, it wasn't taken, and I liked it. So, I'll change the names, fix a few bugs that were found and hopefully release within the week. see ya, eric ----- Original Message ----- From: "Staelin, Carl" To: Sent: Thursday, January 03, 2002 2:26 AM Subject: RE: [SciPy-user] pre-release peek of compiler... and I need help with a name > Why not one of: > py2c > ccompiler > c > > Cheers, > > Carl > > _________________________________________________ > [(hp)] Carl Staelin > Senior Research Scientist > Hewlett-Packard Laboratories > Technion City > Haifa, 32000 > ISRAEL > +972(4)823-1237x221 +972(4)822-0407 fax > staelin at hpl.hp.com > _______http://www.hpl.hp.com/personal/Carl_Staelin_______ > > > > -----Original Message----- > > From: eric [mailto:eric at scipy.org] > > Sent: Thursday, January 03, 2002 1:01 AM > > To: scipy-user at scipy.org; scipy-dev at scipy.org > > Subject: [SciPy-user] pre-release peek of compiler... and I need help > > with a name > > > > > > Hey folks, > > > > Happy new year to all. In celebration, I had planned on > > releasing a new > > version of the scipy.compiler module, but as a stand alone > > package since it > > is useful outside the scientific community. So, the module was to be > > released as 'compiler'. Unfortunately, at the 11th hour, I > > just noticed > > that name is already taken in 2.2. Doh! > > > > So, I need a new name. The module has three main functions > > (or modules). > > > > inline() -- allows you to inline C/C++ code in Python. > > blitz() -- translates python Numeric expressions into Blitz++ (C++) > > expressions for fast execution. > > ext_tools -- Module useful for building Python extension modules. > > > > You can look here at the temporary site (until I get the real name...) > > > > http://www.scipy.org/site_content/compiler/ > > > > Please let me know if you have a suitable name. Travis V. > > has the front > > runner: > > > > cwrap > > > > Putting that along side cow.py is a little much though... so > > we're still > > looking. > > > > Also, there are still tons of typos, etc. in the docs(Trav > > suggests running > > a find/replace on your/you're would help. :| ) Let me know > > how the tests > > fair if anyone gets a chance to try it, and we'll hopefully > > have things > > shaped up with a new name for release by the end of the week. > > > > eric > > > > > > > > _______________________________________________ > > SciPy-user mailing list > > SciPy-user at scipy.net > > http://www.scipy.net/mailman/listinfo/scipy-user > > > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.net > http://www.scipy.net/mailman/listinfo/scipy-user > From jochen at unc.edu Thu Jan 3 12:47:28 2002 From: jochen at unc.edu (Jochen =?iso-8859-1?q?K=FCpper?=) Date: 03 Jan 2002 12:47:28 -0500 Subject: [SciPy-user] pre-release peek of compiler... and I need help with a name In-Reply-To: <041e01c19474$4af4f810$777ba8c0@ericlaptop> References: <40700B4C02ABD5119F00009027876644014B83FB@hplex1.hpl.hp.com> <041e01c19474$4af4f810$777ba8c0@ericlaptop> Message-ID: <864rm31g3z.fsf@bock.chem.unc.edu> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Thu, 3 Jan 2002 11:32:53 -0500 eric wrote: eric> Thanks for all the suggestions everyone. In the end, it came eric> down to 'bicep' and 'weave', and 'weave' won. Why? It weaves eric> snippets of C/C++ into Python, it wasn't taken, and I liked it. 'weave' is already used in the TeX / web world. Well, that's small con 'cause you have somewhat of a name-clash, and a big pro 'cause you you can only profit from TeX's quality name:)) Btw, I like it. Greetings, Jochen - -- University of North Carolina phone: +1-919-962-4403 Department of Chemistry phone: +1-919-962-1579 Venable Hall CB#3290 (Kenan C148) fax: +1-919-843-6041 Chapel Hill, NC 27599, USA GnuPG key: 44BCCD8E -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6-cygwin-fcn-1 (Cygwin) Comment: Processed by Mailcrypt and GnuPG iD8DBQE8NJkxiJ/aUUS8zY4RAkOxAJ9lxBlWSLzApu0k9PI4uPxVcYHIHQCfVr6N wpA+Z+Bvrl7sNZrD41fU+J8= =hs0P -----END PGP SIGNATURE----- From nadavh at envision.co.il Sat Jan 5 17:55:32 2002 From: nadavh at envision.co.il (Nadav Horesh) Date: Sun, 06 Jan 2002 00:55:32 +0200 Subject: [SciPy-user] Installation with Python2.2 on RH7.1 Message-ID: <3C378464.D7682DBA@envision.co.il> Hi! I previously installed scipy with python2.1 and it was smooth. Recently I am trying to move on to python2.2 on the same machines (PIII (one with one CPU, the other with 2) running RH Linux 7.1), and encounter the same problems: 1. Can not find the FFTW single precision libraries (although I compiled FFTW with --enable float) 2. When I removed the single precision libraries from setup.py, the installation went OK, but I am getting the following message when trying to import scipy: >>> import scipy Traceback (most recent call last): File "", line 1, in ? File "/usr/local/lib/python2.2/site-packages/scipy/__init__.py", line 41, in ? from handy import * File "/usr/local/lib/python2.2/site-packages/scipy/handy.py", line 2, in ? import scipy.fft File "/usr/local/lib/python2.2/site-packages/scipy/fft/__init__.py", line 3, in ? from FFT2 import * File "/usr/local/lib/python2.2/site-packages/scipy/fft/FFT2.py", line 3, in ? import Numeric, fftw, types ImportError: /usr/local/lib/python2.2/site-packages/scipy/fft/fftw.so: undefined symbol: fftw_threads_init By the way: In the first trial it could not find fftw-threades (although scipy is installed on those machines with python2.1 with the same fftw libraries), so I recompiled FFTW to add those libraries. Nadav. From h.jansen at fel.tno.nl Mon Jan 7 02:47:41 2002 From: h.jansen at fel.tno.nl (H Jansen) Date: Mon, 07 Jan 2002 08:47:41 +0100 Subject: [SciPy-user] "import scipy" fails with Python-2.2a4 References: Message-ID: <3C39529D.C0972873@fel.tno.nl> Thanks, that helped. Henk. Travis Oliphant wrote: > > > I've compiled the latest SciPy successfully. However, when importing the > > module this is the result: > > > "/fmnt/stbsvr/sdata/apps/inst/lib/python2.2/site-packages/scipy/stats/pstat.py", > > line 176 > > exec execstring > > SyntaxError: unqualified exec is not allowed in function 'colex' it > > contains a nested function with free variables > > >>> > > > > This is a documented error for Python 2.2 which changed the implementation > of exec. The pstat functions using exec must be re-written. Until this > is done, disable the stats package by not including it in the list of > subpackages to be imported in the main __init__.py file > > scipy/__init__.py > > -Travis > > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.net > http://www.scipy.net/mailman/listinfo/scipy-user -------------- next part -------------- A non-text attachment was scrubbed... Name: h.jansen.vcf Type: text/x-vcard Size: 472 bytes Desc: Card for H Jansen URL: From costasm at hotmail.com Mon Jan 7 17:50:48 2002 From: costasm at hotmail.com (Costas Malamas) Date: Mon, 07 Jan 2002 17:50:48 Subject: [SciPy-user] Kmeans help and C source Message-ID: Hi all, I need to use a modified K-means algorithm for a project and I was delighted to discover that SciPy includes a python wrapper for a kmeans() function. However, I am not quite following the kmeans() functionality (I am new to this clustering business, so this maybe a stupid newbie question): my docs tell me that kmeans should partition a dataset into k clusters. So, I expect vq.kmeans(dataset, 2) to return to me dataset split up into two "equivalent" datasets. However, anyway I feed my data into vq.kmeans() this doesn't happen (e.g. I feed it a 5x4 dataset and I get back two 5x1 vectors). My guess is that either this vq.kmeans() does something different --I confess to not understanding the docstring as the observation/codebook terminology has no parallel to the docs I've read-- or that I am not doing something right. Any pointers? Even some documentation on the algorithm would be great help. Secondly, as I mentioned above, I need a modified kmeans. However, I see no C/Fortran code in the src tarball or CVS that seems related to kmeans. Is the base code available? If so, is it hackable by a SWIG newbie? (I am aware of SWIG, but I have never used it for anything serious). Any and all info will be greatly appreciated :-) --and thanks for SciPy! Costas Malamas _________________________________________________________________ Get your FREE download of MSN Explorer at http://explorer.msn.com/intl.asp. From eric at scipy.org Tue Jan 8 10:32:07 2002 From: eric at scipy.org (eric) Date: Tue, 8 Jan 2002 10:32:07 -0500 Subject: [SciPy-user] Kmeans help and C source References: Message-ID: <0be901c19859$b2dbdff0$777ba8c0@ericlaptop> Hey Costas, > Secondly, as I mentioned above, I need a modified kmeans. However, I see no > C/Fortran code in the src tarball or CVS that seems related to kmeans. Is > the base code available? If so, is it hackable by a SWIG newbie? (I am > aware of SWIG, but I have never used it for anything serious). The vq algorithms are all available in Python versions in scipy.cluster.vq. The kmeans algorithm is written in pure Python, and should be easy to modify -- no SWIG necessary. The time consuming part of kmeans is the vector qunatization (vq) algorithm. It has both pure python and C implementations. If your interested in the algorithm, study the Python. The C code does the same thing, but is less obvious. The C version is available in a source distribution of scipy. It lives in scipy/cluster/vq/src. If you need to modify vq also, my advice is to do it in Python. It will be easy to move to C later on. I'm working on a more detailed note about this stuff, and its beginnings are at the end of this note. Here are is an example to at least get you rolling: -------------------------------------------------------- #ex1.py from scipy import * from scipy.cluster import vq def cluster_data(data,cluster_cnt,iter=20,thresh=1e-5): """ Group data into a number of common clusters data -- 2D array of data points. Each point is a row in the array. cluster_cnt -- The number of clusters to use iter -- number of iterations to use for kmeans algorithm thresh -- distortion threshold for kmeans algorithm return -- list of 2D arrays. Each array contains the data points that belong to a specific cluster. Uses kmeans algorithm to find the clusters. """ wh_data = vq.whiten(data) code_book,dist = vq.kmeans(wh_data,cluster_cnt,iter,thresh) code_ids, distortion = vq.vq(wh_data,code_book) clusters = [] for i in range(len(code_book)): cluster = compress(code_ids == i,data,0) clusters.append(cluster) return clusters if __name__ == "__main__": data = array(((400, 79, 5.4), (180, 76, 4.5), (28, 25, 30.), (270, 81, 5.0), (185, 78, 4.6))) clusters = cluster_data(data,2) for i in range(len(clusters)): print 'cluster %d:' % i print clusters[i] -------------------------------------------------------- My (partial) more detailed response follows: > --I confess to not understanding the docstring Sorry for the confusion. I'll try to explain thing more clearly. If it works, will use this as the doc. :) > However, I am not quite following the kmeans() functionality (I am new to > this clustering business, so this maybe a stupid newbie question): my docs > tell me that kmeans should partition a dataset into k clusters. So, I > expect vq.kmeans(dataset, 2) to return to me dataset split up into two > "equivalent" datasets. Splitting the data into two data sets is actually a two or three step process. Here's a complete example. "Observations" are just another name for a data point. The obs matrix is a 2D array of data points. For example if our data set includes height, weight, and 40-yard speed of football players, you might have the following (fictitous) data: obs: lb inches seconds ---------------------------- Refrigerator Perry | 400 79 5.4 Jerry Rice | 180 76 4.5 Zachary Jones | 28 25 30.0 Too Tall Jones | 270 81 5.0 Charlie Joiner | 185 78 4.6 The data above is the "obs" matrix. Each row in the 2D array is a data point, often called an "observation" or "sample". Each column is sometimes called the "features" of a player. Imagine, we want to split this data set into two "clusters", perhaps dividing the data into linemen and receivers. One way to find two "codes", one to represent each of these groups. (I think the term "code" comes from communication theory, but I'm not sure. Perhaps "classes" is more descriptive.) I watched enough games (observed enough players) to make an educated guess as to what these codes might be: possible code book: code lb inches seconds ----------------------------- receiver 0 | 180 75 4.8 lineman 1 | 260 76 5.5 So code 0 stands for a "typical" receiver and code 1 represents your typical lineman. "Vector quantization" is an algorithm that calculates the distance between a data point and every code in the "code book" to find the closest one (i.e. which class is the best match). In scipy.cluster, the vq module houses the vector quantization tools. vq.vq(obs,code_book) returns 2 arrays -- (1) the index (row in the code book) of the code nearest to each data point, and (2) the distance that each data point is from that code. code_book is always a 2D array. If obs is a 2D array, each row is a separate data point. # note I rounded some numbers to save space >>> obs = array(((400, 79, 5.4), ... (180, 76, 4.5), ... (28, 25, 30.), ... (270, 81, 5.0), ... (185, 78, 4.6))) >>> code_book = array(((180, 75, 4.8), ... (260, 76, 5.5))) >>> code_id, distortion = vq.vq(obs,code_book) >>> code_id array([1, 0, 0, 1, 0]) >>> distortion array([ 140.03, 1.045, 161.985, 11.192, 5.834]) code_id now tells what position each of the football players is most likely to play. Distortion is the distance, using sqrt( a^2 + b^2 + c^2), that each player is from the code (typical player) in their category. Low numbers mean the match is good. For example, vq tells us that Jerry Rice is receiver, and the low distortion means that it is pretty dang sure about that. code_id distortion ------------------------------ Refrigerator Perry 1 --> lineman 140.03 Jerry Rice 0 --> receiver 1.04 Zachary Jones 0 --> receiver 161.99 Too Tall Jones 1 --> lineman 11.19 Charlie Joiner 0 --> receiver 5.83 Most of the classifications make sense, but the distortions have some problems. Noteably that my 1 year old son is about as likely to be a receiver as R. Perry is to be a lineman. Looking at the data, it is obvious that R. Perry is a lineman. It isn't obvious where Zach falls because he's small (like a receiver) and slow (like a lineman). So we should be quite a bit more sure about the fact that R. Perry is a lineman. What's wrong? Well, the distortion value's primary contribution comes from the large weight differences between these players and the typical players. Even though Zach's speed is a long way from receiver's speed this doesn't contribute to the distortion much. That's bad because the speed difference carries a lot of useful information. For more accurate distortion values, we would like each feature of a player (weight, height, and speed) to be weighted equally. One way of doing this is to "whiten" the features of both the data and the code book by dividing each feature by the standard deviation of that feature set. The standard deviation of weights and speeds across all the players are: #weight >>> stats.stdev((400,180,28,270,185)) 136.30407183939883 #speed >>> stats.stdev((5.4,4.5,30.0,5.0,4.6)) 11.241885962773329 So the whitened weight and speed distortions for R. Perry from a typical lineman is: # whitened weight >>> abs(400-260)/136.3 1.0271460014673512 # whitened speed >>> abs(5.4-5.5)/11.24 0.0088967971530248789 So the whitened weight and speed distortions for Z. Jones from a typical receiver is: # whitened weight >>> abs(28-180)/136.3 1.1151870873074101 # whitened speed >>> (30.0-4.8)/11.24 2.2419928825622777 It is apparent from these values that Zach's speed difference is gonna have a large affect on the distortion now. The whiten() function handles the chore of whitening a data set. >>> wh_obs = whiten(obs) Usually, the code book is actually calculated from a whitened set of data (via kmeans or some other method), and thus are automatically white. In this case, I specified the code_book in non-whitened units, so they'll need to be normalized by the same standard deviation of the data set. # not normally needed >>> wh_code_book = code_book / std(obs,axis=0) Now, rerunning vq gives: >>> code_id, distortion = vq.vq(wh_obs,wh_code_book) >>> code_id array([1, 0, 0, 1, 0]) >>> distortion array([ 1.034, 0.049, 3.257 , 0.225, 0.131]) code_id whitened distortion --------------------------------------- Refrigerator Perry 1 --> lineman 1.034 Jerry Rice 0 --> receiver 0.049 Zachary Jones 0 --> receiver 3.257 Too Tall Jones 1 --> lineman 0.225 Charlie Joiner 0 --> receiver 0.131 Now Zach's distortion is much higher than everyone elses which makes sense. In the example above, I made an educated guess based on my knowledge of football of what the size and speed of a typical player in a given position might be. This is information I had to supply to the clustering algorithm before it could determine how to classify each player. Suppose you didn't know anything about football, and watched a football gamebut were given a list of players with there position, weight, height, and speed. You could use this information to educate yourself about the traits of a typical receiver, linebacker, lineman, etc. The kmeans algorithm also does this. It takes a set of data, and the number of positions you want to categorize, and spits out an appropriate code book. ----- Original Message ----- From: "Costas Malamas" To: Sent: Monday, January 07, 2002 5:50 PM Subject: [SciPy-user] Kmeans help and C source > Hi all, > > I need to use a modified K-means algorithm for a project and I was delighted > to discover that SciPy includes a python wrapper for a kmeans() function. > > However, I am not quite following the kmeans() functionality (I am new to > this clustering business, so this maybe a stupid newbie question): my docs > tell me that kmeans should partition a dataset into k clusters. So, I > expect vq.kmeans(dataset, 2) to return to me dataset split up into two > "equivalent" datasets. However, anyway I feed my data into vq.kmeans() this > doesn't happen (e.g. I feed it a 5x4 dataset and I get back two 5x1 > vectors). > My guess is that either this vq.kmeans() does something different > --I confess to not understanding the docstring as the observation/codebook > terminology has no parallel to the docs I've read-- or that I am not doing > something right. Any pointers? Even some documentation on the algorithm > would be great help. > > Secondly, as I mentioned above, I need a modified kmeans. However, I see no > C/Fortran code in the src tarball or CVS that seems related to kmeans. Is > the base code available? If so, is it hackable by a SWIG newbie? (I am > aware of SWIG, but I have never used it for anything serious). > > Any and all info will be greatly appreciated :-) --and thanks for SciPy! > > > Costas Malamas > -------------- next part -------------- An HTML attachment was scrubbed... URL: From costasm at hotmail.com Tue Jan 8 17:12:43 2002 From: costasm at hotmail.com (Costas Malamas) Date: Tue, 08 Jan 2002 17:12:43 Subject: [SciPy-user] Kmeans help and C source Message-ID: Eric, This is more lucid than anything I've read so far on the Web. Seriously, this should be in the docstring :-)... I still need to grok where and what to modify, but at least now I know that I don't have to look outside of SciPy ;-) Many thanks, Costas _________________________________________________________________ MSN Photos is the easiest way to share and print your photos: http://photos.msn.com/support/worldwide.aspx From h.jansen at fel.tno.nl Thu Jan 10 08:54:05 2002 From: h.jansen at fel.tno.nl (H Jansen) Date: Thu, 10 Jan 2002 14:54:05 +0100 Subject: [SciPy-user] Nonlinear Constraint Optimization. Message-ID: <3C3D9CFD.C20B0628@fel.tno.nl> I've a nonlinear optimal (trajectory) control problem to solve and have looked around for public domain (C++) code. The best I could find was Omuses (time-continuous front end) that is based on HQP ("Huge" sequential "Quadratic" "Program"/solver that can directly be used for time-discrete problems; it employes sparsity using the sparse meschach code which is in C). Solving a system with Omuses/HQP involves writing a class (that inherits from Omuses_Problem or HQP_Problem) with a set of differential (or difference) equations in C++, linking them with the library; the solution process is initialized/controled with Tcl scripts. Wouldn't it be wonderful if this all could be done from within a Python environment? To that end, the following Python interfaces should be available: 1. One to meschach, and the partially implemented C++ interface, meschach++, may work as a start; 2. One to the Hqp_Program class; 3. One to the Hqp_SeqProgram class; 4. One to the Omuses_Program class; I've tried to make a start with meschach using SWIG which is geared towards C, creating functions that can be wrapped by shadow classes in Python. However, the boost.python library also provides an "interface builder" which provides a tighter coupling between C/C++ and Python. After a while, I had to give up, because I feel I'm lacking the experience to make the correct decisions. Preferably, I would team up with other people that may be interested and join into a shared project in which we can share ideas and build things quicker. Are there people interested? I'm open to all ideas. Kind regards, Henk Jansen. HQP link: http://www.systemtechnik.tu-ilmenau.de/~fg_opt/omuses/hqp.html Omuses link: http://www.systemtechnik.tu-ilmenau.de/~fg_opt/omuses/omuses.html Other links to Optimization Software: http://www-fp.mcs.anl.gov/otc/Guide/SoftwareGuide/index.html -------------- next part -------------- A non-text attachment was scrubbed... Name: h.jansen.vcf Type: text/x-vcard Size: 472 bytes Desc: Card for H Jansen URL: From pearu at cens.ioc.ee Thu Jan 10 09:31:03 2002 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Thu, 10 Jan 2002 16:31:03 +0200 (EET) Subject: [SciPy-user] Nonlinear Constraint Optimization. In-Reply-To: <3C3D9CFD.C20B0628@fel.tno.nl> Message-ID: On Thu, 10 Jan 2002, H Jansen wrote: > Solving a system with Omuses/HQP involves writing a class (that inherits > from Omuses_Problem or HQP_Problem) with a set of differential (or > difference) equations in C++, linking them with the library; the > solution process is initialized/controled with Tcl scripts. > > Wouldn't it be wonderful if this all could be done from within a Python > environment? To that end, the following Python interfaces should be > available: > > 1. One to meschach, and the partially implemented C++ interface, > meschach++, may work as a start; > 2. One to the Hqp_Program class; > 3. One to the Hqp_SeqProgram class; > 4. One to the Omuses_Program class; > > I've tried to make a start with meschach using SWIG which is geared > towards C, creating functions that can be wrapped by shadow classes in > Python. However, the boost.python library also provides an "interface > builder" which provides a tighter coupling between C/C++ and Python. > > After a while, I had to give up, because I feel I'm lacking the > experience to make the correct decisions. Preferably, I would team up > with other people that may be interested and join into a shared project > in which we can share ideas and build things quicker. > > Are there people interested? I'm open to all ideas. If the supporting library for solving the problem in the subject is in C++ then I would recommend boost.python for exposing C++ to Python. Though to get started with boost.python might seem difficult (I am not sure if it is easier with other C++/Python binding tools) but not impossible (I started with many concers about C++ that just fade away with using g++ compiler), with the right design decision of the interface, it will be very small and easy to maintain. Using SWIG for wrapping C++ classes is not very good idea, I think. I have this good experience with interfacing GiNaC C++ library to Python that provides classes for doing symbolic algebra computations. See http://cens.ioc.ee/projects/pyginac/ (I suggest getting it from CVS as the tar balls are very outdated, in case you are interested in the approach). Regards, Pearu From eric at scipy.org Thu Jan 10 12:03:13 2002 From: eric at scipy.org (eric) Date: Thu, 10 Jan 2002 12:03:13 -0500 Subject: [SciPy-user] Nonlinear Constraint Optimization. References: Message-ID: <00bf01c199f8$b059d160$6b01a8c0@ericlaptop> Depending upon what the Omuses/HQP library looks like, SWIG may handle it fine. After all, wxPython uses it, and it is a huge library. Also, David B. has released a new version recently and I think it supports C++ much better. If Omuses/HQP has a large number of overloaded procedures, then SWIG used to not handle this -- at least automatically. I still don't think it is automatic, but it can do it. I'm pretty sure this is why the VTK guys didn't use SWIG as VTK overloads functions many many many times... On the other hand, if the classes don't have many overloaded functions and operators, then SWIG will handle it pretty well. You'll still have to write the typemaps though, which is an adventure no matter how you cut it. I've never used boost. It may be better for the task. I just didn't want SWIG dismissed as not handling C++ well. One other option I've noticed is the Cable project associated with VTK. The gcc-xml tool looks really cool. I doubt this is the fastest way to wrappers -- use one of the above -- but it is worth keeping an eye on. http://public.kitware.com/Cable/ eric ----- Original Message ----- From: "Pearu Peterson" To: "SciPy" Sent: Thursday, January 10, 2002 9:31 AM Subject: Re: [SciPy-user] Nonlinear Constraint Optimization. > > On Thu, 10 Jan 2002, H Jansen wrote: > > > Solving a system with Omuses/HQP involves writing a class (that inherits > > from Omuses_Problem or HQP_Problem) with a set of differential (or > > difference) equations in C++, linking them with the library; the > > solution process is initialized/controled with Tcl scripts. > > > > Wouldn't it be wonderful if this all could be done from within a Python > > environment? To that end, the following Python interfaces should be > > available: > > > > 1. One to meschach, and the partially implemented C++ interface, > > meschach++, may work as a start; > > 2. One to the Hqp_Program class; > > 3. One to the Hqp_SeqProgram class; > > 4. One to the Omuses_Program class; > > > > I've tried to make a start with meschach using SWIG which is geared > > towards C, creating functions that can be wrapped by shadow classes in > > Python. However, the boost.python library also provides an "interface > > builder" which provides a tighter coupling between C/C++ and Python. > > > > After a while, I had to give up, because I feel I'm lacking the > > experience to make the correct decisions. Preferably, I would team up > > with other people that may be interested and join into a shared project > > in which we can share ideas and build things quicker. > > > > Are there people interested? I'm open to all ideas. > > If the supporting library for solving the problem in the subject is in > C++ then I would recommend boost.python for exposing C++ to Python. Though > to get started with boost.python might seem difficult (I am not sure if it > is easier with other C++/Python binding tools) but not impossible (I > started with many concers about C++ that just fade away with using g++ > compiler), with the right design decision of the interface, it will be > very small and easy to maintain. > Using SWIG for wrapping C++ classes is not very good idea, I think. > > I have this good experience with interfacing GiNaC C++ library to > Python that provides classes for doing symbolic algebra computations. > See > http://cens.ioc.ee/projects/pyginac/ > (I suggest getting it from CVS as the tar balls are very outdated, in > case you are interested in the approach). > > > Regards, > Pearu > > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.net > http://www.scipy.net/mailman/listinfo/scipy-user > From eric at scipy.org Thu Jan 10 12:26:46 2002 From: eric at scipy.org (eric) Date: Thu, 10 Jan 2002 12:26:46 -0500 Subject: [SciPy-user] Kmeans help and C source References: Message-ID: <011401c199fb$fb5f1c80$6b01a8c0@ericlaptop> ----- Original Message ----- From: "Costas Malamas" To: Cc: Sent: Tuesday, January 08, 2002 5:12 PM Subject: Re: [SciPy-user] Kmeans help and C source > Eric, > > This is more lucid than anything I've read so far on the Web. Seriously, > this should be in the docstring :-)... > > I still need to grok where and what to modify, but at least now I know that > I don't have to look outside of SciPy ;-) Hey Costas, I've attached a slightly modified version of scipy/cluster/vq.py that takes out the shennanigans used to attempt using the C version of vq. The result is a pure Python kmeans algorithm in 80 lines of code (The rest is doc-strings). Assuming you have SciPy installed, just dump this file into your PYTHONPATH and start hacking. It won't be as fast as the version that uses C, but this shouldn't matter during development. If your modifying your distance metric, then you'll need to modify py_vq(). If your just modifying how the kmeans uses the distortion information, etc. you'll need to modify just the kmeans() or the kmeans_() helper functions. hope that helps, eric ------------------------------------------------- # pure_python_vq.py """ Vector Quantization Module Provides several routines used in creating a code book from a set of observations and comparing a set of observations to a code book. All routines expect an "observation vector" to be stored in each row of the obs matrix. Similarly the codes are stored row wise in the code book matrix. whiten(obs) -- Normalize a group of observations on a per feature basis vq(obs,code_book) -- Calculate code book membership of obs kmeans(obs,k_or_guess,iter=20,thresh=1e-5) -- Train a codebook for mimimum distortion using the kmeans algorithm """ from Numeric import * from fastumath import * from RandomArray import randint import scipy def whiten(obs): """ Normalize a group of observations on a per feature basis Description Before running kmeans algorithms, it is beneficial to "whiten", or scale, the observation data on a per feature basis. This is done by dividing each feature by its standard deviation across all observations. Arguments obs -- 2D array. Each row of the array is an observation. The columns are the "features" seen during each observation # f0 f1 f2 obs = [[ 1., 1., 1.], #o0 [ 2., 2., 2.], #o1 [ 3., 3., 3.], #o2 [ 4., 4., 4.]]) #o3 XXX perhaps should have an axis variable here. Outputs result -- 2D array. Contains the values in obs scaled by the standard devation of each column. Test >>> features = array([[ 1.9,2.3,1.7], ... [ 1.5,2.5,2.2], ... [ 0.8,0.6,1.7,]]) >>> whiten(features) array([[ 3.41250074, 2.20300046, 5.88897275], [ 2.69407953, 2.39456571, 7.62102355], [ 1.43684242, 0.57469577, 5.88897275]]) """ std_dev = scipy.std(obs,axis=0) return obs / std_dev def vq(obs,code_book): """ Vector Quantization: assign features sets to codes in a code book. Description: Vector quantization determines which code in the code book best represents an observation of a target. The features of each observation are compared to each code in the book, and assigned the one closest to it. The observations are contained in the obs array. These features should be "whitened," or nomalized by the standard deviation of all the features before being quantized. The code book can be created using the kmeans algorithm or something similar. Note: This currently forces 32 bit math precision for speed. Anyone know of a situation where this undermines the accuracy of the algorithm? Arguments: obs -- 2D array. Each row of the array is an observation. The columns are the "features" seen during each observation The features must be whitened first using the whiten function or something equivalent. code_book -- 2D array. The code book is usually generated using the kmeans algorithm. Each row of the array holds a different code, and the columns are the features of the code. # c0 c1 c2 c3 code_book = [[ 1., 2., 3., 4.], #f0 [ 1., 2., 3., 4.], #f1 [ 1., 2., 3., 4.]]) #f2 Outputs: code -- 1D array. If obs is a NxM array, then a length M array is returned that holds the selected code book index for each observation. dist -- 1D array. The distortion (distance) between the observation and its nearest code Reference Test >>> code_book = array([[1.,1.,1.], ... [2.,2.,2.]]) >>> features = array([[ 1.9,2.3,1.7], ... [ 1.5,2.5,2.2], ... [ 0.8,0.6,1.7]]) >>> vq(features,code_book) (array([1, 1, 0],'i'), array([ 0.43588989, 0.73484692, 0.83066239])) """ results = py_vq(obs,code_book) return results def py_vq(obs,code_book): """ Python version of vq algorithm. This function is slower than the C versions, but it works for all input types. If the inputs have the wrong types for the C versions of the function, this one is called as a last resort. Its about 20 times slower than the C versions. """ No,Nf = shape(obs) #No = observation count, Nf = feature count # code books and observations should have same number of features assert(Nf == code_book.shape[1]) code = [];min_dist = [] #create a memory block to use for the difference calculations diff = zeros(shape(code_book),_common_type(obs,code_book)) for o in obs: subtract(code_book,o,diff) # faster version of --> diff = code_book - o dist = sqrt(sum(diff*diff,-1)) code.append(argmin(dist)) #something weird here dst does not work reliably because it sometime #returns an array of goofy length. Try except fixes it, but is ugly. dst = minimum.reduce(dist,0) try: dst = dst[0] except: pass min_dist.append(dst) return array(code,typecode=Int), array(min_dist) def kmeans_(obs,guess,thresh=1e-5): """ See kmeans Outputs code_book -- the lowest distortion codebook found. avg_dist -- the average distance a observation is from a code in the book. Lower means the code_book matches the data better. Test Note: not whitened in this example. >>> features = array([[ 1.9,2.3], ... [ 1.5,2.5], ... [ 0.8,0.6], ... [ 0.4,1.8], ... [ 1.0,1.0]]) >>> book = array((features[0],features[2])) >>> kmeans_(features,book) (array([[ 1.7 , 2.4 ], [ 0.73333333, 1.13333333]]), 0.40563916697728591) """ code_book = array(guess,copy=1) Nc = code_book.shape[0] avg_dist=[] diff = thresh+1. while diff>thresh: #compute membership and distances between obs and code_book obs_code, distort = vq(obs,code_book) avg_dist.append(scipy.mean(distort,axis=-1)) #recalc code_book as centroids of associated obs if(diff > thresh): has_members = [] for i in arange(Nc): cell_members = compress(equal(obs_code,i),obs,0) if cell_members.shape[0] > 0: code_book[i] = scipy.mean(cell_members,0) has_members.append(i) #remove code_books that didn't have any members code_book = take(code_book,has_members,0) if len(avg_dist) > 1: diff = avg_dist[-2] - avg_dist[-1] #print avg_dist return code_book, avg_dist[-1] def kmeans(obs,k_or_guess,iter=20,thresh=1e-5): """ Generate a code book with minimum distortion Description Arguments obs -- 2D array Each row of the array is an observation. The columns are the "features" seen during each observation The features must be whitened first using the whiten function or something equivalent. k_or_guess -- integer or 2D array. If integer, it is the number of code book elements. If a 2D array, the array is used as the intial guess for the code book. The array should have k rows, and the same number of columns (features) as the obs array. iter -- integer. The number of times to restart the kmeans algorithm with a new initial guess. If k_or_guess is a 2D array (codebook), this argument is ignored and only 1 iteration is run. thresh -- float Terminate each kmeans run when the distortion change from one iteration to the next is less than this value. Outputs codesbook -- 2D array. The codes that best fit the observation distortion -- float The distortion between the observations and the codes. Reference Test ("Not checked carefully for accuracy..." he said sheepishly) >>> features = array([[ 1.9,2.3], ... [ 1.5,2.5], ... [ 0.8,0.6], ... [ 0.4,1.8], ... [ 0.1,0.1], ... [ 0.2,1.8], ... [ 2.0,0.5], ... [ 0.3,1.5], ... [ 1.0,1.0]]) >>> whitened = whiten(features) >>> book = array((whitened[0],whitened[2])) >>> kmeans(whitened,book) (array([[ 2.3110306 , 2.86287398], [ 0.93218041, 1.24398691]]), 0.85684700941625547) >>> import RandomArray >>> RandomArray.seed(1000,2000) >>> codes = 3 >>> kmeans(whitened,codes) (array([[ 2.3110306 , 2.86287398], [ 1.32544402, 0.65607529], [ 0.40782893, 2.02786907]]), 0.5196582527686241) """ if int(iter) < 1: raise ValueError, 'iter must be >= to 1.' if type(k_or_guess) == type(array([])): guess = k_or_guess result = kmeans_(obs,guess,thresh=thresh) else: best_dist = 100000 #initialize best distance value to a large value No = obs.shape[0] k = k_or_guess #print 'kmeans iter: ', for i in range(iter): #print i, #the intial code book is randomly selected from observations guess = take(obs,randint(0,No,k),0) book,dist = kmeans_(obs,guess,thresh=thresh) if dist < best_dist: best_book = book best_dist = dist #print result = best_book,best_dist return result From costasm at hotmail.com Fri Jan 11 22:02:58 2002 From: costasm at hotmail.com (Costas Malamas) Date: Fri, 11 Jan 2002 22:02:58 Subject: [SciPy-user] Kmeans help and C source Message-ID: Thanks Eric, this will help. I do need to modify the distance metric. However, I am now stuck way before kmeans so it will be a coupla weeks before I attempt it. At some point, I will probably need this in C... I know I am looking too far ahead, but probably my vector-space will not fit into RAM. Does anybody know any tricks for clustering "pages" of the v-space or otherwise fit an 1e8 observations cube into a kmeans algorithm? Thanks, C. _________________________________________________________________ Get your FREE download of MSN Explorer at http://explorer.msn.com/intl.asp. From eric at scipy.org Fri Jan 11 17:17:02 2002 From: eric at scipy.org (eric) Date: Fri, 11 Jan 2002 17:17:02 -0500 Subject: [SciPy-user] Kmeans help and C source References: Message-ID: <01fe01c19aed$b22b18a0$6b00a8c0@ericlaptop> > At some point, I will probably need this in C... That transition will not be difficult. > I know I am looking too far > ahead, but probably my vector-space will not fit into RAM. Does anybody > know any tricks for clustering "pages" of the v-space or otherwise fit an > 1e8 observations cube into a kmeans algorithm? Is your data fairly uniformly distributed, i.e. if you take the first 1e6 values do they represent the entire data set fairly well? If so, you can run kmeans on this subset of data to find your code_book, and then read in the rest of the data in chunks to do your classification. If you need to run kmeans on the entire 1e8 space, then you'll have to do this "chunking" within the kmeans algorithm, and it will become more complicated. Your also talking about long run times. vq is pretty time intensive, and kmeans calls vq a ton of times. One other thing. The new NumArray package that is the next generation of NumPy has memory mapped arrays built into it. A very early .0x release is available at the NumPy site. Probably not usable in production yet at all, but sounds like it might match your needs. see ya, eric From eric at scipy.org Sat Jan 12 16:03:59 2002 From: eric at scipy.org (eric) Date: Sat, 12 Jan 2002 16:03:59 -0500 Subject: [SciPy-user] New tarballs for weave Message-ID: <02fc01c19bac$a8407000$6b00a8c0@ericlaptop> Hello group, New weave tarballs available on: http://www.scipy.org/site_content/weave These tarballs include some attempts at fixes for Prabhu on Debian. They have been tested on Solaris (2.95.1), W2K (gcc 2.95.2), and RH 7.1 (gcc 2.96) and all tests pass. I still haven't found any reason for the "Abort" error reported on Mandrake (bummer). I'd been interested in reports of success or failure of weave.test() on any and all architectures. Instructions for testing are on the page. There is one minor change likely tomorrow and then I'll announce on comp.python.lang on Monday to get a wider testing audience. Thanks to those who have already spent time testing. eric From pearu at cens.ioc.ee Sun Jan 13 04:07:07 2002 From: pearu at cens.ioc.ee (Pearu Peterson) Date: Sun, 13 Jan 2002 11:07:07 +0200 (EET) Subject: [SciPy-user] New F2PY snapshot available In-Reply-To: Message-ID: Hi! I made a new snapshot of F2PY available. It is beta version of the 5th F2PY release. It includes many updates, bug fixes, the latest scipy_distutils etc. See NEWS.txt for details. The most important new feature is *** New -c switch that will run the Fortran wrapping process from Alpha to Omega. For example, if foo.f contains subroutine bar, then running f2py -c foo.f -m car will create car.so in the current directory that can be readily imported to Python: >>> import car >>> print car.bar.__doc__ The snapshot is available for download in http://cens.ioc.ee/projects/f2py2e/2.x/ Regards, Pearu From Michael.Biester at t-online.de Sun Jan 13 13:45:56 2002 From: Michael.Biester at t-online.de (michael biester) Date: Sun, 13 Jan 2002 19:45:56 +0100 Subject: [SciPy-user] comparisions of fft / inverse fft algorithms Message-ID: <000a01c19c62$89a1e0a0$405502d9@mbi> For filter design program I experimented with Fast Fourier Tranforms ( and their inverse). I used algorithms from the Numeric module and from Scipy.fft . It seems, that FFT.fft from Numeric and fft.FFT2.fft perform similar (except for speed of execution and round off errors) However when applying the inverse transformations FFT.inverse_fft(...) from Numeric and fft.FFT2.ifft to the same sequence, results are not the same. While FFT.inverse_fft(...) from Numeric performs as expected ( compared with MATLABs ifft() ) fft.FFT2.ifft gives a sequence which is somehow shifted ... I used this sample script : ------ sample script --------------------------------------------------------- #--- test different versions of fft and its inverse from Numeric import * import FFT from scipy import * import win32com.client L = 500 N = 512 # create sequences with L complex 1's x1 = ones(L)*(1+1j) # use fft from Numeric y1 = FFT.fft(x1,N) # since x1 has L samples transform x1 with zero padding x1i = FFT.inverse_fft(y1) # transform back #-- except for round off errors we expect x1i[0:L] = x1[0:L] # (indeed they are ...) # now use fft and inverse fft from Scipy.fft y2 = fft.FFT2.fft(x1,N) # transform x1 with zero padding x2i = fft.FFT2.ifft(y2) # transform back #-- except for round off errors we expect x2i[0:L] = x1[0:L] # this time however the samples differ; plotting them reveals that # these sequences are similar but shifted versions of each other -------------------------------------------------------------------------------------------- Regards Michael -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at ee.byu.edu Mon Jan 14 11:39:56 2002 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon, 14 Jan 2002 11:39:56 -0500 (EST) Subject: [SciPy-user] comparisions of fft / inverse fft algorithm In-Reply-To: <000a01c19c62$89a1e0a0$405502d9@mbi> Message-ID: Which version of SciPy are you using. The 0.1 release contains a bug in inverse_fft which should be fixed now. I just checked the CVS version and it does not have this error. -Travis From oliphant at ee.byu.edu Mon Jan 14 11:39:56 2002 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Mon, 14 Jan 2002 11:39:56 -0500 (EST) Subject: [SciPy-user] comparisions of fft / inverse fft algorithm In-Reply-To: <000a01c19c62$89a1e0a0$405502d9@mbi> Message-ID: Which version of SciPy are you using. The 0.1 release contains a bug in inverse_fft which should be fixed now. I just checked the CVS version and it does not have this error. -Travis From Michael.Biester at t-online.de Mon Jan 14 15:12:29 2002 From: Michael.Biester at t-online.de (michael biester) Date: Mon, 14 Jan 2002 21:12:29 +0100 Subject: [SciPy-user] comparisions of fft / inverse fft algorithms Message-ID: <000b01c19d37$cbbb5ae0$e86602d9@mbi> Hello , its good to know that a bug in fft.FFT2.ifft(x) has been fixed. However updated modules in the CVS are currently of little use to me. I don't have the tools to compile the source for WIN32. The binary distribution for WIN32 is still version 0.10 and dates back to August 2001. Could anybody indicate, when an update of the binary release will be available ? Best regards Michael mailto:michael.biester at t-online.de From Herbert.W.Schilling at grc.nasa.gov Wed Jan 16 10:59:04 2002 From: Herbert.W.Schilling at grc.nasa.gov (Herb Schilling) Date: Wed, 16 Jan 2002 10:59:04 -0500 Subject: [SciPy-user] problem with using scipy.io.mio on Windows Message-ID: Hello, I am using scipy.io.mio to read in binary PLOT3D files. Everything works fine when I run my code on an SGI but when I run it under Windows 98 I have problems. ( I am using Windows native, not-cygwin Python 2.1 ). Here is the basic code I am using import scipy.io.mio # open the file fileHandle = scipy.io.mio.fopen( filePath, 'r' , format = 'b' ) # read in the total number of blocks in this file numberOfBlocks = fileHandle.fread( 1 , 'l' ) # Read in the size of each block imaxes = [] jmaxes = [] for iBlock in range( numberOfBlocks ) : imax = fileHandle.fread( 1 , 'l' ) jmax = fileHandle.fread( 1 , 'l' ) imaxes.append( imax ) jmaxes.append( jmax ) # Read in the X,Y coordinates of the points in the grids for iBlock in range( numberOfBlocks ) : imax = imaxes[ iBlock ] jmax = jmaxes[ iBlock ] # Note that we do want the transposed form xGrid = fileHandle.fread( ( jmax, imax ) , 'f' , 'f' ) yGrid = fileHandle.fread( ( jmax, imax ) , 'f' , 'f' ) The number of blocks, and the size of each block is read in just fine. When I start to read the X,Y coordinates things go bad. In the case where the file has 5 blocks, I get the following errors and the values for the xGrid and yGrid are, as you would expect, zeros after reading in the first xGrid. numpyio: Warning: 30060 bytes requested, 12 bytes read. numpyio: Warning: 30060 bytes requested, 0 bytes read. numpyio: Warning: 11084 bytes requested, 0 bytes read. numpyio: Warning: 11084 bytes requested, 0 bytes read. numpyio: Warning: 36816 bytes requested, 0 bytes read. numpyio: Warning: 36816 bytes requested, 0 bytes read. numpyio: Warning: 34160 bytes requested, 0 bytes read. numpyio: Warning: 34160 bytes requested, 0 bytes read. numpyio: Warning: 34160 bytes requested, 0 bytes read. numpyio: Warning: 34160 bytes requested, 0 bytes read. I am using the same code and the same file on both the PC and the SGI and I can use "od" to see that there is more data to be read. Even fileHandle.size() returns 292604. Why can't the rest of the bytes be read ? What's different under Windows? Thanks in advance! -- --------------------------------- Herb Schilling NASA Glenn Research Center Cleveland, OH 44135 Herbert.W.Schilling at grc.nasa.gov It is not enough to run, one must start in time. - teabag -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at ee.byu.edu Wed Jan 16 12:00:05 2002 From: oliphant at ee.byu.edu (Travis Oliphant) Date: Wed, 16 Jan 2002 12:00:05 -0500 (EST) Subject: [SciPy-user] problem with using scipy.io.mio on Windows In-Reply-To: Message-ID: On Wed, 16 Jan 2002, Herb Schilling wrote: > Hello, > > I am using scipy.io.mio to read in binary PLOT3D files. Everything > works fine when I run my code on an SGI but when I run it under > Windows 98 I have problems. ( I am using Windows native, not-cygwin > Python 2.1 ). > > > > Here is the basic code I am using > > import scipy.io.mio Note, fopen is defined right under scipy.io (you shouldn't have to import mio). so after an import scipy.io scipy.io.fopen should work > > # open the file > fileHandle = scipy.io.mio.fopen( filePath, 'r' , format = 'b' ) > > # read in the total number of blocks in this file > numberOfBlocks = fileHandle.fread( 1 , 'l' ) > > # Read in the size of each block > imaxes = [] > jmaxes = [] > for iBlock in range( numberOfBlocks ) : > imax = fileHandle.fread( 1 , 'l' ) > jmax = fileHandle.fread( 1 , 'l' ) > imaxes.append( imax ) > jmaxes.append( jmax ) > > # Read in the X,Y coordinates of the points in the grids > for iBlock in range( numberOfBlocks ) : > imax = imaxes[ iBlock ] > jmax = jmaxes[ iBlock ] > > # Note that we do want the transposed form > xGrid = fileHandle.fread( ( jmax, imax ) , 'f' , 'f' ) > yGrid = fileHandle.fread( ( jmax, imax ) , 'f' , 'f' ) > > > numpyio: Warning: 30060 bytes requested, 12 bytes read. > numpyio: Warning: 30060 bytes requested, 0 bytes read. > numpyio: Warning: 11084 bytes requested, 0 bytes read. > numpyio: Warning: 11084 bytes requested, 0 bytes read. > numpyio: Warning: 36816 bytes requested, 0 bytes read. > numpyio: Warning: 36816 bytes requested, 0 bytes read. > numpyio: Warning: 34160 bytes requested, 0 bytes read. > numpyio: Warning: 34160 bytes requested, 0 bytes read. > numpyio: Warning: 34160 bytes requested, 0 bytes read. > numpyio: Warning: 34160 bytes requested, 0 bytes read. > For some reason the C-code is not reading as many bytes as was requested. This warning just alerts you to that fact. One problem I've seen is byte order issues. What is the byte-order on an SGI? It looks like you are opening the file with "big-endian" so that shouldn't be it. Can you send me an example file that I could try to read on Linux? One test you could do is to run fileHandle.tell() to see where the file-pointer is after each read. This should tell you how you are progressing through the file. Sorry I can't help further. -Travis From lluang at yahoo.com Wed Jan 16 18:38:04 2002 From: lluang at yahoo.com (Louis Luangkesorn) Date: Wed, 16 Jan 2002 15:38:04 -0800 (PST) Subject: [SciPy-user] Unit tests for compile_code.py returns ImportError Message-ID: <20020116233804.8753.qmail@web12104.mail.yahoo.com> Hello When I ran the scipy.test_all() after installing everything, I get the failure at the bottom of this message. [ImportError: DLL load failed: Invalid access to memory location.] When I had MINGW installed incorrectly, the compile tests actually succeded (although it did not compile the code and ran those tests as straight python), so the problem is probably with how it interacts with gcc. I'm using gcc 2.95.3-4 (mingw special) which is from mingw-1.0-20010608 Am I missing anything? Thanks Louis <'No test suite found' and 'creating test suite' messages snipped> No test suite found for scipy.special No test suite found for scipy.stats ...................................................................................................... Expression: result[1:-1,1:-1] = (b[1:-1,1:-1] + b[2:,1:-1] + b[:-2,1:-1]+ b[1:-1,2:] + b[1:-1,:-2]) / 5. Run: (10, 10) f Compiling code. This may take a few minutes running build_ext building 'sc_1184ec2786fc765046a8e2929d8c9a2642' extension c:\apps\gccmingw\bin\gcc.exe -mno-cygwin -mdll -O2 -w -IC:\apps\Python\scipy\compiler\blitz-20001213 -IC:\apps\Python\scipy\compiler\scxx -IC:\APPS\Python\Include -c C:\DOCUME~1\kll560\LOCALS~1\Temp\sc_1184ec2786fc765046a8e2929d8c9a2642.cpp -o C:\DOCUME~1\kll560\LOCALS~1\Temp\Release\sc_1184ec2786fc765046a8e2929d8c9a2642.o writing C:\DOCUME~1\kll560\LOCALS~1\Temp\Release\sc_1184ec2786fc765046a8e2929d8c9a2642.def c:\apps\gccmingw\bin\dllwrap.exe --driver-name g++ -mno-cygwin -mdll -static -s C:\DOCUME~1\kll560\LOCALS~1\Temp\Release\sc_1184ec2786fc765046a8e2929d8c9a2642.o C:\DOCUME~1\kll560\LOCALS~1\Temp\Release\sc_1184ec2786fc765046a8e2929d8c9a2642.def -LC:\APPS\Python\libs -lpython21 -o C:\DOCUME~1\kll560\LOCALS~1\Temp\sc_1184ec2786fc765046a8e2929d8c9a2642.pyd finished compiling (sec): 43.7829999924 E..................................................................................... ====================================================================== ERROR: result[1:-1,1:-1] = (b[1:-1,1:-1] + b[2:,1:-1] + b[:-2,1:-1] ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\apps\Python\scipy\compiler\tests\test_compile_code.py", line 280, in check_5point_avg_2d self.generic_2d(expr) File "C:\apps\Python\scipy\compiler\tests\test_compile_code.py", line 256, in generic_2d standard,compiled = self.generic_test(expr,arg_dict,type,size) File "C:\apps\Python\scipy\compiler\tests\test_compile_code.py", line 217, in generic_test scipy.compiler.compile_code.compiled_exec(expr,arg_dict,module_location=None) File "C:\apps\Python\scipy\compiler\compile_code.py", line 134, in compiled_exec exec ('import ' + module_name) File "", line 1, in ? ImportError: DLL load failed: Invalid access to memory location. ---------------------------------------------------------------------- Ran 188 tests in 46.688s FAILED (errors=1) ===== When I consider your heavens, the work of your fingers, the moon and the stars, which you have set in place, what is man that you are mindful of him, the son of man that you care for him? Psalm 8:3-4 Y!M, AOL, jabber.com: lluang __________________________________________________ Do You Yahoo!? Send FREE video emails in Yahoo! Mail! http://promo.yahoo.com/videomail/ From lluang at yahoo.com Wed Jan 16 19:04:58 2002 From: lluang at yahoo.com (Louis Luangkesorn) Date: Wed, 16 Jan 2002 16:04:58 -0800 (PST) Subject: [SciPy-user] Unit tests for compile_code.py returns ImportError Message-ID: <20020117000458.50328.qmail@web12107.mail.yahoo.com> Hello I solved my own problem. It seems that it is only an issue with the latest mingw. When I use the older version (19991024) which is on Mumit Khan's site (and is linked to on the scipy.org site) the unit tests pass OK. I'll add a comment on the Installation page. Louis ===== When I consider your heavens, the work of your fingers, the moon and the stars, which you have set in place, what is man that you are mindful of him, the son of man that you care for him? Psalm 8:3-4 Y!M, AOL, jabber.com: lluang __________________________________________________ Do You Yahoo!? Send FREE video emails in Yahoo! Mail! http://promo.yahoo.com/videomail/ From eric at scipy.org Wed Jan 16 19:38:31 2002 From: eric at scipy.org (eric) Date: Wed, 16 Jan 2002 19:38:31 -0500 Subject: [SciPy-user] Unit tests for compile_code.py returns ImportError References: <20020116233804.8753.qmail@web12104.mail.yahoo.com> Message-ID: <019a01c19eef$49fe75f0$6b01a8c0@ericlaptop> Yes. The scipy.compiler code had some hard coded stuff on windows that doesn't work with newer version. Kinda funny cause the new versions have a much saner set of commands. FYI, compiler has been renamed to weave, and the weave package is much more powerful than compiler and should work with your newer compiler (although I occasionally get some strange errors when testing with multiple versions of gcc + MSVC on a machine). Thanks for putting the note on the installation page. eric ----- Original Message ----- From: "Louis Luangkesorn" To: Sent: Wednesday, January 16, 2002 6:38 PM Subject: [SciPy-user] Unit tests for compile_code.py returns ImportError > Hello > > When I ran the scipy.test_all() after installing everything, I get > the failure at the bottom of this message. > [ImportError: DLL load failed: Invalid access to memory location.] > When I had MINGW installed incorrectly, the compile tests actually > succeded (although it did not compile the code and ran those tests > as straight python), so the problem is probably with how it > interacts with gcc. I'm using gcc 2.95.3-4 (mingw special) which > is from mingw-1.0-20010608 > > Am I missing anything? > > Thanks > Louis > > <'No test suite found' and 'creating test suite' messages snipped> > No test suite found for scipy.special > No test suite found for scipy.stats > ............................................................................ .......................... > Expression: result[1:-1,1:-1] = (b[1:-1,1:-1] + b[2:,1:-1] + > b[:-2,1:-1]+ b[1:-1,2:] + b[1:-1,:-2]) / 5. > Run: (10, 10) f > Compiling code. This may take a few minutes > running build_ext > building 'sc_1184ec2786fc765046a8e2929d8c9a2642' extension > c:\apps\gccmingw\bin\gcc.exe -mno-cygwin -mdll -O2 -w > -IC:\apps\Python\scipy\compiler\blitz-20001213 > -IC:\apps\Python\scipy\compiler\scxx -IC:\APPS\Python\Include -c > C:\DOCUME~1\kll560\LOCALS~1\Temp\sc_1184ec2786fc765046a8e2929d8c9a2642.cpp > -o > C:\DOCUME~1\kll560\LOCALS~1\Temp\Release\sc_1184ec2786fc765046a8e2929d8c9a26 42.o > writing > C:\DOCUME~1\kll560\LOCALS~1\Temp\Release\sc_1184ec2786fc765046a8e2929d8c9a26 42.def > c:\apps\gccmingw\bin\dllwrap.exe --driver-name g++ -mno-cygwin > -mdll -static -s > C:\DOCUME~1\kll560\LOCALS~1\Temp\Release\sc_1184ec2786fc765046a8e2929d8c9a26 42.o > C:\DOCUME~1\kll560\LOCALS~1\Temp\Release\sc_1184ec2786fc765046a8e2929d8c9a26 42.def > -LC:\APPS\Python\libs -lpython21 -o > C:\DOCUME~1\kll560\LOCALS~1\Temp\sc_1184ec2786fc765046a8e2929d8c9a2642.pyd > finished compiling (sec): 43.7829999924 > E........................................................................... .......... > ====================================================================== > ERROR: result[1:-1,1:-1] = (b[1:-1,1:-1] + b[2:,1:-1] + b[:-2,1:-1] > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\apps\Python\scipy\compiler\tests\test_compile_code.py", > line 280, in check_5point_avg_2d > self.generic_2d(expr) > File "C:\apps\Python\scipy\compiler\tests\test_compile_code.py", > line 256, in generic_2d > standard,compiled = self.generic_test(expr,arg_dict,type,size) > File "C:\apps\Python\scipy\compiler\tests\test_compile_code.py", > line 217, in generic_test > > scipy.compiler.compile_code.compiled_exec(expr,arg_dict,module_location=None ) > File "C:\apps\Python\scipy\compiler\compile_code.py", line 134, > in compiled_exec > exec ('import ' + module_name) > File "", line 1, in ? > ImportError: DLL load failed: Invalid access to memory location. > ---------------------------------------------------------------------- > Ran 188 tests in 46.688s > > FAILED (errors=1) > > > > ===== > When I consider your heavens, the work of your fingers, > the moon and the stars, which you have set in place, > what is man that you are mindful of him, the son of man that you care for him? Psalm 8:3-4 > Y!M, AOL, jabber.com: lluang > > __________________________________________________ > Do You Yahoo!? > Send FREE video emails in Yahoo! Mail! > http://promo.yahoo.com/videomail/ > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.net > http://www.scipy.net/mailman/listinfo/scipy-user > From Herbert.W.Schilling at grc.nasa.gov Thu Jan 17 08:48:09 2002 From: Herbert.W.Schilling at grc.nasa.gov (Herb Schilling) Date: Thu, 17 Jan 2002 08:48:09 -0500 Subject: [SciPy-user] problem with using scipy.io.mio on Windows In-Reply-To: References: Message-ID: Hi Travis, > >Note, fopen is defined right under scipy.io (you shouldn't have to import >mio). > >so after an > >import scipy.io > >scipy.io.fopen should work Thanks for the tip! > >> > >For some reason the C-code is not reading as many bytes as was requested. >This warning just alerts you to that fact. > >One problem I've seen is byte order issues. What is the byte-order on an >SGI? Big endian. > It looks like you are opening the file with "big-endian" so that >shouldn't be it. Can you send me an example file that I could try to read >on Linux? Sure. I have attached a file. Sorry for the size. Floating point data files don't compress well. > >One test you could do is to run fileHandle.tell() to see where the >file-pointer is after each read. This should tell you how you are >progressing through the file. Excellent idea! I just tried that and the results are very strange. Calling tell affects the results! Also it is very strange that after reading in one integer that the position leaps to 3588 ! Some of the other positions don't make sense either. Even the values in the warnings don't make sense: numpyio: Warning: 7515 bytes requested, 12 bytes read. This came when I tried to read 501 x 15 floats which should have been 7515 * 4 bytes, not 7515 bytes. Below are the contents of three files: 1. The script I use to read the PLOT3D file I have attached. 2. The output of the script when I am NOT calling tell 3. The output of the script when I am calling tell =============================== script - note that I comment/uncomment a line in the whatIsPosition function to change my call to tell =============================== import sys def whatIsPosition( fileHandle, whatWasJustRead ): pass #print "After " + whatWasJustRead + ", position is " , fileHandle.tell() def readPlot3DBinaryToBlocks( filePath ): '''Read in a binary PLOT3D file to a set of Blocks.''' import scipy.io # Open file fileHandle = scipy.io.fopen( filePath, 'r' , format = 'b' ) print "fileSize is " , fileHandle.size() whatIsPosition( fileHandle , "fopen" ) # read in number of blocks numberOfBlocks = fileHandle.fread( 1 , 'l' ) whatIsPosition( fileHandle , "reading number of blocks" ) print "number of blocks is " , numberOfBlocks # Read in the size of each block imaxes = [] jmaxes = [] for iBlock in range( numberOfBlocks ) : imax = fileHandle.fread( 1 , 'l' ) jmax = fileHandle.fread( 1 , 'l' ) whatIsPosition( fileHandle , "reading imax and jmax" ) print "for block # " , iBlock, "imax and jmax are " , imax , jmax imaxes.append( imax ) jmaxes.append( jmax ) # read in the X and Y values, first comes block 1 X, then block 1 Y, then block 3 X, then block 3 Y blocks = [] for iBlock in range( numberOfBlocks ) : imax = imaxes[ iBlock ] jmax = jmaxes[ iBlock ] # Note that we do want the transposed form xGrid = fileHandle.fread( ( jmax, imax ) , 'f' , 'f' ) yGrid = fileHandle.fread( ( jmax, imax ) , 'f' , 'f' ) print "for block # " , iBlock, "xGrid[ 0: 3 , 0:3 ] is " , xGrid[ 0: 3 , 0:3 ] print "for block # " , iBlock, "yGrid[ 0: 3 , 0:3 ] is " , yGrid[ 0: 3 , 0:3 ] whatIsPosition( fileHandle , "reading xGrid and yGrid" ) return print sys.argv[1] blocks = readPlot3DBinaryToBlocks( sys.argv[1] ) =============================== The output when I am NOT calling tell =============================== binary.p3d fileSize is 292604 number of blocks is 5 for block # 0 imax and jmax are 501 15 for block # 1 imax and jmax are 163 17 for block # 2 imax and jmax are 156 59 for block # 3 imax and jmax are 427 20 for block # 4 imax and jmax are 427 20 for block # 0 xGrid[ 0: 3 , 0:3 ] is [[ 15.99999905 13.77125454 11.87302971] [ 4.51538134 3.98979473 3.54215217] [ 6.70775032 5.85703707 5.13248396]] for block # 0 yGrid[ 0: 3 , 0:3 ] is [[ 0. 0. 0.] [ 0. 0. 0.] [ 0. 0. 0.]] for block # 1 xGrid[ 0: 3 , 0:3 ] is [[ 0. 0. 0.] [ 0. 0. 0.] [ 0. 0. 0.]] for block # 1 yGrid[ 0: 3 , 0:3 ] is [[ 0. 0. 0.] [ 0. 0. 0.] [ 0. 0. 0.]] for block # 2 xGrid[ 0: 3 , 0:3 ] is [[ 0. 0. 0.] [ 0. 0. 0.] [ 0. 0. 0.]] for block # 2 yGrid[ 0: 3 , 0:3 ] is [[ 0. 0. 0.] [ 0. 0. 0.] [ 0. 0. 0.]] for block # 3 xGrid[ 0: 3 , 0:3 ] is [[ 0. 0. 0.] [ 0. 0. 0.] [ 0. 0. 0.]] for block # 3 yGrid[ 0: 3 , 0:3 ] is [[ 0. 0. 0.] [ 0. 0. 0.] [ 0. 0. 0.]] for block # 4 xGrid[ 0: 3 , 0:3 ] is [[ 0. 0. 0.] [ 0. 0. 0.] [ 0. 0. 0.]] for block # 4 yGrid[ 0: 3 , 0:3 ] is [[ 0. 0. 0.] [ 0. 0. 0.] [ 0. 0. 0.]] =============================== The output when I am NOT calling tell =============================== binary.p3d fileSize is 292604 After fopen, position is 0 After reading number of blocks, position is 3588 number of blocks is 5 After reading imax and jmax, position is 3596 for block # 0 imax and jmax are 501 15 After reading imax and jmax, position is 3604 for block # 1 imax and jmax are 163 17 After reading imax and jmax, position is 3612 for block # 2 imax and jmax are 156 59 After reading imax and jmax, position is 3620 for block # 3 imax and jmax are 427 20 After reading imax and jmax, position is 3628 for block # 4 imax and jmax are 427 20 for block # 0 xGrid[ 0: 3 , 0:3 ] is [[ 1.59999990e+001 1.37712545e+001 1.18730297e+001] [ 3.54215217e+000 3.16090393e+000 1.64749781e+021] [ 5.13248396e+000 4.51538134e+000 3.98979473e+000]] for block # 0 yGrid[ 0: 3 , 0:3 ] is [[ 0. 0. 0.] [ 0. 0. 0.] [ 0. 0. 0.]] After reading xGrid and yGrid, position is 32768 for block # 1 xGrid[ 0: 3 , 0:3 ] is [[-0.01891858 -0.01973816 -0.02066956] [-0.03023803 -0.03107315 -0.03175394] [-0.02180799 -0.02299393 -0.02415288]] for block # 1 yGrid[ 0: 3 , 0:3 ] is [[ 0. 0. 0.] [ 0. 0. 0.] [ 0. 0. 0.]] After reading xGrid and yGrid, position is 40960 for block # 2 xGrid[ 0: 3 , 0:3 ] is [[-0.02406131 -0.02338937 -0.02269458] [-0.00455854 -0.00379783 -0.00291636] [ 0.01606247 0.01684676 0.01759628]] for block # 2 yGrid[ 0: 3 , 0:3 ] is [[ 0. 0. 0.] [ 0. 0. 0.] [ 0. 0. 0.]] After reading xGrid and yGrid, position is 73728 for block # 3 xGrid[ 0: 3 , 0:3 ] is [[-0.03620248 -0.03564961 -0.03520766] [-0.03422029 -0.03353022 -0.03296501] [-0.03248635 -0.03199605 -0.03620248]] for block # 3 yGrid[ 0: 3 , 0:3 ] is [[ 0. 0. 0.] [ 0. 0. 0.] [ 0. 0. 0.]] After reading xGrid and yGrid, position is 106496 for block # 4 xGrid[ 0: 3 , 0:3 ] is [[ 0.96958733 0.97537327 0.98071378] [ 1.01128232 1.02077925 1.03193545] [ 1.12465644 1.1539247 1.18828964]] for block # 4 yGrid[ 0: 3 , 0:3 ] is [[ 0. 0. 0.] [ 0. 0. 0.] [ 0. 0. 0.]] After reading xGrid and yGrid, position is 139264 > >Sorry I can't help further. On the contrary, you have given me a lot of help! I was ready to abandon scipy completely and use SWIG to wrap some C code to read PLOT3D files but I thought it would easier to use scipy and it was on UNIX ! Thanks! -- --------------------------------- Herb Schilling NASA Glenn Research Center Cleveland, OH 44135 Herbert.W.Schilling at grc.nasa.gov It is not enough to run, one must start in time. - teabag -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: %binary.p3d.gz Type: application/applefile Size: 123 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: binary.p3d.gz Type: application/octet-stream Size: 252494 bytes Desc: not available URL: From Herbert.W.Schilling at grc.nasa.gov Thu Jan 17 09:35:53 2002 From: Herbert.W.Schilling at grc.nasa.gov (Herb Schilling) Date: Thu, 17 Jan 2002 09:35:53 -0500 Subject: [SciPy-user] very sorry for the sending large file to the list Message-ID: Hi, I just noticed that when I replied to Travis that my message also went to the mailing list ! It would be that bad but I attached a large file. Very sorry about that. I will be more careful in the future. -- --------------------------------- Herb Schilling NASA Glenn Research Center Cleveland, OH 44135 Herbert.W.Schilling at grc.nasa.gov It is not enough to run, one must start in time. - teabag From Herbert.W.Schilling at grc.nasa.gov Fri Jan 18 11:11:14 2002 From: Herbert.W.Schilling at grc.nasa.gov (Herb Schilling) Date: Fri, 18 Jan 2002 11:11:14 -0500 Subject: [SciPy-user] Re: IO problems on windows In-Reply-To: References: Message-ID: >Herb, > >I'm almost sure that opening the file in 'r' mode only is the problem you are >having. I just learned that if you open a binary file with the 'r' >permission instead of 'rb', then the byte '\32' is interpreted as EOF. > >It seems that something like that is happening in your case. > >I hope that fixes your problem. I suppose I could issue a warning if the >permission flag does not have 'b' in it because fopen is primarily used for >binary reading and writing. > Hi Travis, That did it!! Everything works perfectly now. Thank you so much. You saved the day. Maybe the documentation page at http://www.scipy.org/site_content/tutorials/show_ref?input_file=scipy/io/mio_fopen.py.htm could mentioned this ? PS. That page also doesn't mention the fopen's tell method that you told me about. Maybe that could be included too? -- --------------------------------- Herb Schilling NASA Glenn Research Center Cleveland, OH 44135 Herbert.W.Schilling at grc.nasa.gov It is not enough to run, one must start in time. - teabag From lluang at yahoo.com Sun Jan 20 01:59:45 2002 From: lluang at yahoo.com (Louis Luangkesorn) Date: Sat, 19 Jan 2002 22:59:45 -0800 (PST) Subject: [SciPy-user] stats.anova problem Message-ID: <20020120065945.24234.qmail@web12107.mail.yahoo.com> Hello I'm trying to test the anova package. But I'm doing something wrong here. This is what I try. import scipy anovadata = ([[1,'A1','B1',2], [2,'A1','B1',1], [3,'A1','B1',3], [4,'A1','B1',2], [5,'A1','B2',3], [6,'A1','B2',4], [7,'A1','B2',5], [8,'A2','B1',4], [9,'A2','B1',6], [10,'A2','B1',5], [11,'A1','B2',2], [12,'A1','B2',4], [13,'A1','B2',5]]) y=scipy.stats.anova(anovadata) Then, I get: Traceback (most recent call last): File "C:\apps\Python\Pythonwin\pywin\framework\scriptutils.py", line 301, in RunScript exec codeObject in __main__.__dict__ File "C:\Documents and Settings\kll560\My Documents\testdata\anovatest.py", line 16, in ? y=scipy.stats.anova(anovadata) File "C:\apps\Python\scipy\stats\stats.py", line 234, in __call__ return apply(self._dispatch[type(arg1)], (arg1,) + args, kw) File "C:\apps\Python\scipy\stats\stats.py", line 3933, in aanova M = pstat.collapse(data,Bscols,-1,None,None,mean) TypeError: collapse() takes at most 5 arguments (6 given) By the way, in pstats.py, the call for collapse() is in line 183: def collapse (listoflists,keepcols,collapsecols,sterr=None,ns=None): Any ideas? Louis ===== When I consider your heavens, the work of your fingers, the moon and the stars, which you have set in place, what is man that you are mindful of him, the son of man that you care for him? Psalm 8:3-4 Y!M, AOL, jabber.com: lluang __________________________________________________ Do You Yahoo!? Send FREE video emails in Yahoo! Mail! http://promo.yahoo.com/videomail/ From lluang at yahoo.com Sun Jan 20 02:08:16 2002 From: lluang at yahoo.com (Louis Luangkesorn) Date: Sat, 19 Jan 2002 23:08:16 -0800 (PST) Subject: [SciPy-user] unittests for scipy.stats: assert_almost_equal question Message-ID: <20020120070816.24615.qmail@web12107.mail.yahoo.com> Hi I'm writting unittests for the scipy.stats framework (partly to teach myself unit testing, partly to teach myself the stats functions). I'm writing them using Wilkinson's Statistic Quiz (ref. in code attached) for data and comparison numbers. Right now, I get 2 failures and 2 errors (using Numeric arrays). I think the complete framework needs another set of tests for the same data in lists form (if I understand the scipy.stats module correctly) Of the failures/errors, some of them may be in the code, but one of them maybe something I'm missing in the unit testing framework. For one of the tests, I get the following error: ====================================================================== FAIL: check_stdHUGE (__main__.test_basicstats) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Documents and Settings\kll560\My Documents\testdata\test_stats.py", line 150, in check_stdHUGE assert_almost_equal(y,2.738612788e+12) File "C:\apps\Python\scipy\scipy_test.py", line 223, in assert_almost_equal assert desired == actual, msg AssertionError: Items are not equal: DESIRED: 2.738612788e+012 ACTUAL: 2.73861278753e+012 ---------------------------------------------------------------------- The assert_almost_equal function is being called using the default setting of decimal=7 Since here it seems I have 10 digits of accuracy, why did this fail? Louis PS I'm attaching my code. I followed the pattern from test_misc.py ===== When I consider your heavens, the work of your fingers, the moon and the stars, which you have set in place, what is man that you are mindful of him, the son of man that you care for him? Psalm 8:3-4 Y!M, AOL, jabber.com: lluang __________________________________________________ Do You Yahoo!? Send FREE video emails in Yahoo! Mail! http://promo.yahoo.com/videomail/ -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: test_stats.py URL: From eric at scipy.org Sun Jan 20 05:36:31 2002 From: eric at scipy.org (eric) Date: Sun, 20 Jan 2002 05:36:31 -0500 Subject: [SciPy-user] unittests for scipy.stats: assert_almost_equal question References: <20020120070816.24615.qmail@web12107.mail.yahoo.com> Message-ID: <00ec01c1a19e$53a6a760$777ba8c0@ericlaptop> Hey Louis, > I'm writting unittests for the scipy.stats framework (partly to > teach myself unit testing, partly to teach myself the stats > functions). I'm writing them using Wilkinson's Statistic Quiz > (ref. in code attached) for data and comparison numbers. A thousand blessing upon you. I immediately commited it to the CVS! > > Right now, I get 2 failures and 2 errors (using Numeric arrays). I > think the complete framework needs another set of tests for the > same data in lists form (if I understand the scipy.stats module > correctly) Just test the Numeric versions. The list versions are going away entirely. In fact, the interface to the stats functions is due for a *major* overhaul. There is a partially converted version called new_stats.py in the CVS that is destined to replace stats.py, but it needs a ton of work. Notes on the changes needed are here. http://www.scipy.org/Members/eric/stats_notes.html and the current incarnation of the changes are here: http://scipy.net/cgi-bin/viewcvs.cgi/scipy/stats/new_stats.py?rev=1.6&conten t-type=text/vnd.viewcvs-markup This is a project in need of a champion, and anyone who wants to pick it up will get another thousand blessings. Also, this is definitely *not* to discourage you from making unit tests for the current version. If the tests are there, we can convert them to new interfaces later. > > Of the failures/errors, some of them may be in the code, but one of > them maybe something I'm missing in the unit testing framework. > For one of the tests, I get the following error: > > ====================================================================== > FAIL: check_stdHUGE (__main__.test_basicstats) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Documents and Settings\kll560\My > Documents\testdata\test_stats.py", line 150, in check_stdHUGE > assert_almost_equal(y,2.738612788e+12) > File "C:\apps\Python\scipy\scipy_test.py", line 223, in > assert_almost_equal > assert desired == actual, msg > AssertionError: > Items are not equal: > DESIRED: 2.738612788e+012 > ACTUAL: 2.73861278753e+012 > ---------------------------------------------------------------------- > > The assert_almost_equal function is being called using the default > setting of decimal=7 Since here it seems I have 10 digits of > accuracy, why did this fail? 7 decimals is not 7 significant digits but actually 7 decimal places to the left of 0. The way almost_eqaul works is by using the following comparison: round(abs(desired - actual),decimal) == 0 In this case, abs(desired-actual) is 470. rounding it to the 7 digits past the decimal leaves it as 470 which fails the test of being equal to 0. Maybe a test function needs to be added that tests to a specified number of significant digits for cases like these. Thanks tons for working on this, eric From magnus at thinkware.se Sun Jan 20 12:49:07 2002 From: magnus at thinkware.se (Magnus =?iso-8859-1?Q?Lyck=E5?=) Date: Sun, 20 Jan 2002 18:49:07 +0100 Subject: [SciPy-user] stats.anova problem In-Reply-To: <20020120065945.24234.qmail@web12107.mail.yahoo.com> Message-ID: <5.1.0.14.0.20020120184740.03086228@mail.irrblosset.se> At 22:59 2002-01-19 -0800, you wrote: >Hello > >I'm trying to test the anova package. But I'm doing something >wrong here. This is what I try. See http://www.scipy.net/pipermail/scipy-dev/2001-November/date.html and mails with subject "State of stats modules?" -- Magnus Lyck?, Thinkware AB ?lvans v?g 99, SE-907 50 UME? tel: 070-582 80 65, fax: 070-612 80 65 http://www.thinkware.se/ mailto:magnus at thinkware.se From lluang at yahoo.com Sun Jan 20 22:38:03 2002 From: lluang at yahoo.com (Louis Luangkesorn) Date: Sun, 20 Jan 2002 19:38:03 -0800 (PST) Subject: [SciPy-user] unittests for scipy.stats: assert_almost_equal question In-Reply-To: <200201201800.g0KI0BP27274@scipy.org> Message-ID: <20020121033803.41108.qmail@web12108.mail.yahoo.com> Hmmm, I think that I would like to be testing in terms of significant digits as opposed to decimal places, especially when working with floating point. Since you are asking for such a test, here it is. The attached file has a function meant to go into the scipy_test.py module. I wrote assert_approx_equal following the same form as assert_almost_equal assert_approx_equal(actual,desired,significant=7,err_msg='',verbose=1): compares 'actual' and 'desired' and determines the first 'significant' significant digits and checks for accuracy. Let's see, I think I counted significant digits correctly. Can anyone tell? Louis > From: "eric" > To: > Subject: Re: [SciPy-user] unittests for scipy.stats: > assert_almost_equal question > Date: Sun, 20 Jan 2002 05:36:31 -0500 > Organization: enthought > Reply-To: scipy-user at scipy.net > > Hey Louis, > > A thousand blessing upon you. I immediately commited it to the > CVS! > > > > > The assert_almost_equal function is being called using the > default > > setting of decimal=7 Since here it seems I have 10 digits of > > accuracy, why did this fail? > > 7 decimals is not 7 significant digits but actually 7 decimal > places to the > left of 0. The way almost_eqaul works is by using the following > comparison: > round(abs(desired - actual),decimal) == 0 > In this case, abs(desired-actual) is 470. rounding it to the 7 > digits past the decimal leaves it as 470 which fails the test of > being equal to 0. > Maybe a test function needs to be added that tests to a specified > number of significant digits for cases like these. > > Thanks tons for working on this, > > eric > ===== When I consider your heavens, the work of your fingers, the moon and the stars, which you have set in place, what is man that you are mindful of him, the son of man that you care for him? Psalm 8:3-4 Y!M, AOL, jabber.com: lluang __________________________________________________ Do You Yahoo!? Send FREE video emails in Yahoo! Mail! http://promo.yahoo.com/videomail/ -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: assert_approx_equal.py URL: From eric at scipy.org Sun Jan 20 22:38:03 2002 From: eric at scipy.org (eric) Date: Sun, 20 Jan 2002 22:38:03 -0500 Subject: [SciPy-user] unittests for scipy.stats: assert_almost_equal question References: <20020121033803.41108.qmail@web12108.mail.yahoo.com> Message-ID: <001e01c1a22d$083ce0c0$777ba8c0@ericlaptop> Hey Louis, > Hmmm, I think that I would like to be testing in terms of > significant digits as opposed to decimal places, especially when > working with floating point. We need both in different situations. > Since you are asking for such a test, > here it is. Thanks. In my cursory checks, it looks to me like you've counted digits correctly. I'll add this to scipy_test. Does anyone have better names for these. They are quite arbitrary and indistinguishable right now. How about: assert_significant_equal assert_decimal_equal These names give a little more info about what is happening in the function. thoughts? eric From lluang at yahoo.com Tue Jan 22 09:35:58 2002 From: lluang at yahoo.com (Louis Luangkesorn) Date: Tue, 22 Jan 2002 06:35:58 -0800 (PST) Subject: [SciPy-user] stats.anova problem Message-ID: <20020122143558.56204.qmail@web12107.mail.yahoo.com> Hmmm, I see. Does that mean that noone is using the anova functions in SciPy? As an aside, I pulled the stats.py module from Gary Strangman's site. The anova in his version (0.4) runs (he has a test suite that calls all of his functions, somewhat like what the random number generator test suite does.) Something must have happened when stats.py changed to 0.5. Oh, if I follow up to this, I will go to the scipy-dev list. But first I have a proposal to get out. . . Louis >Message: 4 >Date: Sun, 20 Jan 2002 18:49:07 +0100 >To: scipy-user at scipy.org >From: Magnus =?iso-8859-1?Q?Lyck=E5?= >Subject: Re: [SciPy-user] stats.anova problem >Reply-To: scipy-user at scipy.net > >At 22:59 2002-01-19 -0800, you wrote: >>Hello >> >>I'm trying to test the anova package. But I'm doing something >>wrong here. This is what I try. > >See http://www.scipy.net/pipermail/scipy-dev/2001-November/date.html >and mails with subject "State of stats modules?" > > >-- >Magnus Lyck?, Thinkware AB > ===== When I consider your heavens, the work of your fingers, the moon and the stars, which you have set in place, what is man that you are mindful of him, the son of man that you care for him? Psalm 8:3-4 Y!M, AOL, jabber.com: lluang __________________________________________________ Do You Yahoo!? Send FREE video emails in Yahoo! Mail! http://promo.yahoo.com/videomail/ From lluang at yahoo.com Tue Jan 22 10:13:35 2002 From: lluang at yahoo.com (Louis Luangkesorn) Date: Tue, 22 Jan 2002 07:13:35 -0800 (PST) Subject: [SciPy-user] unittests for scipy.stats: In-Reply-To: <200201211800.g0LI02P06758@scipy.org> Message-ID: <20020122151335.61357.qmail@web12107.mail.yahoo.com> Eric, I sent an email to Jim Weirich, who is the maintainer of EiffleUnit. EiffleUnit has an assert_approximate function. His reply is below. My feeling regarding the scipy_test.py is that I rewrite the assert_approx_equals to match the behavior of the EiffelUnit assert_approximate, since it does the same thing that I want, and that we name the scipy function assert_approximate_equals, in the interests of consistancy with other XUnit type modules. Oh, I'm going to have an update on assert_approx_equals regardless. There are a couple of rather embarassing cases I don't account for. Louis ----forwarded message---- Hello Louis. The short form of the assert_approximate function is ... assert_approximate (message: STRING; expected, actual, tolerance: DOUBLE) is -- Assert that two values are approximately equal -- (within a `tolerance'). require message_exists: message /= Void end Assert_approximate uses a tolerance value to specify the value for approximate. This allows the user to choose what close-enough means. For example ... assert_approximate("absolute", expected, actual, 0.00001); assert_approximate("relative", expected, actual, 0.001 * expected); I believe that JUnit has a version of assertEquals that takes two double arguments and a double tolerance. assert_approximate mimics that behavior. I hope this helps. -- -- Jim Weirich jweirich at one.net http://w3.one.net/~jweirich > From: "eric" > To: > Subject: Re: [SciPy-user] unittests for scipy.stats: > assert_almost_equal question > Date: Sun, 20 Jan 2002 22:38:03 -0500 > Organization: enthought > Reply-To: scipy-user at scipy.net > > Hey Louis, > > > Thanks. In my cursory checks, it looks to me like you've counted > digits correctly. I'll add this to scipy_test. Does anyone have > better names for > these. They are quite arbitrary and indistinguishable right now. > How about: > > assert_significant_equal > assert_decimal_equal > > These names give a little more info about what is happening in > the function. > > thoughts? > > eric > > > > > --__--__-- > > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.net > http://www.scipy.net/mailman/listinfo/scipy-user > > > End of SciPy-user Digest ===== When I consider your heavens, the work of your fingers, the moon and the stars, which you have set in place, what is man that you are mindful of him, the son of man that you care for him? Psalm 8:3-4 Y!M, AOL, jabber.com: lluang __________________________________________________ Do You Yahoo!? Send FREE video emails in Yahoo! Mail! http://promo.yahoo.com/videomail/ From YZhu at golder.com Tue Jan 22 18:23:47 2002 From: YZhu at golder.com (Zhu, Yi) Date: Tue, 22 Jan 2002 15:23:47 -0800 Subject: [SciPy-user] SciPy for Python 2.2 Message-ID: <2BDFE2BD9829D31191E3009027729D60CB0857@atlexchange.golder.com> Hi, Is there a SciPy version for Python 2.2? Thanks, Yi From mlerner at umich.edu Wed Jan 23 16:59:41 2002 From: mlerner at umich.edu (Michael George Lerner) Date: Wed, 23 Jan 2002 16:59:41 -0500 (EST) Subject: [SciPy-user] Need help installing with Python 2.2 / 2.1 on Windows Message-ID: Hi, I have both Python 2.2 and Python 2.1 installed on my Windows box (in c:\python22 and c:\21). When I try to install SciPy with the binary installer SciPy-0.1.win32-py2.1.exe, which I downloaded from the website on Jan. 23, the second screen says "Select python installation to choose" and only allows me to choose Python 2.1. It doesn't notice my Python 2.2 installation, so it's not in the list at the top of the screen. Also, I can't edit the text in the text entry box at the bottom of the screen to manually point it at c:\python22. Is there some way to point SciPy at Python 2.2, or will I have to install with version 2.1? thanks, -michael From eric at scipy.org Wed Jan 23 16:45:35 2002 From: eric at scipy.org (eric) Date: Wed, 23 Jan 2002 16:45:35 -0500 Subject: [SciPy-user] Need help installing with Python 2.2 / 2.1 on Windows References: Message-ID: <17e501c1a457$4a1429f0$2b4ffea9@ericlaptop> Hey Michael, I haven't worked with Python2.2 at all yet, but I'm guessing that the .exe installer will only install with python2.1 because that is the system it was built with. There are quite a few extension modules (modules that are built from C code instead of written in Python) in SciPy, and these only work with one version of Python. Hence, the limitation to 2.1. The exe was built with distutils, so I guess the screen to choose a version in its install procedure, but it is also smart enough to only allow installation inot compatible versions of Python. So, for now, your stuck with 2.1. More and more people are asking for a 2.2 version, and it is definitely in the plans. Its safe to say that SciPy-0.2 will have both 2.1 and 2.2 exe available. When 0.2 will be released is still an open question. The main things left for 0.2 are getting the linalg and stats modules cleaned up and better tested. With the work of Travis O. and Pearu, linalg is well on its way. stats still is a bit neglected, but Louis L. is getting some unittests whipped up for it which will help. eric ----- Original Message ----- From: "Michael George Lerner" To: Sent: Wednesday, January 23, 2002 4:59 PM Subject: [SciPy-user] Need help installing with Python 2.2 / 2.1 on Windows > > Hi, > > I have both Python 2.2 and Python 2.1 installed on my Windows box (in > c:\python22 and c:\21). When I try to install SciPy with the binary > installer SciPy-0.1.win32-py2.1.exe, which I downloaded from the website > on Jan. 23, the second screen says "Select python installation to choose" > and only allows me to choose Python 2.1. It doesn't notice my Python 2.2 > installation, so it's not in the list at the top of the screen. Also, I > can't edit the text in the text entry box at the bottom of the screen to > manually point it at c:\python22. > > Is there some way to point SciPy at Python 2.2, or will I have to install > with version 2.1? > > thanks, > > -michael > > _______________________________________________ > SciPy-user mailing list > SciPy-user at scipy.net > http://www.scipy.net/mailman/listinfo/scipy-user > From Jay at BlueLineEngineering.com Thu Jan 24 11:06:15 2002 From: Jay at BlueLineEngineering.com (Jay L. de Halas) Date: Thu, 24 Jan 2002 09:06:15 -0700 Subject: [SciPy-user] Can't import plt, path issues with win32 ??? Message-ID: <3C5030F6.936E1AF6@BlueLineEngineering.com> This one is driving me nuts... I can use all of scipy's functionality within the Python Shell, but I can not import and use the plt function from within my path. However, I can import most of the other functions within the scipy module. I have tried appending the sys.path, changing the registry, modifing the autoexec file. I am way confused !!! Examples: 1. From within the Shell... Python 2.1.1 (#20, Jul 20 2001, 01:19:29) [MSC 32 bit (Intel)] on win32 Type "copyright", "credits" or "license" for more information. IDLE Fork 0.8 -- press F1 for help >>> sys.path ['C:\\Python21\\Tools\\idle_VPython', 'C:\\Python21\\Programs\\Demos', 'C:\\PYTHON21\\DLLs', 'C:\\PYTHON21\\lib', 'C:\\PYTHON21\\lib\\plat-win', 'C:\\PYTHON21\\lib\\lib-tk', 'C:\\PYTHON21', 'C:\\PYTHON21\\Numeric'] >>> from scipy import plt >>> import gui_thread >>> plt.plot((1,2,3)) >>> >>> **** Works Fine **** 2. From my test script..... # import the necessary modules import sys #sys.path.append('C:\Python21\wxPython') sys.path.append('C:\Python21\scipy') #sys.path.append('C:\Python21\scipy\plt') sys.path.append('C:\PYTHON21\scipy\gui_thread') print '\n', sys.path, '\n' import gui_thread from scipy import plt from scipy import * print '\n', dir(scipy), '\n' # create some data to plot x = arange(0,5,.5) y = special.j1(x) plt.plot(x,y) # now add some titles plt.title('First Order Bessel Function') plt.xtitle('x axis') plt.ytitle('y axis') # and x-axis bounds equal to that of the data plt.xaxis('fit') 2a. The output..... ['c:\\python21', 'C:\\Python21\\Programs\\Demos', 'C:\\PYTHON21\\DLLs', 'C:\\PYTHON21\\lib', 'C:\\PYTHON21\\lib\\plat-win', 'C:\\PYTHON21\\lib\\lib-tk', 'C:\\PYTHON21', 'C:\\PYTHON21\\Numeric', 'C:\\Python21\\scipy', 'C:\\PYTHON21\\scipy\\gui_thread'] Traceback (innermost last) File "c:\python21\test02.py", line 14, in ? from scipy import plt SystemError: C:\Code\2.1.1\dist\src\Objects\listobject.c:69: bad argument to internal function Program disconnected. **** Does NOT Work**** Anyhow, I suspect that this is a basic problem, but it has me stumped !!! Thanks !!! and best regards Jay