From ralf.gommers at googlemail.com Mon Aug 1 12:31:09 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Mon, 1 Aug 2011 18:31:09 +0200 Subject: [SciPy-Dev] setup.py develop In-Reply-To: References: Message-ID: On Mon, Aug 1, 2011 at 2:16 AM, Gustavo Goretkin wrote: > Hey Devs, > > I'm making some tiny local changes to SciPy and I'd like to install it in > virtualenv using setup.py develop (as opposed to setup.py install) as I've > done for other projects. What's the way to do this for SciPy? > > http://projects.scipy.org/numpy/wiki/DevelopmentTips Cheers, Ralf > Thanks, > Gustavo > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fperez.net at gmail.com Wed Aug 3 03:40:46 2011 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 3 Aug 2011 00:40:46 -0700 Subject: [SciPy-Dev] [ANN] IPython 0.11 is officially out In-Reply-To: References: Message-ID: On Sun, Jul 31, 2011 at 10:19 AM, Fernando Perez wrote: > Please see our release notes for the full details on everything about > this release: https://github.com/ipython/ipython/zipball/rel-0.11 And embarrassingly, that URL was for a zip download instead (copy/paste error), the detailed release notes are here: http://ipython.org/ipython-doc/rel-0.11/whatsnew/version0.11.html Sorry about the mistake... Cheers, f From ben.root at ou.edu Wed Aug 3 21:31:48 2011 From: ben.root at ou.edu (Benjamin Root) Date: Wed, 3 Aug 2011 20:31:48 -0500 Subject: [SciPy-Dev] setup.py develop In-Reply-To: References: Message-ID: On Mon, Aug 1, 2011 at 11:31 AM, Ralf Gommers wrote: > > > On Mon, Aug 1, 2011 at 2:16 AM, Gustavo Goretkin < > gustavo.goretkin at gmail.com> wrote: > >> Hey Devs, >> >> I'm making some tiny local changes to SciPy and I'd like to install it in >> virtualenv using setup.py develop (as opposed to setup.py install) as I've >> done for other projects. What's the way to do this for SciPy? >> >> http://projects.scipy.org/numpy/wiki/DevelopmentTips > > Cheers, > Ralf > > Just a quick question... how are the two commands: python setup.py build_ext -i python setupegg.py develop --prefix=${HOME} different from just simply doing: python setupegg.py develop --user besides from the obvious that the final location is slightly different (with --user, the location on more recent linuxes is ~/.local). Thanks, Ben Root -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Thu Aug 4 13:11:07 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 4 Aug 2011 19:11:07 +0200 Subject: [SciPy-Dev] setup.py develop In-Reply-To: References: Message-ID: On Thu, Aug 4, 2011 at 3:31 AM, Benjamin Root wrote: > On Mon, Aug 1, 2011 at 11:31 AM, Ralf Gommers > wrote: > >> >> >> On Mon, Aug 1, 2011 at 2:16 AM, Gustavo Goretkin < >> gustavo.goretkin at gmail.com> wrote: >> >>> Hey Devs, >>> >>> I'm making some tiny local changes to SciPy and I'd like to install it in >>> virtualenv using setup.py develop (as opposed to setup.py install) as I've >>> done for other projects. What's the way to do this for SciPy? >>> >>> http://projects.scipy.org/numpy/wiki/DevelopmentTips >> >> Cheers, >> Ralf >> >> > Just a quick question... how are the two commands: > > python setup.py build_ext -i > python setupegg.py develop --prefix=${HOME} > > different from just simply doing: > > python setupegg.py develop --user > > besides from the obvious that the final location is slightly different > (with --user, the location on more recent linuxes is ~/.local). > > --user is not documented ( http://packages.python.org/distribute/setuptools.html#develop-deploy-the-project-source-in-development-mode), so I'm not sure. But of these three options I prefer "python setup.py build_ext -i" since that's the only one that doesn't invoke setuptools/distribute. That's a useful feature... Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Sat Aug 6 04:12:51 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sat, 6 Aug 2011 04:12:51 -0400 Subject: [SciPy-Dev] documenting scipy.special functions Message-ID: This came up in https://github.com/scipy/scipy/pull/52 Josef: One more comment about scipy.special docstrings: From what I have seen they are semiautomatically created. I think eventually we should switch to docstrings following the numpy standard. There are several functions where I would have liked to add more information. The current docstrings of many special functions are awfully uninformative about details. Chris: As for the comments...that's actually not as simple as it sounds. Or at least it wasn't. I just submitted a pull request to numpy to help change that. ufuncs are python objects whose docstrings are read-only. They are set at object creation. The either (1)must be passed in to the C function creating the ufunc (at which point you have to have the docs in some C header file or parse them in from a Python header file with the docs), or (2) you must reset what the ufunc doc pointer points to at the C level. The add_newdoc method doesn't work on ufuncs. (And in fact there are a number of calls to add_newdoc in numpy/add_newdocs.py that don't change the documentation. It isn't noticed because add_newdoc specifically catches all errors and does nothing.) --- Is it possible to improve the docstring for the scipy.special functions (in the long run)? I'm just a consumer of scipy.special, but every once in a while I would like to add some information to special functions. for some it looks possible to edit (and bring them to numpy standard), e.g. http://docs.scipy.org/scipy/docs/scipy.special.orthogonal.hermite/#hermite others, for example http://docs.scipy.org/scipy/docs/scipy.special._cephes.pdtr/#pdtr , cannot be edited, but there is a lot of information hidden in the fortran files, if one looks for it long enough. Josef From jordigh at octave.org Sun Aug 7 16:26:50 2011 From: jordigh at octave.org (=?UTF-8?Q?Jordi_Guti=C3=A9rrez_Hermoso?=) Date: Sun, 7 Aug 2011 15:26:50 -0500 Subject: [SciPy-Dev] ARPACK situation Message-ID: (Mass email, please hit reply-all to keep everyone abreast of the situation. May get some bounces from mailing lists.) I'm writing this email to discuss the future of ARPACK. The problem is this: it's a widely-used library, but it seems abandoned upstream (and upstream, to whom this is addressed, can confirm or deny). This has resulted in the problem of many mini-forks as each organisation distributes ARPACK patches its own way, and very often, for the same bugs. These are the ones I could find: http://hg.savannah.gnu.org/hgweb/octave/file/tip/libcruft/arpack/ http://patch-tracker.debian.org/package/arpack/2.1+parpack96.dfsg-3 https://github.com/inducer/arpack http://mathema.tician.de/software/arpack http://dev.gentoo.org/~bicatali/ http://pkgs.org/centos-5-rhel-5/epel-x86_64/arpack-2.1-13.el5.x86_64.rpm.html https://github.com/scipy/scipy/tree/fa21e840ad69fbac7ff600a7ef2b36929c18b975/scipy/sparse/linalg/eigen/arpack http://bazaar.launchpad.net/~igraph/igraph/0.5-main/files/1139.1.143/src/arpack/ Additionally, the Mathworks (they make Matlab) probably also has their own version of ARPACK, but I wasn't able to find a public version of it, nor an email to send them questions to. If someone could contact them, it would be nice to let them know. These all seem to have modified ARPACK in some way, with minor or major bugfixes, and as far as I can tell, have mostly done so independently. To me, this seems like unnecessary work, if we're all patching the library again and again and making our own private forks. What I therefore propose is to have some sort of central location for it and we all pool our efforts on this one location. I think it would be easiest to use Andreas Kl?ckner's existing fork on github, since this requires the least maintenance and work from anyone. All that it requires for now is for each of the people above to see what patches they have made and transplant them to the git repo. It would be helpful if upstream could confirm that they are happy with ARPACK development continuing on github and mention this on the ARPACK webpage, so that new people who are interested on ARPACK can be redirected. Thanks, - Jordi G. H. GNU Octave developer From jordigh at octave.org Mon Aug 8 10:45:21 2011 From: jordigh at octave.org (=?UTF-8?Q?Jordi_Guti=C3=A9rrez_Hermoso?=) Date: Mon, 8 Aug 2011 09:45:21 -0500 Subject: [SciPy-Dev] ARPACK situation In-Reply-To: References: Message-ID: (If you missed my first email, this is the context: http://jordi.inversethought.com/blog/arpack-situation/ and you're receiving this because I thought you were involved with ARPACK distribution or maintenance.) On 7 August 2011 22:04, Lehoucq, Richard wrote: > Thank you for your well articulated email. ARPACK is the property of > Rice University and so any "upstream" decision ultimately resides > with Rice. I understand. Can you give us contact information for someone at Rice University who may could be in a position to decide to change the ARPACK website to point to a hypothetical community-maintained version of the software? Addresses like arpack at caam.rice.edu obtained from this website have bounced. Thanks again, - Jordi G. H. GNU Octave Developer From bdeb at willmore.eu Tue Aug 9 12:09:48 2011 From: bdeb at willmore.eu (Ben Willmore) Date: Tue, 9 Aug 2011 17:09:48 +0100 Subject: [SciPy-Dev] installation of scipy on Mac OS X 10.7 Message-ID: I have spent a day attempting to get a version of scipy that passes unit tests on Mac OS X 10.7. I hope the results of my investigations (below) are useful to others trying to do the same thing. Ben == Introduction I have attempted to obtain a version of scipy on Mac OS X 10.7 that passes unit tests, both by downloading it, or by compiling against the system's python, using the three C compilers available, and various compile flags. == Conclusions 1. Both Enthought Python 7.1 and the scipy superpack crash during scipy unit tests. 2. Compilation with the default C compiler (llvm-gcc-4.2) also leads to a crash during unit tests. 3. Compilation using either of the other C compilers (gcc-4.2 or clang) allows scipy to complete unit tests, only if the fortran flag --ff2c is used. In both cases, scipy passes almost all tests -- producing errors on one, and failing two more. With gcc-4.2, numpy passes all tests. With clang, numpy fails one test. 4. The following recipe was most successful for me: * Remove everything in /usr/local and /Library/Python/2.7/site-packages (BEWARE! These directories may well contain stuff you care about) * Install gfortran from here: http://r.research.att.com/gfortran-lion-5666-3.pkg mkdir ~/tmp cd ~/tmp git clone git://github.com/numpy/numpy.git cd numpy export CC=gcc-4.2 export CXX=g++-4.2 export FFLAGS=-ff2c python setupegg.py build --fcompiler=gfortran sudo python setupegg.py install cd ~/tmp git clone git://github.com/scipy/scipy.git cd scipy export CC=gcc-4.2 export CXX=g++-4.2 export FFLAGS=-ff2c python setupegg.py build --fcompiler=gfortran sudo python setupegg.py install (repeat for matplotlib, ipython ...) == Details of test results = Enthought python 7.1 numpy.test(): OK scipy.test(): Crash (Segmentation fault) = Scipy superpack numpy.test(): 3 failures: FAIL: test_umath_complex.TestCsqrt.test_special_values(, 1, inf, inf, inf) FAIL: test_umath_complex.TestCsqrt.test_special_values(, -1, inf, inf, inf) FAIL: test_umath_complex.TestCsqrt.test_special_values(, 0.0, inf, inf, inf) FAIL: test_umath_complex.TestCsqrt.test_special_values(, -0.0, inf, inf, inf) FAIL: test_umath_complex.TestCsqrt.test_special_values(, inf, inf, inf, inf) FAIL: test_umath_complex.TestCsqrt.test_special_values(, -inf, inf, inf, inf) FAIL: test_umath_complex.TestCsqrt.test_special_values(, nan, inf, inf, inf) FAIL: test_umath_complex.TestCsqrt.test_special_values(, -inf, 1, 0.0, inf) scipy.test(): Crash (Abort trap) = Compilation from scratch (see above for recipe) = llvm-gcc-4.2 = (no compiler flags) numpy.test(): OK scipy.test(): Crash (Segmentation fault: 11) = gcc-4.2 = CC=gcc-4.2 CXX=g++-4.2 numpy.test(): OK scipy.test(): Hang (during ARPACK tests) = clang = CC=clang CXX=clang++ numpy.test(): 1 failure: FAIL: Test basic arithmetic function errors AssertionError: Type did not raise fpe error ''. scipy.test(): Hang (during ARPACK tests) = llvm-gcc-4.2 and -ff2c = FFLAGS=-ff2c numpy.test(): OK scipy.test(): Crash (Segmentation fault: 11) = gcc-4.2 and -ff2c = CC=gcc-4.2 CXX=g++-4.2 FFLAGS=-ff2c numpy.test(): OK scipy.test(): 1 error, 2 failures: ERROR: test_arpack.test_hermitian_modes(True, , 'F', 2, 'SA', None, None, ) FAIL: test_iterative.test_convergence(, ) FAIL: test_expon (test_morestats.TestAnderson) = clang and -ff2c = CC=clang CXX=clang++ FFLAGS=-ff2c numpy.test(): 1 failure: FAIL: Test basic arithmetic function errors AssertionError: Type did not raise fpe error ''. scipy.test(): 1 error, 2 failures: ERROR: test_arpack.test_hermitian_modes(True, , 'F', 2, 'SA', None, None, ) FAIL: test_iterative.test_convergence(, ) FAIL: test_expon (test_morestats.TestAnderson) From cimrman3 at ntc.zcu.cz Wed Aug 10 08:55:50 2011 From: cimrman3 at ntc.zcu.cz (Robert Cimrman) Date: Wed, 10 Aug 2011 14:55:50 +0200 Subject: [SciPy-Dev] ANN: SfePy 2011.3 Message-ID: <4E427FD6.5000707@ntc.zcu.cz> I am pleased to announce release 2011.3 of SfePy. Description ----------- SfePy (simple finite elements in Python) is a software for solving systems of coupled partial differential equations by the finite element method. The code is based on NumPy and SciPy packages. It is distributed under the new BSD license. Home page: http://sfepy.org Mailing lists, issue tracking: http://code.google.com/p/sfepy/ Git (source) repository: http://github.com/sfepy Documentation: http://docs.sfepy.org/doc Highlights of this release -------------------------- - major update of terms aiming at easier usage and definition while retaining original C functions - overriding problem description items on command line - improved developer guide - Primer tutorial - a step-by-step walk-through of the process to solve a simple mechanics problem For more information on this release, see http://sfepy.googlecode.com/svn/web/releases/2011.3_RELEASE_NOTES.txt (full release notes, rather long and technical). Best regards, Robert Cimrman and Contributors (*) (*) Contributors to this release (alphabetical order): Vladim?r Luke?, Maty?? Nov?k, Andre Smit From martin.teichmann at lkb.ens.fr Thu Aug 11 03:00:55 2011 From: martin.teichmann at lkb.ens.fr (Martin Teichmann) Date: Thu, 11 Aug 2011 09:00:55 +0200 Subject: [SciPy-Dev] QR-decomposition with Q but not Q Message-ID: Hi list, A QR decomposition is done in two steps: firstly, R is calculated and an intermediate form of Q (the so called "elementary reflectors"), and in a second step, Q is actually calculated. The second step, however, can be costly and thus is not done if you are not interested in Q. The QR decomposition code already allows for not calculating Q. Very often, however, one is interested in Q only in order to multiply it with a vector c. This can be done without ever calculating Q. I just wrote a patch and submitted it to github, it's here: https://github.com/scipy/scipy/pull/55 which does just this. I'm not sure if I should write both on scipy-dev and github, scipy-dev just seems to have faster responses... Greetings Martin -- Max-Born-Institut Max-Born-Stra?e 2a 12489 Berlin +49 30 6392 1234 From lkb.teichmann at gmail.com Thu Aug 11 03:02:21 2011 From: lkb.teichmann at gmail.com (Martin Teichmann) Date: Thu, 11 Aug 2011 09:02:21 +0200 Subject: [SciPy-Dev] QR-decomposition with Q but not Q In-Reply-To: References: Message-ID: Hi list, A QR decomposition is done in two steps: firstly, R is calculated and an intermediate form of Q (the so called "elementary reflectors"), and in a second step, Q is actually calculated. The second step, however, can be costly and thus is not done if you are not interested in Q. The QR decomposition code already allows for not calculating Q. Very often, however, one is interested in Q only in order to multiply it with a vector c. This can be done without ever calculating Q. I just wrote a patch and submitted it to github, it's here: https://github.com/scipy/scipy/pull/55 which does just this. I'm not sure if I should write both on scipy-dev and github, scipy-dev just seems to have faster responses... Greetings Martin -- Max-Born-Institut Max-Born-Stra?e 2a 12489 Berlin +49 30 6392 1234 From njs at pobox.com Thu Aug 11 11:47:50 2011 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 11 Aug 2011 08:47:50 -0700 Subject: [SciPy-Dev] QR-decomposition with Q but not Q In-Reply-To: References: Message-ID: On Thu, Aug 11, 2011 at 12:00 AM, Martin Teichmann wrote: > Hi list, > > A QR decomposition is done in two steps: firstly, > R is calculated and an intermediate form of Q > (the so called "elementary reflectors"), and in a second > step, Q is actually calculated. > > The second step, however, can be costly and > thus is not done if you are not interested in Q. The > QR decomposition code already allows for not > calculating Q. > > Very often, however, one is interested in Q only in > order to multiply it with a vector c. This can be done > without ever calculating Q. I made a comment there on an infelicity in the docs, but here are some more general comments: I was a little confused by your removing the scipy tril/triu functions. Certainly they're redundant with the ones in numpy, but does this have any consequences for compatibility? Do they need to be deprecated for a cycle first? Does the "economic" versus "full" distinction still make sense when working with the reflectors? Because if so, then you should add a new argument (like 'reflectors=True/False') instead of overloading the mode= argument. IIUC, the two options you add are (1) get out the elementary reflections themselves, (2) get out the product of those elementary reflections with some matrix c. Wouldn't it make more sense to have just one option, for getting out the elementary reflections, and then have a separate function that let you compute the product of some elementary reflections with some matrix? It seems like you need that anyway if anyone is to make any use of the elementary reflections from option (1) -- and it allows more flexible use, like if you have lots of c matrices that you need to multiply Q by -- but then once you have such a function then option (2) is redundant and over-complicated. -- Nathaniel From lkb.teichmann at gmail.com Thu Aug 11 14:01:08 2011 From: lkb.teichmann at gmail.com (Martin Teichmann) Date: Thu, 11 Aug 2011 20:01:08 +0200 Subject: [SciPy-Dev] QR-decomposition with Q but not Q In-Reply-To: References: Message-ID: Hi list, Hi Nathaniel, > I was a little confused by your removing the scipy tril/triu > functions. Certainly they're redundant with the ones in numpy, but > does this have any consequences for compatibility? Do they need to be > deprecated for a cycle first? Well, yes. But I guess it's much simpler to just import them from numpy, than repeating the code. Some "from numpy import triu, tril" should do the job, I am just not sure where to put it. > Does the "economic" versus "full" distinction still make sense when > working with the reflectors? No. The reflectors always have one fixed size. (Let me double check that I got the right one...) > IIUC, the two options you add are (1) get out the elementary > reflections themselves, (2) get out the product of those elementary > reflections with some matrix c. Wouldn't it make more sense to have > just one option, for getting out the elementary reflections, and then > have a separate function that let you compute the product of some > elementary reflections with some matrix? I thought about that too, and I am not sure. I opted for the way I did it because I think it gives more readable code, especially for beginners. You don't need to explain what elementary reflectors are, and what to do with them. In theory, one could make an entire object-oriented approach with a class that automatically does the right job, but I think that's just overkill. The problem that one might want to multiply with Q several times is normally not an issue: in this case it is more efficient to calculate Q and just use dot to do the multiplication. Only if Q is large and c is small the internal multiplication is an advantage. Greetings Martin From sturla at molden.no Thu Aug 11 17:01:30 2011 From: sturla at molden.no (Sturla Molden) Date: Thu, 11 Aug 2011 23:01:30 +0200 Subject: [SciPy-Dev] QR-decomposition with Q but not Q In-Reply-To: References: Message-ID: <4E44432A.7010600@molden.no> Regarding the "costlyness" of QR and SVD: Very often we are interested in QR or SVD only to compute a least-squares fit. To save time and memory instead of computing the full QR or SVD, LAPACK has specialized drivers for least-squares problems: *GELS lstsq with QR *GELSS lstsq with SVD *GELSD lstsq with SVD (divide and conquer) *GGGLM general linear model with QR Some of these are exposed in SciPy or statmodels (I think, but I like to do it myself...) Anyway, I don't think it's worth the effort to duplicate these drivers with specialized QR code, if we only need the QR for least-squares. If anyone is interested, I'm attaching some Fortran 95 code I've used for the purpose (porting to Python/NumPy is easy). Array arguments and shapes are similar to standard convention in statistics, c.f. Montgomery & Peck (1992), Introduction to linear regression analysis. Sturla Den 11.08.2011 09:00, skrev Martin Teichmann: > Hi list, > > A QR decomposition is done in two steps: firstly, > R is calculated and an intermediate form of Q > (the so called "elementary reflectors"), and in a second > step, Q is actually calculated. > > The second step, however, can be costly and > thus is not done if you are not interested in Q. The > QR decomposition code already allows for not > calculating Q. > > Very often, however, one is interested in Q only in > order to multiply it with a vector c. This can be done > without ever calculating Q. > > I just wrote a patch and submitted it to github, it's here: > https://github.com/scipy/scipy/pull/55 > which does just this. I'm not sure if I should write both > on scipy-dev and github, scipy-dev just seems to > have faster responses... > > Greetings > > Martin > -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: linalg.f90 URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: linear_regression.f95 URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: numtypes.f95 URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: test_regression.f95 URL: From martin.teichmann at mbi-berlin.de Fri Aug 12 03:07:30 2011 From: martin.teichmann at mbi-berlin.de (Martin Teichmann) Date: Fri, 12 Aug 2011 09:07:30 +0200 Subject: [SciPy-Dev] QR-decomposition with Q but not Q In-Reply-To: <4E44432A.7010600@molden.no> References: <4E44432A.7010600@molden.no> Message-ID: Hi List, Hi Sturla, > Some of these are exposed in SciPy or statmodels (I think, but I like to do > it myself...) Anyway, I don't think it's worth the effort to duplicate these > drivers with specialized QR code, if we only need the QR for least-squares. so, I'm working on the Marquard-Levenberg fitting algorithm, and indeed, I need not much more than least-squares solving. In princple, I think, I could use the LAPACK routines for that. However, I definitely prefer the readability of python over FORTRAN. I'm not interested in the last bit of speed, my computer is fast enough. Profiling my all-python (with numpy and scipy for sure) ML fit, I realized that most time is spent in the QR factorization. By doing this in FORTRAN, I would be able to speed up everything fast enough to not have to wait for fit results anymore. > If anyone is interested, I'm attaching some Fortran 95 code I've used for > the purpose (porting to Python/NumPy is easy). I personally do not think we should start writing our own FORTRAN code in scipy. FORTRAN is ugly. Python is pretty. And with cython you can get as fast as FORTRAN. Sure, it does make sense to use already existing FORTRAN subroutines, it would be ridiculous to discard some 50 years worth of optimization, and yes, calling FORTRAN code from cython is not so simple, but that can be changed. Greetings Martin From sturla at molden.no Fri Aug 12 14:01:05 2011 From: sturla at molden.no (Sturla Molden) Date: Fri, 12 Aug 2011 20:01:05 +0200 Subject: [SciPy-Dev] QR-decomposition with Q but not Q In-Reply-To: References: <4E44432A.7010600@molden.no> Message-ID: <4E456A61.3050702@molden.no> Den 12.08.2011 09:07, skrev Martin Teichmann: > so, I'm working on the Marquard-Levenberg fitting algorithm, and > indeed, I need not much more than least-squares solving. In princple, > I think, I could use the LAPACK routines for that. Levenberg-Marquard is better implemented with LAPACK method DGELS than using QR manually. Also consider that scipy already has Levenberg-Marquard in scipy.optimize.leastsq, which is a wrapper for MINPACK routines LMDER and LMDIF. Sturla From charlesr.harris at gmail.com Sat Aug 13 19:54:19 2011 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 13 Aug 2011 17:54:19 -0600 Subject: [SciPy-Dev] QR-decomposition with Q but not Q In-Reply-To: <4E456A61.3050702@molden.no> References: <4E44432A.7010600@molden.no> <4E456A61.3050702@molden.no> Message-ID: On Fri, Aug 12, 2011 at 12:01 PM, Sturla Molden wrote: > Den 12.08.2011 09:07, skrev Martin Teichmann: > > > so, I'm working on the Marquard-Levenberg fitting algorithm, and > > indeed, I need not much more than least-squares solving. In princple, > > I think, I could use the LAPACK routines for that. > > Levenberg-Marquard is better implemented with LAPACK method DGELS than > using QR manually. Also consider that scipy already has > Levenberg-Marquard in scipy.optimize.leastsq, which is a wrapper for > MINPACK routines LMDER and LMDIF. > > Although it would be nice if someone added the ability to mask some variables. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Wed Aug 17 12:11:50 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 17 Aug 2011 18:11:50 +0200 Subject: [SciPy-Dev] scipy.interpolate.Rbf default epsilon In-Reply-To: <1311252457.2283.72.camel@razi> References: <1311252457.2283.72.camel@razi> Message-ID: I opened http://projects.scipy.org/scipy/ticket/1495 to keep track of this. Ralf On Thu, Jul 21, 2011 at 2:47 PM, J.J. Green wrote: > Hi all, > > I believe that there is a problem with the default value > of the "epsilon" parameter in the scipy.interpolate.Rbf > initialisation function. > > In the original documentation for the library from which > this package is derived (Matlab code by Alex Chirokov) > the author states that epsilon should be "the average > distance between nodes", but means "the average distance > between consecutive nodes" (slide 10 of the ppt included > with the matlab package, available here > > > http://www.mathworks.com/matlabcentral/fileexchange/10056-scattered-data-interpolation-and-approximation-using-radial-base-functions > ). > > The distinction is not clear as the author deals with the > one dimensional case in his example. > > The version in scipy.interpolate.Rbf takes this advice literally > and calculates the mean distance between nodes, so uses a > value of epsilon which is *much* too large in general. > > I noticed this when getting bad results interpolating from > 300 data-points on a region 4,500 square, the mean distance > between *nearest* data-points is around 330, but the default > epsilon used by the package was 2,500. > > To fix this one could use the method from the original package > (matlab) > > (prod(max(x')-min(x'))/nXCount)^(1/nXDim); > > ie, take the volume of the bounding cube, divide by the number > of points and take the n-th root (n = number of dimensions). > > This approach would give 260 for my example above. > > Cheers > > Jim > -- > J.J. Green > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at enthought.com Wed Aug 17 19:08:12 2011 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Wed, 17 Aug 2011 18:08:12 -0500 Subject: [SciPy-Dev] Target date for 0.10 release Message-ID: Just checking: the roadmap (http://projects.scipy.org/scipy/roadmap) says Sept. 30 for the release of 0.10. Is that still the plan? Warren -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Wed Aug 17 20:13:50 2011 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 18 Aug 2011 00:13:50 +0000 (UTC) Subject: [SciPy-Dev] Target date for 0.10 release References: Message-ID: On Wed, 17 Aug 2011 18:08:12 -0500, Warren Weckesser wrote: > Just checking: the roadmap (http://projects.scipy.org/scipy/roadmap) > says Sept. 30 for the release of 0.10. Is that still the plan? Possibly -- there are already some new features and bugfixes. However, the crash-bugs that occur in the iterative solvers on OSX (but not on other platforms) in my opinion are a blocker that should be solved before release. Pauli From ralf.gommers at googlemail.com Thu Aug 18 02:58:51 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 18 Aug 2011 08:58:51 +0200 Subject: [SciPy-Dev] Target date for 0.10 release In-Reply-To: References: Message-ID: On Thu, Aug 18, 2011 at 2:13 AM, Pauli Virtanen wrote: > On Wed, 17 Aug 2011 18:08:12 -0500, Warren Weckesser wrote: > > Just checking: the roadmap (http://projects.scipy.org/scipy/roadmap) > > says Sept. 30 for the release of 0.10. Is that still the plan? > > Possibly -- there are already some new features and bugfixes. The 0.10 release notes are still a bit sparse. A few things that should be in there but aren't yet: - Lomb-Scargle periodogram - firwin types III/IV - update to CODATA 2010 - deprecated lib.blas & some cleanup in linalg.blas There are also quite a few open pull requests. Here are some that I think are in good shape and would be good to merge before a first beta: 4: more stable BLAS routines for euclidean norm 11: support for Harwell-Boeing format 21: discrete state-space, transfer function and LTI systems 35: add description of public API and import guidelines 48: chkfinite flags for linalg functions 52: logit/expit functions > However, > the crash-bugs that occur in the iterative solvers on OSX (but not on > other platforms) in my opinion are a blocker that should be solved > before release. > Agreed. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Thu Aug 18 04:16:01 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 18 Aug 2011 10:16:01 +0200 Subject: [SciPy-Dev] Target date for 0.10 release In-Reply-To: References: Message-ID: On Thu, Aug 18, 2011 at 8:58 AM, Ralf Gommers wrote: > > > On Thu, Aug 18, 2011 at 2:13 AM, Pauli Virtanen wrote: > >> On Wed, 17 Aug 2011 18:08:12 -0500, Warren Weckesser wrote: >> > Just checking: the roadmap (http://projects.scipy.org/scipy/roadmap) >> > says Sept. 30 for the release of 0.10. Is that still the plan? >> >> Possibly -- there are already some new features and bugfixes. > > > The 0.10 release notes are still a bit sparse. A few things that should be > in there but aren't yet: > - Lomb-Scargle periodogram > - firwin types III/IV > - update to CODATA 2010 > - deprecated lib.blas & some cleanup in linalg.blas > > I also checked the pending deprecations/removals. The only deprecated items in 0.9 were the following nonlinear solvers: - ``broyden_modified`` (bad performance) - ``broyden1_modified`` (bad performance) - ``broyden_generalized`` (equivalent to ``anderson``) - ``anderson2`` (equivalent to ``anderson``) - ``broyden3`` (obsoleted by new limited-memory broyden methods) - ``vackar`` (renamed to ``diagbroyden``) Can these be removed? Ralf There are also quite a few open pull requests. Here are some that I think > are in good shape and would be good to merge before a first beta: > > 4: more stable BLAS routines for euclidean norm > 11: support for Harwell-Boeing format > 21: discrete state-space, transfer function and LTI systems > 35: add description of public API and import guidelines > 48: chkfinite flags for linalg functions > 52: logit/expit functions > > > >> However, >> the crash-bugs that occur in the iterative solvers on OSX (but not on >> other platforms) in my opinion are a blocker that should be solved >> before release. >> > > Agreed. > > Ralf > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From anton.soppelsa at igi.cnr.it Fri Aug 19 11:28:02 2011 From: anton.soppelsa at igi.cnr.it (Soppelsa Anton) Date: Fri, 19 Aug 2011 17:28:02 +0200 Subject: [SciPy-Dev] Problem building Scipy: unrecognized command line option "-mp" In-Reply-To: References: Message-ID: <4E4E8102.4070307@igi.cnr.it> Dear all, I'm experiencing nasty problems building SciPy 0.9.0 on a amd64 machine running Linux using gcc and gfortran. In particular I get an error by gfortran: f951: error: unrecognized command line option "-mp". BTW, I noticed that the string "-openmp -mp -I/usr/local/vni/imsl/fnl600/lnxin100e64/include" comes always together (sometimes two times!?!). I looked for the "-openmp" and/or "-mp" string in the entire numpy source code without being able to find it. My questions are: from where does come these options? How can I disable the option -mp? With this configuration I was able to build Python and NumPy, but not SciPy. The details follow. Sincerely, A. Soppelsa [System]: uname -a Linux ra22.igi.cnr.it 2.6.18-53.el5 #1 SMP Wed Oct 10 16:34:19 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux cat /etc/*-release Red Hat Enterprise Linux Server release 5.1 (Tikanga) cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 15 model name : Intel(R) Xeon(R) CPU E7330 @ 2.40GHz stepping : 11 cpu MHz : 2400.093 cache size : 3072 KB physical id : 0 siblings : 4 core id : 0 cpu cores : 4 fpu : yes fpu_exception : yes cpuid level : 10 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall lm constant_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr lahf_lm bogomips : 4803.75 clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management: 16x of them. [Environment]: export PATH=/usr/local/bin:/usr/bin:/bin export LD_LIBRARY_PATH=/usr/lib64 I polished the PATH and LD_LIBRARY_PATH because distutil wasn't able to use gfortran after having correctly found it. (Drop distutils, please) [Site.cfg]: [DEFAULT] #library_dirs = /usr/local/lib #include_dirs = /usr/local/include library_dirs = /home/soppelsa/local/lib include_dirs = /home/soppelsa/local/include # Optimized BLAS and LAPACK # ------------------------- # Use the blas_opt and lapack_opt sections to give any settings that are # required to link against your chosen BLAS and LAPACK, including the regular # FORTRAN reference BLAS and also ATLAS. Some other sections still exist for # linking against certain optimized libraries (e.g. [atlas], [lapack_atlas]), # however, they are now deprecated and should not be used. # # These are typical configurations for ATLAS (assuming that the library and # include directories have already been set in [DEFAULT]; the include directory # is important for the BLAS C interface): # [blas_opt] library_dirs = /usr/local/intel/mkl/10.1.0.015/lib/em64t/ #libraries = mkl_intel_lp64 mkl_intel_thread libraries = mkl [lapack_opt] library_dirs = /usr/local/intel/mkl/10.1.0.015/lib/em64t/ #libraries = mkl_lapack mkl_core guide pthread libraries = mkl_lapack # UMFPACK # ------- # The UMFPACK library is used in scikits.umfpack to factor large sparse matrices. # It, in turn, depends on the AMD library for reordering the matrices for # better performance. Note that the AMD library has nothing to do with AMD # (Advanced Micro Devices), the CPU company. # # UMFPACK is not needed for numpy or scipy. # # http://www.cise.ufl.edu/research/sparse/umfpack/ # http://www.cise.ufl.edu/research/sparse/amd/ # http://scikits.appspot.com/umfpack # [amd] #src_dirs = /home/soppelsa/local/src/AMD/Source #include_dirs = /home/soppelsa/local/src/AMD/Include /home/soppelsa/local/src/UFconfig #library_dirs = /home/soppelsa/local/src/AMD/Lib amd_libs = amd # [umfpack] #src_dirs = /home/soppelsa/local/src/UMFPACK/Source #include_dirs = /home/soppelsa/local/src/UMFPACK/Include /home/soppelsa/local/src/UFconfig #library_dirs = /home/soppelsa/local/src/UMFPACK/Lib umfpack_libs = umfpack # FFT libraries # ------------- # There are two FFT libraries that we can configure here: FFTW (2 and 3) and djbfft. # Note that these libraries are not needed for numpy or scipy. # # http://fftw.org/ # http://cr.yp.to/djbfft.html # # Given only this section, numpy.distutils will try to figure out which version # of FFTW you are using. #[fftw] #libraries = fftw3 # # For djbfft, numpy.distutils will look for either djbfft.a or libdjbfft.a . #[djbfft] #include_dirs = /usr/local/djbfft/include #library_dirs = /usr/local/djbfft/lib # MKL #---- # For recent (9.0.21, for example) mkl, you need to change the names of the # lapack library. Assuming you installed the mkl in /opt, for a 32 bits cpu: # [mkl] # library_dirs = /opt/intel/mkl/9.1.023/lib/32/ # lapack_libs = mkl_lapack # # For 10.*, on 32 bits machines: [mkl] library_dirs = /usr/local/intel/mkl/10.1.0.015/lib/em64t/ lapack_libs = mkl_lapack mkl_libs = mkl, guide #mkl_libs = mkl_intel_lp64 mkl_intel_thread mkl_lapack mkl_core guide pthread [Build command and output]: ~/local/bin/python setup.py build --fcompiler=gnu95 --compiler=unix install Warning: No configuration returned, assuming unavailable.blas_opt_info: blas_mkl_info: FOUND: libraries = ['mkl', 'guide', 'pthread'] library_dirs = ['/usr/local/intel/mkl/10.1.0.015/lib/em64t/'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/home/soppelsa/local/include'] FOUND: libraries = ['mkl', 'guide', 'pthread'] library_dirs = ['/usr/local/intel/mkl/10.1.0.015/lib/em64t/'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/home/soppelsa/local/include'] non-existing path in 'scipy/io': 'docs' lapack_opt_info: lapack_mkl_info: mkl_info: FOUND: libraries = ['mkl', 'guide', 'pthread'] library_dirs = ['/usr/local/intel/mkl/10.1.0.015/lib/em64t/'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/home/soppelsa/local/include'] FOUND: libraries = ['mkl_lapack', 'mkl', 'guide', 'pthread'] library_dirs = ['/usr/local/intel/mkl/10.1.0.015/lib/em64t/'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/home/soppelsa/local/include'] FOUND: libraries = ['mkl_lapack', 'mkl', 'guide', 'pthread'] library_dirs = ['/usr/local/intel/mkl/10.1.0.015/lib/em64t/'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/home/soppelsa/local/include'] umfpack_info: amd_info: FOUND: libraries = ['amd'] library_dirs = ['/home/soppelsa/local/lib'] swig_opts = ['-I/home/soppelsa/local/include'] define_macros = [('SCIPY_AMD_H', None)] include_dirs = ['/home/soppelsa/local/include'] FOUND: libraries = ['umfpack', 'amd'] library_dirs = ['/home/soppelsa/local/lib'] swig_opts = ['-I/home/soppelsa/local/include', '-I/home/soppelsa/local/include'] define_macros = [('SCIPY_UMFPACK_H', None), ('SCIPY_AMD_H', None)] include_dirs = ['/home/soppelsa/local/include'] running build running config_cc unifing config_cc, config, build_clib, build_ext, build commands --compiler options running config_fc unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options running build_src build_src building py_modules sources building library "dfftpack" sources building library "fftpack" sources building library "linpack_lite" sources building library "mach" sources building library "quadpack" sources building library "odepack" sources building library "dop" sources building library "fitpack" sources building library "odrpack" sources building library "minpack" sources building library "rootfind" sources building library "superlu_src" sources building library "arpack" sources building library "qhull" sources building library "sc_c_misc" sources building library "sc_cephes" sources building library "sc_mach" sources building library "sc_toms" sources building library "sc_amos" sources building library "sc_cdf" sources building library "sc_specfun" sources building library "statlib" sources building extension "scipy.cluster._vq" sources building extension "scipy.cluster._hierarchy_wrap" sources building extension "scipy.fftpack._fftpack" sources f2py options: [] adding 'build/src.linux-x86_64-2.7/fortranobject.c' to sources. adding 'build/src.linux-x86_64-2.7' to include_dirs. building extension "scipy.fftpack.convolve" sources f2py options: [] adding 'build/src.linux-x86_64-2.7/fortranobject.c' to sources. adding 'build/src.linux-x86_64-2.7' to include_dirs. building extension "scipy.integrate._quadpack" sources building extension "scipy.integrate._odepack" sources building extension "scipy.integrate.vode" sources f2py options: [] adding 'build/src.linux-x86_64-2.7/fortranobject.c' to sources. adding 'build/src.linux-x86_64-2.7' to include_dirs. building extension "scipy.integrate._dop" sources f2py options: [] adding 'build/src.linux-x86_64-2.7/fortranobject.c' to sources. adding 'build/src.linux-x86_64-2.7' to include_dirs. building extension "scipy.interpolate.interpnd" sources building extension "scipy.interpolate._fitpack" sources building extension "scipy.interpolate.dfitpack" sources f2py options: [] adding 'build/src.linux-x86_64-2.7/fortranobject.c' to sources. adding 'build/src.linux-x86_64-2.7' to include_dirs. adding 'build/src.linux-x86_64-2.7/scipy/interpolate/src/dfitpack-f2pywrappers.f' to sources. building extension "scipy.interpolate._interpolate" sources building extension "scipy.io.matlab.streams" sources building extension "scipy.io.matlab.mio_utils" sources building extension "scipy.io.matlab.mio5_utils" sources building extension "scipy.lib.blas.fblas" sources f2py options: ['skip:', ':'] adding 'build/src.linux-x86_64-2.7/fortranobject.c' to sources. adding 'build/src.linux-x86_64-2.7' to include_dirs. adding 'build/src.linux-x86_64-2.7/build/src.linux-x86_64-2.7/scipy/lib/blas/fblas-f2pywrappers.f' to sources. building extension "scipy.lib.blas.cblas" sources adding 'build/src.linux-x86_64-2.7/scipy/lib/blas/cblas.pyf' to sources. f2py options: ['skip:', ':'] adding 'build/src.linux-x86_64-2.7/fortranobject.c' to sources. adding 'build/src.linux-x86_64-2.7' to include_dirs. building extension "scipy.lib.lapack.flapack" sources f2py options: ['skip:', ':'] adding 'build/src.linux-x86_64-2.7/fortranobject.c' to sources. adding 'build/src.linux-x86_64-2.7' to include_dirs. building extension "scipy.lib.lapack.clapack" sources adding 'build/src.linux-x86_64-2.7/scipy/lib/lapack/clapack.pyf' to sources. f2py options: ['skip:', ':'] adding 'build/src.linux-x86_64-2.7/fortranobject.c' to sources. adding 'build/src.linux-x86_64-2.7' to include_dirs. building extension "scipy.lib.lapack.calc_lwork" sources f2py options: [] adding 'build/src.linux-x86_64-2.7/fortranobject.c' to sources. adding 'build/src.linux-x86_64-2.7' to include_dirs. building extension "scipy.lib.lapack.atlas_version" sources building extension "scipy.linalg.fblas" sources adding 'build/src.linux-x86_64-2.7/scipy/linalg/fblas.pyf' to sources. f2py options: [] adding 'build/src.linux-x86_64-2.7/fortranobject.c' to sources. adding 'build/src.linux-x86_64-2.7' to include_dirs. adding 'build/src.linux-x86_64-2.7/build/src.linux-x86_64-2.7/scipy/linalg/fblas-f2pywrappers.f' to sources. building extension "scipy.linalg.cblas" sources adding 'build/src.linux-x86_64-2.7/scipy/linalg/cblas.pyf' to sources. f2py options: [] adding 'build/src.linux-x86_64-2.7/fortranobject.c' to sources. adding 'build/src.linux-x86_64-2.7' to include_dirs. building extension "scipy.linalg.flapack" sources adding 'build/src.linux-x86_64-2.7/scipy/linalg/flapack.pyf' to sources. f2py options: [] adding 'build/src.linux-x86_64-2.7/fortranobject.c' to sources. adding 'build/src.linux-x86_64-2.7' to include_dirs. adding 'build/src.linux-x86_64-2.7/build/src.linux-x86_64-2.7/scipy/linalg/flapack-f2pywrappers.f' to sources. building extension "scipy.linalg.clapack" sources adding 'build/src.linux-x86_64-2.7/scipy/linalg/clapack.pyf' to sources. f2py options: [] adding 'build/src.linux-x86_64-2.7/fortranobject.c' to sources. adding 'build/src.linux-x86_64-2.7' to include_dirs. building extension "scipy.linalg._flinalg" sources f2py options: [] adding 'build/src.linux-x86_64-2.7/fortranobject.c' to sources. adding 'build/src.linux-x86_64-2.7' to include_dirs. building extension "scipy.linalg.calc_lwork" sources f2py options: [] adding 'build/src.linux-x86_64-2.7/fortranobject.c' to sources. adding 'build/src.linux-x86_64-2.7' to include_dirs. building extension "scipy.linalg.atlas_version" sources building extension "scipy.odr.__odrpack" sources building extension "scipy.optimize._minpack" sources building extension "scipy.optimize._zeros" sources building extension "scipy.optimize._lbfgsb" sources f2py options: [] adding 'build/src.linux-x86_64-2.7/fortranobject.c' to sources. adding 'build/src.linux-x86_64-2.7' to include_dirs. building extension "scipy.optimize.moduleTNC" sources building extension "scipy.optimize._cobyla" sources f2py options: [] adding 'build/src.linux-x86_64-2.7/fortranobject.c' to sources. adding 'build/src.linux-x86_64-2.7' to include_dirs. building extension "scipy.optimize.minpack2" sources f2py options: [] adding 'build/src.linux-x86_64-2.7/fortranobject.c' to sources. adding 'build/src.linux-x86_64-2.7' to include_dirs. building extension "scipy.optimize._slsqp" sources f2py options: [] adding 'build/src.linux-x86_64-2.7/fortranobject.c' to sources. adding 'build/src.linux-x86_64-2.7' to include_dirs. building extension "scipy.optimize._nnls" sources f2py options: [] adding 'build/src.linux-x86_64-2.7/fortranobject.c' to sources. adding 'build/src.linux-x86_64-2.7' to include_dirs. building extension "scipy.signal.sigtools" sources building extension "scipy.signal.spline" sources building extension "scipy.sparse.linalg.isolve._iterative" sources f2py options: [] adding 'build/src.linux-x86_64-2.7/fortranobject.c' to sources. adding 'build/src.linux-x86_64-2.7' to include_dirs. building extension "scipy.sparse.linalg.dsolve._superlu" sources building extension "scipy.sparse.linalg.dsolve.umfpack.__umfpack" sources adding 'scipy/sparse/linalg/dsolve/umfpack/umfpack.i' to sources. building extension "scipy.sparse.linalg.eigen.arpack._arpack" sources f2py options: [] adding 'build/src.linux-x86_64-2.7/fortranobject.c' to sources. adding 'build/src.linux-x86_64-2.7' to include_dirs. adding 'build/src.linux-x86_64-2.7/build/src.linux-x86_64-2.7/scipy/sparse/linalg/eigen/arpack/_arpack-f2pywrappers.f' to sources. building extension "scipy.sparse.sparsetools._csr" sources building extension "scipy.sparse.sparsetools._csc" sources building extension "scipy.sparse.sparsetools._coo" sources building extension "scipy.sparse.sparsetools._bsr" sources building extension "scipy.sparse.sparsetools._dia" sources building extension "scipy.sparse.sparsetools._csgraph" sources building extension "scipy.spatial.qhull" sources building extension "scipy.spatial.ckdtree" sources building extension "scipy.spatial._distance_wrap" sources building extension "scipy.special._cephes" sources building extension "scipy.special.specfun" sources f2py options: ['--no-wrap-functions'] adding 'build/src.linux-x86_64-2.7/fortranobject.c' to sources. adding 'build/src.linux-x86_64-2.7' to include_dirs. building extension "scipy.special.orthogonal_eval" sources building extension "scipy.special.lambertw" sources building extension "scipy.stats.statlib" sources f2py options: ['--no-wrap-functions'] adding 'build/src.linux-x86_64-2.7/fortranobject.c' to sources. adding 'build/src.linux-x86_64-2.7' to include_dirs. building extension "scipy.stats.vonmises_cython" sources building extension "scipy.stats.futil" sources f2py options: [] adding 'build/src.linux-x86_64-2.7/fortranobject.c' to sources. adding 'build/src.linux-x86_64-2.7' to include_dirs. building extension "scipy.stats.mvn" sources f2py options: [] adding 'build/src.linux-x86_64-2.7/fortranobject.c' to sources. adding 'build/src.linux-x86_64-2.7' to include_dirs. adding 'build/src.linux-x86_64-2.7/scipy/stats/mvn-f2pywrappers.f' to sources. building extension "scipy.ndimage._nd_image" sources building data_files sources build_src: building npy-pkg config files running build_py copying scipy/version.py -> build/lib.linux-x86_64-2.7/scipy copying build/src.linux-x86_64-2.7/scipy/__config__.py -> build/lib.linux-x86_64-2.7/scipy running build_clib customize UnixCCompiler customize UnixCCompiler using build_clib customize Gnu95FCompiler Could not locate executable ifort Found executable /usr/bin/gfortran customize Gnu95FCompiler using build_clib building 'dfftpack' library compiling Fortran sources Fortran f77 compiler: /usr/bin/gfortran -Wall -ffixed-form -fno-second-underscore -openmp -mp -I/usr/local/vni/imsl/fnl600/lnxin100e64/include -fPIC -O3 -funroll-loops Fortran f90 compiler: ifort -openmp -mp -I/usr/local/vni/imsl/fnl600/lnxin100e64/include -openmp -mp -I/usr/local/vni/imsl/fnl600/lnxin100e64/include -fPIC -O3 -funroll-loops Fortran fix compiler: ifort -Wall -ffixed-form -fno-second-underscore -openmp -mp -I/usr/local/vni/imsl/fnl600/lnxin100e64/include -openmp -mp -I/usr/local/vni/imsl/fnl600/lnxin100e64/include -fPIC -O3 -funroll-loops compile options: '-I/home/soppelsa/local/lib/python2.7/site-packages/numpy/core/include -c' gfortran:f77: scipy/fftpack/src/dfftpack/dffti.f f951: error: unrecognized command line option "-mp" f951: error: unrecognized command line option "-mp" error: Command "/usr/bin/gfortran -Wall -ffixed-form -fno-second-underscore -openmp -mp -I/usr/local/vni/imsl/fnl600/lnxin100e64/include -fPIC -O3 -funroll-loops -I/home/soppelsa/local/lib/python2.7/site-packages/numpy/core/include -c -c scipy/fftpack/src/dfftpack/dffti.f -o build/temp.linux-x86_64-2.7/scipy/fftpack/src/dfftpack/dffti.o" failed with exit status 1 From robert.kern at gmail.com Fri Aug 19 13:24:14 2011 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 19 Aug 2011 12:24:14 -0500 Subject: [SciPy-Dev] Problem building Scipy: unrecognized command line option "-mp" In-Reply-To: <4E4E8102.4070307@igi.cnr.it> References: <4E4E8102.4070307@igi.cnr.it> Message-ID: On Fri, Aug 19, 2011 at 10:28, Soppelsa Anton wrote: > Dear all, > > I'm experiencing nasty problems building SciPy 0.9.0 on a amd64 machine > running Linux using gcc and gfortran. > > In particular I get an error by gfortran: f951: error: unrecognized > command line option "-mp". > > BTW, I noticed that the string "-openmp -mp > -I/usr/local/vni/imsl/fnl600/lnxin100e64/include" comes always together > (sometimes two times!?!). > > I looked for the "-openmp" and/or "-mp" string in the entire numpy > source code without being able to find it. > > My questions are: from where does come these options? How can I disable > the option -mp? They probably come from environment variables that you have set. Try this: $ env | grep mp -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From josef.pktd at gmail.com Fri Aug 19 23:19:01 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 19 Aug 2011 23:19:01 -0400 Subject: [SciPy-Dev] life expectancy of scipy.stats nan statistics Message-ID: I'm just looking at http://projects.scipy.org/scipy/ticket/1200 I agree with Ralf that the bias keyword should be changed to ddof as in the numpy functions. For functions in scipy.stats, and statistics in general, I prefer the usual axis=0 default. However, I think these functions, like scipy.stats.nanstd, should be replaced by corresponding numpy functions, which might happen relatively soon. But how soon? Is it worth deprecating bias in scipy 0.10, and then deprecate again for removal in 0.11 or 0.12? Josef From lkb.teichmann at gmail.com Sat Aug 20 11:10:34 2011 From: lkb.teichmann at gmail.com (Martin Teichmann) Date: Sat, 20 Aug 2011 17:10:34 +0200 Subject: [SciPy-Dev] QR-decomposition with Q but not Q In-Reply-To: References: <4E44432A.7010600@molden.no> <4E456A61.3050702@molden.no> Message-ID: Hi List, Hi Sturla, Hi Charles, > using QR manually. Also consider that scipy already has > Levenberg-Marquard in scipy.optimize.leastsq, which is a wrapper for > MINPACK routines LMDER and LMDIF. Yep, I considered that, I'm using them all the time. But they have notorious problems with exceptions raised in the functions, and threading. I also have a fitting function where the fitting is annoyingly unstable, but can be stabilized by tweaking some parameters in lmdif. No to talk about things like masking parameters as Charles proposed. So I thought the best way to solve all of that would be to do the actual lm algorithm in python, it's not what's taking most of the time, so that I'm very flexible, while doing all the compute-intensive stuff with LAPACK. Thanks thus for hinting to DGELS, I'll try to use that. Greetings Martin From bsouthey at gmail.com Sat Aug 20 21:22:40 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Sat, 20 Aug 2011 20:22:40 -0500 Subject: [SciPy-Dev] [Numpy-discussion] life expectancy of scipy.stats nan statistics In-Reply-To: References: Message-ID: On Fri, Aug 19, 2011 at 10:19 PM, wrote: > I'm just looking at http://projects.scipy.org/scipy/ticket/1200 Cringe, I wrote that poor code... > > I agree with Ralf that the bias keyword should be changed to ddof as > in the numpy functions. For functions in scipy.stats, and statistics > in general, I prefer the usual axis=0 default. I agree for the keyword but not the axis since a user should expect the same default behavior as in numpy for the 'same' function. > > However, I think these functions, like scipy.stats.nanstd, should be > replaced by corresponding numpy functions, which might happen > relatively soon. But how soon? Anyhow, I like Mark's NA stuff for the little that I have done with it and there still is the masked stuff. But it is still going to be a while for it to be mainstream yet a really good numpy 2.0 feature. If the 'skipna' could be more general then these separate na handling functions would be unnecessary. > > Is it worth deprecating bias in scipy 0.10, and then deprecate again > for removal in 0.11 or 0.12? > > Josef +1 Bruce From miguel-py at astro.su.se Sun Aug 21 07:04:26 2011 From: miguel-py at astro.su.se (Miguel de Val-Borro) Date: Sun, 21 Aug 2011 13:04:26 +0200 Subject: [SciPy-Dev] docs.scipy.org wiki edit rights Message-ID: <20110821110426.GB2467@poincare.pc.linmpi.mpg.de> Hi, I would like to fix a couple of typos that I have found in scipy docstrings. Could I get edit permission on the wiki at docs.scipy.org? My username is miguel. Thanks, Miguel From gael.varoquaux at normalesup.org Sun Aug 21 07:06:57 2011 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sun, 21 Aug 2011 13:06:57 +0200 Subject: [SciPy-Dev] docs.scipy.org wiki edit rights In-Reply-To: <20110821110426.GB2467@poincare.pc.linmpi.mpg.de> References: <20110821110426.GB2467@poincare.pc.linmpi.mpg.de> Message-ID: <20110821110657.GE28879@phare.normalesup.org> Hi, On Sun, Aug 21, 2011 at 01:04:26PM +0200, Miguel de Val-Borro wrote: > I would like to fix a couple of typos that I have found in scipy > docstrings. Could I get edit permission on the wiki at docs.scipy.org? > My username is miguel. You should now be good to go. Thanks a lot, this is very useful. Gael From anton.soppelsa at igi.cnr.it Mon Aug 22 08:16:19 2011 From: anton.soppelsa at igi.cnr.it (Soppelsa Anton) Date: Mon, 22 Aug 2011 14:16:19 +0200 Subject: [SciPy-Dev] Problem building Scipy: unrecognized command line option "-mp" In-Reply-To: References: <4E4E8102.4070307@igi.cnr.it> Message-ID: <4E524893.8050206@igi.cnr.it> On 19/08/2011 19.24, Robert Kern wrote: > > They probably come from environment variables that you have set. Try this: > > $ env | grep mp > So simple! The environment where I'm trying to compile is densely populated and it also defines the F90FLAGS=-openmp -mp -I/usr/local/vni/imsl/fnl600/lnxin100e64/include FFLAGS=-openmp -mp -I/usr/local/vni/imsl/fnl600/lnxin100e64/include variables. Your answer was very useful (searched openmp, as mp is a rather common sequence of caracters :) ). Thank you very much for the help! Sincerely, A. Soppelsa From ralf.gommers at googlemail.com Wed Aug 24 07:04:42 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 24 Aug 2011 13:04:42 +0200 Subject: [SciPy-Dev] [Numpy-discussion] life expectancy of scipy.stats nan statistics In-Reply-To: References: Message-ID: On Sun, Aug 21, 2011 at 3:22 AM, Bruce Southey wrote: > On Fri, Aug 19, 2011 at 10:19 PM, wrote: > > I'm just looking at http://projects.scipy.org/scipy/ticket/1200 > Cringe, I wrote that poor code... > > > > I agree with Ralf that the bias keyword should be changed to ddof as > > in the numpy functions. For functions in scipy.stats, and statistics > > in general, I prefer the usual axis=0 default. > > I agree for the keyword but not the axis since a user should expect > the same default behavior as in numpy for the 'same' function. > > In this case I agree with Bruce, because then it'll be easy to switch to a future numpy nanstd implementation. > > > > > However, I think these functions, like scipy.stats.nanstd, should be > > replaced by corresponding numpy functions, which might happen > > relatively soon. But how soon? > > Anyhow, I like Mark's NA stuff for the little that I have done with it > and there still is the masked stuff. But it is still going to be a > while for it to be mainstream yet a really good numpy 2.0 feature. If > the 'skipna' could be more general then these separate na handling > functions would be unnecessary. > > > > > Is it worth deprecating bias in scipy 0.10, and then deprecate again > > for removal in 0.11 or 0.12? > > > > Josef > > +1 > > OK, I'll do this then. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From collinstocks at gmail.com Thu Aug 25 15:10:37 2011 From: collinstocks at gmail.com (Collin Stocks) Date: Thu, 25 Aug 2011 15:10:37 -0400 Subject: [SciPy-Dev] Pull request 62 Message-ID: <1314299437.6365.4.camel@SietchTabr> List, I'd like to draw your attention to pull request 62. It is a minor correction to the test code I wrote that was accepted from pull request 44. I had accidentally named two of the new test functions the same as the ones I was modeling them against. As it is right now, two of my test functions override two of the existing ones. Thanks, Collin -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part URL: From ralf.gommers at googlemail.com Fri Aug 26 05:47:14 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Fri, 26 Aug 2011 11:47:14 +0200 Subject: [SciPy-Dev] adding chkfinite flags to linalg functions Message-ID: Hi, https://github.com/scipy/scipy/pull/48 adds a chkfinite flag to scipy.linalg functions that allows for disabling a check on whether there are infs/nans in the array, which can be a speed gain if you already know that there's no infs/nans in your input array. Is anyone opposed to merging this? Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From collinstocks at gmail.com Fri Aug 26 07:15:16 2011 From: collinstocks at gmail.com (Collin Stocks) Date: Fri, 26 Aug 2011 07:15:16 -0400 Subject: [SciPy-Dev] adding chkfinite flags to linalg functions In-Reply-To: References: Message-ID: <1314357316.6365.15.camel@SietchTabr> I am not opposed to making this change, but perhaps in the future (if someone gets around to it -- maybe me?) it would be cleaner for arrays to have a flag attached to them that records whether they are known to (or not to) contain infinite or NaN values. The flag would have to update itself on assignment to the array, or on calls that could possibly add NaN's or Inf's to the array. If the above change were in place, chkfinite would (in most cases) just look at the flag. Only if the flag said that it did not know would it have to check the entire array. Of course, this would require some major changes to NumPy, so I'm in favor of the current solution proposed in pull request 48 for now. -- Collin -------------- next part -------------- An embedded message was scrubbed... From: Ralf Gommers Subject: [SciPy-Dev] adding chkfinite flags to linalg functions Date: Fri, 26 Aug 2011 11:47:14 +0200 Size: 4018 URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part URL: From gael.varoquaux at normalesup.org Fri Aug 26 08:42:24 2011 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Fri, 26 Aug 2011 14:42:24 +0200 Subject: [SciPy-Dev] adding chkfinite flags to linalg functions In-Reply-To: <1314357316.6365.15.camel@SietchTabr> References: <1314357316.6365.15.camel@SietchTabr> Message-ID: <1314362544.8481.17.camel@Nokia-N900-51-1> ----- Original message ----- > I am not opposed to making this change, but perhaps in the future (if > someone gets around to it -- maybe me?) it would be cleaner for arrays > to have a flag attached to them that records whether they are known to > (or not to) contain infinite or NaN values. The flag would have to > update itself on assignment to the array, or on calls that could > possibly add NaN's or Inf's to the array. I don't think that this can work: I can easily take a view of an array without Python even knowing it, and then when the view gets modified, it can gather NaNs and the flag of the original array will not be modified. My 2 cents, Gael From njs at pobox.com Fri Aug 26 10:40:19 2011 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 26 Aug 2011 07:40:19 -0700 Subject: [SciPy-Dev] adding chkfinite flags to linalg functions In-Reply-To: References: Message-ID: Would calling it check_finite instead be too horrible? - Nathaniel On Aug 26, 2011 2:47 AM, "Ralf Gommers" wrote: > Hi, > > https://github.com/scipy/scipy/pull/48 adds a chkfinite flag to scipy.linalg > functions that allows for disabling a check on whether there are infs/nans > in the array, which can be a speed gain if you already know that there's no > infs/nans in your input array. > > Is anyone opposed to merging this? > > Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsouthey at gmail.com Fri Aug 26 11:38:50 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Fri, 26 Aug 2011 10:38:50 -0500 Subject: [SciPy-Dev] adding chkfinite flags to linalg functions In-Reply-To: References: Message-ID: <4E57BE0A.90506@gmail.com> On 08/26/2011 09:40 AM, Nathaniel Smith wrote: > > Would calling it check_finite instead be too horrible? > > - Nathaniel > > On Aug 26, 2011 2:47 AM, "Ralf Gommers" > wrote: > > Hi, > > > > https://github.com/scipy/scipy/pull/48 adds a chkfinite flag to > scipy.linalg > > functions that allows for disabling a check on whether there are > infs/nans > > in the array, which can be a speed gain if you already know that > there's no > > infs/nans in your input array. > > > > Is anyone opposed to merging this? > > > > Ralf > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev I do oppose this because I do not think it is the correct solution to an apparent issue of computational speed. I do say apparent because there is no code and no timings to support it. The proposed argument is really not a check but to avoid something that has to be done in order to obtain a valid outcome. Hence a possible reason why the original author put this check in these functions. If we want to avoid this then we need to change the workflow by breaking this into two steps rather than this approach. All this patch does is say if you want to go 'fast' then set 'chkfinite=False' regardless of type of input and fails to address any issue. We quite often see on numpy/scipy lists where a user has made incorrect assumptions about their input which is will not catch or catch at the end. I do not know if Chris provided any tickets for the issues he pointed out in: http://mail.scipy.org/pipermail/scipy-dev/2011-July/016386.html "But if you pass a numpy array with inf's in it, then it hangs." This should have a ticket with an clear easy example because it highlights a large problem. I also agree with Nathaniel that the name of the argument is bad but check_finite would not solve my issue with the name. Yes, I am being negative because this is a community and it needs to be shown that this patch will improve scipy as whole. Bruce -------------- next part -------------- An HTML attachment was scrubbed... URL: From cjordan1 at uw.edu Fri Aug 26 12:51:28 2011 From: cjordan1 at uw.edu (Christopher Jordan-Squire) Date: Fri, 26 Aug 2011 11:51:28 -0500 Subject: [SciPy-Dev] adding chkfinite flags to linalg functions In-Reply-To: References: Message-ID: On Fri, Aug 26, 2011 at 9:40 AM, Nathaniel Smith wrote: > Would calling it check_finite instead be too horrible? > It's called chkfinite in accordance with the numpy function that is called. -Chris JS > - Nathaniel > > On Aug 26, 2011 2:47 AM, "Ralf Gommers" wrote: >> Hi, >> >> https://github.com/scipy/scipy/pull/48 adds a chkfinite flag to >> scipy.linalg >> functions that allows for disabling a check on whether there are infs/nans >> in the array, which can be a speed gain if you already know that there's >> no >> infs/nans in your input array. >> >> Is anyone opposed to merging this? >> >> Ralf > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > > From cjordan1 at uw.edu Fri Aug 26 13:02:28 2011 From: cjordan1 at uw.edu (Christopher Jordan-Squire) Date: Fri, 26 Aug 2011 12:02:28 -0500 Subject: [SciPy-Dev] adding chkfinite flags to linalg functions In-Reply-To: <4E57BE0A.90506@gmail.com> References: <4E57BE0A.90506@gmail.com> Message-ID: On Fri, Aug 26, 2011 at 10:38 AM, Bruce Southey wrote: > On 08/26/2011 09:40 AM, Nathaniel Smith wrote: > > Would calling it check_finite instead be too horrible? > > - Nathaniel > > On Aug 26, 2011 2:47 AM, "Ralf Gommers" wrote: >> Hi, >> >> https://github.com/scipy/scipy/pull/48 adds a chkfinite flag to >> scipy.linalg >> functions that allows for disabling a check on whether there are infs/nans >> in the array, which can be a speed gain if you already know that there's >> no >> infs/nans in your input array. >> >> Is anyone opposed to merging this? >> >> Ralf > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > > I do oppose this because I do not think it is the correct solution to an > apparent issue of computational speed. I do say apparent because there is no > code and no timings to support it. > Some timing data is at the pull request link. (I was comparing the then-current chkfinite against a modification Ralf suggested.) "I tried that (calling it chkfinite_mod) on an ndarray A of shape (10000,500). (The extreme size was mostly to make the differences clear. The issues in my application had been appearing when A was (3000,2).) The results of timeit are as follows. np.asarray_chkfinite(A): 153 ms chkfinite_mod(A): 59.3 ms np.asarray(A): 1.9 us So asarray_chkfinite is 75,000 times as slow, and chkfinite_mod is 30,000 times as slow. That's enough of a change that I'm going to make the small modification to numpy and make a pull request, but it still isn't big enough. To give a feel for the sort of issue I'm thinking of, let's say that A and b are ndarrays of shape (100000,5) and b is (100000,). Then currently a call to lstsqr produces (using prun in ipython) that the entire solve takes 0.060s, and 0.017 seconds is spent in asarray_chkfinite. That's a huge amount of time wasted verifying the input, proportionally, so I'd really like orders of magnitude reduction in the time taken to pass the input on to the solver. Especially since I want to use these things in situations where this solver is being called thousands of times over and over on data that's already been verified as finite. " > The proposed argument is really not a check but to avoid something that has > to be done in order to obtain a valid outcome.? Hence a possible reason why > the original author put this check in these functions.? If we want to avoid > this then we need to change the workflow by breaking this into two steps > rather than this approach. > The check's aren't something that must be done to obtain a valid outcome. They're a way to ensure the input is valid. If you know for other reasons that your input is valid then it's a waste of resources. Breaking it into two steps has its own issues associated with it as well. Then the user must verify their own inputs before passing them to scipy linalg methods. If I understand your suggestion correctly. Why would that be a better approach? > All this patch does is say if you want to go 'fast' then set > 'chkfinite=False' regardless of type of input and fails to address any > issue. We quite often see on numpy/scipy lists where a user has made > incorrect assumptions about their input which is will not catch or catch at > the end. > > I do not know if Chris provided any tickets for the issues he pointed out > in: > http://mail.scipy.org/pipermail/scipy-dev/2011-July/016386.html > "But if you pass a numpy array with inf's in it, then it hangs." > > This should have a ticket with an clear easy example because it highlights a > large problem. > It has to do with the underlying Lapack call, if IIRC. Not something in numpy/scipy. > > I also agree with Nathaniel that the name of the argument is bad but > check_finite would not solve my issue with the name. > chkfinite was used because of the underlying numpy function being bypassed. I don't have a strong preference, but that does make it easier to relate the option to the actual code. > Yes, I am being negative because this is a community and it needs to be > shown that this patch will improve scipy as whole. > > Bruce > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > > From bsouthey at gmail.com Fri Aug 26 14:45:45 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Fri, 26 Aug 2011 13:45:45 -0500 Subject: [SciPy-Dev] adding chkfinite flags to linalg functions In-Reply-To: References: <4E57BE0A.90506@gmail.com> Message-ID: <4E57E9D9.7010009@gmail.com> On 08/26/2011 12:02 PM, Christopher Jordan-Squire wrote: > On Fri, Aug 26, 2011 at 10:38 AM, Bruce Southey wrote: >> On 08/26/2011 09:40 AM, Nathaniel Smith wrote: >> >> Would calling it check_finite instead be too horrible? >> >> - Nathaniel >> >> On Aug 26, 2011 2:47 AM, "Ralf Gommers" wrote: >>> Hi, >>> >>> https://github.com/scipy/scipy/pull/48 adds a chkfinite flag to >>> scipy.linalg >>> functions that allows for disabling a check on whether there are infs/nans >>> in the array, which can be a speed gain if you already know that there's >>> no >>> infs/nans in your input array. >>> >>> Is anyone opposed to merging this? >>> >>> Ralf >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-dev >> >> I do oppose this because I do not think it is the correct solution to an >> apparent issue of computational speed. I do say apparent because there is no >> code and no timings to support it. >> > Some timing data is at the pull request link. (I was comparing the > then-current chkfinite against a modification Ralf suggested.) > > "I tried that (calling it chkfinite_mod) on an ndarray A of shape > (10000,500). (The extreme size was mostly to make the differences > clear. The issues in my application had been appearing when A was > (3000,2).) The results of timeit are as follows. > > np.asarray_chkfinite(A): 153 ms > chkfinite_mod(A): 59.3 ms > np.asarray(A): 1.9 us > > So asarray_chkfinite is 75,000 times as slow, and chkfinite_mod is > 30,000 times as slow. That's enough of a change that I'm going to make > the small modification to numpy and make a pull request, but it still > isn't big enough. There is no example or timing code so this is meaningless. Also 'asarray_chkfinite' is numpy not scipy so it is not relevant here. Also how many different computers have you tested this on that gave this difference? Hint one is not enough. > To give a feel for the sort of issue I'm thinking of, let's say that A > and b are ndarrays of shape (100000,5) and b is (100000,). Then > currently a call to lstsqr produces (using prun in ipython) that the > entire solve takes 0.060s, and 0.017 seconds is spent in > asarray_chkfinite. That's a huge amount of time wasted verifying the > input, proportionally, so I'd really like orders of magnitude > reduction in the time taken to pass the input on to the solver. > Especially since I want to use these things in situations where this > solver is being called thousands of times over and over on data that's > already been verified as finite. " I do understand the time issue but that is your very specific situation. You have to provide the sample code that replicates this as are using it might be just your environment and computer. >> The proposed argument is really not a check but to avoid something that has >> to be done in order to obtain a valid outcome. Hence a possible reason why >> the original author put this check in these functions. If we want to avoid >> this then we need to change the workflow by breaking this into two steps >> rather than this approach. >> > The check's aren't something that must be done to obtain a valid > outcome. They're a way to ensure the input is valid. If you know for > other reasons that your input is valid then it's a waste of resources. Yes, I do understand that but I do not think that you are the average scipy user nor is your situation typical of scipy. > Breaking it into two steps has its own issues associated with it as > well. Then the user must verify their own inputs before passing them > to scipy linalg methods. If I understand your suggestion correctly. That is exactly what your patch is doing because people will misuse that argument just because they perceive it as faster. > Why would that be a better approach? I currently have no problems with the scipy approach as what you indicate below regarding lapack. >> All this patch does is say if you want to go 'fast' then set >> 'chkfinite=False' regardless of type of input and fails to address any >> issue. We quite often see on numpy/scipy lists where a user has made >> incorrect assumptions about their input which is will not catch or catch at >> the end. >> >> I do not know if Chris provided any tickets for the issues he pointed out >> in: >> http://mail.scipy.org/pipermail/scipy-dev/2011-July/016386.html >> "But if you pass a numpy array with inf's in it, then it hangs." >> >> This should have a ticket with an clear easy example because it highlights a >> large problem. >> > It has to do with the underlying Lapack call, if IIRC. Not something > in numpy/scipy. So what? File a bug report so people know it is a problem and can fix it! If it is in Lapack then it needs to be pushed upstream. So in the meantime we do have a work around it. But this patch permits users to incorrectly remove that work around. >> I also agree with Nathaniel that the name of the argument is bad but >> check_finite would not solve my issue with the name. >> > chkfinite was used because of the underlying numpy function being > bypassed. I don't have a strong preference, but that does make it > easier to relate the option to the actual code. That is poor reason for a name as only someone who knows the code will understand the meaning - users don't read documentation :-) >> Yes, I am being negative because this is a community and it needs to be >> shown that this patch will improve scipy as whole. >> >> Bruce >> >> Bruce From matthew.brett at gmail.com Fri Aug 26 15:09:34 2011 From: matthew.brett at gmail.com (Matthew Brett) Date: Fri, 26 Aug 2011 12:09:34 -0700 Subject: [SciPy-Dev] adding chkfinite flags to linalg functions In-Reply-To: <4E57E9D9.7010009@gmail.com> References: <4E57BE0A.90506@gmail.com> <4E57E9D9.7010009@gmail.com> Message-ID: Hi, On Fri, Aug 26, 2011 at 11:45 AM, Bruce Southey wrote: > On 08/26/2011 12:02 PM, Christopher Jordan-Squire wrote: >> Breaking it into two steps has its own issues associated with it as >> well. Then the user must verify their own inputs before passing them >> to scipy linalg methods. If I understand your suggestion correctly. > That is exactly what your patch is doing because people will misuse that > argument just because they perceive it as faster. I object to this objection. There's a long tradition of letting people do dangerous things if they want to. Here the option is sensibly labeled and it's not the default. Best, Matthew From bsouthey at gmail.com Fri Aug 26 15:30:21 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Fri, 26 Aug 2011 14:30:21 -0500 Subject: [SciPy-Dev] adding chkfinite flags to linalg functions In-Reply-To: References: <4E57BE0A.90506@gmail.com> <4E57E9D9.7010009@gmail.com> Message-ID: <4E57F44D.3030507@gmail.com> On 08/26/2011 02:09 PM, Matthew Brett wrote: > Hi, > > On Fri, Aug 26, 2011 at 11:45 AM, Bruce Southey wrote: >> On 08/26/2011 12:02 PM, Christopher Jordan-Squire wrote: >>> Breaking it into two steps has its own issues associated with it as >>> well. Then the user must verify their own inputs before passing them >>> to scipy linalg methods. If I understand your suggestion correctly. >> That is exactly what your patch is doing because people will misuse that >> argument just because they perceive it as faster. > I object to this objection. > > There's a long tradition of letting people do dangerous things if they > want to. Here the option is sensibly labeled and it's not the > default. > > Best, > > Matthew > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev So why not have the patch remove all the calls to asarray_chkfinite()? Bruce From cjordan1 at uw.edu Fri Aug 26 15:38:59 2011 From: cjordan1 at uw.edu (Christopher Jordan-Squire) Date: Fri, 26 Aug 2011 14:38:59 -0500 Subject: [SciPy-Dev] adding chkfinite flags to linalg functions In-Reply-To: <4E57F44D.3030507@gmail.com> References: <4E57BE0A.90506@gmail.com> <4E57E9D9.7010009@gmail.com> <4E57F44D.3030507@gmail.com> Message-ID: On Fri, Aug 26, 2011 at 2:30 PM, Bruce Southey wrote: > On 08/26/2011 02:09 PM, Matthew Brett wrote: >> Hi, >> >> On Fri, Aug 26, 2011 at 11:45 AM, Bruce Southey ?wrote: >>> On 08/26/2011 12:02 PM, Christopher Jordan-Squire wrote: >>>> Breaking it into two steps has its own issues associated with it as >>>> well. Then the user must verify their own inputs before passing them >>>> to scipy linalg methods. If I understand your suggestion correctly. >>> That is exactly what your patch is doing because people will misuse that >>> argument just because they perceive it as faster. >> I object to this objection. >> >> There's a long tradition of letting people do dangerous things if they >> want to. ?Here the option is sensibly labeled and it's not the >> default. >> >> Best, >> >> Matthew >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-dev > So why not have the patch remove all the calls to asarray_chkfinite()? > Because it's a sensible default option. It's just not appropriate for all circumstances. -Chris > Bruce > > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > From matthew.brett at gmail.com Fri Aug 26 15:39:34 2011 From: matthew.brett at gmail.com (Matthew Brett) Date: Fri, 26 Aug 2011 12:39:34 -0700 Subject: [SciPy-Dev] adding chkfinite flags to linalg functions In-Reply-To: <4E57F44D.3030507@gmail.com> References: <4E57BE0A.90506@gmail.com> <4E57E9D9.7010009@gmail.com> <4E57F44D.3030507@gmail.com> Message-ID: On Fri, Aug 26, 2011 at 12:30 PM, Bruce Southey wrote: > On 08/26/2011 02:09 PM, Matthew Brett wrote: >> Hi, >> >> On Fri, Aug 26, 2011 at 11:45 AM, Bruce Southey ?wrote: >>> On 08/26/2011 12:02 PM, Christopher Jordan-Squire wrote: >>>> Breaking it into two steps has its own issues associated with it as >>>> well. Then the user must verify their own inputs before passing them >>>> to scipy linalg methods. If I understand your suggestion correctly. >>> That is exactly what your patch is doing because people will misuse that >>> argument just because they perceive it as faster. >> I object to this objection. >> >> There's a long tradition of letting people do dangerous things if they >> want to. ?Here the option is sensibly labeled and it's not the >> default. >> >> Best, >> >> Matthew >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at scipy.org >> http://mail.scipy.org/mailman/listinfo/scipy-dev > So why not have the patch remove all the calls to asarray_chkfinite()? Perhaps I misunderstood, sorry if so. My understanding was that the routine, by default, will check for non-finite values before running, and error otherwise. The chkfinite=False flag would be a specialized case where the user is sure there are no non-finite values. If my understanding is correct, it seems to me perfectly reasonable to allow the user to shoot themselves in the foot when changing a well-labelled and described option. To answer your question then, changing the default behavior would obviously be silly. Adding an extra non-default behavior is not, in my view. Best, Matthew From robert.kern at gmail.com Fri Aug 26 15:41:18 2011 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 26 Aug 2011 14:41:18 -0500 Subject: [SciPy-Dev] adding chkfinite flags to linalg functions In-Reply-To: <4E57F44D.3030507@gmail.com> References: <4E57BE0A.90506@gmail.com> <4E57E9D9.7010009@gmail.com> <4E57F44D.3030507@gmail.com> Message-ID: On Fri, Aug 26, 2011 at 14:30, Bruce Southey wrote: > On 08/26/2011 02:09 PM, Matthew Brett wrote: >> Hi, >> >> On Fri, Aug 26, 2011 at 11:45 AM, Bruce Southey ?wrote: >>> On 08/26/2011 12:02 PM, Christopher Jordan-Squire wrote: >>>> Breaking it into two steps has its own issues associated with it as >>>> well. Then the user must verify their own inputs before passing them >>>> to scipy linalg methods. If I understand your suggestion correctly. >>> That is exactly what your patch is doing because people will misuse that >>> argument just because they perceive it as faster. >> I object to this objection. >> >> There's a long tradition of letting people do dangerous things if they >> want to. ?Here the option is sensibly labeled and it's not the >> default. > > So why not have the patch remove all the calls to asarray_chkfinite()? Because we want the *default* to be safe for all inputs. We want to *allow* people to avoid those checks when they have extra information about the inputs. Adding a flag is a more convenient way to do this than any other alternative. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From cjordan1 at uw.edu Fri Aug 26 16:13:47 2011 From: cjordan1 at uw.edu (Christopher Jordan-Squire) Date: Fri, 26 Aug 2011 15:13:47 -0500 Subject: [SciPy-Dev] adding chkfinite flags to linalg functions In-Reply-To: <4E57E9D9.7010009@gmail.com> References: <4E57BE0A.90506@gmail.com> <4E57E9D9.7010009@gmail.com> Message-ID: On Fri, Aug 26, 2011 at 1:45 PM, Bruce Southey wrote: > On 08/26/2011 12:02 PM, Christopher Jordan-Squire wrote: >> On Fri, Aug 26, 2011 at 10:38 AM, Bruce Southey ?wrote: >>> On 08/26/2011 09:40 AM, Nathaniel Smith wrote: >>> >>> Would calling it check_finite instead be too horrible? >>> >>> - Nathaniel >>> >>> On Aug 26, 2011 2:47 AM, "Ralf Gommers" ?wrote: >>>> Hi, >>>> >>>> https://github.com/scipy/scipy/pull/48 adds a chkfinite flag to >>>> scipy.linalg >>>> functions that allows for disabling a check on whether there are infs/nans >>>> in the array, which can be a speed gain if you already know that there's >>>> no >>>> infs/nans in your input array. >>>> >>>> Is anyone opposed to merging this? >>>> >>>> Ralf >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev at scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-dev >>> >>> I do oppose this because I do not think it is the correct solution to an >>> apparent issue of computational speed. I do say apparent because there is no >>> code and no timings to support it. >>> >> Some timing data is at the pull request link. (I was comparing the >> then-current chkfinite against a modification Ralf suggested.) >> >> "I tried that (calling it chkfinite_mod) on an ndarray A of shape >> (10000,500). (The extreme size was mostly to make the differences >> clear. The issues in my application had been appearing when A was >> (3000,2).) The results of timeit are as follows. >> >> np.asarray_chkfinite(A): 153 ms >> chkfinite_mod(A): 59.3 ms >> np.asarray(A): 1.9 us >> >> So asarray_chkfinite is 75,000 times as slow, and chkfinite_mod is >> 30,000 times as slow. That's enough of a change that I'm going to make >> the small modification to numpy and make a pull request, but it still >> isn't big enough. > > There is no example or timing code so this is meaningless. > Also ?'asarray_chkfinite' is numpy not scipy so it is not relevant here. > Also how many different computers have you tested this on that gave this > difference? Hint one is not enough. > Please don't be rude. I cut and pasted the text from the github pull request, where asarray_chkfinite was part of a short exchange with Ralf. I added code to the pull request, but it simple translates the above description directly into code. I wasn't trying to hide any fanciness or massage numbers. >> To give a feel for the sort of issue I'm thinking of, let's say that A >> and b are ndarrays of shape (100000,5) and b is (100000,). Then >> currently a call to lstsqr produces (using prun in ipython) that the >> entire solve takes 0.060s, and 0.017 seconds is spent in >> asarray_chkfinite. That's a huge amount of time wasted verifying the >> input, proportionally, so I'd really like orders of magnitude >> reduction in the time taken to pass the input on to the solver. >> Especially since I want to use these things in situations where this >> solver is being called thousands of times over and over on data that's >> already been verified as finite. " > I do understand the time issue but that is your very specific > situation. ?You have to provide the sample code that replicates this as > are using it might be just your environment and computer. > >>> The proposed argument is really not a check but to avoid something that has >>> to be done in order to obtain a valid outcome. ?Hence a possible reason why >>> the original author put this check in these functions. ?If we want to avoid >>> this then we need to change the workflow by breaking this into two steps >>> rather than this approach. >>> >> The check's aren't something that must be done to obtain a valid >> outcome. They're a way to ensure the input is valid. If you know for >> other reasons that your input is valid then it's a waste of resources. > Yes, I do understand that but I do not think that you are the average > scipy user nor is your situation typical of scipy. I think it's a situation anyone with long, thin matrices will run into, particularly if they're iterating over many replications of some computation. The situation is typical for fitting statistical models with a small number of parameters on a large dataset. This is the context where I ran into the problem. Specifically when I was fitting a linear model repeatedly on large subsamples of a very large dataset. Both the general problem and that particular instance seemed like common and generic use cases. >> Breaking it into two steps has its own issues associated with it as >> well. Then the user must verify their own inputs before passing them >> to scipy linalg methods. If I understand your suggestion correctly. > That is exactly what your patch is doing because people will misuse that > argument just because they perceive it as faster. > If it works then the user wasn't misusing it. If it fails then the user made the wrong choice because they didn't understand their algorithm. I don't think I understand your point. >> Why would that be a better approach? > I currently have no problems with the scipy approach as what you > indicate below regarding lapack. >>> All this patch does is say if you want to go 'fast' then set >>> 'chkfinite=False' regardless of type of input and fails to address any >>> issue. We quite often see on numpy/scipy lists where a user has made >>> incorrect assumptions about their input which is will not catch or catch at >>> the end. >>> >>> I do not know if Chris provided any tickets for the issues he pointed out >>> in: >>> http://mail.scipy.org/pipermail/scipy-dev/2011-July/016386.html >>> "But if you pass a numpy array with inf's in it, then it hangs." >>> >>> This should have a ticket with an clear easy example because it highlights a >>> large problem. >>> >> It has to do with the underlying Lapack call, if IIRC. Not something >> in numpy/scipy. > So what? > File a bug report so people know it is a problem and can fix it! > If it is in Lapack then it needs to be pushed upstream. So in the > meantime we do have a work around it. But this patch permits users to > incorrectly remove that work around. > >>> I also agree with Nathaniel that the name of the argument is bad but >>> check_finite would not solve my issue with the name. >>> >> chkfinite was used because of the underlying numpy function being >> bypassed. I don't have a strong preference, but that does make it >> easier to relate the option to the actual code. > That is poor reason for a name as only someone who knows the code will > understand the meaning - users don't read documentation :-) > But users do use google. And google will recognize chkfinite faster than check_finite. Besides, the default is chkfinite=True, so they'd have to read the documentation to know what to change. -Chris JS >>> Yes, I am being negative because this is a community and it needs to be >>> shown that this patch will improve scipy as whole. >>> >>> Bruce >>> >>> > > Bruce > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > From bsouthey at gmail.com Fri Aug 26 16:24:06 2011 From: bsouthey at gmail.com (Bruce Southey) Date: Fri, 26 Aug 2011 15:24:06 -0500 Subject: [SciPy-Dev] adding chkfinite flags to linalg functions In-Reply-To: References: <4E57BE0A.90506@gmail.com> <4E57E9D9.7010009@gmail.com> <4E57F44D.3030507@gmail.com> Message-ID: <4E5800E6.9010805@gmail.com> On 08/26/2011 02:41 PM, Robert Kern wrote: > On Fri, Aug 26, 2011 at 14:30, Bruce Southey wrote: >> On 08/26/2011 02:09 PM, Matthew Brett wrote: >>> Hi, >>> >>> On Fri, Aug 26, 2011 at 11:45 AM, Bruce Southey wrote: >>>> On 08/26/2011 12:02 PM, Christopher Jordan-Squire wrote: >>>>> Breaking it into two steps has its own issues associated with it as >>>>> well. Then the user must verify their own inputs before passing them >>>>> to scipy linalg methods. If I understand your suggestion correctly. >>>> That is exactly what your patch is doing because people will misuse that >>>> argument just because they perceive it as faster. >>> I object to this objection. >>> >>> There's a long tradition of letting people do dangerous things if they >>> want to. Here the option is sensibly labeled and it's not the >>> default. >> So why not have the patch remove all the calls to asarray_chkfinite()? > Because we want the *default* to be safe for all inputs. We want to > *allow* people to avoid those checks when they have extra information > about the inputs. Adding a flag is a more convenient way to do this > than any other alternative. > You and others are totally ignoring the author has stated (without example) that there is a major roadblock when that assumption is wrong. I am very aware of that many functions will provide something even if the input was not correct. So I agree with Matthew about letting people live dangerously as the output will tell them that something was wrong. That means there should be no need for asarray_chkfinite() here. Bruce From collinstocks at gmail.com Sat Aug 27 00:37:19 2011 From: collinstocks at gmail.com (Collin Stocks) Date: Sat, 27 Aug 2011 00:37:19 -0400 Subject: [SciPy-Dev] adding chkfinite flags to linalg functions In-Reply-To: <1314362544.8481.17.camel@Nokia-N900-51-1> References: <1314357316.6365.15.camel@SietchTabr> <1314362544.8481.17.camel@Nokia-N900-51-1> Message-ID: <1314419839.6365.19.camel@SietchTabr> Thanks for this comment. I'm not that familiar with how arrays get accessed by modules written in other languages. Yet. -- Collin -------------- next part -------------- An embedded message was scrubbed... From: Gael Varoquaux Subject: Re: [SciPy-Dev] adding chkfinite flags to linalg functions Date: Fri, 26 Aug 2011 14:42:24 +0200 Size: 2738 URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part URL: From ralf.gommers at googlemail.com Sat Aug 27 04:36:40 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sat, 27 Aug 2011 10:36:40 +0200 Subject: [SciPy-Dev] adding chkfinite flags to linalg functions In-Reply-To: <4E5800E6.9010805@gmail.com> References: <4E57BE0A.90506@gmail.com> <4E57E9D9.7010009@gmail.com> <4E57F44D.3030507@gmail.com> <4E5800E6.9010805@gmail.com> Message-ID: On Fri, Aug 26, 2011 at 10:24 PM, Bruce Southey wrote: > On 08/26/2011 02:41 PM, Robert Kern wrote: > > On Fri, Aug 26, 2011 at 14:30, Bruce Southey wrote: > >> On 08/26/2011 02:09 PM, Matthew Brett wrote: > >>> Hi, > >>> > >>> On Fri, Aug 26, 2011 at 11:45 AM, Bruce Southey > wrote: > >>>> On 08/26/2011 12:02 PM, Christopher Jordan-Squire wrote: > >>>>> Breaking it into two steps has its own issues associated with it as > >>>>> well. Then the user must verify their own inputs before passing them > >>>>> to scipy linalg methods. If I understand your suggestion correctly. > >>>> That is exactly what your patch is doing because people will misuse > that > >>>> argument just because they perceive it as faster. > >>> I object to this objection. > >>> > >>> There's a long tradition of letting people do dangerous things if they > >>> want to. Here the option is sensibly labeled and it's not the > >>> default. > >> So why not have the patch remove all the calls to asarray_chkfinite()? > > Because we want the *default* to be safe for all inputs. We want to > > *allow* people to avoid those checks when they have extra information > > about the inputs. Adding a flag is a more convenient way to do this > > than any other alternative. > > OK, so this idea seems to be getting a positive reception. The thing that still needs to be resolved is whether to call the keyword chkfinite or check_finite. I have a slight preference for the latter, because it's more readable (googling a a keyword seems less important). > You and others are totally ignoring the author has stated (without > example) that there is a major roadblock when that assumption is wrong. > > I am very aware of that many functions will provide something even if > the input was not correct. So I agree with Matthew about letting people > live dangerously as the output will tell them that something was wrong. > That means there should be no need for asarray_chkfinite() here. > It's not just that functions may return incorrect results; they may hang in an infinite loop, or crash inside a lapack call for example. Removing asarray_chkfinite() is not possible. In addition it would break backwards compatibility. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Sat Aug 27 06:28:14 2011 From: cournape at gmail.com (David Cournapeau) Date: Sat, 27 Aug 2011 12:28:14 +0200 Subject: [SciPy-Dev] Removing numscons, adding bento scripts to main branch ? Message-ID: Hi there, I am finally at a stage where bento can do most of what numscons could do. I would rather avoid having 3 different set of build scripts (distutils+bento+numscons) to maintain in the long term, so I would favor removing numscons scripts from numpy and scipy. I was thinking about keeping maybe numscons scripts for one release for both numpy/scipy, with a warning about their deprecation, and then removing them one release later. Does that sound ok with everyone ? cheers, David From josef.pktd at gmail.com Sat Aug 27 07:05:54 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sat, 27 Aug 2011 07:05:54 -0400 Subject: [SciPy-Dev] adding chkfinite flags to linalg functions In-Reply-To: References: <4E57BE0A.90506@gmail.com> <4E57E9D9.7010009@gmail.com> <4E57F44D.3030507@gmail.com> <4E5800E6.9010805@gmail.com> Message-ID: On Sat, Aug 27, 2011 at 4:36 AM, Ralf Gommers wrote: > > > On Fri, Aug 26, 2011 at 10:24 PM, Bruce Southey wrote: >> >> On 08/26/2011 02:41 PM, Robert Kern wrote: >> > On Fri, Aug 26, 2011 at 14:30, Bruce Southey ?wrote: >> >> On 08/26/2011 02:09 PM, Matthew Brett wrote: >> >>> Hi, >> >>> >> >>> On Fri, Aug 26, 2011 at 11:45 AM, Bruce Southey >> >>> ?wrote: >> >>>> On 08/26/2011 12:02 PM, Christopher Jordan-Squire wrote: >> >>>>> Breaking it into two steps has its own issues associated with it as >> >>>>> well. Then the user must verify their own inputs before passing them >> >>>>> to scipy linalg methods. If I understand your suggestion correctly. >> >>>> That is exactly what your patch is doing because people will misuse >> >>>> that >> >>>> argument just because they perceive it as faster. >> >>> I object to this objection. >> >>> >> >>> There's a long tradition of letting people do dangerous things if they >> >>> want to. ?Here the option is sensibly labeled and it's not the >> >>> default. >> >> So why not have the patch remove all the calls to asarray_chkfinite()? >> > Because we want the *default* to be safe for all inputs. We want to >> > *allow* people to avoid those checks when they have extra information >> > about the inputs. Adding a flag is a more convenient way to do this >> > than any other alternative. >> > OK, so this idea seems to be getting a positive reception. The thing that > still needs to be resolved is whether to call the keyword chkfinite or > check_finite. I have a slight preference for the latter, because it's more > readable (googling a a keyword seems less important). > >> >> You and others are totally ignoring the author has stated (without >> example) that there is a major roadblock when that assumption is wrong. >> >> I am very aware of that many functions will provide something even if >> the input was not correct. So I agree with Matthew about letting people >> live dangerously as the output will tell them that something was wrong. >> That means there should be no need for asarray_chkfinite() here. > > It's not just that functions may return incorrect results; they may hang in > an infinite loop, or crash inside a lapack call for example. Removing > asarray_chkfinite() is not possible. In addition it would break backwards > compatibility. Crashing or hanging makes this quite dangerous to have in a library package. I'm all in favor of living dangerously and do ex-post checks if something is wrong in the style of try ... except. But having python crash after some long calculations or never return is no fun. scipy had crashing on wrong inputs in ndimage, and on some machines or with some version combinations some optimizers never return. Cheers, Josef > > Cheers, > Ralf > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > > From ralf.gommers at googlemail.com Sat Aug 27 07:31:17 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sat, 27 Aug 2011 13:31:17 +0200 Subject: [SciPy-Dev] [Numpy-discussion] Removing numscons, adding bento scripts to main branch ? In-Reply-To: References: Message-ID: On Sat, Aug 27, 2011 at 12:28 PM, David Cournapeau wrote: > Hi there, > > I am finally at a stage where bento can do most of what numscons could > do. I would rather avoid having 3 different set of build scripts > (distutils+bento+numscons) to maintain in the long term, so I would > favor removing numscons scripts from numpy and scipy. > > That's awesome! > I was thinking about keeping maybe numscons scripts for one release > for both numpy/scipy, with a warning about their deprecation, and then > removing them one release later. > > Does that sound ok with everyone ? > > Sounds like the right thing to do. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From tomasz at kotarba.net Sat Aug 27 08:52:09 2011 From: tomasz at kotarba.net (Tomasz J. Kotarba) Date: Sat, 27 Aug 2011 13:52:09 +0100 Subject: [SciPy-Dev] scipy.spatial.distance.pdist - unnecessarily limited functionality and a suggestion of a possible solution Message-ID: Hello, I would like to suggest a small improvement to pdist(). I have just implemented some new similarity/distance metrics which work with heterogeneous arrays containing both numerical, categorical and set-valued data and tried to use pdist() for calculating the condensed distance matrix by passing the new metric functions as intended (i.e. using the 'metric' argument). Unfortunately, it turned out that I had to implement my own pdist function as the original pdist's functionality is unnecessarily limited to numerical data because of a conversion attempt in line 1089 (shown below): 1088 # The C code doesn't do striding. 1089 [X] = _copy_arrays_if_base_present([_convert_to_double(X)]) I understand the need for that conversion but propose that it either be moved down (to be applied only in case of distance measures which require it) or made possible to switch off (e.g. using a new function call parameter with a default value so that the change does not break the backward compatibility). The rationale behind this is that the intended purpose of pdist() does not require it to access individual values stored in the array - it is the job of a metric function - so decoupling pdist from the data would improve its design (and increase the number of its potential usage scenarios). Regards, T From njs at pobox.com Sat Aug 27 11:18:40 2011 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 27 Aug 2011 08:18:40 -0700 Subject: [SciPy-Dev] adding chkfinite flags to linalg functions In-Reply-To: References: <4E57BE0A.90506@gmail.com> <4E57E9D9.7010009@gmail.com> <4E57F44D.3030507@gmail.com> <4E5800E6.9010805@gmail.com> Message-ID: On Sat, Aug 27, 2011 at 1:36 AM, Ralf Gommers wrote: >> On 08/26/2011 02:41 PM, Robert Kern wrote: >> > Because we want the *default* to be safe for all inputs. We want to >> > *allow* people to avoid those checks when they have extra information >> > about the inputs. Adding a flag is a more convenient way to do this >> > than any other alternative. >> > OK, so this idea seems to be getting a positive reception. The thing that > still needs to be resolved is whether to call the keyword chkfinite or > check_finite. I have a slight preference for the latter, because it's more > readable (googling a a keyword seems less important). As per the observation upthread that these names say what the implementation does, rather than why it does it, which may leave people using it as a generic go-faster knob without realizing the risks involved... what about something like "assume_finite"? Assumptions are things that bite you when they are false. -- Nathaniel From cjordan1 at uw.edu Sun Aug 28 13:07:25 2011 From: cjordan1 at uw.edu (Christopher Jordan-Squire) Date: Sun, 28 Aug 2011 13:07:25 -0400 Subject: [SciPy-Dev] adding chkfinite flags to linalg functions In-Reply-To: References: <4E57BE0A.90506@gmail.com> <4E57E9D9.7010009@gmail.com> <4E57F44D.3030507@gmail.com> <4E5800E6.9010805@gmail.com> Message-ID: Fabian says he's working on a more efficient numpy asarray_chkfinite implementation, and will hopefully have something within a week. https://github.com/scipy/scipy/pull/48 Perhaps this discussion should be put on hold until we see if a faster and more memory-efficient asarray_chkfinite makes this pull request moot. -Chris JS On Sat, Aug 27, 2011 at 11:18 AM, Nathaniel Smith wrote: > On Sat, Aug 27, 2011 at 1:36 AM, Ralf Gommers > wrote: >>> On 08/26/2011 02:41 PM, Robert Kern wrote: >>> > Because we want the *default* to be safe for all inputs. We want to >>> > *allow* people to avoid those checks when they have extra information >>> > about the inputs. Adding a flag is a more convenient way to do this >>> > than any other alternative. >>> >> OK, so this idea seems to be getting a positive reception. The thing that >> still needs to be resolved is whether to call the keyword chkfinite or >> check_finite. I have a slight preference for the latter, because it's more >> readable (googling a a keyword seems less important). > > As per the observation upthread that these names say what the > implementation does, rather than why it does it, which may leave > people using it as a generic go-faster knob without realizing the > risks involved... what about something like "assume_finite"? > Assumptions are things that bite you when they are false. > > -- Nathaniel > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > From ajo.fod at gmail.com Sun Aug 28 13:22:20 2011 From: ajo.fod at gmail.com (Ajo Fod) Date: Sun, 28 Aug 2011 10:22:20 -0700 Subject: [SciPy-Dev] On fmin_slsqp in scipy optimize Message-ID: I think the acc variable is a red herring ... it probably should be frozen at 1e-6 or not be allowed to be below it. I observed that It works best when set to 1e-6 (or larger). When below this it fails and creates unstable solutions. My guess is that in the code: http://www.netlib.org/toms/733 The comments give a clue: * accuracy of optimizer acc = 1.0d-06 * accuracy of simulator ac1 = 1.0d-08 Perhaps the simulator's accuracy needs to be linked to the accuracy of the optimizer? -Ajo From forkandwait at gmail.com Sun Aug 28 19:01:00 2011 From: forkandwait at gmail.com (fork) Date: Sun, 28 Aug 2011 23:01:00 +0000 (UTC) Subject: [SciPy-Dev] =?utf-8?q?import_scipy=2Estats_hangs_in_mod=5Fpython_?= =?utf-8?q?application?= Message-ID: Hi all, This may be really tricky to diagnose, given all the rather weird parameters, but here goes: I have a mod_python application that takes a bunch of mortality rates via a form, and processes them with scipy, numpy, and matplotlib. It is running on a Gentoo box. (Gentoo + mod_python seemed like a good idea at the time, but so did the Edsel... anyway, I still have to maintain this beast...) In the multitude of files that get loaded when a mod_python based page is called, there is a line "import scipy.stats as ST". That line works just great if I am running python at the command line, but it hangs forever when in the modpython application; I know this because I have run it both with an exception before it (barfs on the exception) and after it (just sits there, no errors to the apache log, etc). scipy.__version__ 0.9.0 python is 2.7.1 Anybody have any ideas how to fix this or even diagnose it? From robert.kern at gmail.com Sun Aug 28 19:24:35 2011 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 28 Aug 2011 18:24:35 -0500 Subject: [SciPy-Dev] import scipy.stats hangs in mod_python application In-Reply-To: References: Message-ID: On Sun, Aug 28, 2011 at 18:01, fork wrote: > Hi all, > > This may be really tricky to diagnose, given all the rather weird parameters, > but here goes: > > I have a mod_python application that takes a bunch of mortality rates via a > form, and processes them with scipy, numpy, and matplotlib. It is running on a > Gentoo box. ?(Gentoo + mod_python seemed like a good idea at the time, but so > did the Edsel... anyway, I still have to maintain this beast...) > > In the multitude of files that get loaded when a mod_python based page is > called, there is a line "import scipy.stats as ST". ?That line works just great > if I am running python at the command line, but it hangs forever when in the > modpython application; I know this because I have run it both with an exception > before it (barfs on the exception) and after it (just sits there, no errors to > the apache log, etc). > > scipy.__version__ 0.9.0 > > python is 2.7.1 > > Anybody have any ideas how to fix this or even diagnose it? So, has your program ever worked under mod_python? Is this an intermittent problem, or is it reliably repeatable? In your program, do you import numpy before the scipy.stats import? IIRC, under certain configurations mod_python tries to instantiate multiple interpreters in the same process. For pure Python code, this is usually not too atrocious, even though it is officially unsupported. However, numpy maintains some global pointer tables, and people have run into problems trying to run multiple Python interpreters in the same process when the programs use numpy. There may be mod_python configuration options that control this behavior. First, I would convert the entry point into your application into a WSGI application. Since it looks like there is only one entry point, doing basic form processing before passing the inputs to the computational code, this should be trivial. It's really just a matter of converting the mod_python idioms into WSGI idioms. http://wsgi.org/wsgi/ Now that you have a WSGI application, there are a *lot* of different ways to deploy it. For Apache, mod_wsgi is much preferred over mod_python these days, although since it was written by the same author under very similar constraints as mod_python, he may have used the same multiple interpreters trick. You can use mod_fcgid or mod_scgi to communicate with a separate process running your scipy code fronted by flup: http://trac.saddi.com/flup Using mod_proxy, you can have Apache sit on the front end and communicate with any number of WSGI-specific HTTP servers. gunicorn is a nice one: http://gunicorn.org/ You can look at other options here: http://wsgi.org/wsgi/Servers -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From howarth at bromo.med.uc.edu Sun Aug 28 19:26:04 2011 From: howarth at bromo.med.uc.edu (Jack Howarth) Date: Sun, 28 Aug 2011 19:26:04 -0400 Subject: [SciPy-Dev] suitearse vs scipy Message-ID: <20110828232604.GA18746@bromo.med.uc.edu> I've asked this question before without any responses. What are the supported versions of suitesparse for use with scipy? The current scipy 0.9.0 release doesn't appear to be fully compatible with suitesparse 3.6.1 and causes testsuite failures in scipy. Any information on what is the latest version of suitesparse known to be fully compatible with the current scipy release would be most appreciated. Jack ps I am rather surprised that this information isn't part of the scipy documentation or tested for within setup.py of scipy itself. From josef.pktd at gmail.com Sun Aug 28 19:26:08 2011 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 28 Aug 2011 19:26:08 -0400 Subject: [SciPy-Dev] import scipy.stats hangs in mod_python application In-Reply-To: References: Message-ID: On Sun, Aug 28, 2011 at 7:01 PM, fork wrote: > Hi all, > > This may be really tricky to diagnose, given all the rather weird parameters, > but here goes: > > I have a mod_python application that takes a bunch of mortality rates via a > form, and processes them with scipy, numpy, and matplotlib. It is running on a > Gentoo box. ?(Gentoo + mod_python seemed like a good idea at the time, but so > did the Edsel... anyway, I still have to maintain this beast...) > > In the multitude of files that get loaded when a mod_python based page is > called, there is a line "import scipy.stats as ST". ?That line works just great > if I am running python at the command line, but it hangs forever when in the > modpython application; I know this because I have run it both with an exception > before it (barfs on the exception) and after it (just sits there, no errors to > the apache log, etc). > > scipy.__version__ 0.9.0 > > python is 2.7.1 > > Anybody have any ideas how to fix this or even diagnose it? I have no idea about the actual problem, but scipy.stats requires or loads many of the other scipy subpackages. Since stats is relatively light in compiled extension modules, maybe trying to load first the other subpackages might help to narrow it down. import scipy import scipy.special sparse, optimize, ... Josef > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > From forkandwait at gmail.com Sun Aug 28 21:38:45 2011 From: forkandwait at gmail.com (fork) Date: Mon, 29 Aug 2011 01:38:45 +0000 (UTC) Subject: [SciPy-Dev] =?utf-8?q?import_scipy=2Estats_hangs_in_mod=5Fpython_?= =?utf-8?q?application?= References: Message-ID: Robert Kern gmail.com> writes: > So, has your program ever worked under mod_python? Is this an > intermittent problem, or is it reliably repeatable? In your program, > do you import numpy before the scipy.stats import? Yeah, worked great until last week. > > IIRC, under certain configurations mod_python tries to instantiate > multiple interpreters in the same process. For pure Python code, this > is usually not too atrocious, even though it is officially > unsupported. However, numpy maintains some global pointer tables, and > people have run into problems trying to run multiple Python > interpreters in the same process when the programs use numpy. There > may be mod_python configuration options that control this behavior. Huh. Interesting. > First, I would convert the entry point into your application into a > WSGI application. Since it looks like there is only one entry point, > doing basic form processing before passing the inputs to the > computational code, this should be trivial. It's really just a matter > of converting the mod_python idioms into WSGI idioms. > > http://wsgi.org/wsgi/ The problem is that I use a lot of the convenience dispatching functions in mod_python, and I really, really don't want to rewrite them. Speed isn't the real issue, actually. If I could do it over again, I would have stuck with simple cgi, since the real computational expense is in a bunch of linear algebra not in input/ output. From robert.kern at gmail.com Sun Aug 28 21:57:11 2011 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 28 Aug 2011 20:57:11 -0500 Subject: [SciPy-Dev] import scipy.stats hangs in mod_python application In-Reply-To: References: Message-ID: On Sun, Aug 28, 2011 at 20:38, fork wrote: > Robert Kern gmail.com> writes: > >> So, has your program ever worked under mod_python? Is this an >> intermittent problem, or is it reliably repeatable? In your program, >> do you import numpy before the scipy.stats import? > > Yeah, worked great until last week. Would you mind answering all of the questions rather than just the first? :-) Can you identify what changed last week? Did you change your program? Did you upgrade Apache/mod_python? That's going to be the only way to figure out what is wrong. >> First, I would convert the entry point into your application into a >> WSGI application. Since it looks like there is only one entry point, >> doing basic form processing before passing the inputs to the >> computational code, this should be trivial. It's really just a matter >> of converting the mod_python idioms into WSGI idioms. >> >> ? http://wsgi.org/wsgi/ > > The problem is that I use a lot of the convenience dispatching functions in > mod_python, and I really, really don't want to rewrite them. You may want to take a quick look at a couple of the WSGI micro-frameworks around to see if they might be able to quickly implement the dispatching you need. http://bottlepy.org/docs/dev/ http://flask.pocoo.org/ -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From forkandwait at gmail.com Mon Aug 29 01:32:20 2011 From: forkandwait at gmail.com (fork) Date: Mon, 29 Aug 2011 05:32:20 +0000 (UTC) Subject: [SciPy-Dev] =?utf-8?q?import_scipy=2Estats_hangs_in_mod=5Fpython_?= =?utf-8?q?application?= References: Message-ID: Robert Kern gmail.com> writes: > > On Sun, Aug 28, 2011 at 20:38, fork gmail.com> wrote: > > Robert Kern gmail.com> writes: > > > >> So, has your program ever worked under mod_python? Is this an > >> intermittent problem, or is it reliably repeatable? In your program, > >> do you import numpy before the scipy.stats import? > > > > Yeah, worked great until last week. > > Would you mind answering all of the questions rather than just the first? Sorry (hehe) ... worked great for years, reliable problem, import numpy (and scipy) before scipy.stats. > Can you identify what changed last week? Did you change your program? Did you > upgrade Apache/mod_python? That's going to be the only way to figure out what > is wrong. We upgraded the default python on gentoo from 2.6 to 2.7, so, like, everything changed and it is almost impossible to say what exactly. My partner helped upgrade a bunch of math libraries, but I wasn't part of that so don't know exactly (yet...). I don't *believe* that apache got upgraded, and mod_python is definitely not upgraded, but it was reinstalled. > You may want to take a quick look at a couple of the WSGI > micro-frameworks around to see if they might be able to quickly > implement the dispatching you need. > > http://bottlepy.org/docs/dev/ > http://flask.pocoo.org/ That is great advice, but (1) I am still smarting from depending on a framework that got dropped from current development (three years ago cherrypy would have been in the list above), and (2) I don't have the 100 or so hours to refactor everything to fix this one (new) problem on code that probably won't see a next generation anyway. So, ... probably not gonna happen. On a fresher project, ... well, I would avoid frameworks like the plague and make it all cgi. (I hope I am not coming off obnoxious, but we probably don't need to discuss this any more.) From forkandwait at gmail.com Mon Aug 29 01:49:50 2011 From: forkandwait at gmail.com (fork) Date: Mon, 29 Aug 2011 05:49:50 +0000 (UTC) Subject: [SciPy-Dev] =?utf-8?q?import_scipy=2Estats_hangs_in_mod=5Fpython_?= =?utf-8?q?application?= References: Message-ID: fork gmail.com> writes: > Sorry (hehe) ... worked great for years, reliable problem, import numpy (and > scipy) before scipy.stats. And I just put scipy.stats ahead of everything, same problem. From robert.kern at gmail.com Mon Aug 29 02:16:39 2011 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 29 Aug 2011 01:16:39 -0500 Subject: [SciPy-Dev] import scipy.stats hangs in mod_python application In-Reply-To: References: Message-ID: On Mon, Aug 29, 2011 at 00:32, fork wrote: > Robert Kern gmail.com> writes: > >> >> On Sun, Aug 28, 2011 at 20:38, fork gmail.com> wrote: >> > Robert Kern gmail.com> writes: >> > >> >> So, has your program ever worked under mod_python? Is this an >> >> intermittent problem, or is it reliably repeatable? In your program, >> >> do you import numpy before the scipy.stats import? >> > >> > Yeah, worked great until last week. >> >> Would you mind answering all of the questions rather than just the first? > > Sorry (hehe) ... worked great for years, reliable problem, import numpy (and > scipy) before scipy.stats. Hmm. Check what shared libraries are being linked to all of the scipy extension modules. There might be environment variables affecting how the shared libraries get resolved. >> Can you identify what changed last week? Did you change your program? ?Did you >> upgrade Apache/mod_python? That's going to be the only way to figure out what >> is wrong. > > We upgraded the default python on gentoo from 2.6 to 2.7, so, like, everything > changed and it is almost impossible to say what exactly. ?My partner helped > upgrade a bunch of math libraries, but I wasn't part of that so don't know > exactly (yet...). ?I don't *believe* that apache got upgraded, and mod_python is > definitely not upgraded, but it was reinstalled. > >> You may want to take a quick look at a couple of the WSGI >> micro-frameworks around to see if they might be able to quickly >> implement the dispatching you need. >> >> ? http://bottlepy.org/docs/dev/ >> ? http://flask.pocoo.org/ > > That is great advice, but (1) I am still smarting from depending on a framework > that got dropped from current development (three years ago cherrypy would have > been in the list above), and FWIW, CherryPy is still being developed: http://www.cherrypy.org/timeline -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From pav at iki.fi Mon Aug 29 04:36:42 2011 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 29 Aug 2011 08:36:42 +0000 (UTC) Subject: [SciPy-Dev] suitesparse vs scipy References: <20110828232604.GA18746@bromo.med.uc.edu> Message-ID: Sun, 28 Aug 2011 19:26:04 -0400, Jack Howarth wrote: > I've asked this question before without any responses. What are the > supported versions of suitesparse for use with scipy? The current scipy > 0.9.0 release doesn't appear to be fully compatible with suitesparse > 3.6.1 and causes testsuite failures in scipy. Can you paste the test failures you see? Scipy does not use suitesparse except for the optional UMFPACK wrapper, which many people do not use. Suitesparse is still at 3.4.0 in Debian/Ubuntu, and I suppose newer ones have not been tested. -- Pauli Virtanen From benny.malengier at gmail.com Mon Aug 29 06:02:02 2011 From: benny.malengier at gmail.com (Benny Malengier) Date: Mon, 29 Aug 2011 12:02:02 +0200 Subject: [SciPy-Dev] scikits web cleanup Message-ID: Can somebody with sufficient rights go to http://projects.scipy.org/scikits/wiki and in section Scikits projects, delete everything and instead put a link to the new page: http://scikits.appspot.com/scikits The Opt packages are not on the new links, but then, they had no link anyway, and nearest neighbor is also not present, but that is kdtree Thanks, Benny -------------- next part -------------- An HTML attachment was scrubbed... URL: From howarth at bromo.med.uc.edu Mon Aug 29 09:02:35 2011 From: howarth at bromo.med.uc.edu (Jack Howarth) Date: Mon, 29 Aug 2011 09:02:35 -0400 Subject: [SciPy-Dev] suitesparse vs scipy In-Reply-To: References: <20110828232604.GA18746@bromo.med.uc.edu> Message-ID: <20110829130235.GA23043@bromo.med.uc.edu> On Mon, Aug 29, 2011 at 08:36:42AM +0000, Pauli Virtanen wrote: > Sun, 28 Aug 2011 19:26:04 -0400, Jack Howarth wrote: > > > I've asked this question before without any responses. What are the > > supported versions of suitesparse for use with scipy? The current scipy > > 0.9.0 release doesn't appear to be fully compatible with suitesparse > > 3.6.1 and causes testsuite failures in scipy. > > Can you paste the test failures you see? > > Scipy does not use suitesparse except for the optional UMFPACK wrapper, > which many people do not use. Suitesparse is still at 3.4.0 in > Debian/Ubuntu, and I suppose newer ones have not been tested. Pauli, My mistake. This isn't a testsuite failure but rather a build failure on x86_64-apple-darwin11... building 'scipy.sparse.linalg.dsolve.umfpack.__umfpack' extension compiling C sources C compiler: gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/sw/include creating build/temp.macosx-10.7-x86_64-2.6/build/src.macosx-10.7-x86_64-2.6/scipy/sparse/linalg/dsolve creating build/temp.macosx-10.7-x86_64-2.6/build/src.macosx-10.7-x86_64-2.6/scipy/sparse/linalg/dsolve/umfpack compile options: '-DNO_ATLAS_INFO=3 -I/sw/lib/python2.6/site-packages/numpy/core/include -I/sw/include/python2.6 -c' extra options: '-msse3 -I/System/Library/Frameworks/vecLib.framework/Headers' gcc: build/src.macosx-10.7-x86_64-2.6/scipy/sparse/linalg/dsolve/umfpack/_umfpack_wrap.c build/src.macosx-10.7-x86_64-2.6/scipy/sparse/linalg/dsolve/umfpack/_umfpack_wrap.c:2351:23: warning: explicitly assigning a variable of type 'int' to itself [-Wself-assign] res = SWIG_AddCast(res); ~~~ ^ ~~~ build/src.macosx-10.7-x86_64-2.6/scipy/sparse/linalg/dsolve/umfpack/_umfpack_wrap.c:2354:23: warning: explicitly assigning a variable of type 'int' to itself [-Wself-assign] res = SWIG_AddCast(res); ~~~ ^ ~~~ build/src.macosx-10.7-x86_64-2.6/scipy/sparse/linalg/dsolve/umfpack/_umfpack_wrap.c:6941:14: warning: explicitly assigning a variable of type 'void *' to itself [-Wself-assign] clientdata = clientdata; ~~~~~~~~~~ ^ ~~~~~~~~~~ 3 warnings generated. gcc -L/sw/lib -bundle -L/sw/lib/python2.6/config -lpython2.6 -I/sw/include build/temp.macosx-10.7-x86_64-2.6/build/src.macosx-10.7-x86_64-2.6/scipy/sparse/linalg/dsolve/umfpack/_umfpack_wrap.o -L/opt/local/lib -Lbuild/temp.macosx-10.7-x86_64-2.6 -lumfpack -lamd -o build/lib.macosx-10.7-x86_64-2.6/scipy/sparse/linalg/dsolve/umfpack/__umfpack.so -Wl,-framework -Wl,Accelerate Undefined symbols for architecture x86_64: "_cholmod_start", referenced from: _umf_i_cholmod in libumfpack.a(umf_i_cholmod.o) "_cholmod_transpose", referenced from: _umf_i_cholmod in libumfpack.a(umf_i_cholmod.o) "_cholmod_analyze", referenced from: _umf_i_cholmod in libumfpack.a(umf_i_cholmod.o) "_cholmod_free_sparse", referenced from: _umf_i_cholmod in libumfpack.a(umf_i_cholmod.o) "_cholmod_free_factor", referenced from: _umf_i_cholmod in libumfpack.a(umf_i_cholmod.o) "_cholmod_print_common", referenced from: _umf_i_cholmod in libumfpack.a(umf_i_cholmod.o) "_cholmod_finish", referenced from: _umf_i_cholmod in libumfpack.a(umf_i_cholmod.o) "_cholmod_l_start", referenced from: _umf_l_cholmod in libumfpack.a(umf_l_cholmod.o) "_cholmod_l_transpose", referenced from: _umf_l_cholmod in libumfpack.a(umf_l_cholmod.o) "_cholmod_l_analyze", referenced from: _umf_l_cholmod in libumfpack.a(umf_l_cholmod.o) "_cholmod_l_free_sparse", referenced from: _umf_l_cholmod in libumfpack.a(umf_l_cholmod.o) "_cholmod_l_free_factor", referenced from: _umf_l_cholmod in libumfpack.a(umf_l_cholmod.o) "_cholmod_l_print_common", referenced from: _umf_l_cholmod in libumfpack.a(umf_l_cholmod.o) "_cholmod_l_finish", referenced from: _umf_l_cholmod in libumfpack.a(umf_l_cholmod.o) ld: symbol(s) not found for architecture x86_64 clang: error: linker command failed with exit code 1 (use -v to see invocation) Undefined symbols for architecture x86_64: "_cholmod_start", referenced from: _umf_i_cholmod in libumfpack.a(umf_i_cholmod.o) "_cholmod_transpose", referenced from: _umf_i_cholmod in libumfpack.a(umf_i_cholmod.o) "_cholmod_analyze", referenced from: _umf_i_cholmod in libumfpack.a(umf_i_cholmod.o) "_cholmod_free_sparse", referenced from: _umf_i_cholmod in libumfpack.a(umf_i_cholmod.o) "_cholmod_free_factor", referenced from: _umf_i_cholmod in libumfpack.a(umf_i_cholmod.o) "_cholmod_print_common", referenced from: _umf_i_cholmod in libumfpack.a(umf_i_cholmod.o) "_cholmod_finish", referenced from: _umf_i_cholmod in libumfpack.a(umf_i_cholmod.o) "_cholmod_l_start", referenced from: _umf_l_cholmod in libumfpack.a(umf_l_cholmod.o) "_cholmod_l_transpose", referenced from: _umf_l_cholmod in libumfpack.a(umf_l_cholmod.o) "_cholmod_l_analyze", referenced from: _umf_l_cholmod in libumfpack.a(umf_l_cholmod.o) "_cholmod_l_free_sparse", referenced from: _umf_l_cholmod in libumfpack.a(umf_l_cholmod.o) "_cholmod_l_free_factor", referenced from: _umf_l_cholmod in libumfpack.a(umf_l_cholmod.o) "_cholmod_l_print_common", referenced from: _umf_l_cholmod in libumfpack.a(umf_l_cholmod.o) "_cholmod_l_finish", referenced from: _umf_l_cholmod in libumfpack.a(umf_l_cholmod.o) ld: symbol(s) not found for architecture x86_64 clang: error: linker command failed with exit code 1 (use -v to see invocation) error: Command "gcc -L/sw/lib -bundle -L/sw/lib/python2.6/config -lpython2.6 -I/sw/include build/temp.macosx-10.7-x86_64-2.6/build/src.macosx-10.7-x86_64-2.6/scipy/sparse/linalg/dsolve/umfpack/_umfpack_wrap.o -L/opt/local/lib -Lbuild/temp.macosx-10.7-x86_64-2.6 -lumfpack -lamd -o build/lib.macosx-10.7-x86_64-2.6/scipy/sparse/linalg/dsolve/umfpack/__umfpack.so -Wl,-framework -Wl,Accelerate" failed with exit status 1 The initial output from setup.py in scipy 0.9 is... #!/bin/sh -ev unset LDFLAGS /sw/bin/python2.6 setup.py build --fcompiler=gnu95 Warning: No configuration returned, assuming unavailable.blas_opt_info: FOUND: extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] define_macros = [('NO_ATLAS_INFO', 3)] extra_compile_args = ['-msse3', '-I/System/Library/Frameworks/vecLib.framework/Headers'] non-existing path in 'scipy/io': 'docs' lapack_opt_info: FOUND: extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] define_macros = [('NO_ATLAS_INFO', 3)] extra_compile_args = ['-msse3'] umfpack_info: libraries umfpack not found in /usr/local/lib libraries umfpack not found in /usr/lib amd_info: libraries amd not found in /usr/local/lib libraries amd not found in /usr/lib FOUND: libraries = ['amd'] library_dirs = ['/opt/local/lib'] FOUND: libraries = ['umfpack', 'amd'] library_dirs = ['/opt/local/lib'] running build running config_cc unifing config_cc, config, build_clib, build_ext, build commands --compiler options running config_fc unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options running build_src build_src building py_modules sources creating build creating build/src.macosx-10.7-x86_64-2.6 creating build/src.macosx-10.7-x86_64-2.6/scipy building library "dfftpack" sources building library "fftpack" sources building library "linpack_lite" sources building library "mach" sources building library "quadpack" sources building library "odepack" sources building library "dop" sources building library "fitpack" sources building library "odrpack" sources building library "minpack" sources building library "rootfind" sources building library "superlu_src" sources building library "arpack" sources building library "qhull" sources building library "sc_c_misc" sources building library "sc_cephes" sources building library "sc_mach" sources building library "sc_toms" sources building library "sc_amos" sources building library "sc_cdf" sources building library "sc_specfun" sources building library "statlib" sources building extension "scipy.cluster._vq" sources building extension "scipy.cluster._hierarchy_wrap" sources building extension "scipy.fftpack._fftpack" sources creating build/src.macosx-10.7-x86_64-2.6/scipy/fftpack creating build/src.macosx-10.7-x86_64-2.6/scipy/fftpack/src conv_template:> build/src.macosx-10.7-x86_64-2.6/scipy/fftpack/src/dct.c f2py options: [] f2py: scipy/fftpack/fftpack.pyf Reading fortran codes... Reading file 'scipy/fftpack/fftpack.pyf' (format:free) Line #86 in scipy/fftpack/fftpack.pyf:" /* Single precision version */" crackline:2: No pattern for line > > -- > Pauli Virtanen > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev From howarth at bromo.med.uc.edu Mon Aug 29 10:55:21 2011 From: howarth at bromo.med.uc.edu (Jack Howarth) Date: Mon, 29 Aug 2011 10:55:21 -0400 Subject: [SciPy-Dev] suitesparse vs scipy In-Reply-To: <20110829130235.GA23043@bromo.med.uc.edu> References: <20110828232604.GA18746@bromo.med.uc.edu> <20110829130235.GA23043@bromo.med.uc.edu> Message-ID: <20110829145521.GA23834@bromo.med.uc.edu> On Mon, Aug 29, 2011 at 09:02:35AM -0400, Jack Howarth wrote: > On Mon, Aug 29, 2011 at 08:36:42AM +0000, Pauli Virtanen wrote: > > Sun, 28 Aug 2011 19:26:04 -0400, Jack Howarth wrote: > > > > > I've asked this question before without any responses. What are the > > > supported versions of suitesparse for use with scipy? The current scipy > > > 0.9.0 release doesn't appear to be fully compatible with suitesparse > > > 3.6.1 and causes testsuite failures in scipy. > > > > Can you paste the test failures you see? > > > > Scipy does not use suitesparse except for the optional UMFPACK wrapper, > > which many people do not use. Suitesparse is still at 3.4.0 in > > Debian/Ubuntu, and I suppose newer ones have not been tested. > > Pauli, > My mistake. This isn't a testsuite failure but rather a build failure on > x86_64-apple-darwin11... > > building 'scipy.sparse.linalg.dsolve.umfpack.__umfpack' extension > compiling C sources > C compiler: gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/sw/include > > creating build/temp.macosx-10.7-x86_64-2.6/build/src.macosx-10.7-x86_64-2.6/scipy/sparse/linalg/dsolve > creating build/temp.macosx-10.7-x86_64-2.6/build/src.macosx-10.7-x86_64-2.6/scipy/sparse/linalg/dsolve/umfpack > compile options: '-DNO_ATLAS_INFO=3 -I/sw/lib/python2.6/site-packages/numpy/core/include -I/sw/include/python2.6 -c' > extra options: '-msse3 -I/System/Library/Frameworks/vecLib.framework/Headers' > gcc: build/src.macosx-10.7-x86_64-2.6/scipy/sparse/linalg/dsolve/umfpack/_umfpack_wrap.c > build/src.macosx-10.7-x86_64-2.6/scipy/sparse/linalg/dsolve/umfpack/_umfpack_wrap.c:2351:23: warning: explicitly assigning a variable of type 'int' to itself [-Wself-assign] > res = SWIG_AddCast(res); > ~~~ ^ ~~~ > build/src.macosx-10.7-x86_64-2.6/scipy/sparse/linalg/dsolve/umfpack/_umfpack_wrap.c:2354:23: warning: explicitly assigning a variable of type 'int' to itself [-Wself-assign] > res = SWIG_AddCast(res); > ~~~ ^ ~~~ > build/src.macosx-10.7-x86_64-2.6/scipy/sparse/linalg/dsolve/umfpack/_umfpack_wrap.c:6941:14: warning: explicitly assigning a variable of type 'void *' to itself [-Wself-assign] > clientdata = clientdata; > ~~~~~~~~~~ ^ ~~~~~~~~~~ > 3 warnings generated. > gcc -L/sw/lib -bundle -L/sw/lib/python2.6/config -lpython2.6 -I/sw/include build/temp.macosx-10.7-x86_64-2.6/build/src.macosx-10.7-x86_64-2.6/scipy/sparse/linalg/dsolve/umfpack/_umfpack_wrap.o -L/opt/local/lib -Lbuild/temp.macosx-10.7-x86_64-2.6 -lumfpack -lamd -o build/lib.macosx-10.7-x86_64-2.6/scipy/sparse/linalg/dsolve/umfpack/__umfpack.so -Wl,-framework -Wl,Accelerate > Undefined symbols for architecture x86_64: > "_cholmod_start", referenced from: > _umf_i_cholmod in libumfpack.a(umf_i_cholmod.o) > "_cholmod_transpose", referenced from: > _umf_i_cholmod in libumfpack.a(umf_i_cholmod.o) > "_cholmod_analyze", referenced from: > _umf_i_cholmod in libumfpack.a(umf_i_cholmod.o) > "_cholmod_free_sparse", referenced from: > _umf_i_cholmod in libumfpack.a(umf_i_cholmod.o) > "_cholmod_free_factor", referenced from: > _umf_i_cholmod in libumfpack.a(umf_i_cholmod.o) > "_cholmod_print_common", referenced from: > _umf_i_cholmod in libumfpack.a(umf_i_cholmod.o) > "_cholmod_finish", referenced from: > _umf_i_cholmod in libumfpack.a(umf_i_cholmod.o) > "_cholmod_l_start", referenced from: > _umf_l_cholmod in libumfpack.a(umf_l_cholmod.o) > "_cholmod_l_transpose", referenced from: > _umf_l_cholmod in libumfpack.a(umf_l_cholmod.o) > "_cholmod_l_analyze", referenced from: > _umf_l_cholmod in libumfpack.a(umf_l_cholmod.o) > "_cholmod_l_free_sparse", referenced from: > _umf_l_cholmod in libumfpack.a(umf_l_cholmod.o) > "_cholmod_l_free_factor", referenced from: > _umf_l_cholmod in libumfpack.a(umf_l_cholmod.o) > "_cholmod_l_print_common", referenced from: > _umf_l_cholmod in libumfpack.a(umf_l_cholmod.o) > "_cholmod_l_finish", referenced from: > _umf_l_cholmod in libumfpack.a(umf_l_cholmod.o) > ld: symbol(s) not found for architecture x86_64 > clang: error: linker command failed with exit code 1 (use -v to see invocation) > Undefined symbols for architecture x86_64: > "_cholmod_start", referenced from: > _umf_i_cholmod in libumfpack.a(umf_i_cholmod.o) > "_cholmod_transpose", referenced from: > _umf_i_cholmod in libumfpack.a(umf_i_cholmod.o) > "_cholmod_analyze", referenced from: > _umf_i_cholmod in libumfpack.a(umf_i_cholmod.o) > "_cholmod_free_sparse", referenced from: > _umf_i_cholmod in libumfpack.a(umf_i_cholmod.o) > "_cholmod_free_factor", referenced from: > _umf_i_cholmod in libumfpack.a(umf_i_cholmod.o) > "_cholmod_print_common", referenced from: > _umf_i_cholmod in libumfpack.a(umf_i_cholmod.o) > "_cholmod_finish", referenced from: > _umf_i_cholmod in libumfpack.a(umf_i_cholmod.o) > "_cholmod_l_start", referenced from: > _umf_l_cholmod in libumfpack.a(umf_l_cholmod.o) > "_cholmod_l_transpose", referenced from: > _umf_l_cholmod in libumfpack.a(umf_l_cholmod.o) > "_cholmod_l_analyze", referenced from: > _umf_l_cholmod in libumfpack.a(umf_l_cholmod.o) > "_cholmod_l_free_sparse", referenced from: > _umf_l_cholmod in libumfpack.a(umf_l_cholmod.o) > "_cholmod_l_free_factor", referenced from: > _umf_l_cholmod in libumfpack.a(umf_l_cholmod.o) > "_cholmod_l_print_common", referenced from: > _umf_l_cholmod in libumfpack.a(umf_l_cholmod.o) > "_cholmod_l_finish", referenced from: > _umf_l_cholmod in libumfpack.a(umf_l_cholmod.o) > ld: symbol(s) not found for architecture x86_64 > clang: error: linker command failed with exit code 1 (use -v to see invocation) > error: Command "gcc -L/sw/lib -bundle -L/sw/lib/python2.6/config -lpython2.6 -I/sw/include build/temp.macosx-10.7-x86_64-2.6/build/src.macosx-10.7-x86_64-2.6/scipy/sparse/linalg/dsolve/umfpack/_umfpack_wrap.o -L/opt/local/lib -Lbuild/temp.macosx-10.7-x86_64-2.6 -lumfpack -lamd -o build/lib.macosx-10.7-x86_64-2.6/scipy/sparse/linalg/dsolve/umfpack/__umfpack.so -Wl,-framework -Wl,Accelerate" failed with exit status 1 These missing symbols seem to exist in libcholmod.a which isn't being added to the linkage. Any idea of the correct way to fix this for the newer suitesparse? > > The initial output from setup.py in scipy 0.9 is... > > #!/bin/sh -ev > unset LDFLAGS > /sw/bin/python2.6 setup.py build --fcompiler=gnu95 > Warning: No configuration returned, assuming unavailable.blas_opt_info: > FOUND: > extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] > define_macros = [('NO_ATLAS_INFO', 3)] > extra_compile_args = ['-msse3', '-I/System/Library/Frameworks/vecLib.framework/Headers'] > > non-existing path in 'scipy/io': 'docs' > lapack_opt_info: > FOUND: > extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] > define_macros = [('NO_ATLAS_INFO', 3)] > extra_compile_args = ['-msse3'] > > umfpack_info: > libraries umfpack not found in /usr/local/lib > libraries umfpack not found in /usr/lib > amd_info: > libraries amd not found in /usr/local/lib > libraries amd not found in /usr/lib > FOUND: > libraries = ['amd'] > library_dirs = ['/opt/local/lib'] > > FOUND: > libraries = ['umfpack', 'amd'] > library_dirs = ['/opt/local/lib'] > > running build > running config_cc > unifing config_cc, config, build_clib, build_ext, build commands --compiler options > running config_fc > unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options > running build_src > build_src > building py_modules sources > creating build > creating build/src.macosx-10.7-x86_64-2.6 > creating build/src.macosx-10.7-x86_64-2.6/scipy > building library "dfftpack" sources > building library "fftpack" sources > building library "linpack_lite" sources > building library "mach" sources > building library "quadpack" sources > building library "odepack" sources > building library "dop" sources > building library "fitpack" sources > building library "odrpack" sources > building library "minpack" sources > building library "rootfind" sources > building library "superlu_src" sources > building library "arpack" sources > building library "qhull" sources > building library "sc_c_misc" sources > building library "sc_cephes" sources > building library "sc_mach" sources > building library "sc_toms" sources > building library "sc_amos" sources > building library "sc_cdf" sources > building library "sc_specfun" sources > building library "statlib" sources > building extension "scipy.cluster._vq" sources > building extension "scipy.cluster._hierarchy_wrap" sources > building extension "scipy.fftpack._fftpack" sources > creating build/src.macosx-10.7-x86_64-2.6/scipy/fftpack > creating build/src.macosx-10.7-x86_64-2.6/scipy/fftpack/src > conv_template:> build/src.macosx-10.7-x86_64-2.6/scipy/fftpack/src/dct.c > f2py options: [] > f2py: scipy/fftpack/fftpack.pyf > Reading fortran codes... > Reading file 'scipy/fftpack/fftpack.pyf' (format:free) > Line #86 in scipy/fftpack/fftpack.pyf:" /* Single precision version */" > crackline:2: No pattern for line > > > > > -- > > Pauli Virtanen > > > > _______________________________________________ > > SciPy-Dev mailing list > > SciPy-Dev at scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-dev > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev From pav at iki.fi Mon Aug 29 11:00:00 2011 From: pav at iki.fi (Pauli Virtanen) Date: Mon, 29 Aug 2011 15:00:00 +0000 (UTC) Subject: [SciPy-Dev] suitesparse vs scipy References: <20110828232604.GA18746@bromo.med.uc.edu> <20110829130235.GA23043@bromo.med.uc.edu> Message-ID: Mon, 29 Aug 2011 09:02:35 -0400, Jack Howarth wrote: [clip] > "_cholmod_l_finish", referenced from: > _umf_l_cholmod in libumfpack.a(umf_l_cholmod.o) > ld: symbol(s) not found for architecture x86_64 clang: error: linker > command failed with exit code 1 (use -v to see invocation) error: > Command "gcc -L/sw/lib -bundle -L/sw/lib/python2.6/config -lpython2.6 > -I/sw/include > build/temp.macosx-10.7-x86_64-2.6/build/src.macosx-10.7-x86_64-2.6/scipy/sparse/linalg/dsolve/umfpack/_umfpack_wrap.o > -L/opt/local/lib -Lbuild/temp.macosx-10.7-x86_64-2.6 -lumfpack -lamd -o > build/lib.macosx-10.7-x86_64-2.6/scipy/sparse/linalg/dsolve/umfpack/__umfpack.so > -Wl,-framework -Wl,Accelerate" failed with exit status 1 Apparently, umfpack started to depend on cholmod. Try adding this to site.cfg before building scipy: [umfpack] umfpack_libs = umfpack, cholmod, amd From howarth at bromo.med.uc.edu Mon Aug 29 12:22:50 2011 From: howarth at bromo.med.uc.edu (Jack Howarth) Date: Mon, 29 Aug 2011 12:22:50 -0400 Subject: [SciPy-Dev] suitesparse vs scipy In-Reply-To: References: <20110828232604.GA18746@bromo.med.uc.edu> <20110829130235.GA23043@bromo.med.uc.edu> Message-ID: <20110829162250.GA24364@bromo.med.uc.edu> On Mon, Aug 29, 2011 at 03:00:00PM +0000, Pauli Virtanen wrote: > Mon, 29 Aug 2011 09:02:35 -0400, Jack Howarth wrote: > [clip] > > "_cholmod_l_finish", referenced from: > > _umf_l_cholmod in libumfpack.a(umf_l_cholmod.o) > > ld: symbol(s) not found for architecture x86_64 clang: error: linker > > command failed with exit code 1 (use -v to see invocation) error: > > Command "gcc -L/sw/lib -bundle -L/sw/lib/python2.6/config -lpython2.6 > > -I/sw/include > > build/temp.macosx-10.7-x86_64-2.6/build/src.macosx-10.7-x86_64-2.6/scipy/sparse/linalg/dsolve/umfpack/_umfpack_wrap.o > > -L/opt/local/lib -Lbuild/temp.macosx-10.7-x86_64-2.6 -lumfpack -lamd -o > > build/lib.macosx-10.7-x86_64-2.6/scipy/sparse/linalg/dsolve/umfpack/__umfpack.so > > -Wl,-framework -Wl,Accelerate" failed with exit status 1 > > Apparently, umfpack started to depend on cholmod. > Try adding this to site.cfg before building scipy: > > [umfpack] > umfpack_libs = umfpack, cholmod, amd > Pauli, Actually, a site.cfg with... [umfpack] umfpack_libs = umfpack, colamd, cholmod, amd is required to resolve all of the cholmod related symbols when building scipy 0.9.0 against suitesparse 3.6.1. Jack > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev From ralf.gommers at googlemail.com Mon Aug 29 13:11:04 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Mon, 29 Aug 2011 19:11:04 +0200 Subject: [SciPy-Dev] scipy.spatial.distance.pdist - unnecessarily limited functionality and a suggestion of a possible solution In-Reply-To: References: Message-ID: On Sat, Aug 27, 2011 at 2:52 PM, Tomasz J. Kotarba wrote: > Hello, > I would like to suggest a small improvement to pdist(). I have just > implemented some new similarity/distance metrics which work with > heterogeneous arrays containing both numerical, categorical and > set-valued data and tried to use pdist() for calculating the condensed > distance matrix by passing the new metric functions as intended (i.e. > using the 'metric' argument). Unfortunately, it turned out that I had > to implement my own pdist function as the original pdist's > functionality is unnecessarily limited to numerical data because of a > conversion attempt in line 1089 (shown below): > > 1088 # The C code doesn't do striding. > 1089 [X] = _copy_arrays_if_base_present([_convert_to_double(X)]) > > I understand the need for that conversion but propose that it either > be moved down (to be applied only in case of distance measures which > require it) or made possible to switch off (e.g. using a new function > call parameter with a default value so that the change does not break > the backward compatibility). The rationale behind this is that the > intended purpose of pdist() does not require it to access individual > values stored in the array - it is the job of a metric function - so > decoupling pdist from the data would improve its design (and increase > the number of its potential usage scenarios). > > It looks like you could move the _convert_to_double call into each of these if/elif cases: if metric == minkowski: def dfun(u, v): return minkowski(u, v, p) elif metric == wminkowski: def dfun(u, v): return wminkowski(u, v, p, w) elif metric == seuclidean: def dfun(u, v): return seuclidean(u, v, V) elif metric == mahalanobis: def dfun(u, v): return mahalanobis(u, v, V) I don't think you can move _copy_arrays_if_base_present(). Could you put together a patch including a test that covers your use case? Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Mon Aug 29 13:56:44 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Mon, 29 Aug 2011 19:56:44 +0200 Subject: [SciPy-Dev] Target date for 0.10 release In-Reply-To: References: Message-ID: On Thu, Aug 18, 2011 at 10:16 AM, Ralf Gommers wrote: > > > On Thu, Aug 18, 2011 at 8:58 AM, Ralf Gommers > wrote: > >> >> >> On Thu, Aug 18, 2011 at 2:13 AM, Pauli Virtanen wrote: >> >>> On Wed, 17 Aug 2011 18:08:12 -0500, Warren Weckesser wrote: >>> > Just checking: the roadmap (http://projects.scipy.org/scipy/roadmap) >>> > says Sept. 30 for the release of 0.10. Is that still the plan? >>> >>> Possibly -- there are already some new features and bugfixes. >> >> >> The 0.10 release notes are still a bit sparse. A few things that should be >> in there but aren't yet: >> - Lomb-Scargle periodogram >> - firwin types III/IV >> - update to CODATA 2010 >> - deprecated lib.blas & some cleanup in linalg.blas >> >> I also checked the pending deprecations/removals. The only deprecated > items in 0.9 were the following nonlinear solvers: > - ``broyden_modified`` (bad performance) > - ``broyden1_modified`` (bad performance) > - ``broyden_generalized`` (equivalent to ``anderson``) > - ``anderson2`` (equivalent to ``anderson``) > - ``broyden3`` (obsoleted by new limited-memory broyden methods) > - ``vackar`` (renamed to ``diagbroyden``) > > Can these be removed? > > Any opinions on this? If no one has a good reason for keeping them around a little longer I'll remove them. > > > There are also quite a few open pull requests. Here are some that I think >> are in good shape and would be good to merge before a first beta: >> >> 4: more stable BLAS routines for euclidean norm >> 11: support for Harwell-Boeing format >> 21: discrete state-space, transfer function and LTI systems >> 35: add description of public API and import guidelines >> 48: chkfinite flags for linalg functions >> 52: logit/expit functions >> >> > >> >>> However, >>> the crash-bugs that occur in the iterative solvers on OSX (but not on >>> other platforms) in my opinion are a blocker that should be solved >>> before release. >>> >> >> This is fixed now, so there are no show stoppers anymore. Just a number of pull request and tickets to close. Most of the tickets marked for 0.9.1 or 0.10.0 in Trac are about the signal and stats modules. Warren and Josef, could you please have a look at those? If they're not straightforward and important to fix they can be set to Unscheduled. There are several pull requests that are (almost) ready to go in; of the ones that still require some work or review I think especially #21 (discrete state-space, transfer function and LTI systems) and #55 (QR decomposition with Q multiplication) would be nice to have merged. Here is a proposed schedule: Sun 11 Sep: beta 1 Sun 25 Sep: rc 1 Sun 02 Oct: rc 2 (if needed) Sun 09 Oct: final release Does that work for everyone? Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From vanforeest at gmail.com Mon Aug 29 13:57:07 2011 From: vanforeest at gmail.com (nicky van foreest) Date: Mon, 29 Aug 2011 19:57:07 +0200 Subject: [SciPy-Dev] small typo in scipy.stats.uniform doc strign Message-ID: Hi, I encountered the following string in http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.uniform.html : This distribution is constant between loc and loc = scale. I am almost sure that the intention is "loc and loc+scale", rather than the = Nicky From robert.kern at gmail.com Mon Aug 29 14:08:50 2011 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 29 Aug 2011 13:08:50 -0500 Subject: [SciPy-Dev] small typo in scipy.stats.uniform doc strign In-Reply-To: References: Message-ID: On Mon, Aug 29, 2011 at 12:57, nicky van foreest wrote: > Hi, > > I encountered the following string in > http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.uniform.html > : > > This distribution is constant between loc and loc = scale. > > I am almost sure that the intention is "loc and loc+scale", rather than the = Yes. Fixed through the web editor. Thank you! -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco From stefan at sun.ac.za Tue Aug 30 07:28:11 2011 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Tue, 30 Aug 2011 13:28:11 +0200 Subject: [SciPy-Dev] Target date for 0.10 release In-Reply-To: References: Message-ID: On Mon, Aug 29, 2011 at 7:56 PM, Ralf Gommers wrote: > Here is a proposed schedule: > Sun 11 Sep: beta 1 > Sun 25 Sep: rc 1 > Sun 02 Oct: rc 2 (if needed) > Sun 09 Oct: final release > > Does that work for everyone? I'll be out of touch for the next 10 days, but if I can help with any patch reviews / reworking after that, let me know. The schedule looks good! Regards St?fan From howarth at bromo.med.uc.edu Tue Aug 30 08:55:15 2011 From: howarth at bromo.med.uc.edu (Jack Howarth) Date: Tue, 30 Aug 2011 08:55:15 -0400 Subject: [SciPy-Dev] suitesparse vs scipy In-Reply-To: References: <20110828232604.GA18746@bromo.med.uc.edu> Message-ID: <20110830125515.GA1309@bromo.med.uc.edu> On Mon, Aug 29, 2011 at 08:36:42AM +0000, Pauli Virtanen wrote: > Sun, 28 Aug 2011 19:26:04 -0400, Jack Howarth wrote: > > > I've asked this question before without any responses. What are the > > supported versions of suitesparse for use with scipy? The current scipy > > 0.9.0 release doesn't appear to be fully compatible with suitesparse > > 3.6.1 and causes testsuite failures in scipy. > > Can you paste the test failures you see? > > Scipy does not use suitesparse except for the optional UMFPACK wrapper, > which many people do not use. Suitesparse is still at 3.4.0 in > Debian/Ubuntu, and I suppose newer ones have not been tested. > > -- > Pauli Virtanen Pauli, On x86_64-apple-darwin11, building scypy 0.9.0 against python2.7 using the clang compilers and gfortran from FSF gcc 4.6.1, I get the following testsuite results... [MacPro:~] howarth% /sw/bin/python2.7 Python 2.7.2 (default, Jul 21 2011, 22:39:54) [GCC 4.2.1 Compatible Apple Clang 2.1 (tags/Apple/clang-163.7.1)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import scipy >>> scipy.test() Running unit tests for scipy NumPy version 1.6.0 NumPy is installed in /sw/lib/python2.7/site-packages/numpy SciPy version 0.9.0 SciPy is installed in /sw/lib/python2.7/site-packages/scipy Python version 2.7.2 (default, Jul 21 2011, 22:39:54) [GCC 4.2.1 Compatible Apple Clang 2.1 (tags/Apple/clang-163.7.1)] nose version 1.1.2 .............................................................................................................................................................................................................K......................................................................................................................K..K.........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................F..FF......................................................................................................................................SSSSSS......SSSSSS......SSSS.....................................................F....FF.....S................................................................................................................................................................................................................K...........................................................................................................................................................................................SSSSS.........S.........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................../sw/lib/python2.7/site-packages/scipy/sparse/linalg/dsolve/linsolve.py:259: DeprecationWarning: scipy.sparse.linalg.dsolve.umfpack will be removed, install scikits.umfpack instead ' install scikits.umfpack instead', DeprecationWarning ) ../sw/lib/python2.7/site-packages/scipy/sparse/linalg/dsolve/linsolve.py:75: DeprecationWarning: scipy.sparse.linalg.dsolve.umfpack will be removed, install scikits.umfpack instead ' install scikits.umfpack instead', DeprecationWarning ) ................K.K..KK.KK...........................................................................................................................................K...............................................................K...........................................................................................................................................................KK...............................................................................................................................................................................................................................................................................................................................................................................................................................K.K........................................................................................................................................................................................................................................................................................................................................................................................K........K.........SSSSSSS................................................................................................................................................................................................................................................................................................................................................................................................../sw/lib/python2.7/site-packages/scipy/stats/distributions.py:3546: RuntimeWarning: overflow encountered in exp return exp(c*x-exp(x)-gamln(c)) ................................./sw/lib/python2.7/site-packages/scipy/stats/distributions.py:3955: RuntimeWarning: invalid value encountered in sqrt vals = 2*(bt+1.0)*sqrt(b-2.0)/((b-3.0)*sqrt(b)) .........................................................................................................................................................................................................................................................................................................S..............................................................................................................................................................................................F............................................................................................................................................................................................................................................................................................................................................................................................................. ====================================================================== FAIL: test_asum (test_blas.TestFBLAS1Simple) ---------------------------------------------------------------------- Traceback (most recent call last): File "/sw/lib/python2.7/site-packages/scipy/lib/blas/tests/test_blas.py", line 58, in test_asum assert_almost_equal(f([3,-4,5]),12) File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 468, in assert_almost_equal raise AssertionError(msg) AssertionError: Arrays are not almost equal to 7 decimals ACTUAL: 0.0 DESIRED: 12 ====================================================================== FAIL: test_dot (test_blas.TestFBLAS1Simple) ---------------------------------------------------------------------- Traceback (most recent call last): File "/sw/lib/python2.7/site-packages/scipy/lib/blas/tests/test_blas.py", line 67, in test_dot assert_almost_equal(f([3,-4,5],[2,5,1]),-9) File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 468, in assert_almost_equal raise AssertionError(msg) AssertionError: Arrays are not almost equal to 7 decimals ACTUAL: 0.0 DESIRED: -9 ====================================================================== FAIL: test_nrm2 (test_blas.TestFBLAS1Simple) ---------------------------------------------------------------------- Traceback (most recent call last): File "/sw/lib/python2.7/site-packages/scipy/lib/blas/tests/test_blas.py", line 78, in test_nrm2 assert_almost_equal(f([3,-4,5]),math.sqrt(50)) File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 468, in assert_almost_equal raise AssertionError(msg) AssertionError: Arrays are not almost equal to 7 decimals ACTUAL: 0.0 DESIRED: 7.0710678118654755 ====================================================================== FAIL: test_asum (test_blas.TestFBLAS1Simple) ---------------------------------------------------------------------- Traceback (most recent call last): File "/sw/lib/python2.7/site-packages/scipy/linalg/tests/test_blas.py", line 57, in test_asum assert_almost_equal(f([3,-4,5]),12) File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 468, in assert_almost_equal raise AssertionError(msg) AssertionError: Arrays are not almost equal to 7 decimals ACTUAL: 0.0 DESIRED: 12 ====================================================================== FAIL: test_dot (test_blas.TestFBLAS1Simple) ---------------------------------------------------------------------- Traceback (most recent call last): File "/sw/lib/python2.7/site-packages/scipy/linalg/tests/test_blas.py", line 66, in test_dot assert_almost_equal(f([3,-4,5],[2,5,1]),-9) File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 468, in assert_almost_equal raise AssertionError(msg) AssertionError: Arrays are not almost equal to 7 decimals ACTUAL: 0.0 DESIRED: -9 ====================================================================== FAIL: test_nrm2 (test_blas.TestFBLAS1Simple) ---------------------------------------------------------------------- Traceback (most recent call last): File "/sw/lib/python2.7/site-packages/scipy/linalg/tests/test_blas.py", line 83, in test_nrm2 assert_almost_equal(f([3,-4,5]),math.sqrt(50)) File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 468, in assert_almost_equal raise AssertionError(msg) AssertionError: Arrays are not almost equal to 7 decimals ACTUAL: 0.0 DESIRED: 7.0710678118654755 ====================================================================== FAIL: test_expon (test_morestats.TestAnderson) ---------------------------------------------------------------------- Traceback (most recent call last): File "/sw/lib/python2.7/site-packages/scipy/stats/tests/test_morestats.py", line 72, in test_expon assert_array_less(crit[:-1], A) File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 869, in assert_array_less header='Arrays are not less-ordered') File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 613, in assert_array_compare chk_same_position(x_id, y_id, hasval='inf') File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 588, in chk_same_position raise AssertionError(msg) AssertionError: Arrays are not less-ordered x and y inf location mismatch: x: array([ 0.911, 1.065, 1.325, 1.587]) y: array(inf) ---------------------------------------------------------------------- Ran 4722 tests in 63.364s FAILED (KNOWNFAIL=18, SKIP=31, failures=7) >>> FYI, I see the same seven failures if I use Apple's gcc-4.2 compilers instead of the clang compilers. Jack > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev From pav at iki.fi Tue Aug 30 09:32:09 2011 From: pav at iki.fi (Pauli Virtanen) Date: Tue, 30 Aug 2011 13:32:09 +0000 (UTC) Subject: [SciPy-Dev] suitesparse vs scipy References: <20110828232604.GA18746@bromo.med.uc.edu> <20110830125515.GA1309@bromo.med.uc.edu> Message-ID: Tue, 30 Aug 2011 08:55:15 -0400, Jack Howarth wrote: [clip] > On x86_64-apple-darwin11, building scypy 0.9.0 against python2.7 > using the clang compilers and gfortran from FSF gcc 4.6.1, I get the following > testsuite results... [clip] > ====================================================================== > FAIL: test_dot (test_blas.TestFBLAS1Simple) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > "/sw/lib/python2.7/site-packages/scipy/lib/blas/tests/test_blas.py", > line 67, in test_dot > assert_almost_equal(f([3,-4,5],[2,5,1]),-9) > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line > 468, in assert_almost_equal > raise AssertionError(msg) > AssertionError: > Arrays are not almost equal to 7 decimals > ACTUAL: 0.0 > DESIRED: -9 Does Numpy's test suite pass? Either the BLAS you linked Scipy against is broken, or there are some compatibility problems due to compilers or mixing compiler versions. You could check which libraries your `fblas.so` and `cblas.so` in the Scipy installation are linked against. I've never used OSX, though, so I do not know how to do this there. You can see the same information in the beginning of the build logs, though. Maybe someone who is more familiar than me with OSX knows more. -- Pauli Virtanen From bdeb at willmore.eu Tue Aug 30 10:02:48 2011 From: bdeb at willmore.eu (Ben Willmore) Date: Tue, 30 Aug 2011 15:02:48 +0100 Subject: [SciPy-Dev] suitesparse vs scipy In-Reply-To: <20110830125515.GA1309@bromo.med.uc.edu> References: <20110828232604.GA18746@bromo.med.uc.edu> <20110830125515.GA1309@bromo.med.uc.edu> Message-ID: Hi Jack, > Pauli, > On x86_64-apple-darwin11, building scypy 0.9.0 against python2.7 using > the clang compilers and gfortran from FSF gcc 4.6.1, I get the following testsuite > results... I'm a scipy novice, probably hanging out on the wrong list, but I know a little bit about most of the errors you're seeing. There are some differences in calling conventions between Apple's veclib and scipy's expectations which mean that various single-precision functions fail, see here: http://www.macresearch.org/lapackblas-fortran-106 This can be fixed by setting 'export FFLAGS=-ff2c' before running/compiling numpy and scipy (*). For me, this fixed all test_blas failures, using a recent git clone. As far as I can tell, it doesn't introduce other problems (i.e. most tests pass). I tried this with all the standard Lion C compilers, and a couple of different versions of gfortran, with similar results (though llvm has it's own problems (**), and you shouldn't use it). So this is not a question of choosing the right compiler. Ben (*) The results of my experiments are here: http://mail.scipy.org/pipermail/scipy-dev/2011-August/016396.html (**) For instance, ifft(fft(signal)) != signal when using scipy.fftpack From ralf.gommers at googlemail.com Tue Aug 30 12:48:20 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Tue, 30 Aug 2011 18:48:20 +0200 Subject: [SciPy-Dev] removing netcdf_variable from netcdf.__all__ Message-ID: Hi, Question for users of scipy.io.netcdf: did you ever use netcdf_variable directly, or only via netcdf_file.createVariable? The documentation says that the latter is the only intended use, which means that this class shouldn't be in __all__. https://github.com/scipy/scipy/pull/67 removes it (and changes its __init__ in a non-backwards compatible way) - the patch looks correct but I want to double check that no one is using this class. Thanks, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From howarth at bromo.med.uc.edu Tue Aug 30 12:59:12 2011 From: howarth at bromo.med.uc.edu (Jack Howarth) Date: Tue, 30 Aug 2011 12:59:12 -0400 Subject: [SciPy-Dev] suitesparse vs scipy In-Reply-To: References: <20110828232604.GA18746@bromo.med.uc.edu> <20110830125515.GA1309@bromo.med.uc.edu> Message-ID: <20110830165912.GA5769@bromo.med.uc.edu> On Tue, Aug 30, 2011 at 01:32:09PM +0000, Pauli Virtanen wrote: > Tue, 30 Aug 2011 08:55:15 -0400, Jack Howarth wrote: > [clip] > > On x86_64-apple-darwin11, building scypy 0.9.0 against python2.7 > > using the clang compilers and gfortran from FSF gcc 4.6.1, I get the following > > testsuite results... > [clip] > > ====================================================================== > > FAIL: test_dot (test_blas.TestFBLAS1Simple) > > ---------------------------------------------------------------------- > > Traceback (most recent call last): > > File > > "/sw/lib/python2.7/site-packages/scipy/lib/blas/tests/test_blas.py", > > line 67, in test_dot > > assert_almost_equal(f([3,-4,5],[2,5,1]),-9) > > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line > > 468, in assert_almost_equal > > raise AssertionError(msg) > > AssertionError: > > Arrays are not almost equal to 7 decimals > > ACTUAL: 0.0 > > DESIRED: -9 > > Does Numpy's test suite pass? On x86_64-apple-darwin10, numpy passes its testsuite... [frodo:~] howarth% python2.7 Python 2.7.2 (default, Aug 4 2011, 09:22:24) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> numpy.test() Running unit tests for numpy NumPy version 1.6.0 NumPy is installed in /sw/lib/python2.7/site-packages/numpy Python version 2.7.2 (default, Aug 4 2011, 09:22:24) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] nose version 1.1.2 ....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................../sw/lib/python2.7/site-packages/numpy/core/numeric.py:1920: RuntimeWarning: invalid value encountered in absolute return all(less_equal(absolute(x-y), atol + rtol * absolute(y))) ....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K.................................................................................................K......................K...............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................S..................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................... ---------------------------------------------------------------------- Ran 3529 tests in 34.328s OK (KNOWNFAIL=3, SKIP=1) >>> whereas on x86_64-apple-darwin11 (building with clang), numpy fails... [GCC 4.2.1 Compatible Apple Clang 2.1 (tags/Apple/clang-163.7.1)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> numpy.test() Running unit tests for numpy NumPy version 1.6.0 NumPy is installed in /sw/lib/python2.7/site-packages/numpy Python version 2.7.2 (default, Jul 21 2011, 22:39:54) [GCC 4.2.1 Compatible Apple Clang 2.1 (tags/Apple/clang-163.7.1)] nose version 1.1.2 ................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................F.........................................................................................................................................................................................................................................................................................................................................................................................................................K.................................................................................................K......................K...........FFFFFFF...F./sw/lib/python2.7/site-packages/numpy/core/tests/test_umath_complex.py:364: RuntimeWarning: invalid value encountered in sqrt z = np.sqrt(np.array(np.complex(-np.inf, np.nan))) ........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................S..................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................... ====================================================================== FAIL: Test basic arithmetic function errors ---------------------------------------------------------------------- Traceback (most recent call last): File "/sw/lib/python2.7/site-packages/numpy/testing/decorators.py", line 215, in knownfailer return f(*args, **kwargs) File "/sw/lib/python2.7/site-packages/numpy/core/tests/test_numeric.py", line 321, in test_floating_exceptions lambda a,b:a/b, ft_tiny, ft_max) File "/sw/lib/python2.7/site-packages/numpy/core/tests/test_numeric.py", line 271, in assert_raises_fpe "Type %s did not raise fpe error '%s'." % (ftype, fpeerr)) File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 34, in assert_ raise AssertionError(msg) AssertionError: Type did not raise fpe error ''. ====================================================================== FAIL: test_umath_complex.TestCsqrt.test_special_values(, 1, inf, inf, inf) ---------------------------------------------------------------------- Traceback (most recent call last): File "/sw/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/sw/lib/python2.7/site-packages/numpy/core/tests/test_umath_complex.py", line 578, in check_complex_value assert_equal(f(z1), z2) File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 256, in assert_equal return assert_array_equal(actual, desired, err_msg, verbose) File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 707, in assert_array_equal verbose=verbose, header='Arrays are not equal') File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 605, in assert_array_compare chk_same_position(x_id, y_id, hasval='nan') File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 588, in chk_same_position raise AssertionError(msg) AssertionError: Arrays are not equal x and y nan location mismatch: x: array([ nan+infj]) y: array((inf+infj)) ====================================================================== FAIL: test_umath_complex.TestCsqrt.test_special_values(, -1, inf, inf, inf) ---------------------------------------------------------------------- Traceback (most recent call last): File "/sw/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/sw/lib/python2.7/site-packages/numpy/core/tests/test_umath_complex.py", line 578, in check_complex_value assert_equal(f(z1), z2) File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 256, in assert_equal return assert_array_equal(actual, desired, err_msg, verbose) File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 707, in assert_array_equal verbose=verbose, header='Arrays are not equal') File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 605, in assert_array_compare chk_same_position(x_id, y_id, hasval='nan') File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 588, in chk_same_position raise AssertionError(msg) AssertionError: Arrays are not equal x and y nan location mismatch: x: array([ nan+infj]) y: array((inf+infj)) ====================================================================== FAIL: test_umath_complex.TestCsqrt.test_special_values(, 0.0, inf, inf, inf) ---------------------------------------------------------------------- Traceback (most recent call last): File "/sw/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/sw/lib/python2.7/site-packages/numpy/core/tests/test_umath_complex.py", line 578, in check_complex_value assert_equal(f(z1), z2) File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 256, in assert_equal return assert_array_equal(actual, desired, err_msg, verbose) File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 707, in assert_array_equal verbose=verbose, header='Arrays are not equal') File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 605, in assert_array_compare chk_same_position(x_id, y_id, hasval='nan') File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 588, in chk_same_position raise AssertionError(msg) AssertionError: Arrays are not equal x and y nan location mismatch: x: array([ nan+infj]) y: array((inf+infj)) ====================================================================== FAIL: test_umath_complex.TestCsqrt.test_special_values(, -0.0, inf, inf, inf) ---------------------------------------------------------------------- Traceback (most recent call last): File "/sw/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/sw/lib/python2.7/site-packages/numpy/core/tests/test_umath_complex.py", line 578, in check_complex_value assert_equal(f(z1), z2) File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 256, in assert_equal return assert_array_equal(actual, desired, err_msg, verbose) File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 707, in assert_array_equal verbose=verbose, header='Arrays are not equal') File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 605, in assert_array_compare chk_same_position(x_id, y_id, hasval='nan') File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 588, in chk_same_position raise AssertionError(msg) AssertionError: Arrays are not equal x and y nan location mismatch: x: array([ nan+infj]) y: array((inf+infj)) ====================================================================== FAIL: test_umath_complex.TestCsqrt.test_special_values(, inf, inf, inf, inf) ---------------------------------------------------------------------- Traceback (most recent call last): File "/sw/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/sw/lib/python2.7/site-packages/numpy/core/tests/test_umath_complex.py", line 578, in check_complex_value assert_equal(f(z1), z2) File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 256, in assert_equal return assert_array_equal(actual, desired, err_msg, verbose) File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 707, in assert_array_equal verbose=verbose, header='Arrays are not equal') File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 605, in assert_array_compare chk_same_position(x_id, y_id, hasval='nan') File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 588, in chk_same_position raise AssertionError(msg) AssertionError: Arrays are not equal x and y nan location mismatch: x: array([ nan+infj]) y: array((inf+infj)) ====================================================================== FAIL: test_umath_complex.TestCsqrt.test_special_values(, -inf, inf, inf, inf) ---------------------------------------------------------------------- Traceback (most recent call last): File "/sw/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/sw/lib/python2.7/site-packages/numpy/core/tests/test_umath_complex.py", line 578, in check_complex_value assert_equal(f(z1), z2) File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 256, in assert_equal return assert_array_equal(actual, desired, err_msg, verbose) File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 707, in assert_array_equal verbose=verbose, header='Arrays are not equal') File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 605, in assert_array_compare chk_same_position(x_id, y_id, hasval='nan') File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 588, in chk_same_position raise AssertionError(msg) AssertionError: Arrays are not equal x and y nan location mismatch: x: array([ nan+infj]) y: array((inf+infj)) ====================================================================== FAIL: test_umath_complex.TestCsqrt.test_special_values(, nan, inf, inf, inf) ---------------------------------------------------------------------- Traceback (most recent call last): File "/sw/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/sw/lib/python2.7/site-packages/numpy/core/tests/test_umath_complex.py", line 578, in check_complex_value assert_equal(f(z1), z2) File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 256, in assert_equal return assert_array_equal(actual, desired, err_msg, verbose) File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 707, in assert_array_equal verbose=verbose, header='Arrays are not equal') File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 605, in assert_array_compare chk_same_position(x_id, y_id, hasval='nan') File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 588, in chk_same_position raise AssertionError(msg) AssertionError: Arrays are not equal x and y nan location mismatch: x: array([ nan+infj]) y: array((inf+infj)) ====================================================================== FAIL: test_umath_complex.TestCsqrt.test_special_values(, -inf, 1, 0.0, inf) ---------------------------------------------------------------------- Traceback (most recent call last): File "/sw/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/sw/lib/python2.7/site-packages/numpy/core/tests/test_umath_complex.py", line 578, in check_complex_value assert_equal(f(z1), z2) File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 256, in assert_equal return assert_array_equal(actual, desired, err_msg, verbose) File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 707, in assert_array_equal verbose=verbose, header='Arrays are not equal') File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 605, in assert_array_compare chk_same_position(x_id, y_id, hasval='nan') File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 588, in chk_same_position raise AssertionError(msg) AssertionError: Arrays are not equal x and y nan location mismatch: x: array([ nan+infj]) y: array(infj) ---------------------------------------------------------------------- Ran 3529 tests in 38.236s FAILED (KNOWNFAIL=3, SKIP=1, failures=9) In both cases (darwin10 and darwin11), the blas used is from /System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate. http://developer.apple.com/performance/accelerateframework.html I doubt the system blas is at fault and more likely this is an issue with the clang compilers. I know python required an explicit addition of -fwrapv since its test for that option is confused by clang. Perhaps numpy/scipy requires -fwrapv as well? Jack > > Either the BLAS you linked Scipy against is broken, or there are some > compatibility problems due to compilers or mixing compiler versions. > > You could check which libraries your `fblas.so` and `cblas.so` in the > Scipy installation are linked against. I've never used OSX, though, > so I do not know how to do this there. You can see the same information > in the beginning of the build logs, though. > > Maybe someone who is more familiar than me with OSX knows more. > > -- > Pauli Virtanen > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev From nwagner at iam.uni-stuttgart.de Tue Aug 30 14:23:09 2011 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Tue, 30 Aug 2011 20:23:09 +0200 Subject: [SciPy-Dev] scipy.test() errors - SciPy version 0.10.0.dev7179 Message-ID: ====================================================================== ERROR: Failure: IndentationError (unexpected indent (dia.py, line 105)) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/loader.py", line 382, in loadTestsFromName addr.filename, addr.module) File "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/importer.py", line 39, in importFromPath return self.importFromDir(dir_path, fqname) File "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/importer.py", line 86, in importFromDir mod = load_module(part_fqname, fh, filename, desc) File "/home/nwagner/local/lib64/python2.6/site-packages/scipy/cluster/__init__.py", line 26, in import vq, hierarchy File "/home/nwagner/local/lib64/python2.6/site-packages/scipy/cluster/hierarchy.py", line 179, in import scipy.spatial.distance as distance File "/home/nwagner/local/lib64/python2.6/site-packages/scipy/spatial/__init__.py", line 25, in from kdtree import * File "/home/nwagner/local/lib64/python2.6/site-packages/scipy/spatial/kdtree.py", line 6, in import scipy.sparse File "/home/nwagner/local/lib64/python2.6/site-packages/scipy/sparse/__init__.py", line 175, in from csr import * File "/home/nwagner/local/lib64/python2.6/site-packages/scipy/sparse/csr.py", line 12, in from sparsetools import csr_tocsc, csr_tobsr, csr_count_blocks, \ File "/home/nwagner/local/lib64/python2.6/site-packages/scipy/sparse/sparsetools/__init__.py", line 7, in from dia import * File "/home/nwagner/local/lib64/python2.6/site-packages/scipy/sparse/sparsetools/dia.py", line 105 return _dia.dia_matvec(*args) ^ IndentationError: unexpected indent ====================================================================== FAIL: test_lu (test_decomp.TestLUSolve) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/nwagner/local/lib64/python2.6/site-packages/scipy/linalg/tests/test_decomp.py", line 728, in test_lu assert_array_equal(x1,x2) File "/home/nwagner/local/lib64/python2.6/site-packages/numpy/testing/utils.py", line 753, in assert_array_equal verbose=verbose, header='Arrays are not equal') File "/home/nwagner/local/lib64/python2.6/site-packages/numpy/testing/utils.py", line 677, in assert_array_compare raise AssertionError(msg) AssertionError: Arrays are not equal (mismatch 90.0%) x: array([-0.36278861, 0.75920432, 0.19844271, 0.65479681, 0.40510514, 0.79604687, -1.05344321, -0.25597019, -0.34997286, 0.07080989]) y: array([-0.36278861, 0.75920432, 0.19844271, 0.65479681, 0.40510514, 0.79604687, -1.05344321, -0.25597019, -0.34997286, 0.07080989]) From ralf.gommers at googlemail.com Tue Aug 30 15:00:09 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Tue, 30 Aug 2011 21:00:09 +0200 Subject: [SciPy-Dev] scipy.test() errors - SciPy version 0.10.0.dev7179 In-Reply-To: References: Message-ID: On Tue, Aug 30, 2011 at 8:23 PM, Nils Wagner wrote: Fixed the indentation error, sorry about that. I don't see the second error, is it reproducible? If so, can you open a ticket with details on your OS, etc.? Ralf > ====================================================================== > ERROR: Failure: IndentationError (unexpected indent > (dia.py, line 105)) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > > "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/loader.py", > line 382, in loadTestsFromName > addr.filename, addr.module) > File > > "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/importer.py", > line 39, in importFromPath > return self.importFromDir(dir_path, fqname) > File > > "/home/nwagner/local/lib64/python2.6/site-packages/nose-0.11.2.dev-py2.6.egg/nose/importer.py", > line 86, in importFromDir > mod = load_module(part_fqname, fh, filename, desc) > File > > "/home/nwagner/local/lib64/python2.6/site-packages/scipy/cluster/__init__.py", > line 26, in > import vq, hierarchy > File > > "/home/nwagner/local/lib64/python2.6/site-packages/scipy/cluster/hierarchy.py", > line 179, in > import scipy.spatial.distance as distance > File > > "/home/nwagner/local/lib64/python2.6/site-packages/scipy/spatial/__init__.py", > line 25, in > from kdtree import * > File > > "/home/nwagner/local/lib64/python2.6/site-packages/scipy/spatial/kdtree.py", > line 6, in > import scipy.sparse > File > > "/home/nwagner/local/lib64/python2.6/site-packages/scipy/sparse/__init__.py", > line 175, in > from csr import * > File > "/home/nwagner/local/lib64/python2.6/site-packages/scipy/sparse/csr.py", > line 12, in > from sparsetools import csr_tocsc, csr_tobsr, > csr_count_blocks, \ > File > > "/home/nwagner/local/lib64/python2.6/site-packages/scipy/sparse/sparsetools/__init__.py", > line 7, in > from dia import * > File > > "/home/nwagner/local/lib64/python2.6/site-packages/scipy/sparse/sparsetools/dia.py", > line 105 > return _dia.dia_matvec(*args) > ^ > IndentationError: unexpected indent > > ====================================================================== > FAIL: test_lu (test_decomp.TestLUSolve) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File > > "/home/nwagner/local/lib64/python2.6/site-packages/scipy/linalg/tests/test_decomp.py", > line 728, in test_lu > assert_array_equal(x1,x2) > File > "/home/nwagner/local/lib64/python2.6/site-packages/numpy/testing/utils.py", > line 753, in assert_array_equal > verbose=verbose, header='Arrays are not equal') > File > "/home/nwagner/local/lib64/python2.6/site-packages/numpy/testing/utils.py", > line 677, in assert_array_compare > raise AssertionError(msg) > AssertionError: > Arrays are not equal > > (mismatch 90.0%) > x: array([-0.36278861, 0.75920432, 0.19844271, > 0.65479681, 0.40510514, > 0.79604687, -1.05344321, -0.25597019, > -0.34997286, 0.07080989]) > y: array([-0.36278861, 0.75920432, 0.19844271, > 0.65479681, 0.40510514, > 0.79604687, -1.05344321, -0.25597019, > -0.34997286, 0.07080989]) > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Tue Aug 30 15:04:18 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Tue, 30 Aug 2011 21:04:18 +0200 Subject: [SciPy-Dev] suitesparse vs scipy In-Reply-To: <20110830165912.GA5769@bromo.med.uc.edu> References: <20110828232604.GA18746@bromo.med.uc.edu> <20110830125515.GA1309@bromo.med.uc.edu> <20110830165912.GA5769@bromo.med.uc.edu> Message-ID: On Tue, Aug 30, 2011 at 6:59 PM, Jack Howarth wrote: > On Tue, Aug 30, 2011 at 01:32:09PM +0000, Pauli Virtanen wrote: > > Tue, 30 Aug 2011 08:55:15 -0400, Jack Howarth wrote: > > [clip] > > > On x86_64-apple-darwin11, building scypy 0.9.0 against python2.7 > > > using the clang compilers and gfortran from FSF gcc 4.6.1, I get the > following > > > testsuite results... > > [clip] > > > ====================================================================== > > > FAIL: test_dot (test_blas.TestFBLAS1Simple) > > > ---------------------------------------------------------------------- > > > Traceback (most recent call last): > > > File > > > "/sw/lib/python2.7/site-packages/scipy/lib/blas/tests/test_blas.py", > > > line 67, in test_dot > > > assert_almost_equal(f([3,-4,5],[2,5,1]),-9) > > > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line > > > 468, in assert_almost_equal > > > raise AssertionError(msg) > > > AssertionError: > > > Arrays are not almost equal to 7 decimals > > > ACTUAL: 0.0 > > > DESIRED: -9 > > > > Does Numpy's test suite pass? > > On x86_64-apple-darwin10, numpy passes its testsuite... > > [frodo:~] howarth% python2.7 > Python 2.7.2 (default, Aug 4 2011, 09:22:24) > [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin > Type "help", "copyright", "credits" or "license" for more information. > >>> import numpy > >>> numpy.test() > Running unit tests for numpy > NumPy version 1.6.0 > NumPy is installed in /sw/lib/python2.7/site-packages/numpy > Python version 2.7.2 (default, Aug 4 2011, 09:22:24) [GCC 4.2.1 (Apple > Inc. build 5666) (dot 3)] > nose version 1.1.2 > ....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................../sw/lib/python2.7/site-packages/numpy/core/numeric.py:1920: > RuntimeWarning: invalid value encountered in absolute > return all(less_equal(absolute(x-y), atol + rtol * absolute(y))) > > ....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K.................................................................................................K......................K................................................................................................................................................................................................................................................................................................................................................................................................ > > ............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................. > > ..................................................................................................................................................................S.......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................... > > ........................................................................................... > ---------------------------------------------------------------------- > Ran 3529 tests in 34.328s > > OK (KNOWNFAIL=3, SKIP=1) > > >>> > > whereas on x86_64-apple-darwin11 (building with clang), numpy fails... > None of those failures is a problem, nor is it relevant to your problem with scipy. If anyone feels like writing a patch to disable those test_special_values on all platforms except linux, that would be appreciated. Ralf > [GCC 4.2.1 Compatible Apple Clang 2.1 (tags/Apple/clang-163.7.1)] on darwin > Type "help", "copyright", "credits" or "license" for more information. > >>> import numpy > >>> numpy.test() > Running unit tests for numpy > NumPy version 1.6.0 > NumPy is installed in /sw/lib/python2.7/site-packages/numpy > Python version 2.7.2 (default, Jul 21 2011, 22:39:54) [GCC 4.2.1 Compatible > Apple Clang 2.1 (tags/Apple/clang-163.7.1)] > nose version 1.1.2 > > ................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................F.........................................................................................................................................................................................................................................................................................................................................................................................................................K................................... > ..............................................................K......................K...........FFFFFFF...F./sw/lib/python2.7/site-packages/numpy/core/tests/test_umath_complex.py:364: > RuntimeWarning: invalid value encountered in sqrt > z = np.sqrt(np.array(np.complex(-np.inf, np.nan))) > > .............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................. > > ..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................S.................................................................................................................................................................................................................................................................................................................................................................................................................................................................................. > > ................................................................................................................................................................................................................................................................................................................................................................................................................................................................... > ====================================================================== > FAIL: Test basic arithmetic function errors > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/sw/lib/python2.7/site-packages/numpy/testing/decorators.py", line > 215, in knownfailer > return f(*args, **kwargs) > File "/sw/lib/python2.7/site-packages/numpy/core/tests/test_numeric.py", > line 321, in test_floating_exceptions > lambda a,b:a/b, ft_tiny, ft_max) > File "/sw/lib/python2.7/site-packages/numpy/core/tests/test_numeric.py", > line 271, in assert_raises_fpe > "Type %s did not raise fpe error '%s'." % (ftype, fpeerr)) > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 34, in > assert_ > raise AssertionError(msg) > AssertionError: Type did not raise fpe error ''. > > ====================================================================== > FAIL: test_umath_complex.TestCsqrt.test_special_values(, 1, > inf, inf, inf) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/sw/lib/python2.7/site-packages/nose/case.py", line 197, in runTest > self.test(*self.arg) > File > "/sw/lib/python2.7/site-packages/numpy/core/tests/test_umath_complex.py", > line 578, in check_complex_value > assert_equal(f(z1), z2) > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 256, > in assert_equal > return assert_array_equal(actual, desired, err_msg, verbose) > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 707, > in assert_array_equal > verbose=verbose, header='Arrays are not equal') > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 605, > in assert_array_compare > chk_same_position(x_id, y_id, hasval='nan') > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 588, > in chk_same_position > raise AssertionError(msg) > AssertionError: > Arrays are not equal > > x and y nan location mismatch: > x: array([ nan+infj]) > y: array((inf+infj)) > > ====================================================================== > FAIL: test_umath_complex.TestCsqrt.test_special_values(, -1, > inf, inf, inf) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/sw/lib/python2.7/site-packages/nose/case.py", line 197, in runTest > self.test(*self.arg) > File > "/sw/lib/python2.7/site-packages/numpy/core/tests/test_umath_complex.py", > line 578, in check_complex_value > assert_equal(f(z1), z2) > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 256, > in assert_equal > return assert_array_equal(actual, desired, err_msg, verbose) > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 707, > in assert_array_equal > verbose=verbose, header='Arrays are not equal') > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 605, > in assert_array_compare > chk_same_position(x_id, y_id, hasval='nan') > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 588, > in chk_same_position > raise AssertionError(msg) > AssertionError: > Arrays are not equal > > x and y nan location mismatch: > x: array([ nan+infj]) > y: array((inf+infj)) > > ====================================================================== > FAIL: test_umath_complex.TestCsqrt.test_special_values(, 0.0, > inf, inf, inf) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/sw/lib/python2.7/site-packages/nose/case.py", line 197, in runTest > self.test(*self.arg) > File > "/sw/lib/python2.7/site-packages/numpy/core/tests/test_umath_complex.py", > line 578, in check_complex_value > assert_equal(f(z1), z2) > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 256, > in assert_equal > return assert_array_equal(actual, desired, err_msg, verbose) > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 707, > in assert_array_equal > verbose=verbose, header='Arrays are not equal') > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 605, > in assert_array_compare > chk_same_position(x_id, y_id, hasval='nan') > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 588, > in chk_same_position > raise AssertionError(msg) > AssertionError: > Arrays are not equal > > x and y nan location mismatch: > x: array([ nan+infj]) > y: array((inf+infj)) > > ====================================================================== > FAIL: test_umath_complex.TestCsqrt.test_special_values(, > -0.0, inf, inf, inf) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/sw/lib/python2.7/site-packages/nose/case.py", line 197, in runTest > self.test(*self.arg) > File > "/sw/lib/python2.7/site-packages/numpy/core/tests/test_umath_complex.py", > line 578, in check_complex_value > assert_equal(f(z1), z2) > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 256, > in assert_equal > return assert_array_equal(actual, desired, err_msg, verbose) > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 707, > in assert_array_equal > verbose=verbose, header='Arrays are not equal') > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 605, > in assert_array_compare > chk_same_position(x_id, y_id, hasval='nan') > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 588, > in chk_same_position > raise AssertionError(msg) > AssertionError: > Arrays are not equal > > x and y nan location mismatch: > x: array([ nan+infj]) > y: array((inf+infj)) > > ====================================================================== > FAIL: test_umath_complex.TestCsqrt.test_special_values(, inf, > inf, inf, inf) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/sw/lib/python2.7/site-packages/nose/case.py", line 197, in runTest > self.test(*self.arg) > File > "/sw/lib/python2.7/site-packages/numpy/core/tests/test_umath_complex.py", > line 578, in check_complex_value > assert_equal(f(z1), z2) > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 256, > in assert_equal > return assert_array_equal(actual, desired, err_msg, verbose) > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 707, > in assert_array_equal > verbose=verbose, header='Arrays are not equal') > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 605, > in assert_array_compare > chk_same_position(x_id, y_id, hasval='nan') > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 588, > in chk_same_position > raise AssertionError(msg) > AssertionError: > Arrays are not equal > > x and y nan location mismatch: > x: array([ nan+infj]) > y: array((inf+infj)) > > ====================================================================== > FAIL: test_umath_complex.TestCsqrt.test_special_values(, > -inf, inf, inf, inf) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/sw/lib/python2.7/site-packages/nose/case.py", line 197, in runTest > self.test(*self.arg) > File > "/sw/lib/python2.7/site-packages/numpy/core/tests/test_umath_complex.py", > line 578, in check_complex_value > assert_equal(f(z1), z2) > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 256, > in assert_equal > return assert_array_equal(actual, desired, err_msg, verbose) > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 707, > in assert_array_equal > verbose=verbose, header='Arrays are not equal') > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 605, > in assert_array_compare > chk_same_position(x_id, y_id, hasval='nan') > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 588, > in chk_same_position > raise AssertionError(msg) > AssertionError: > Arrays are not equal > > x and y nan location mismatch: > x: array([ nan+infj]) > y: array((inf+infj)) > > ====================================================================== > FAIL: test_umath_complex.TestCsqrt.test_special_values(, nan, > inf, inf, inf) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/sw/lib/python2.7/site-packages/nose/case.py", line 197, in runTest > self.test(*self.arg) > File > "/sw/lib/python2.7/site-packages/numpy/core/tests/test_umath_complex.py", > line 578, in check_complex_value > assert_equal(f(z1), z2) > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 256, > in assert_equal > return assert_array_equal(actual, desired, err_msg, verbose) > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 707, > in assert_array_equal > verbose=verbose, header='Arrays are not equal') > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 605, > in assert_array_compare > chk_same_position(x_id, y_id, hasval='nan') > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 588, > in chk_same_position > raise AssertionError(msg) > AssertionError: > Arrays are not equal > > x and y nan location mismatch: > x: array([ nan+infj]) > y: array((inf+infj)) > > ====================================================================== > FAIL: test_umath_complex.TestCsqrt.test_special_values(, > -inf, 1, 0.0, inf) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/sw/lib/python2.7/site-packages/nose/case.py", line 197, in runTest > self.test(*self.arg) > File > "/sw/lib/python2.7/site-packages/numpy/core/tests/test_umath_complex.py", > line 578, in check_complex_value > assert_equal(f(z1), z2) > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 256, > in assert_equal > return assert_array_equal(actual, desired, err_msg, verbose) > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 707, > in assert_array_equal > verbose=verbose, header='Arrays are not equal') > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 605, > in assert_array_compare > chk_same_position(x_id, y_id, hasval='nan') > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 588, > in chk_same_position > raise AssertionError(msg) > AssertionError: > Arrays are not equal > > x and y nan location mismatch: > x: array([ nan+infj]) > y: array(infj) > > ---------------------------------------------------------------------- > Ran 3529 tests in 38.236s > > FAILED (KNOWNFAIL=3, SKIP=1, failures=9) > > > In both cases (darwin10 and darwin11), the blas used is from > /System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate. > > http://developer.apple.com/performance/accelerateframework.html > > I doubt the system blas is at fault and more likely this is an issue with > the clang compilers. I know python required an explicit > addition of -fwrapv since its test for that option is confused by clang. > Perhaps numpy/scipy requires -fwrapv as well? > Jack > > > > > Either the BLAS you linked Scipy against is broken, or there are some > > compatibility problems due to compilers or mixing compiler versions. > > > > You could check which libraries your `fblas.so` and `cblas.so` in the > > Scipy installation are linked against. I've never used OSX, though, > > so I do not know how to do this there. You can see the same information > > in the beginning of the build logs, though. > > > > Maybe someone who is more familiar than me with OSX knows more. > > > > -- > > Pauli Virtanen > > > > _______________________________________________ > > SciPy-Dev mailing list > > SciPy-Dev at scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-dev > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nwagner at iam.uni-stuttgart.de Tue Aug 30 15:44:04 2011 From: nwagner at iam.uni-stuttgart.de (Nils Wagner) Date: Tue, 30 Aug 2011 21:44:04 +0200 Subject: [SciPy-Dev] scipy.test() errors - SciPy version 0.10.0.dev7179 In-Reply-To: References: Message-ID: On Tue, 30 Aug 2011 21:00:09 +0200 Ralf Gommers wrote: > On Tue, Aug 30, 2011 at 8:23 PM, Nils Wagner > wrote: > >Fixed the indentation error, sorry about that. I don't >see the second error, > is it reproducible? If so, can you open a ticket with >details on your OS, > etc.? > > Ralf > Done. See http://projects.scipy.org/scipy/ticket/1504 Nils From josh.k.lawrence at gmail.com Tue Aug 30 16:02:56 2011 From: josh.k.lawrence at gmail.com (Josh Lawrence) Date: Tue, 30 Aug 2011 16:02:56 -0400 Subject: [SciPy-Dev] scipy.test() errors - SciPy version 0.10.0.dev7179 In-Reply-To: References: Message-ID: <4E5D41F0.2070809@gmail.com> This isn't exactly on topic, but since scipy has been moved to git, how is it you reference a specific version number? On Tue Aug 30 15:44:04 2011, Nils Wagner wrote: > On Tue, 30 Aug 2011 21:00:09 +0200 > Ralf Gommers wrote: >> On Tue, Aug 30, 2011 at 8:23 PM, Nils Wagner >> wrote: >> >> Fixed the indentation error, sorry about that. I don't >> see the second error, >> is it reproducible? If so, can you open a ticket with >> details on your OS, >> etc.? >> >> Ralf >> > > Done. See http://projects.scipy.org/scipy/ticket/1504 > > Nils > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev From ralf.gommers at googlemail.com Tue Aug 30 16:49:29 2011 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Tue, 30 Aug 2011 22:49:29 +0200 Subject: [SciPy-Dev] scipy.test() errors - SciPy version 0.10.0.dev7179 In-Reply-To: <4E5D41F0.2070809@gmail.com> References: <4E5D41F0.2070809@gmail.com> Message-ID: On Tue, Aug 30, 2011 at 10:02 PM, Josh Lawrence wrote: > This isn't exactly on topic, but since scipy has been moved to git, how > is it you reference a specific version number? > > When you build from source (not a released version) the first seven characters of the git commit hash are appended to the full version: In [5]: np.__version__ Out[5]: '2.0.0.dev-d91521e' Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From josh.k.lawrence at gmail.com Tue Aug 30 19:48:05 2011 From: josh.k.lawrence at gmail.com (Josh Lawrence) Date: Tue, 30 Aug 2011 19:48:05 -0400 Subject: [SciPy-Dev] scipy.test() errors - SciPy version 0.10.0.dev7179 In-Reply-To: References: <4E5D41F0.2070809@gmail.com> Message-ID: <4E5D76B5.3020108@gmail.com> This is what I get: In [402]: scipy.__version__ Out[402]: '0.10.0.dev' On Tue Aug 30 16:49:29 2011, Ralf Gommers wrote: > On Tue, Aug 30, 2011 at 10:02 PM, Josh Lawrence > > wrote: > > This isn't exactly on topic, but since scipy has been moved to git, how > is it you reference a specific version number? > > When you build from source (not a released version) the first seven > characters of the git commit hash are appended to the full version: > > In [5]: np.__version__ > Out[5]: '2.0.0.dev-d91521e' > > Ralf > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev From hofsaess at ifb.uni-stuttgart.de Wed Aug 31 04:20:14 2011 From: hofsaess at ifb.uni-stuttgart.de (=?ISO-8859-15?Q?Martin_Hofs=E4=DF?=) Date: Wed, 31 Aug 2011 10:20:14 +0200 Subject: [SciPy-Dev] problems building scipy with scons & mkl Message-ID: <4E5DEEBE.2040807@ifb.uni-stuttgart.de> Hi all, I updated my git repository today and want to rebuild scipy with scons and mkl support. But I get errors (please see attachment). Can anyone help me? Regards Martin -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: log.txt URL: From cournape at gmail.com Wed Aug 31 05:22:00 2011 From: cournape at gmail.com (David Cournapeau) Date: Wed, 31 Aug 2011 11:22:00 +0200 Subject: [SciPy-Dev] problems building scipy with scons & mkl In-Reply-To: <4E5DEEBE.2040807@ifb.uni-stuttgart.de> References: <4E5DEEBE.2040807@ifb.uni-stuttgart.de> Message-ID: Hi Martin, On Wed, Aug 31, 2011 at 10:20 AM, Martin Hofs?? wrote: > Hi all, > > I updated my git repository today and want to rebuild scipy with scons and > mkl support. Did you clean up the build tree after updating scipy ? It seems that the generated fblasmodule.c is invalid (my German is rusty, but it looks mostly like syntax errors). If you still get some errors, please attach the generated fblasemodule.c cheers, David From scott.sinclair.za at gmail.com Wed Aug 31 05:49:49 2011 From: scott.sinclair.za at gmail.com (Scott Sinclair) Date: Wed, 31 Aug 2011 11:49:49 +0200 Subject: [SciPy-Dev] scipy.test() errors - SciPy version 0.10.0.dev7179 In-Reply-To: <4E5D76B5.3020108@gmail.com> References: <4E5D41F0.2070809@gmail.com> <4E5D76B5.3020108@gmail.com> Message-ID: On 31 August 2011 01:48, Josh Lawrence wrote: > This is what I get: > > In [402]: scipy.__version__ > Out[402]: '0.10.0.dev' There's a pull request to fix this at https://github.com/scipy/scipy/pull/68 Cheers, Scott From hofsaess at ifb.uni-stuttgart.de Wed Aug 31 05:31:25 2011 From: hofsaess at ifb.uni-stuttgart.de (=?UTF-8?B?TWFydGluIEhvZnPDpMOf?=) Date: Wed, 31 Aug 2011 11:31:25 +0200 Subject: [SciPy-Dev] problems building scipy with scons & mkl In-Reply-To: References: <4E5DEEBE.2040807@ifb.uni-stuttgart.de> Message-ID: <4E5DFF6D.7010004@ifb.uni-stuttgart.de> Hi David, I cleand the build tree, but it has no effect. The errors are the same. I attached the generated flasemodule.c. I hope this will help. Martin Am 31.08.2011 11:22, schrieb David Cournapeau: > Hi Martin, > > On Wed, Aug 31, 2011 at 10:20 AM, Martin Hofs?? > wrote: >> Hi all, >> >> I updated my git repository today and want to rebuild scipy with scons and >> mkl support. > Did you clean up the build tree after updating scipy ? It seems that > the generated fblasmodule.c is invalid (my German is rusty, but it > looks mostly like syntax errors). If you still get some errors, please > attach the generated fblasemodule.c > > cheers, > > David > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: fblasmodule.c URL: From howarth at bromo.med.uc.edu Wed Aug 31 09:33:04 2011 From: howarth at bromo.med.uc.edu (Jack Howarth) Date: Wed, 31 Aug 2011 09:33:04 -0400 Subject: [SciPy-Dev] suitesparse vs scipy In-Reply-To: References: <20110828232604.GA18746@bromo.med.uc.edu> <20110830125515.GA1309@bromo.med.uc.edu> Message-ID: <20110831133304.GA29045@bromo.med.uc.edu> On Tue, Aug 30, 2011 at 03:02:48PM +0100, Ben Willmore wrote: > Hi Jack, > > > Pauli, > > On x86_64-apple-darwin11, building scypy 0.9.0 against python2.7 using > > the clang compilers and gfortran from FSF gcc 4.6.1, I get the following testsuite > > results... > > I'm a scipy novice, probably hanging out on the wrong list, but I know a little bit about most of the errors you're seeing. > > There are some differences in calling conventions between Apple's veclib and scipy's expectations which mean that various single-precision functions fail, see here: > > http://www.macresearch.org/lapackblas-fortran-106 > > This can be fixed by setting 'export FFLAGS=-ff2c' before running/compiling numpy and scipy (*). For me, this fixed all test_blas failures, using a recent git clone. As far as I can tell, it doesn't introduce other problems (i.e. most tests pass). > > I tried this with all the standard Lion C compilers, and a couple of different versions of gfortran, with similar results (though llvm has it's own problems (**), and you shouldn't use it). So this is not a question of choosing the right compiler. > > Ben This doesn't make any sense for numpy. I tried you approach of using... export FFLAGS=-ff2c /sw/bin/python2.7 setup.py build --fcompiler=gfortran and while I see.. customize Gnu95FCompiler Found executable /sw/bin/gfortran customize Gnu95FCompiler customize Gnu95FCompiler using config at no point during the numpy build is anything compiled with gfortran. The resulting numpy still fails the test_special_values tests. Note that these are claimed to be non-critical and should be disabled on darwin... http://mail.scipy.org/pipermail/scipy-dev/2011-August/016472.html Jack > > > (*) The results of my experiments are here: > > http://mail.scipy.org/pipermail/scipy-dev/2011-August/016396.html > > (**) For instance, ifft(fft(signal)) != signal when using scipy.fftpack > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev From bdeb at willmore.eu Wed Aug 31 09:56:07 2011 From: bdeb at willmore.eu (Ben Willmore) Date: Wed, 31 Aug 2011 14:56:07 +0100 Subject: [SciPy-Dev] suitesparse vs scipy In-Reply-To: <20110831133304.GA29045@bromo.med.uc.edu> References: <20110828232604.GA18746@bromo.med.uc.edu> <20110830125515.GA1309@bromo.med.uc.edu> <20110831133304.GA29045@bromo.med.uc.edu> Message-ID: On 31 Aug 2011, at 14:33, Jack Howarth wrote: > This doesn't make any sense for numpy. I tried you approach of using... > > export FFLAGS=-ff2c > /sw/bin/python2.7 setup.py build --fcompiler=gfortran '-ff2c' fixes the test_blas errors you saw in scipy. It doesn't do anything about the test_special_values errors you saw with numpy. I'm sure those should be disabled as recommended by Ralf. Ben From cgohlke at uci.edu Wed Aug 31 11:17:49 2011 From: cgohlke at uci.edu (Christoph Gohlke) Date: Wed, 31 Aug 2011 08:17:49 -0700 Subject: [SciPy-Dev] problems building scipy with scons & mkl In-Reply-To: <4E5DEEBE.2040807@ifb.uni-stuttgart.de> References: <4E5DEEBE.2040807@ifb.uni-stuttgart.de> Message-ID: <4E5E509D.8030901@uci.edu> On 8/31/2011 1:20 AM, Martin Hofs?? wrote: > Hi all, > > I updated my git repository today and want to rebuild scipy with scons > and mkl support. > > But I get errors (please see attachment). > > Can anyone help me? > > Regards > > Martin > Hi, Try change line 350 in scipy/linalg/fblas_l1.pyf.src to fortranname F_FUNC(iamax,IAMAX) Christoph From howarth at bromo.med.uc.edu Wed Aug 31 13:02:05 2011 From: howarth at bromo.med.uc.edu (Jack Howarth) Date: Wed, 31 Aug 2011 13:02:05 -0400 Subject: [SciPy-Dev] suitesparse vs scipy In-Reply-To: References: <20110828232604.GA18746@bromo.med.uc.edu> <20110830125515.GA1309@bromo.med.uc.edu> <20110830165912.GA5769@bromo.med.uc.edu> Message-ID: <20110831170205.GA30235@bromo.med.uc.edu> On Tue, Aug 30, 2011 at 09:04:18PM +0200, Ralf Gommers wrote: > On Tue, Aug 30, 2011 at 6:59 PM, Jack Howarth wrote: > > > On Tue, Aug 30, 2011 at 01:32:09PM +0000, Pauli Virtanen wrote: > > > Tue, 30 Aug 2011 08:55:15 -0400, Jack Howarth wrote: > > > [clip] > > > > On x86_64-apple-darwin11, building scypy 0.9.0 against python2.7 > > > > using the clang compilers and gfortran from FSF gcc 4.6.1, I get the > > following > > > > testsuite results... > > > [clip] > > > > ====================================================================== > > > > FAIL: test_dot (test_blas.TestFBLAS1Simple) > > > > ---------------------------------------------------------------------- > > > > Traceback (most recent call last): > > > > File > > > > "/sw/lib/python2.7/site-packages/scipy/lib/blas/tests/test_blas.py", > > > > line 67, in test_dot > > > > assert_almost_equal(f([3,-4,5],[2,5,1]),-9) > > > > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line > > > > 468, in assert_almost_equal > > > > raise AssertionError(msg) > > > > AssertionError: > > > > Arrays are not almost equal to 7 decimals > > > > ACTUAL: 0.0 > > > > DESIRED: -9 > > > > > > Does Numpy's test suite pass? > > > > On x86_64-apple-darwin10, numpy passes its testsuite... > > > > [frodo:~] howarth% python2.7 > > Python 2.7.2 (default, Aug 4 2011, 09:22:24) > > [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin > > Type "help", "copyright", "credits" or "license" for more information. > > >>> import numpy > > >>> numpy.test() > > Running unit tests for numpy > > NumPy version 1.6.0 > > NumPy is installed in /sw/lib/python2.7/site-packages/numpy > > Python version 2.7.2 (default, Aug 4 2011, 09:22:24) [GCC 4.2.1 (Apple > > Inc. build 5666) (dot 3)] > > nose version 1.1.2 > > ....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................../sw/lib/python2.7/site-packages/numpy/core/numeric.py:1920: > > RuntimeWarning: invalid value encountered in absolute > > return all(less_equal(absolute(x-y), atol + rtol * absolute(y))) > > > > ....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K.................................................................................................K......................K................................................................................................................................................................................................................................................................................................................................................................................................ > > > > ............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................. > > > > ..................................................................................................................................................................S.......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................... > > > > ........................................................................................... > > ---------------------------------------------------------------------- > > Ran 3529 tests in 34.328s > > > > OK (KNOWNFAIL=3, SKIP=1) > > > > >>> > > > > whereas on x86_64-apple-darwin11 (building with clang), numpy fails... > > > > None of those failures is a problem, nor is it relevant to your problem with > scipy. > > If anyone feels like writing a patch to disable those test_special_values on > all platforms except linux, that would be appreciated. Ralf, It seems very odd that these test_special_values tests pass in numpy on darwin10 but fail on darwin11. Doesn't that imply a possible regression in Lion? If so it might be wise to open a radar with Apple on the issue. Remember that Lion is the first darwin release built with llvm-based compilers and this change may well cause subtle regressions. For example, ppl exposed the fact that llvm-gcc silently ignores -frounding-math (while clang at least admits to doing so). Jack > > Ralf > > > > > [GCC 4.2.1 Compatible Apple Clang 2.1 (tags/Apple/clang-163.7.1)] on darwin > > Type "help", "copyright", "credits" or "license" for more information. > > >>> import numpy > > >>> numpy.test() > > Running unit tests for numpy > > NumPy version 1.6.0 > > NumPy is installed in /sw/lib/python2.7/site-packages/numpy > > Python version 2.7.2 (default, Jul 21 2011, 22:39:54) [GCC 4.2.1 Compatible > > Apple Clang 2.1 (tags/Apple/clang-163.7.1)] > > nose version 1.1.2 > > > > ................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................F.........................................................................................................................................................................................................................................................................................................................................................................................................................K................................... > > ..............................................................K......................K...........FFFFFFF...F./sw/lib/python2.7/site-packages/numpy/core/tests/test_umath_complex.py:364: > > RuntimeWarning: invalid value encountered in sqrt > > z = np.sqrt(np.array(np.complex(-np.inf, np.nan))) > > > > .............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................. > > > > ..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................S.................................................................................................................................................................................................................................................................................................................................................................................................................................................................................. > > > > ................................................................................................................................................................................................................................................................................................................................................................................................................................................................... > > ====================================================================== > > FAIL: Test basic arithmetic function errors > > ---------------------------------------------------------------------- > > Traceback (most recent call last): > > File "/sw/lib/python2.7/site-packages/numpy/testing/decorators.py", line > > 215, in knownfailer > > return f(*args, **kwargs) > > File "/sw/lib/python2.7/site-packages/numpy/core/tests/test_numeric.py", > > line 321, in test_floating_exceptions > > lambda a,b:a/b, ft_tiny, ft_max) > > File "/sw/lib/python2.7/site-packages/numpy/core/tests/test_numeric.py", > > line 271, in assert_raises_fpe > > "Type %s did not raise fpe error '%s'." % (ftype, fpeerr)) > > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 34, in > > assert_ > > raise AssertionError(msg) > > AssertionError: Type did not raise fpe error ''. > > > > ====================================================================== > > FAIL: test_umath_complex.TestCsqrt.test_special_values(, 1, > > inf, inf, inf) > > ---------------------------------------------------------------------- > > Traceback (most recent call last): > > File "/sw/lib/python2.7/site-packages/nose/case.py", line 197, in runTest > > self.test(*self.arg) > > File > > "/sw/lib/python2.7/site-packages/numpy/core/tests/test_umath_complex.py", > > line 578, in check_complex_value > > assert_equal(f(z1), z2) > > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 256, > > in assert_equal > > return assert_array_equal(actual, desired, err_msg, verbose) > > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 707, > > in assert_array_equal > > verbose=verbose, header='Arrays are not equal') > > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 605, > > in assert_array_compare > > chk_same_position(x_id, y_id, hasval='nan') > > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 588, > > in chk_same_position > > raise AssertionError(msg) > > AssertionError: > > Arrays are not equal > > > > x and y nan location mismatch: > > x: array([ nan+infj]) > > y: array((inf+infj)) > > > > ====================================================================== > > FAIL: test_umath_complex.TestCsqrt.test_special_values(, -1, > > inf, inf, inf) > > ---------------------------------------------------------------------- > > Traceback (most recent call last): > > File "/sw/lib/python2.7/site-packages/nose/case.py", line 197, in runTest > > self.test(*self.arg) > > File > > "/sw/lib/python2.7/site-packages/numpy/core/tests/test_umath_complex.py", > > line 578, in check_complex_value > > assert_equal(f(z1), z2) > > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 256, > > in assert_equal > > return assert_array_equal(actual, desired, err_msg, verbose) > > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 707, > > in assert_array_equal > > verbose=verbose, header='Arrays are not equal') > > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 605, > > in assert_array_compare > > chk_same_position(x_id, y_id, hasval='nan') > > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 588, > > in chk_same_position > > raise AssertionError(msg) > > AssertionError: > > Arrays are not equal > > > > x and y nan location mismatch: > > x: array([ nan+infj]) > > y: array((inf+infj)) > > > > ====================================================================== > > FAIL: test_umath_complex.TestCsqrt.test_special_values(, 0.0, > > inf, inf, inf) > > ---------------------------------------------------------------------- > > Traceback (most recent call last): > > File "/sw/lib/python2.7/site-packages/nose/case.py", line 197, in runTest > > self.test(*self.arg) > > File > > "/sw/lib/python2.7/site-packages/numpy/core/tests/test_umath_complex.py", > > line 578, in check_complex_value > > assert_equal(f(z1), z2) > > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 256, > > in assert_equal > > return assert_array_equal(actual, desired, err_msg, verbose) > > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 707, > > in assert_array_equal > > verbose=verbose, header='Arrays are not equal') > > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 605, > > in assert_array_compare > > chk_same_position(x_id, y_id, hasval='nan') > > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 588, > > in chk_same_position > > raise AssertionError(msg) > > AssertionError: > > Arrays are not equal > > > > x and y nan location mismatch: > > x: array([ nan+infj]) > > y: array((inf+infj)) > > > > ====================================================================== > > FAIL: test_umath_complex.TestCsqrt.test_special_values(, > > -0.0, inf, inf, inf) > > ---------------------------------------------------------------------- > > Traceback (most recent call last): > > File "/sw/lib/python2.7/site-packages/nose/case.py", line 197, in runTest > > self.test(*self.arg) > > File > > "/sw/lib/python2.7/site-packages/numpy/core/tests/test_umath_complex.py", > > line 578, in check_complex_value > > assert_equal(f(z1), z2) > > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 256, > > in assert_equal > > return assert_array_equal(actual, desired, err_msg, verbose) > > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 707, > > in assert_array_equal > > verbose=verbose, header='Arrays are not equal') > > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 605, > > in assert_array_compare > > chk_same_position(x_id, y_id, hasval='nan') > > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 588, > > in chk_same_position > > raise AssertionError(msg) > > AssertionError: > > Arrays are not equal > > > > x and y nan location mismatch: > > x: array([ nan+infj]) > > y: array((inf+infj)) > > > > ====================================================================== > > FAIL: test_umath_complex.TestCsqrt.test_special_values(, inf, > > inf, inf, inf) > > ---------------------------------------------------------------------- > > Traceback (most recent call last): > > File "/sw/lib/python2.7/site-packages/nose/case.py", line 197, in runTest > > self.test(*self.arg) > > File > > "/sw/lib/python2.7/site-packages/numpy/core/tests/test_umath_complex.py", > > line 578, in check_complex_value > > assert_equal(f(z1), z2) > > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 256, > > in assert_equal > > return assert_array_equal(actual, desired, err_msg, verbose) > > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 707, > > in assert_array_equal > > verbose=verbose, header='Arrays are not equal') > > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 605, > > in assert_array_compare > > chk_same_position(x_id, y_id, hasval='nan') > > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 588, > > in chk_same_position > > raise AssertionError(msg) > > AssertionError: > > Arrays are not equal > > > > x and y nan location mismatch: > > x: array([ nan+infj]) > > y: array((inf+infj)) > > > > ====================================================================== > > FAIL: test_umath_complex.TestCsqrt.test_special_values(, > > -inf, inf, inf, inf) > > ---------------------------------------------------------------------- > > Traceback (most recent call last): > > File "/sw/lib/python2.7/site-packages/nose/case.py", line 197, in runTest > > self.test(*self.arg) > > File > > "/sw/lib/python2.7/site-packages/numpy/core/tests/test_umath_complex.py", > > line 578, in check_complex_value > > assert_equal(f(z1), z2) > > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 256, > > in assert_equal > > return assert_array_equal(actual, desired, err_msg, verbose) > > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 707, > > in assert_array_equal > > verbose=verbose, header='Arrays are not equal') > > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 605, > > in assert_array_compare > > chk_same_position(x_id, y_id, hasval='nan') > > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 588, > > in chk_same_position > > raise AssertionError(msg) > > AssertionError: > > Arrays are not equal > > > > x and y nan location mismatch: > > x: array([ nan+infj]) > > y: array((inf+infj)) > > > > ====================================================================== > > FAIL: test_umath_complex.TestCsqrt.test_special_values(, nan, > > inf, inf, inf) > > ---------------------------------------------------------------------- > > Traceback (most recent call last): > > File "/sw/lib/python2.7/site-packages/nose/case.py", line 197, in runTest > > self.test(*self.arg) > > File > > "/sw/lib/python2.7/site-packages/numpy/core/tests/test_umath_complex.py", > > line 578, in check_complex_value > > assert_equal(f(z1), z2) > > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 256, > > in assert_equal > > return assert_array_equal(actual, desired, err_msg, verbose) > > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 707, > > in assert_array_equal > > verbose=verbose, header='Arrays are not equal') > > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 605, > > in assert_array_compare > > chk_same_position(x_id, y_id, hasval='nan') > > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 588, > > in chk_same_position > > raise AssertionError(msg) > > AssertionError: > > Arrays are not equal > > > > x and y nan location mismatch: > > x: array([ nan+infj]) > > y: array((inf+infj)) > > > > ====================================================================== > > FAIL: test_umath_complex.TestCsqrt.test_special_values(, > > -inf, 1, 0.0, inf) > > ---------------------------------------------------------------------- > > Traceback (most recent call last): > > File "/sw/lib/python2.7/site-packages/nose/case.py", line 197, in runTest > > self.test(*self.arg) > > File > > "/sw/lib/python2.7/site-packages/numpy/core/tests/test_umath_complex.py", > > line 578, in check_complex_value > > assert_equal(f(z1), z2) > > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 256, > > in assert_equal > > return assert_array_equal(actual, desired, err_msg, verbose) > > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 707, > > in assert_array_equal > > verbose=verbose, header='Arrays are not equal') > > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 605, > > in assert_array_compare > > chk_same_position(x_id, y_id, hasval='nan') > > File "/sw/lib/python2.7/site-packages/numpy/testing/utils.py", line 588, > > in chk_same_position > > raise AssertionError(msg) > > AssertionError: > > Arrays are not equal > > > > x and y nan location mismatch: > > x: array([ nan+infj]) > > y: array(infj) > > > > ---------------------------------------------------------------------- > > Ran 3529 tests in 38.236s > > > > FAILED (KNOWNFAIL=3, SKIP=1, failures=9) > > > > > > In both cases (darwin10 and darwin11), the blas used is from > > /System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate. > > > > http://developer.apple.com/performance/accelerateframework.html > > > > I doubt the system blas is at fault and more likely this is an issue with > > the clang compilers. I know python required an explicit > > addition of -fwrapv since its test for that option is confused by clang. > > Perhaps numpy/scipy requires -fwrapv as well? > > Jack > > > > > > > > Either the BLAS you linked Scipy against is broken, or there are some > > > compatibility problems due to compilers or mixing compiler versions. > > > > > > You could check which libraries your `fblas.so` and `cblas.so` in the > > > Scipy installation are linked against. I've never used OSX, though, > > > so I do not know how to do this there. You can see the same information > > > in the beginning of the build logs, though. > > > > > > Maybe someone who is more familiar than me with OSX knows more. > > > > > > -- > > > Pauli Virtanen > > > > > > _______________________________________________ > > > SciPy-Dev mailing list > > > SciPy-Dev at scipy.org > > > http://mail.scipy.org/mailman/listinfo/scipy-dev > > _______________________________________________ > > SciPy-Dev mailing list > > SciPy-Dev at scipy.org > > http://mail.scipy.org/mailman/listinfo/scipy-dev > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev at scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev