From tjhnson at gmail.com Thu Apr 1 00:28:34 2010 From: tjhnson at gmail.com (T J) Date: Wed, 31 Mar 2010 21:28:34 -0700 Subject: [Numpy-discussion] Bug in logaddexp2.reduce In-Reply-To: References: Message-ID: On Wed, Mar 31, 2010 at 7:06 PM, Charles R Harris wrote: > > That is a 32 bit kernel, right? > Correct. Regarding the config.h, which config.h? I have a numpyconfig.h. Which compilation options should I obtain and how? When I run setup.py, I see: C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC compile options: '-Inumpy/core/include -Ibuild/src.linux-i686-2.6/numpy/core/include/numpy -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/include -I/usr/include/python2.6 -Ibuild/src.linux-i686-2.6/numpy/core/src/multiarray -Ibuild/src.linux-i686-2.6/numpy/core/src/umath -c' Here is my show_config(). atlas_threads_info: NOT AVAILABLE blas_opt_info: libraries = ['f77blas', 'cblas', 'atlas'] library_dirs = ['/usr/lib/sse2'] define_macros = [('ATLAS_INFO', '"\\"3.6.0\\""')] language = c include_dirs = ['/usr/include'] atlas_blas_threads_info: NOT AVAILABLE lapack_opt_info: libraries = ['lapack', 'f77blas', 'cblas', 'atlas'] library_dirs = ['/usr/lib/sse2/atlas', '/usr/lib/sse2'] define_macros = [('ATLAS_INFO', '"\\"3.6.0\\""')] language = f77 include_dirs = ['/usr/include'] atlas_info: libraries = ['lapack', 'f77blas', 'cblas', 'atlas'] library_dirs = ['/usr/lib/sse2/atlas', '/usr/lib/sse2'] language = f77 include_dirs = ['/usr/include'] lapack_mkl_info: NOT AVAILABLE blas_mkl_info: NOT AVAILABLE atlas_blas_info: libraries = ['f77blas', 'cblas', 'atlas'] library_dirs = ['/usr/lib/sse2'] language = c include_dirs = ['/usr/include'] mkl_info: NOT AVAILABLE $ gcc --version gcc (Ubuntu 4.4.1-4ubuntu9) 4.4.1 Copyright (C) 2009 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. From charlesr.harris at gmail.com Thu Apr 1 01:02:23 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 31 Mar 2010 23:02:23 -0600 Subject: [Numpy-discussion] Bug in logaddexp2.reduce In-Reply-To: References: Message-ID: On Wed, Mar 31, 2010 at 10:28 PM, T J wrote: > On Wed, Mar 31, 2010 at 7:06 PM, Charles R Harris > wrote: > > > > That is a 32 bit kernel, right? > > > > Correct. > > Regarding the config.h, which config.h? I have a numpyconfig.h. > Which compilation options should I obtain and how? When I run > setup.py, I see: > > C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 > -Wall -Wstrict-prototypes -fPIC > > compile options: '-Inumpy/core/include > -Ibuild/src.linux-i686-2.6/numpy/core/include/numpy > -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core > -Inumpy/core/src/npymath -Inumpy/core/src/multiarray > -Inumpy/core/src/umath -Inumpy/core/include -I/usr/include/python2.6 > -Ibuild/src.linux-i686-2.6/numpy/core/src/multiarray > -Ibuild/src.linux-i686-2.6/numpy/core/src/umath -c' > > Here is my show_config(). > > atlas_threads_info: > NOT AVAILABLE > blas_opt_info: > libraries = ['f77blas', 'cblas', 'atlas'] > library_dirs = ['/usr/lib/sse2'] > define_macros = [('ATLAS_INFO', '"\\"3.6.0\\""')] > language = c > include_dirs = ['/usr/include'] > atlas_blas_threads_info: > NOT AVAILABLE > lapack_opt_info: > libraries = ['lapack', 'f77blas', 'cblas', 'atlas'] > library_dirs = ['/usr/lib/sse2/atlas', '/usr/lib/sse2'] > define_macros = [('ATLAS_INFO', '"\\"3.6.0\\""')] > language = f77 > include_dirs = ['/usr/include'] > atlas_info: > libraries = ['lapack', 'f77blas', 'cblas', 'atlas'] > library_dirs = ['/usr/lib/sse2/atlas', '/usr/lib/sse2'] > language = f77 > include_dirs = ['/usr/include'] > lapack_mkl_info: > NOT AVAILABLE > blas_mkl_info: > NOT AVAILABLE > atlas_blas_info: > libraries = ['f77blas', 'cblas', 'atlas'] > library_dirs = ['/usr/lib/sse2'] > language = c > include_dirs = ['/usr/include'] > mkl_info: > NOT AVAILABLE > > I'm running the same distro, but 64 bits instead of 32. Strange, maybe sse2 has something to do with it. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From peridot.faceted at gmail.com Thu Apr 1 01:03:06 2010 From: peridot.faceted at gmail.com (Anne Archibald) Date: Thu, 1 Apr 2010 01:03:06 -0400 Subject: [Numpy-discussion] Bug in logaddexp2.reduce In-Reply-To: References: Message-ID: On 31 March 2010 16:21, Charles R Harris wrote: > > > On Wed, Mar 31, 2010 at 11:38 AM, T J wrote: >> >> On Wed, Mar 31, 2010 at 10:30 AM, T J wrote: >> > Hi, >> > >> > I'm getting some strange behavior with logaddexp2.reduce: >> > >> > from itertools import permutations >> > import numpy as np >> > x = np.array([-53.584962500721154, -1.5849625007211563, >> > -0.5849625007211563]) >> > for p in permutations([0,1,2]): >> > ? ?print p, np.logaddexp2.reduce(x[list(p)]) >> > >> > Essentially, the result depends on the order of the array...and we get >> > nans in the "bad" orders. ?Likely, this also affects logaddexp. >> > >> >> Sorry, forgot version information: >> >> $ python -c "import numpy;print numpy.__version__" >> 1.5.0.dev8106 >> __ > > Looks like roundoff error. No. The whole point of logaddexp and logaddexp2 is that they function correctly - in particular, avoid unnecessary underflow and overflow - in this sort of situation. This is a genuine problem. I see it using the stock numpy in Ubuntu karmic: In [3]: np.logaddexp2(-0.5849625007211563, -53.584962500721154) Out[3]: nan In [4]: np.logaddexp2(-53.584962500721154, -0.5849625007211563) Out[4]: nan In [5]: np.log2(2**(-53.584962500721154) + 2**(-0.5849625007211563)) Out[5]: -0.58496250072115608 In [6]: numpy.__version__ Out[6]: '1.3.0' In [8]: !uname -a Linux octopus 2.6.31-20-generic #58-Ubuntu SMP Fri Mar 12 05:23:09 UTC 2010 i686 GNU/Linux (the machine is a 32-bit Pentium M laptop.) I do not see a similar problem with logaddexp. Anne > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From charlesr.harris at gmail.com Thu Apr 1 01:06:04 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 31 Mar 2010 23:06:04 -0600 Subject: [Numpy-discussion] "Match" two arrays In-Reply-To: References: Message-ID: On Wed, Mar 31, 2010 at 9:24 PM, Shailendra wrote: > Hi All, > I want to make a function which should be like this > > cordinates1=(x1,y1) # x1 and y1 are x-cord and y-cord of a large > number of points > cordinates2=(x2,y2) # similar to condinates1 > indices1,indices2= match_cordinates(cordinates1,cordinates2) > > (x1[indices1],y1[indices1]) "matches" (x2[indices2],y2[indices2]) > > where definition of "match" is such that : > If A is closest point to B and distance between A and B is less that > delta than it is a "match". > If A is closest point to B and distance between A and B is more that > delta than there is no match. > Every point has either 1 "match"(closest point) or none > > Also, the size of the cordinates1 and cordinates2 are quite large and > "outer" should not be used. I can think of only C style code to > achieve this. Can any one suggest pythonic way of doing this? > > Take a look at KdTree in scipy spatial for finding nearest neighbors. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Apr 1 01:11:23 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 31 Mar 2010 23:11:23 -0600 Subject: [Numpy-discussion] Bug in logaddexp2.reduce In-Reply-To: References: Message-ID: On Wed, Mar 31, 2010 at 11:03 PM, Anne Archibald wrote: > On 31 March 2010 16:21, Charles R Harris > wrote: > > > > > > On Wed, Mar 31, 2010 at 11:38 AM, T J wrote: > >> > >> On Wed, Mar 31, 2010 at 10:30 AM, T J wrote: > >> > Hi, > >> > > >> > I'm getting some strange behavior with logaddexp2.reduce: > >> > > >> > from itertools import permutations > >> > import numpy as np > >> > x = np.array([-53.584962500721154, -1.5849625007211563, > >> > -0.5849625007211563]) > >> > for p in permutations([0,1,2]): > >> > print p, np.logaddexp2.reduce(x[list(p)]) > >> > > >> > Essentially, the result depends on the order of the array...and we get > >> > nans in the "bad" orders. Likely, this also affects logaddexp. > >> > > >> > >> Sorry, forgot version information: > >> > >> $ python -c "import numpy;print numpy.__version__" > >> 1.5.0.dev8106 > >> __ > > > > Looks like roundoff error. > > No. The whole point of logaddexp and logaddexp2 is that they function > correctly - in particular, avoid unnecessary underflow and overflow - > in this sort of situation. This is a genuine problem. > > Well, it did function correctly for me ;) Just some roundoff error. > I see it using the stock numpy in Ubuntu karmic: > In [3]: np.logaddexp2(-0.5849625007211563, -53.584962500721154) > Out[3]: nan > > In [4]: np.logaddexp2(-53.584962500721154, -0.5849625007211563) > Out[4]: nan > > In [5]: np.log2(2**(-53.584962500721154) + 2**(-0.5849625007211563)) > Out[5]: -0.58496250072115608 > > In [6]: numpy.__version__ > Out[6]: '1.3.0' > > In [8]: !uname -a > Linux octopus 2.6.31-20-generic #58-Ubuntu SMP Fri Mar 12 05:23:09 UTC > 2010 i686 GNU/Linux > > (the machine is a 32-bit Pentium M laptop.) > > I think this is related to 32 bits somehow. The code for the logaddexp and logaddexp2 functions is almost identical. Maybe the compiler? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Apr 1 01:17:08 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 31 Mar 2010 23:17:08 -0600 Subject: [Numpy-discussion] Bug in logaddexp2.reduce In-Reply-To: References: <4BB3DCB3.7040402@enthought.com> Message-ID: On Wed, Mar 31, 2010 at 6:08 PM, wrote: > On Wed, Mar 31, 2010 at 7:37 PM, Warren Weckesser > wrote: > > T J wrote: > >> On Wed, Mar 31, 2010 at 1:21 PM, Charles R Harris > >> wrote: > >> > >>> Looks like roundoff error. > >>> > >>> > >> > >> So this is "expected" behavior? > >> > >> In [1]: np.logaddexp2(-1.5849625007211563, -53.584962500721154) > >> Out[1]: -1.5849625007211561 > >> > >> In [2]: np.logaddexp2(-0.5849625007211563, -53.584962500721154) > >> Out[2]: nan > >> > > > > Is any able to reproduce this? I don't get 'nan' in either 1.4.0 or > > 2.0.0.dev8313 (32 bit Mac OSX). In an earlier email T J reported using > > 1.5.0.dev8106. > > > > >>> np.logaddexp2(-0.5849625007211563, -53.584962500721154) > nan > >>> np.logaddexp2(-1.5849625007211563, -53.584962500721154) > -1.5849625007211561 > > >>> np.version.version > '1.4.0' > > WindowsXP 32 > > What compiler? Mingw? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From peridot.faceted at gmail.com Thu Apr 1 01:17:03 2010 From: peridot.faceted at gmail.com (Anne Archibald) Date: Thu, 1 Apr 2010 01:17:03 -0400 Subject: [Numpy-discussion] Bug in logaddexp2.reduce In-Reply-To: References: Message-ID: On 1 April 2010 01:11, Charles R Harris wrote: > > > On Wed, Mar 31, 2010 at 11:03 PM, Anne Archibald > wrote: >> >> On 31 March 2010 16:21, Charles R Harris >> wrote: >> > >> > >> > On Wed, Mar 31, 2010 at 11:38 AM, T J wrote: >> >> >> >> On Wed, Mar 31, 2010 at 10:30 AM, T J wrote: >> >> > Hi, >> >> > >> >> > I'm getting some strange behavior with logaddexp2.reduce: >> >> > >> >> > from itertools import permutations >> >> > import numpy as np >> >> > x = np.array([-53.584962500721154, -1.5849625007211563, >> >> > -0.5849625007211563]) >> >> > for p in permutations([0,1,2]): >> >> > ? ?print p, np.logaddexp2.reduce(x[list(p)]) >> >> > >> >> > Essentially, the result depends on the order of the array...and we >> >> > get >> >> > nans in the "bad" orders. ?Likely, this also affects logaddexp. >> >> > >> >> >> >> Sorry, forgot version information: >> >> >> >> $ python -c "import numpy;print numpy.__version__" >> >> 1.5.0.dev8106 >> >> __ >> > >> > Looks like roundoff error. >> >> No. The whole point of logaddexp and logaddexp2 is that they function >> correctly - in particular, avoid unnecessary underflow and overflow - >> in this sort of situation. This is a genuine problem. >> > > Well, it did function correctly for me ;) Just some roundoff error. > >> >> I see it using the stock numpy in Ubuntu karmic: >> In [3]: np.logaddexp2(-0.5849625007211563, -53.584962500721154) >> Out[3]: nan >> >> In [4]: np.logaddexp2(-53.584962500721154, -0.5849625007211563) >> Out[4]: nan >> >> In [5]: np.log2(2**(-53.584962500721154) + 2**(-0.5849625007211563)) >> Out[5]: -0.58496250072115608 >> >> In [6]: numpy.__version__ >> Out[6]: '1.3.0' >> >> In [8]: !uname -a >> Linux octopus 2.6.31-20-generic #58-Ubuntu SMP Fri Mar 12 05:23:09 UTC >> 2010 i686 GNU/Linux >> >> (the machine is a 32-bit Pentium M laptop.) >> > > I think this is related to 32 bits somehow. The code for the logaddexp and > logaddexp2 functions is almost identical. Maybe the compiler? Possibly, or possibly I just didn't find values that trigger the bug for logaddexp. I wonder if the problem is the handling of denormalized floats? Anne > Chuck > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From josef.pktd at gmail.com Thu Apr 1 01:22:11 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 1 Apr 2010 01:22:11 -0400 Subject: [Numpy-discussion] Bug in logaddexp2.reduce In-Reply-To: References: <4BB3DCB3.7040402@enthought.com> Message-ID: On Thu, Apr 1, 2010 at 1:17 AM, Charles R Harris wrote: > > > On Wed, Mar 31, 2010 at 6:08 PM, wrote: >> >> On Wed, Mar 31, 2010 at 7:37 PM, Warren Weckesser >> wrote: >> > T J wrote: >> >> On Wed, Mar 31, 2010 at 1:21 PM, Charles R Harris >> >> wrote: >> >> >> >>> Looks like roundoff error. >> >>> >> >>> >> >> >> >> So this is "expected" behavior? >> >> >> >> In [1]: np.logaddexp2(-1.5849625007211563, -53.584962500721154) >> >> Out[1]: -1.5849625007211561 >> >> >> >> In [2]: np.logaddexp2(-0.5849625007211563, -53.584962500721154) >> >> Out[2]: nan >> >> >> > >> > Is any able to reproduce this? ?I don't get 'nan' in either 1.4.0 or >> > 2.0.0.dev8313 (32 bit Mac OSX). ?In an earlier email T J reported using >> > 1.5.0.dev8106. >> >> >> >> >>> np.logaddexp2(-0.5849625007211563, -53.584962500721154) >> nan >> >>> np.logaddexp2(-1.5849625007211563, -53.584962500721154) >> -1.5849625007211561 >> >> >>> np.version.version >> '1.4.0' >> >> WindowsXP 32 >> > > What compiler? Mingw? yes, mingw 3.4.5. , official binaries release 1.4.0 by David Josef > > Chuck > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From charlesr.harris at gmail.com Thu Apr 1 01:23:29 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 31 Mar 2010 23:23:29 -0600 Subject: [Numpy-discussion] Bug in logaddexp2.reduce In-Reply-To: References: Message-ID: On Wed, Mar 31, 2010 at 11:17 PM, Anne Archibald wrote: > On 1 April 2010 01:11, Charles R Harris wrote: > > > > > > On Wed, Mar 31, 2010 at 11:03 PM, Anne Archibald < > peridot.faceted at gmail.com> > > wrote: > >> > >> On 31 March 2010 16:21, Charles R Harris > >> wrote: > >> > > >> > > >> > On Wed, Mar 31, 2010 at 11:38 AM, T J wrote: > >> >> > >> >> On Wed, Mar 31, 2010 at 10:30 AM, T J wrote: > >> >> > Hi, > >> >> > > >> >> > I'm getting some strange behavior with logaddexp2.reduce: > >> >> > > >> >> > from itertools import permutations > >> >> > import numpy as np > >> >> > x = np.array([-53.584962500721154, -1.5849625007211563, > >> >> > -0.5849625007211563]) > >> >> > for p in permutations([0,1,2]): > >> >> > print p, np.logaddexp2.reduce(x[list(p)]) > >> >> > > >> >> > Essentially, the result depends on the order of the array...and we > >> >> > get > >> >> > nans in the "bad" orders. Likely, this also affects logaddexp. > >> >> > > >> >> > >> >> Sorry, forgot version information: > >> >> > >> >> $ python -c "import numpy;print numpy.__version__" > >> >> 1.5.0.dev8106 > >> >> __ > >> > > >> > Looks like roundoff error. > >> > >> No. The whole point of logaddexp and logaddexp2 is that they function > >> correctly - in particular, avoid unnecessary underflow and overflow - > >> in this sort of situation. This is a genuine problem. > >> > > > > Well, it did function correctly for me ;) Just some roundoff error. > > > >> > >> I see it using the stock numpy in Ubuntu karmic: > >> In [3]: np.logaddexp2(-0.5849625007211563, -53.584962500721154) > >> Out[3]: nan > >> > >> In [4]: np.logaddexp2(-53.584962500721154, -0.5849625007211563) > >> Out[4]: nan > >> > >> In [5]: np.log2(2**(-53.584962500721154) + 2**(-0.5849625007211563)) > >> Out[5]: -0.58496250072115608 > >> > >> In [6]: numpy.__version__ > >> Out[6]: '1.3.0' > >> > >> In [8]: !uname -a > >> Linux octopus 2.6.31-20-generic #58-Ubuntu SMP Fri Mar 12 05:23:09 UTC > >> 2010 i686 GNU/Linux > >> > >> (the machine is a 32-bit Pentium M laptop.) > >> > > > > I think this is related to 32 bits somehow. The code for the logaddexp > and > > logaddexp2 functions is almost identical. Maybe the compiler? > > Possibly, or possibly I just didn't find values that trigger the bug > for logaddexp. I wonder if the problem is the handling of denormalized > floats? > > I'm suspecting the log2/exp2 function in the 32 bit gcc library or possibly sse2. The exponents aren't large enough to denormalize the numbers. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Thu Apr 1 01:25:02 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 1 Apr 2010 01:25:02 -0400 Subject: [Numpy-discussion] Bug in logaddexp2.reduce In-Reply-To: References: <4BB3DCB3.7040402@enthought.com> Message-ID: On Thu, Apr 1, 2010 at 1:22 AM, wrote: > On Thu, Apr 1, 2010 at 1:17 AM, Charles R Harris > wrote: >> >> >> On Wed, Mar 31, 2010 at 6:08 PM, wrote: >>> >>> On Wed, Mar 31, 2010 at 7:37 PM, Warren Weckesser >>> wrote: >>> > T J wrote: >>> >> On Wed, Mar 31, 2010 at 1:21 PM, Charles R Harris >>> >> wrote: >>> >> >>> >>> Looks like roundoff error. >>> >>> >>> >>> >>> >> >>> >> So this is "expected" behavior? >>> >> >>> >> In [1]: np.logaddexp2(-1.5849625007211563, -53.584962500721154) >>> >> Out[1]: -1.5849625007211561 >>> >> >>> >> In [2]: np.logaddexp2(-0.5849625007211563, -53.584962500721154) >>> >> Out[2]: nan >>> >> >>> > >>> > Is any able to reproduce this? ?I don't get 'nan' in either 1.4.0 or >>> > 2.0.0.dev8313 (32 bit Mac OSX). ?In an earlier email T J reported using >>> > 1.5.0.dev8106. >>> >>> >>> >>> >>> np.logaddexp2(-0.5849625007211563, -53.584962500721154) >>> nan >>> >>> np.logaddexp2(-1.5849625007211563, -53.584962500721154) >>> -1.5849625007211561 >>> >>> >>> np.version.version >>> '1.4.0' >>> >>> WindowsXP 32 >>> >> >> What compiler? Mingw? > > yes, mingw 3.4.5. , official binaries release 1.4.0 by David sse2 Pentium M > > Josef > >> >> Chuck >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > From charlesr.harris at gmail.com Thu Apr 1 01:40:06 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 31 Mar 2010 23:40:06 -0600 Subject: [Numpy-discussion] Bug in logaddexp2.reduce In-Reply-To: References: <4BB3DCB3.7040402@enthought.com> Message-ID: On Wed, Mar 31, 2010 at 11:25 PM, wrote: > On Thu, Apr 1, 2010 at 1:22 AM, wrote: > > On Thu, Apr 1, 2010 at 1:17 AM, Charles R Harris > > wrote: > >> > >> > >> On Wed, Mar 31, 2010 at 6:08 PM, wrote: > >>> > >>> On Wed, Mar 31, 2010 at 7:37 PM, Warren Weckesser > >>> wrote: > >>> > T J wrote: > >>> >> On Wed, Mar 31, 2010 at 1:21 PM, Charles R Harris > >>> >> wrote: > >>> >> > >>> >>> Looks like roundoff error. > >>> >>> > >>> >>> > >>> >> > >>> >> So this is "expected" behavior? > >>> >> > >>> >> In [1]: np.logaddexp2(-1.5849625007211563, -53.584962500721154) > >>> >> Out[1]: -1.5849625007211561 > >>> >> > >>> >> In [2]: np.logaddexp2(-0.5849625007211563, -53.584962500721154) > >>> >> Out[2]: nan > >>> >> > >>> > > >>> > Is any able to reproduce this? I don't get 'nan' in either 1.4.0 or > >>> > 2.0.0.dev8313 (32 bit Mac OSX). In an earlier email T J reported > using > >>> > 1.5.0.dev8106. > >>> > >>> > >>> > >>> >>> np.logaddexp2(-0.5849625007211563, -53.584962500721154) > >>> nan > >>> >>> np.logaddexp2(-1.5849625007211563, -53.584962500721154) > >>> -1.5849625007211561 > >>> > >>> >>> np.version.version > >>> '1.4.0' > >>> > >>> WindowsXP 32 > >>> > >> > >> What compiler? Mingw? > > > > yes, mingw 3.4.5. , official binaries release 1.4.0 by David > > sse2 Pentium M > > Can you try the exp2/log2 functions with the problem data and see if something goes wrong? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From peridot.faceted at gmail.com Thu Apr 1 01:46:12 2010 From: peridot.faceted at gmail.com (Anne Archibald) Date: Thu, 1 Apr 2010 01:46:12 -0400 Subject: [Numpy-discussion] Bug in logaddexp2.reduce In-Reply-To: References: <4BB3DCB3.7040402@enthought.com> Message-ID: On 1 April 2010 01:40, Charles R Harris wrote: > > > On Wed, Mar 31, 2010 at 11:25 PM, wrote: >> >> On Thu, Apr 1, 2010 at 1:22 AM, ? wrote: >> > On Thu, Apr 1, 2010 at 1:17 AM, Charles R Harris >> > wrote: >> >> >> >> >> >> On Wed, Mar 31, 2010 at 6:08 PM, wrote: >> >>> >> >>> On Wed, Mar 31, 2010 at 7:37 PM, Warren Weckesser >> >>> wrote: >> >>> > T J wrote: >> >>> >> On Wed, Mar 31, 2010 at 1:21 PM, Charles R Harris >> >>> >> wrote: >> >>> >> >> >>> >>> Looks like roundoff error. >> >>> >>> >> >>> >>> >> >>> >> >> >>> >> So this is "expected" behavior? >> >>> >> >> >>> >> In [1]: np.logaddexp2(-1.5849625007211563, -53.584962500721154) >> >>> >> Out[1]: -1.5849625007211561 >> >>> >> >> >>> >> In [2]: np.logaddexp2(-0.5849625007211563, -53.584962500721154) >> >>> >> Out[2]: nan >> >>> >> >> >>> > >> >>> > Is any able to reproduce this? ?I don't get 'nan' in either 1.4.0 or >> >>> > 2.0.0.dev8313 (32 bit Mac OSX). ?In an earlier email T J reported >> >>> > using >> >>> > 1.5.0.dev8106. >> >>> >> >>> >> >>> >> >>> >>> np.logaddexp2(-0.5849625007211563, -53.584962500721154) >> >>> nan >> >>> >>> np.logaddexp2(-1.5849625007211563, -53.584962500721154) >> >>> -1.5849625007211561 >> >>> >> >>> >>> np.version.version >> >>> '1.4.0' >> >>> >> >>> WindowsXP 32 >> >>> >> >> >> >> What compiler? Mingw? >> > >> > yes, mingw 3.4.5. , official binaries release 1.4.0 by David >> >> sse2 Pentium M >> > > Can you try the exp2/log2 functions with the problem data and see if > something goes wrong? Works fine for me. If it helps clarify things, the difference between the two problem input values is exactly 53 (and that's what logaddexp2 does an exp2 of); so I can provide a simpler example: In [23]: np.logaddexp2(0, -53) Out[23]: nan Of course, for me it fails in both orders. Anne From charlesr.harris at gmail.com Thu Apr 1 01:59:16 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 31 Mar 2010 23:59:16 -0600 Subject: [Numpy-discussion] Bug in logaddexp2.reduce In-Reply-To: References: <4BB3DCB3.7040402@enthought.com> Message-ID: On Wed, Mar 31, 2010 at 11:46 PM, Anne Archibald wrote: > On 1 April 2010 01:40, Charles R Harris wrote: > > > > > > On Wed, Mar 31, 2010 at 11:25 PM, wrote: > >> > >> On Thu, Apr 1, 2010 at 1:22 AM, wrote: > >> > On Thu, Apr 1, 2010 at 1:17 AM, Charles R Harris > >> > wrote: > >> >> > >> >> > >> >> On Wed, Mar 31, 2010 at 6:08 PM, wrote: > >> >>> > >> >>> On Wed, Mar 31, 2010 at 7:37 PM, Warren Weckesser > >> >>> wrote: > >> >>> > T J wrote: > >> >>> >> On Wed, Mar 31, 2010 at 1:21 PM, Charles R Harris > >> >>> >> wrote: > >> >>> >> > >> >>> >>> Looks like roundoff error. > >> >>> >>> > >> >>> >>> > >> >>> >> > >> >>> >> So this is "expected" behavior? > >> >>> >> > >> >>> >> In [1]: np.logaddexp2(-1.5849625007211563, -53.584962500721154) > >> >>> >> Out[1]: -1.5849625007211561 > >> >>> >> > >> >>> >> In [2]: np.logaddexp2(-0.5849625007211563, -53.584962500721154) > >> >>> >> Out[2]: nan > >> >>> >> > >> >>> > > >> >>> > Is any able to reproduce this? I don't get 'nan' in either 1.4.0 > or > >> >>> > 2.0.0.dev8313 (32 bit Mac OSX). In an earlier email T J reported > >> >>> > using > >> >>> > 1.5.0.dev8106. > >> >>> > >> >>> > >> >>> > >> >>> >>> np.logaddexp2(-0.5849625007211563, -53.584962500721154) > >> >>> nan > >> >>> >>> np.logaddexp2(-1.5849625007211563, -53.584962500721154) > >> >>> -1.5849625007211561 > >> >>> > >> >>> >>> np.version.version > >> >>> '1.4.0' > >> >>> > >> >>> WindowsXP 32 > >> >>> > >> >> > >> >> What compiler? Mingw? > >> > > >> > yes, mingw 3.4.5. , official binaries release 1.4.0 by David > >> > >> sse2 Pentium M > >> > > > > Can you try the exp2/log2 functions with the problem data and see if > > something goes wrong? > > Works fine for me. > > If it helps clarify things, the difference between the two problem > input values is exactly 53 (and that's what logaddexp2 does an exp2 > of); so I can provide a simpler example: > > In [23]: np.logaddexp2(0, -53) > Out[23]: nan > > Of course, for me it fails in both orders. > > Ah, that's progress then ;) The effective number of bits in a double is 53 (52 + implicit bit). That shouldn't cause problems but it sure looks suspicious. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From peridot.faceted at gmail.com Thu Apr 1 02:04:52 2010 From: peridot.faceted at gmail.com (Anne Archibald) Date: Thu, 1 Apr 2010 02:04:52 -0400 Subject: [Numpy-discussion] Bug in logaddexp2.reduce In-Reply-To: References: <4BB3DCB3.7040402@enthought.com> Message-ID: On 1 April 2010 01:59, Charles R Harris wrote: > > > On Wed, Mar 31, 2010 at 11:46 PM, Anne Archibald > wrote: >> >> On 1 April 2010 01:40, Charles R Harris wrote: >> > >> > >> > On Wed, Mar 31, 2010 at 11:25 PM, wrote: >> >> >> >> On Thu, Apr 1, 2010 at 1:22 AM, ? wrote: >> >> > On Thu, Apr 1, 2010 at 1:17 AM, Charles R Harris >> >> > wrote: >> >> >> >> >> >> >> >> >> On Wed, Mar 31, 2010 at 6:08 PM, wrote: >> >> >>> >> >> >>> On Wed, Mar 31, 2010 at 7:37 PM, Warren Weckesser >> >> >>> wrote: >> >> >>> > T J wrote: >> >> >>> >> On Wed, Mar 31, 2010 at 1:21 PM, Charles R Harris >> >> >>> >> wrote: >> >> >>> >> >> >> >>> >>> Looks like roundoff error. >> >> >>> >>> >> >> >>> >>> >> >> >>> >> >> >> >>> >> So this is "expected" behavior? >> >> >>> >> >> >> >>> >> In [1]: np.logaddexp2(-1.5849625007211563, -53.584962500721154) >> >> >>> >> Out[1]: -1.5849625007211561 >> >> >>> >> >> >> >>> >> In [2]: np.logaddexp2(-0.5849625007211563, -53.584962500721154) >> >> >>> >> Out[2]: nan >> >> >>> >> >> >> >>> > >> >> >>> > Is any able to reproduce this? ?I don't get 'nan' in either 1.4.0 >> >> >>> > or >> >> >>> > 2.0.0.dev8313 (32 bit Mac OSX). ?In an earlier email T J reported >> >> >>> > using >> >> >>> > 1.5.0.dev8106. >> >> >>> >> >> >>> >> >> >>> >> >> >>> >>> np.logaddexp2(-0.5849625007211563, -53.584962500721154) >> >> >>> nan >> >> >>> >>> np.logaddexp2(-1.5849625007211563, -53.584962500721154) >> >> >>> -1.5849625007211561 >> >> >>> >> >> >>> >>> np.version.version >> >> >>> '1.4.0' >> >> >>> >> >> >>> WindowsXP 32 >> >> >>> >> >> >> >> >> >> What compiler? Mingw? >> >> > >> >> > yes, mingw 3.4.5. , official binaries release 1.4.0 by David >> >> >> >> sse2 Pentium M >> >> >> > >> > Can you try the exp2/log2 functions with the problem data and see if >> > something goes wrong? >> >> Works fine for me. >> >> If it helps clarify things, the difference between the two problem >> input values is exactly 53 (and that's what logaddexp2 does an exp2 >> of); so I can provide a simpler example: >> >> In [23]: np.logaddexp2(0, -53) >> Out[23]: nan >> >> Of course, for me it fails in both orders. >> > > Ah, that's progress then ;) The effective number of bits in a double is 53 > (52 + implicit bit). That shouldn't cause problems but it sure looks > suspicious. Indeed, that's what led me to the totally wrong suspicion that denormals have something to do with the problem. More data points: In [38]: np.logaddexp2(63.999, 0) Out[38]: nan In [39]: np.logaddexp2(64, 0) Out[39]: 64.0 In [42]: np.logaddexp2(52.999, 0) Out[42]: 52.999000000000002 In [43]: np.logaddexp2(53, 0) Out[43]: nan It looks to me like perhaps the NaNs are appearing when the smaller term affects only the "extra" bits provided by the FPU's internal larger-than-double representation. Some such issue would explain why the problem seems to be hardware- and compiler-dependent. Anne From ranavishal at gmail.com Thu Apr 1 02:13:54 2010 From: ranavishal at gmail.com (Vishal Rana) Date: Wed, 31 Mar 2010 23:13:54 -0700 Subject: [Numpy-discussion] Can this be done more efficiently using numpy? Message-ID: Hi, A calculation which goes like this... n = 5 a = np.arange(1000) b = np.arange(n - 1, 1000) l = [] for i in range(b.size): # Absolute difference of n a elements and nth b element x = np.abs(a[i:i + n] - b[i]) # Average of x y = x.sum() / n l.append(y) It takes a while if I have like 200K records! Is there a more efficient way to do this using numpy? Thanks Vishal Rana -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Apr 1 02:24:26 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 1 Apr 2010 00:24:26 -0600 Subject: [Numpy-discussion] Bug in logaddexp2.reduce In-Reply-To: References: <4BB3DCB3.7040402@enthought.com> Message-ID: On Thu, Apr 1, 2010 at 12:04 AM, Anne Archibald wrote: > On 1 April 2010 01:59, Charles R Harris wrote: > > > > > > On Wed, Mar 31, 2010 at 11:46 PM, Anne Archibald < > peridot.faceted at gmail.com> > > wrote: > >> > >> On 1 April 2010 01:40, Charles R Harris > wrote: > >> > > >> > > >> > On Wed, Mar 31, 2010 at 11:25 PM, wrote: > >> >> > >> >> On Thu, Apr 1, 2010 at 1:22 AM, wrote: > >> >> > On Thu, Apr 1, 2010 at 1:17 AM, Charles R Harris > >> >> > wrote: > >> >> >> > >> >> >> > >> >> >> On Wed, Mar 31, 2010 at 6:08 PM, wrote: > >> >> >>> > >> >> >>> On Wed, Mar 31, 2010 at 7:37 PM, Warren Weckesser > >> >> >>> wrote: > >> >> >>> > T J wrote: > >> >> >>> >> On Wed, Mar 31, 2010 at 1:21 PM, Charles R Harris > >> >> >>> >> wrote: > >> >> >>> >> > >> >> >>> >>> Looks like roundoff error. > >> >> >>> >>> > >> >> >>> >>> > >> >> >>> >> > >> >> >>> >> So this is "expected" behavior? > >> >> >>> >> > >> >> >>> >> In [1]: np.logaddexp2(-1.5849625007211563, > -53.584962500721154) > >> >> >>> >> Out[1]: -1.5849625007211561 > >> >> >>> >> > >> >> >>> >> In [2]: np.logaddexp2(-0.5849625007211563, > -53.584962500721154) > >> >> >>> >> Out[2]: nan > >> >> >>> >> > >> >> >>> > > >> >> >>> > Is any able to reproduce this? I don't get 'nan' in either > 1.4.0 > >> >> >>> > or > >> >> >>> > 2.0.0.dev8313 (32 bit Mac OSX). In an earlier email T J > reported > >> >> >>> > using > >> >> >>> > 1.5.0.dev8106. > >> >> >>> > >> >> >>> > >> >> >>> > >> >> >>> >>> np.logaddexp2(-0.5849625007211563, -53.584962500721154) > >> >> >>> nan > >> >> >>> >>> np.logaddexp2(-1.5849625007211563, -53.584962500721154) > >> >> >>> -1.5849625007211561 > >> >> >>> > >> >> >>> >>> np.version.version > >> >> >>> '1.4.0' > >> >> >>> > >> >> >>> WindowsXP 32 > >> >> >>> > >> >> >> > >> >> >> What compiler? Mingw? > >> >> > > >> >> > yes, mingw 3.4.5. , official binaries release 1.4.0 by David > >> >> > >> >> sse2 Pentium M > >> >> > >> > > >> > Can you try the exp2/log2 functions with the problem data and see if > >> > something goes wrong? > >> > >> Works fine for me. > >> > >> If it helps clarify things, the difference between the two problem > >> input values is exactly 53 (and that's what logaddexp2 does an exp2 > >> of); so I can provide a simpler example: > >> > >> In [23]: np.logaddexp2(0, -53) > >> Out[23]: nan > >> > >> Of course, for me it fails in both orders. > >> > > > > Ah, that's progress then ;) The effective number of bits in a double is > 53 > > (52 + implicit bit). That shouldn't cause problems but it sure looks > > suspicious. > > Indeed, that's what led me to the totally wrong suspicion that > denormals have something to do with the problem. More data points: > > In [38]: np.logaddexp2(63.999, 0) > Out[38]: nan > > In [39]: np.logaddexp2(64, 0) > Out[39]: 64.0 > > In [42]: np.logaddexp2(52.999, 0) > Out[42]: 52.999000000000002 > > In [43]: np.logaddexp2(53, 0) > Out[43]: nan > > It looks to me like perhaps the NaNs are appearing when the smaller > term affects only the "extra" bits provided by the FPU's internal > larger-than-double representation. Some such issue would explain why > the problem seems to be hardware- and compiler-dependent. > > Hmm, that 63.999 is kinda strange. Here is a bit of code that might confuse a compiler working with different size mantissas: @type@ npy_log2_1p at c@(@type@ x) { @type@ u = 1 + x; if (u == 1) { return LOG2E*x; } else { return npy_log2 at c@(u) * x / (u - 1); } } It might be that u != 1 does not imply u-1 != 0. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From peridot.faceted at gmail.com Thu Apr 1 02:46:20 2010 From: peridot.faceted at gmail.com (Anne Archibald) Date: Thu, 1 Apr 2010 02:46:20 -0400 Subject: [Numpy-discussion] Bug in logaddexp2.reduce In-Reply-To: References: Message-ID: On 1 April 2010 02:24, Charles R Harris wrote: > > > On Thu, Apr 1, 2010 at 12:04 AM, Anne Archibald > wrote: >> >> On 1 April 2010 01:59, Charles R Harris wrote: >> > >> > >> > On Wed, Mar 31, 2010 at 11:46 PM, Anne Archibald >> > >> > wrote: >> >> >> >> On 1 April 2010 01:40, Charles R Harris >> >> wrote: >> >> > >> >> > >> >> > On Wed, Mar 31, 2010 at 11:25 PM, wrote: >> >> >> >> >> >> On Thu, Apr 1, 2010 at 1:22 AM, ? wrote: >> >> >> > On Thu, Apr 1, 2010 at 1:17 AM, Charles R Harris >> >> >> > wrote: >> >> >> >> >> >> >> >> >> >> >> >> On Wed, Mar 31, 2010 at 6:08 PM, wrote: >> >> >> >>> >> >> >> >>> On Wed, Mar 31, 2010 at 7:37 PM, Warren Weckesser >> >> >> >>> wrote: >> >> >> >>> > T J wrote: >> >> >> >>> >> On Wed, Mar 31, 2010 at 1:21 PM, Charles R Harris >> >> >> >>> >> wrote: >> >> >> >>> >> >> >> >> >>> >>> Looks like roundoff error. >> >> >> >>> >>> >> >> >> >>> >>> >> >> >> >>> >> >> >> >> >>> >> So this is "expected" behavior? >> >> >> >>> >> >> >> >> >>> >> In [1]: np.logaddexp2(-1.5849625007211563, >> >> >> >>> >> -53.584962500721154) >> >> >> >>> >> Out[1]: -1.5849625007211561 >> >> >> >>> >> >> >> >> >>> >> In [2]: np.logaddexp2(-0.5849625007211563, >> >> >> >>> >> -53.584962500721154) >> >> >> >>> >> Out[2]: nan >> >> >> >>> >> >> >> >> >>> > >> >> >> >>> > Is any able to reproduce this? ?I don't get 'nan' in either >> >> >> >>> > 1.4.0 >> >> >> >>> > or >> >> >> >>> > 2.0.0.dev8313 (32 bit Mac OSX). ?In an earlier email T J >> >> >> >>> > reported >> >> >> >>> > using >> >> >> >>> > 1.5.0.dev8106. >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >>> np.logaddexp2(-0.5849625007211563, -53.584962500721154) >> >> >> >>> nan >> >> >> >>> >>> np.logaddexp2(-1.5849625007211563, -53.584962500721154) >> >> >> >>> -1.5849625007211561 >> >> >> >>> >> >> >> >>> >>> np.version.version >> >> >> >>> '1.4.0' >> >> >> >>> >> >> >> >>> WindowsXP 32 >> >> >> >>> >> >> >> >> >> >> >> >> What compiler? Mingw? >> >> >> > >> >> >> > yes, mingw 3.4.5. , official binaries release 1.4.0 by David >> >> >> >> >> >> sse2 Pentium M >> >> >> >> >> > >> >> > Can you try the exp2/log2 functions with the problem data and see if >> >> > something goes wrong? >> >> >> >> Works fine for me. >> >> >> >> If it helps clarify things, the difference between the two problem >> >> input values is exactly 53 (and that's what logaddexp2 does an exp2 >> >> of); so I can provide a simpler example: >> >> >> >> In [23]: np.logaddexp2(0, -53) >> >> Out[23]: nan >> >> >> >> Of course, for me it fails in both orders. >> >> >> > >> > Ah, that's progress then ;) The effective number of bits in a double is >> > 53 >> > (52 + implicit bit). That shouldn't cause problems but it sure looks >> > suspicious. >> >> Indeed, that's what led me to the totally wrong suspicion that >> denormals have something to do with the problem. More data points: >> >> In [38]: np.logaddexp2(63.999, 0) >> Out[38]: nan >> >> In [39]: np.logaddexp2(64, 0) >> Out[39]: 64.0 >> >> In [42]: np.logaddexp2(52.999, 0) >> Out[42]: 52.999000000000002 >> >> In [43]: np.logaddexp2(53, 0) >> Out[43]: nan >> >> It looks to me like perhaps the NaNs are appearing when the smaller >> term affects only the "extra" bits provided by the FPU's internal >> larger-than-double representation. Some such issue would explain why >> the problem seems to be hardware- and compiler-dependent. >> > > Hmm, that 63.999 is kinda strange. Here is a bit of code that might confuse > a compiler working with different size mantissas: > > @type@ npy_log2_1p at c@(@type@ x) > { > ??? @type@ u = 1 + x; > ??? if (u == 1) { > ??????? return LOG2E*x; > ??? } else { > ??????? return npy_log2 at c@(u) * x / (u - 1); > ??? } > } > > It might be that u != 1 does not imply u-1 != 0. That does indeed look highly suspicious. I'm not entirely sure how to work around it. GSL uses a volatile declaration: http://www.google.ca/codesearch/p?hl=en#p9nGS4eQGUI/gnu/gsl/gsl-1.8.tar.gz%7C8VCQSLJ5jR8/gsl-1.8/sys/log1p.c&q=log1p On the other hand boost declares itself defeated by optimizing compilers and uses a Taylor series: http://www.google.ca/codesearch/p?hl=en#sdP2GRSfgKo/dcplusplus/trunk/boost/boost/math/special_functions/log1p.hpp&q=log1p&sa=N&cd=7&ct=rc While R makes no mention of the corrected formula or optimizing compilers but takes the same approach, only with Chebyshev series: http://www.google.ca/codesearch/p?hl=en#gBBSWbwZmuk/src/base/R-2/R-2.3.1.tar.gz%7CVuh8XhRbUi8/R-2.3.1/src/nmath/log1p.c&q=log1p Since, at least on my machine, ordinary log1p appears to work fine, is there any reason not to have log2_1p call it and scale the result? Or does the compiler make a hash of our log1p too? Anne > Chuck > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From david at silveregg.co.jp Thu Apr 1 02:49:27 2010 From: david at silveregg.co.jp (David Cournapeau) Date: Thu, 01 Apr 2010 15:49:27 +0900 Subject: [Numpy-discussion] Bug in logaddexp2.reduce In-Reply-To: References: <4BB3DCB3.7040402@enthought.com> Message-ID: <4BB441F7.3010804@silveregg.co.jp> Charles R Harris wrote: > On Thu, Apr 1, 2010 at 12:04 AM, Anne Archibald > wrote: > >> On 1 April 2010 01:59, Charles R Harris wrote: >>> >>> On Wed, Mar 31, 2010 at 11:46 PM, Anne Archibald < >> peridot.faceted at gmail.com> >>> wrote: >>>> On 1 April 2010 01:40, Charles R Harris >> wrote: >>>>> >>>>> On Wed, Mar 31, 2010 at 11:25 PM, wrote: >>>>>> On Thu, Apr 1, 2010 at 1:22 AM, wrote: >>>>>>> On Thu, Apr 1, 2010 at 1:17 AM, Charles R Harris >>>>>>> wrote: >>>>>>>> >>>>>>>> On Wed, Mar 31, 2010 at 6:08 PM, wrote: >>>>>>>>> On Wed, Mar 31, 2010 at 7:37 PM, Warren Weckesser >>>>>>>>> wrote: >>>>>>>>>> T J wrote: >>>>>>>>>>> On Wed, Mar 31, 2010 at 1:21 PM, Charles R Harris >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Looks like roundoff error. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> So this is "expected" behavior? >>>>>>>>>>> >>>>>>>>>>> In [1]: np.logaddexp2(-1.5849625007211563, >> -53.584962500721154) >>>>>>>>>>> Out[1]: -1.5849625007211561 >>>>>>>>>>> >>>>>>>>>>> In [2]: np.logaddexp2(-0.5849625007211563, >> -53.584962500721154) >>>>>>>>>>> Out[2]: nan >>>>>>>>>>> >>>>>>>>>> Is any able to reproduce this? I don't get 'nan' in either >> 1.4.0 >>>>>>>>>> or >>>>>>>>>> 2.0.0.dev8313 (32 bit Mac OSX). In an earlier email T J >> reported >>>>>>>>>> using >>>>>>>>>> 1.5.0.dev8106. >>>>>>>>> >>>>>>>>> >>>>>>>>>>>> np.logaddexp2(-0.5849625007211563, -53.584962500721154) >>>>>>>>> nan >>>>>>>>>>>> np.logaddexp2(-1.5849625007211563, -53.584962500721154) >>>>>>>>> -1.5849625007211561 >>>>>>>>> >>>>>>>>>>>> np.version.version >>>>>>>>> '1.4.0' >>>>>>>>> >>>>>>>>> WindowsXP 32 >>>>>>>>> >>>>>>>> What compiler? Mingw? >>>>>>> yes, mingw 3.4.5. , official binaries release 1.4.0 by David >>>>>> sse2 Pentium M >>>>>> >>>>> Can you try the exp2/log2 functions with the problem data and see if >>>>> something goes wrong? >>>> Works fine for me. >>>> >>>> If it helps clarify things, the difference between the two problem >>>> input values is exactly 53 (and that's what logaddexp2 does an exp2 >>>> of); so I can provide a simpler example: >>>> >>>> In [23]: np.logaddexp2(0, -53) >>>> Out[23]: nan >>>> >>>> Of course, for me it fails in both orders. >>>> >>> Ah, that's progress then ;) The effective number of bits in a double is >> 53 >>> (52 + implicit bit). That shouldn't cause problems but it sure looks >>> suspicious. >> Indeed, that's what led me to the totally wrong suspicion that >> denormals have something to do with the problem. More data points: >> >> In [38]: np.logaddexp2(63.999, 0) >> Out[38]: nan >> >> In [39]: np.logaddexp2(64, 0) >> Out[39]: 64.0 >> >> In [42]: np.logaddexp2(52.999, 0) >> Out[42]: 52.999000000000002 >> >> In [43]: np.logaddexp2(53, 0) >> Out[43]: nan >> >> It looks to me like perhaps the NaNs are appearing when the smaller >> term affects only the "extra" bits provided by the FPU's internal >> larger-than-double representation. Some such issue would explain why >> the problem seems to be hardware- and compiler-dependent. >> >> > Hmm, that 63.999 is kinda strange. Here is a bit of code that might confuse > a compiler working with different size mantissas: > > @type@ npy_log2_1p at c@(@type@ x) > { > @type@ u = 1 + x; > if (u == 1) { > return LOG2E*x; > } else { > return npy_log2 at c@(u) * x / (u - 1); > } > } > > It might be that u != 1 does not imply u-1 != 0. I don't think that's true for floating point, since the spacing at 1 and at 0 are quite different, and that's what defines whether two floating point numbers are close or not. But I think you're right about the problem. I played a bit on my netbook where I have the same problem (Ubuntu 10.04 beta, gc 4.4.3, 32 bits). The problem is in npy_log2_1p (for example, try npy_log2_1p(1e-18) vs npy_log2_p1(1e-15)). If I compile with -ffloat-store, the problem also disappears. I think the fix is to do : @type@ npy_log2_1p at c@(@type@ x) { @type@ u = 1 + x; if ((u-1) == 0) { return LOG2E*x; } else { return npy_log2 at c@(u) * x / (u - 1); } } this works for a separate C program, but for some reason, once I try this in numpy, it does not work, and I have no battery anymore, cheers, David From peridot.faceted at gmail.com Thu Apr 1 02:56:37 2010 From: peridot.faceted at gmail.com (Anne Archibald) Date: Thu, 1 Apr 2010 02:56:37 -0400 Subject: [Numpy-discussion] Bug in logaddexp2.reduce In-Reply-To: <4BB441F7.3010804@silveregg.co.jp> References: <4BB441F7.3010804@silveregg.co.jp> Message-ID: On 1 April 2010 02:49, David Cournapeau wrote: > Charles R Harris wrote: >> On Thu, Apr 1, 2010 at 12:04 AM, Anne Archibald >> wrote: >> >>> On 1 April 2010 01:59, Charles R Harris wrote: >>>> >>>> On Wed, Mar 31, 2010 at 11:46 PM, Anne Archibald < >>> peridot.faceted at gmail.com> >>>> wrote: >>>>> On 1 April 2010 01:40, Charles R Harris >>> wrote: >>>>>> >>>>>> On Wed, Mar 31, 2010 at 11:25 PM, wrote: >>>>>>> On Thu, Apr 1, 2010 at 1:22 AM, ? wrote: >>>>>>>> On Thu, Apr 1, 2010 at 1:17 AM, Charles R Harris >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> On Wed, Mar 31, 2010 at 6:08 PM, wrote: >>>>>>>>>> On Wed, Mar 31, 2010 at 7:37 PM, Warren Weckesser >>>>>>>>>> wrote: >>>>>>>>>>> T J wrote: >>>>>>>>>>>> On Wed, Mar 31, 2010 at 1:21 PM, Charles R Harris >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Looks like roundoff error. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> So this is "expected" behavior? >>>>>>>>>>>> >>>>>>>>>>>> In [1]: np.logaddexp2(-1.5849625007211563, >>> -53.584962500721154) >>>>>>>>>>>> Out[1]: -1.5849625007211561 >>>>>>>>>>>> >>>>>>>>>>>> In [2]: np.logaddexp2(-0.5849625007211563, >>> -53.584962500721154) >>>>>>>>>>>> Out[2]: nan >>>>>>>>>>>> >>>>>>>>>>> Is any able to reproduce this? ?I don't get 'nan' in either >>> 1.4.0 >>>>>>>>>>> or >>>>>>>>>>> 2.0.0.dev8313 (32 bit Mac OSX). ?In an earlier email T J >>> reported >>>>>>>>>>> using >>>>>>>>>>> 1.5.0.dev8106. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>> np.logaddexp2(-0.5849625007211563, -53.584962500721154) >>>>>>>>>> nan >>>>>>>>>>>>> np.logaddexp2(-1.5849625007211563, -53.584962500721154) >>>>>>>>>> -1.5849625007211561 >>>>>>>>>> >>>>>>>>>>>>> np.version.version >>>>>>>>>> '1.4.0' >>>>>>>>>> >>>>>>>>>> WindowsXP 32 >>>>>>>>>> >>>>>>>>> What compiler? Mingw? >>>>>>>> yes, mingw 3.4.5. , official binaries release 1.4.0 by David >>>>>>> sse2 Pentium M >>>>>>> >>>>>> Can you try the exp2/log2 functions with the problem data and see if >>>>>> something goes wrong? >>>>> Works fine for me. >>>>> >>>>> If it helps clarify things, the difference between the two problem >>>>> input values is exactly 53 (and that's what logaddexp2 does an exp2 >>>>> of); so I can provide a simpler example: >>>>> >>>>> In [23]: np.logaddexp2(0, -53) >>>>> Out[23]: nan >>>>> >>>>> Of course, for me it fails in both orders. >>>>> >>>> Ah, that's progress then ;) The effective number of bits in a double is >>> 53 >>>> (52 + implicit bit). That shouldn't cause problems but it sure looks >>>> suspicious. >>> Indeed, that's what led me to the totally wrong suspicion that >>> denormals have something to do with the problem. More data points: >>> >>> In [38]: np.logaddexp2(63.999, 0) >>> Out[38]: nan >>> >>> In [39]: np.logaddexp2(64, 0) >>> Out[39]: 64.0 >>> >>> In [42]: np.logaddexp2(52.999, 0) >>> Out[42]: 52.999000000000002 >>> >>> In [43]: np.logaddexp2(53, 0) >>> Out[43]: nan >>> >>> It looks to me like perhaps the NaNs are appearing when the smaller >>> term affects only the "extra" bits provided by the FPU's internal >>> larger-than-double representation. Some such issue would explain why >>> the problem seems to be hardware- and compiler-dependent. >>> >>> >> Hmm, that 63.999 is kinda strange. Here is a bit of code that might confuse >> a compiler working with different size mantissas: >> >> @type@ npy_log2_1p at c@(@type@ x) >> { >> ? ? @type@ u = 1 + x; >> ? ? if (u == 1) { >> ? ? ? ? return LOG2E*x; >> ? ? } else { >> ? ? ? ? return npy_log2 at c@(u) * x / (u - 1); >> ? ? } >> } >> >> It might be that u != 1 does not imply u-1 != 0. > > I don't think that's true for floating point, since the spacing at 1 and > at 0 are quite different, and that's what defines whether two floating > point numbers are close or not. > > But I think you're right about the problem. I played a bit on my netbook > where I have the same problem (Ubuntu 10.04 beta, gc 4.4.3, 32 bits). > The problem is in npy_log2_1p (for example, try npy_log2_1p(1e-18) vs > npy_log2_p1(1e-15)). If I compile with -ffloat-store, the problem also > disappears. > > I think the fix is to do : > > @type@ npy_log2_1p at c@(@type@ x) > { > ? ? ?@type@ u = 1 + x; > ? ? ?if ((u-1) == 0) { > ? ? ? ? ?return LOG2E*x; > ? ? ?} else { > ? ? ? ? ?return npy_log2 at c@(u) * x / (u - 1); > ? ? ?} > } Particularly given the comments in the boost source code, I'm leery of this fix; who knows what an optimizing compiler will do with it? What if, for example, u-1 and u get treated differently - one gets truncated to double, but the other doesn't, for example? Now the "correction" is completely bogus. Or, of course, the compiler might rewrite the expressions. At the least an explicit "volatile" variable or two is necessary, and even so making it compiler-proof seems like a real challenge. At the least it's letting us in for all sorts of subtle trouble down the road (though at least unit tests can catch it). Since this is a subtle issue, I vote for delegating it (and log10_1p) to log1p. If *that* has trouble, well, at least it'll only be on non-C99 machines, and we can try compiler gymnastics. > this works for a separate C program, but for some reason, once I try > this in numpy, it does not work, and I have no battery anymore, Clearly the optimizing compiler is inserting the DRDB (drain David's battery) opcode. Anne From david at silveregg.co.jp Thu Apr 1 03:15:11 2010 From: david at silveregg.co.jp (David Cournapeau) Date: Thu, 01 Apr 2010 16:15:11 +0900 Subject: [Numpy-discussion] Bug in logaddexp2.reduce In-Reply-To: References: <4BB441F7.3010804@silveregg.co.jp> Message-ID: <4BB447FF.4080507@silveregg.co.jp> Anne Archibald wrote: > > Particularly given the comments in the boost source code, I'm leery of > this fix; who knows what an optimizing compiler will do with it? But the current code *is* wrong: it is not true that u == 1 implies u - 1 == 0 (and that (u-1) != 0 -> u != 1), because the spacing between two consecutive floats is much bigger at 1 than at 0. And the current code relies on this wrong assumption: at least with the correction, we test for what we care about. So I would say the code correction I suggest is at least better in that respect. > Now the > "correction" is completely bogus. I am not sure to understand why ? It is at least not worse than the current code. > Since this is a subtle issue, I vote for delegating it (and log10_1p) > to log1p. If *that* has trouble, well, at least it'll only be on > non-C99 machines, and we can try compiler gymnastics. Yes, we could do that. I note that on glibc, the function called is an intrinsic for log1p (FYL2XP1) if x is sufficiently small. > Clearly the optimizing compiler is inserting the DRDB (drain David's > battery) opcode. :) David From lists at dipole-consulting.com Thu Apr 1 04:21:01 2010 From: lists at dipole-consulting.com (Didrik Pinte) Date: Thu, 01 Apr 2010 10:21:01 +0200 Subject: [Numpy-discussion] Can this be done more efficiently using numpy? In-Reply-To: References: Message-ID: <1270110061.2845.20.camel@homer.dipole.lan> On Wed, 2010-03-31 at 23:13 -0700, Vishal Rana wrote: > Hi, > > > A calculation which goes like this... > > > n = 5 > a = np.arange(1000) > b = np.arange(n - 1, 1000) > > > l = [] > for i in range(b.size): > # Absolute difference of n a elements and nth b element > x = np.abs(a[i:i + n] - b[i]) > > # Average of x > y = x.sum() / n > > l.append(y) > > > It takes a while if I have like 200K records! > Is there a more efficient way to do this using numpy? Like this ? import numpy from numpy.lib.stride_tricks import as_strided n = 5 a = numpy.arange(1000) b = numpy.arange(n - 1, 1000) def sum_1(n, a, b): # original method l = [] for i in range(b.size): # Absolute difference of n a elements and nth b element x = numpy.abs(a[i:i + n] - b[i]) # Average of x y = x.sum() / n l.append(y) return l def sum_numpy(n, a, b): ast = as_strided(a, shape=(1000-n+1, n), strides=(8,8)) diff = numpy.abs(ast - b[:,None]) y = diff.sum(axis=1) /n return y test1 = sum_1(n,a,b) test2 = sum_numpy(n,a,b) assert(numpy.alltrue(test2 == test1)) -- Didrik -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part URL: From peridot.faceted at gmail.com Thu Apr 1 04:39:09 2010 From: peridot.faceted at gmail.com (Anne Archibald) Date: Thu, 1 Apr 2010 04:39:09 -0400 Subject: [Numpy-discussion] Bug in logaddexp2.reduce In-Reply-To: <4BB447FF.4080507@silveregg.co.jp> References: <4BB441F7.3010804@silveregg.co.jp> <4BB447FF.4080507@silveregg.co.jp> Message-ID: On 1 April 2010 03:15, David Cournapeau wrote: > Anne Archibald wrote: > >> >> Particularly given the comments in the boost source code, I'm leery of >> this fix; who knows what an optimizing compiler will do with it? > > But the current code *is* wrong: it is not true that u == 1 implies u - > 1 == 0 (and that (u-1) != 0 -> u != 1), because the spacing between two > consecutive floats is much bigger at 1 than at 0. And the current code > relies on this wrong assumption: at least with the correction, we test > for what we care about. I don't think this is true for IEEE floats, at least in the case we're interested in where u is approximately 1. After all, the issue is representation: if you take an x very close to zero and add 1 to it, you get exactly 1 (which is what you're pointing out); but if you then subtract 1 again, you get exactly zero. After all, u==1 means that the bit patterns for u and 1 are identical, so if you subtract 1 from u you get exactly zero. If u is not equal to 1 (and we're talking about IEEE floats) and you subtract 1, you will get something that is not zero. The problem is that what we're dealing with is not IEEE floating-point: it resembles IEEE floating-point, but arithmetic is often done with extra digits, which are discarded or not at the whim of the compiler. If the extra digits were kept we'd still have (u-1)==0 iff u==1, but if the compiler discards the extra digits in one expression but not the other - which is a question about FPU register scheduling - funny things can happen. > So I would say the code correction I suggest is at least better in that > respect. > >> Now the >> "correction" is completely bogus. > > I am not sure to understand why ? It is at least not worse than the > current code. Well, it would be hard to be worse than inappropriate NaNs. But the point of the correction is that, through some deviousness I don't fully understand, the roundoff error involved in the log is compensated for by the roundoff error in the calculation of (1+x)-1. If the compiler uses a different number of digits to calculate the log than it uses to caiculate (1+x)-1, the "correction" will make a hash of things. >> Since this is a subtle issue, I vote for delegating it (and log10_1p) >> to log1p. If *that* has trouble, well, at least it'll only be on >> non-C99 machines, and we can try compiler gymnastics. > > Yes, we could do that. I note that on glibc, the function called is an > intrinsic for log1p (FYL2XP1) if x is sufficiently small. Even if we end up having to implement this function by hand, we should at least implement it only once - log1p - and not three times (log1p, log10_1p, log2_1p). >> Clearly the optimizing compiler is inserting the DRDB (drain David's >> battery) opcode. > > :) That was tongue-in-cheek, but there was a real point: optimizing compilers do all sorts of peculiar things, particularly with floating-point. For one thing, many Intel CPUs have two (logically-)separate units for doing double-precision calculations - the FPU, which keeps extra digits, and SSE2, which doesn't. Who knows which one gets used for which operations? -ffloat-store is supposed to force the constant discarding of extra digits, but we certainly don't want to apply that to all of numpy. So if we have to get log1p working ourselves it'll be an exercise in suppressing specific optimizations in all optimizing compilers that might compile numpy. Anne From neilcrighton at gmail.com Thu Apr 1 05:45:48 2010 From: neilcrighton at gmail.com (Neil Crighton) Date: Thu, 1 Apr 2010 09:45:48 +0000 (UTC) Subject: [Numpy-discussion] "Match" two arrays References: Message-ID: Shailendra gmail.com> writes: > > Hi All, > I want to make a function which should be like this > > cordinates1=(x1,y1) # x1 and y1 are x-cord and y-cord of a large > number of points > cordinates2=(x2,y2) # similar to condinates1 > indices1,indices2= match_cordinates(cordinates1,cordinates2) > > (x1[indices1],y1[indices1]) "matches" (x2[indices2],y2[indices2]) > > where definition of "match" is such that : > If A is closest point to B and distance between A and B is less that > delta than it is a "match". > If A is closest point to B and distance between A and B is more that > delta than there is no match. > Every point has either 1 "match"(closest point) or none > > Also, the size of the cordinates1 and cordinates2 are quite large and > "outer" should not be used. I can think of only C style code to > achieve this. Can any one suggest pythonic way of doing this? > > Thanks, > Shailendra > A similar problem comes up when you have to match astronomical coordinates. I wrote a python + numpy function that is fast enough for my use cases - you might be able to adapt it: http://bitbucket.org/nhmc/pyserpens/src/tip/coord.py The matching function starts on line 166. Disclaimer: I haven't looked at the kdtree code yet, that might be a better approach. Neil From david at silveregg.co.jp Thu Apr 1 05:52:43 2010 From: david at silveregg.co.jp (David Cournapeau) Date: Thu, 01 Apr 2010 18:52:43 +0900 Subject: [Numpy-discussion] Bug in logaddexp2.reduce In-Reply-To: References: <4BB441F7.3010804@silveregg.co.jp> <4BB447FF.4080507@silveregg.co.jp> Message-ID: <4BB46CEB.6050902@silveregg.co.jp> Anne Archibald wrote: > On 1 April 2010 03:15, David Cournapeau wrote: >> Anne Archibald wrote: >> >>> Particularly given the comments in the boost source code, I'm leery of >>> this fix; who knows what an optimizing compiler will do with it? >> But the current code *is* wrong: it is not true that u == 1 implies u - >> 1 == 0 (and that (u-1) != 0 -> u != 1), because the spacing between two >> consecutive floats is much bigger at 1 than at 0. And the current code >> relies on this wrong assumption: at least with the correction, we test >> for what we care about. > > I don't think this is true for IEEE floats, at least in the case we're > interested in where u is approximately 1. Yes, sorry, you're right. For log1p, we can use the msun code, it is claimed to be such as the error is bounded by 1 ulp, does not rely on ASM, and we already have all the necessary macros in npymath so that the code should be easy to integrate for single and double precision. I don't see code for the long double version, though. We can use the boost test data to see if we get something sensible there . > Who knows > which one gets used for which operations? -ffloat-store is supposed to > force the constant discarding of extra digits, but we certainly don't > want to apply that to all of numpy. I think relying on -ffloat-store is buggy anyway, and it would be a maintenance hell. It is useful to know when it makes a difference, though. cheers, David From e.antero.tammi at gmail.com Thu Apr 1 06:54:47 2010 From: e.antero.tammi at gmail.com (eat) Date: Thu, 1 Apr 2010 10:54:47 +0000 (UTC) Subject: [Numpy-discussion] "Match" two arrays References: Message-ID: Shailendra gmail.com> writes: > > Hi All, > I want to make a function which should be like this > > cordinates1=(x1,y1) # x1 and y1 are x-cord and y-cord of a large > number of points > cordinates2=(x2,y2) # similar to condinates1 > indices1,indices2= match_cordinates(cordinates1,cordinates2) > > (x1[indices1],y1[indices1]) "matches" (x2[indices2],y2[indices2]) > > where definition of "match" is such that : > If A is closest point to B and distance between A and B is less that > delta than it is a "match". > If A is closest point to B and distance between A and B is more that > delta than there is no match. > Every point has either 1 "match"(closest point) or none This logic is problematic in general case. See below. You may need to be able to handle several pairs of 'closest points'! > > Also, the size of the cordinates1 and cordinates2 are quite large and > "outer" should not be used. I can think of only C style code to > achieve this. Can any one suggest pythonic way of doing this? > > Thanks, > Shailendra > This is straightforward implementation as a starting point. eat import numpy as np def dist(p1, p2): return np.sqrt(np.sum((p1- p2)** 2, 0)) def cdist(p1, p2, trh): """Expects 2d arrays p1 and p2, with combatible first dimesions and a threshold. Returns indicies of points close to each other -ind[:, 0], array of p1 indicies -ind[:, 1], array of p2 indicies -ambi, list of list of ambiquous situations (where more than 1 pair of points are 'equally close') The indicies are aranged such that dist(p1[:, ind[k, 0]], p2[:, ind[k, 1]])< trh is true for all k. """ ind= [] ambi= [] for k in range(p2.shape[1]): d= dist(p1, p2[:, None, k]) i= np.where(d< trh)[0] if 0< len(i): m= np.where(d[i]== d[i].min())[0] # problematic i= i[m].tolist() ind.append([i[0], k]) if 1< len(m): ambi.append([ind[-1], i]) return np.array(ind), ambi if __name__ == '__main__': n= 10 trh= 2e-1 p1= np.round(np.random.rand(2, n), 1) p2= np.round(p1+ 1e-1* np.random.randn(2, n), 1) ind, ambi= cdist(p1, p2, trh) print 'points close to each other:' if 0< len(ind): print 'p1:' print p1[:, ind[:, 0]], ind[:, 0] print 'p2:' print p2[:, ind[:, 1]], ind[:, 1] print 'valid:' print dist(p1[:, ind[:, 0]], p2[:, ind[:, 1]])< trh print 'with ambiguous situation(s):' if ambi: print ambi else: print 'None' else: print 'None' import timeit n= 1e2 trh= 2e-1 rep= 5 p1= np.random.rand(2, 1e3* n) p2= np.random.randn(2, n) def perf(): ind, ambi= cdist(p1, p2, trh) print 'performance:' t= np.array(timeit.repeat(perf, repeat= rep, number= 1))/ rep print 'min: ', t.min(), 'mean: ', t.mean(), 'max: ', t.max() From e.antero.tammi at gmail.com Thu Apr 1 07:59:43 2010 From: e.antero.tammi at gmail.com (eat) Date: Thu, 1 Apr 2010 11:59:43 +0000 (UTC) Subject: [Numpy-discussion] &quot;Match&quot; two arrays References: Message-ID: Oops. Wrongly timed. > t= np.array(timeit.repeat(perf, repeat= rep, number= 1))/ rep should be t= np.array(timeit.repeat(perf, repeat= rep, number= 1)) eat From charlesr.harris at gmail.com Thu Apr 1 10:16:06 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 1 Apr 2010 08:16:06 -0600 Subject: [Numpy-discussion] Bug in logaddexp2.reduce In-Reply-To: <4BB46CEB.6050902@silveregg.co.jp> References: <4BB441F7.3010804@silveregg.co.jp> <4BB447FF.4080507@silveregg.co.jp> <4BB46CEB.6050902@silveregg.co.jp> Message-ID: On Thu, Apr 1, 2010 at 3:52 AM, David Cournapeau wrote: > Anne Archibald wrote: > > On 1 April 2010 03:15, David Cournapeau wrote: > >> Anne Archibald wrote: > >> > >>> Particularly given the comments in the boost source code, I'm leery of > >>> this fix; who knows what an optimizing compiler will do with it? > >> But the current code *is* wrong: it is not true that u == 1 implies u - > >> 1 == 0 (and that (u-1) != 0 -> u != 1), because the spacing between two > >> consecutive floats is much bigger at 1 than at 0. And the current code > >> relies on this wrong assumption: at least with the correction, we test > >> for what we care about. > > > > I don't think this is true for IEEE floats, at least in the case we're > > interested in where u is approximately 1. > > > Yes, sorry, you're right. > > For log1p, we can use the msun code, it is claimed to be such as the > error is bounded by 1 ulp, does not rely on ASM, and we already have all > the necessary macros in npymath so that the code should be easy to > integrate for single and double precision. I don't see code for the long > double version, though. > > We can use the boost test data to see if we get something sensible there . > > I've had fixing these log_1p functions in the back of my mind since 1.3, mostly because I didn't trust their accuracy. I'm inclined to go with one of the series approaches, there are several out there. Note that the complex versions need to be fixed also. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Apr 1 10:37:47 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 1 Apr 2010 08:37:47 -0600 Subject: [Numpy-discussion] Bug in logaddexp2.reduce In-Reply-To: References: Message-ID: On Thu, Apr 1, 2010 at 12:46 AM, Anne Archibald wrote: > On 1 April 2010 02:24, Charles R Harris wrote: > > > > > > On Thu, Apr 1, 2010 at 12:04 AM, Anne Archibald < > peridot.faceted at gmail.com> > > wrote: > >> > >> On 1 April 2010 01:59, Charles R Harris > wrote: > >> > > >> > > >> > On Wed, Mar 31, 2010 at 11:46 PM, Anne Archibald > >> > > >> > wrote: > >> >> > >> >> On 1 April 2010 01:40, Charles R Harris > >> >> wrote: > >> >> > > >> >> > > >> >> > On Wed, Mar 31, 2010 at 11:25 PM, wrote: > >> >> >> > >> >> >> On Thu, Apr 1, 2010 at 1:22 AM, wrote: > >> >> >> > On Thu, Apr 1, 2010 at 1:17 AM, Charles R Harris > >> >> >> > wrote: > >> >> >> >> > >> >> >> >> > >> >> >> >> On Wed, Mar 31, 2010 at 6:08 PM, wrote: > >> >> >> >>> > >> >> >> >>> On Wed, Mar 31, 2010 at 7:37 PM, Warren Weckesser > >> >> >> >>> wrote: > >> >> >> >>> > T J wrote: > >> >> >> >>> >> On Wed, Mar 31, 2010 at 1:21 PM, Charles R Harris > >> >> >> >>> >> wrote: > >> >> >> >>> >> > >> >> >> >>> >>> Looks like roundoff error. > >> >> >> >>> >>> > >> >> >> >>> >>> > >> >> >> >>> >> > >> >> >> >>> >> So this is "expected" behavior? > >> >> >> >>> >> > >> >> >> >>> >> In [1]: np.logaddexp2(-1.5849625007211563, > >> >> >> >>> >> -53.584962500721154) > >> >> >> >>> >> Out[1]: -1.5849625007211561 > >> >> >> >>> >> > >> >> >> >>> >> In [2]: np.logaddexp2(-0.5849625007211563, > >> >> >> >>> >> -53.584962500721154) > >> >> >> >>> >> Out[2]: nan > >> >> >> >>> >> > >> >> >> >>> > > >> >> >> >>> > Is any able to reproduce this? I don't get 'nan' in either > >> >> >> >>> > 1.4.0 > >> >> >> >>> > or > >> >> >> >>> > 2.0.0.dev8313 (32 bit Mac OSX). In an earlier email T J > >> >> >> >>> > reported > >> >> >> >>> > using > >> >> >> >>> > 1.5.0.dev8106. > >> >> >> >>> > >> >> >> >>> > >> >> >> >>> > >> >> >> >>> >>> np.logaddexp2(-0.5849625007211563, -53.584962500721154) > >> >> >> >>> nan > >> >> >> >>> >>> np.logaddexp2(-1.5849625007211563, -53.584962500721154) > >> >> >> >>> -1.5849625007211561 > >> >> >> >>> > >> >> >> >>> >>> np.version.version > >> >> >> >>> '1.4.0' > >> >> >> >>> > >> >> >> >>> WindowsXP 32 > >> >> >> >>> > >> >> >> >> > >> >> >> >> What compiler? Mingw? > >> >> >> > > >> >> >> > yes, mingw 3.4.5. , official binaries release 1.4.0 by David > >> >> >> > >> >> >> sse2 Pentium M > >> >> >> > >> >> > > >> >> > Can you try the exp2/log2 functions with the problem data and see > if > >> >> > something goes wrong? > >> >> > >> >> Works fine for me. > >> >> > >> >> If it helps clarify things, the difference between the two problem > >> >> input values is exactly 53 (and that's what logaddexp2 does an exp2 > >> >> of); so I can provide a simpler example: > >> >> > >> >> In [23]: np.logaddexp2(0, -53) > >> >> Out[23]: nan > >> >> > >> >> Of course, for me it fails in both orders. > >> >> > >> > > >> > Ah, that's progress then ;) The effective number of bits in a double > is > >> > 53 > >> > (52 + implicit bit). That shouldn't cause problems but it sure looks > >> > suspicious. > >> > >> Indeed, that's what led me to the totally wrong suspicion that > >> denormals have something to do with the problem. More data points: > >> > >> In [38]: np.logaddexp2(63.999, 0) > >> Out[38]: nan > >> > >> In [39]: np.logaddexp2(64, 0) > >> Out[39]: 64.0 > >> > >> In [42]: np.logaddexp2(52.999, 0) > >> Out[42]: 52.999000000000002 > >> > >> In [43]: np.logaddexp2(53, 0) > >> Out[43]: nan > >> > >> It looks to me like perhaps the NaNs are appearing when the smaller > >> term affects only the "extra" bits provided by the FPU's internal > >> larger-than-double representation. Some such issue would explain why > >> the problem seems to be hardware- and compiler-dependent. > >> > > > > Hmm, that 63.999 is kinda strange. Here is a bit of code that might > confuse > > a compiler working with different size mantissas: > > > > @type@ npy_log2_1p at c@(@type@ x) > > { > > @type@ u = 1 + x; > > if (u == 1) { > > return LOG2E*x; > > } else { > > return npy_log2 at c@(u) * x / (u - 1); > > } > > } > > > > It might be that u != 1 does not imply u-1 != 0. > > That does indeed look highly suspicious. I'm not entirely sure how to > work around it. GSL uses a volatile declaration: > > http://www.google.ca/codesearch/p?hl=en#p9nGS4eQGUI/gnu/gsl/gsl-1.8.tar.gz%7C8VCQSLJ5jR8/gsl-1.8/sys/log1p.c&q=log1p > On the other hand boost declares itself defeated by optimizing > compilers and uses a Taylor series: > > http://www.google.ca/codesearch/p?hl=en#sdP2GRSfgKo/dcplusplus/trunk/boost/boost/math/special_functions/log1p.hpp&q=log1p&sa=N&cd=7&ct=rc > While R makes no mention of the corrected formula or optimizing > compilers but takes the same approach, only with Chebyshev series: > > http://www.google.ca/codesearch/p?hl=en#gBBSWbwZmuk/src/base/R-2/R-2.3.1.tar.gz%7CVuh8XhRbUi8/R-2.3.1/src/nmath/log1p.c&q=log1p > > Since, at least on my machine, ordinary log1p appears to work fine, is > there any reason not to have log2_1p call it and scale the result? Or > does the compiler make a hash of our log1p too? > > Calling log1p and scaling looks like the right thing to do here. And our log1p needs improvement. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From jlewi at intellisis.com Thu Apr 1 11:52:44 2010 From: jlewi at intellisis.com (Jeremy Lewi) Date: Thu, 1 Apr 2010 08:52:44 -0700 Subject: [Numpy-discussion] Re Dealloat Numy arrays in C In-Reply-To: References: Message-ID: <000301cad1b3$5f6d4bf0$1e47e3d0$@com> Thanks Robert. I ended up letting Python handle the memory management. In my C-extension I just call Py_INCREF to prevent Python from deallocating it while I'm using it. I then call Py_DECREF once I'm done with it. Jeremy From charlesr.harris at gmail.com Thu Apr 1 13:38:03 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 1 Apr 2010 11:38:03 -0600 Subject: [Numpy-discussion] Bug in logaddexp2.reduce In-Reply-To: References: Message-ID: On Thu, Apr 1, 2010 at 8:37 AM, Charles R Harris wrote: > > > On Thu, Apr 1, 2010 at 12:46 AM, Anne Archibald > wrote: > >> On 1 April 2010 02:24, Charles R Harris >> wrote: >> > >> > >> > On Thu, Apr 1, 2010 at 12:04 AM, Anne Archibald < >> peridot.faceted at gmail.com> >> > wrote: >> >> >> >> On 1 April 2010 01:59, Charles R Harris >> wrote: >> >> > >> >> > >> >> > On Wed, Mar 31, 2010 at 11:46 PM, Anne Archibald >> >> > >> >> > wrote: >> >> >> >> >> >> On 1 April 2010 01:40, Charles R Harris >> >> >> wrote: >> >> >> > >> >> >> > >> >> >> > On Wed, Mar 31, 2010 at 11:25 PM, wrote: >> >> >> >> >> >> >> >> On Thu, Apr 1, 2010 at 1:22 AM, wrote: >> >> >> >> > On Thu, Apr 1, 2010 at 1:17 AM, Charles R Harris >> >> >> >> > wrote: >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> On Wed, Mar 31, 2010 at 6:08 PM, >> wrote: >> >> >> >> >>> >> >> >> >> >>> On Wed, Mar 31, 2010 at 7:37 PM, Warren Weckesser >> >> >> >> >>> wrote: >> >> >> >> >>> > T J wrote: >> >> >> >> >>> >> On Wed, Mar 31, 2010 at 1:21 PM, Charles R Harris >> >> >> >> >>> >> wrote: >> >> >> >> >>> >> >> >> >> >> >>> >>> Looks like roundoff error. >> >> >> >> >>> >>> >> >> >> >> >>> >>> >> >> >> >> >>> >> >> >> >> >> >>> >> So this is "expected" behavior? >> >> >> >> >>> >> >> >> >> >> >>> >> In [1]: np.logaddexp2(-1.5849625007211563, >> >> >> >> >>> >> -53.584962500721154) >> >> >> >> >>> >> Out[1]: -1.5849625007211561 >> >> >> >> >>> >> >> >> >> >> >>> >> In [2]: np.logaddexp2(-0.5849625007211563, >> >> >> >> >>> >> -53.584962500721154) >> >> >> >> >>> >> Out[2]: nan >> >> >> >> >>> >> >> >> >> >> >>> > >> >> >> >> >>> > Is any able to reproduce this? I don't get 'nan' in either >> >> >> >> >>> > 1.4.0 >> >> >> >> >>> > or >> >> >> >> >>> > 2.0.0.dev8313 (32 bit Mac OSX). In an earlier email T J >> >> >> >> >>> > reported >> >> >> >> >>> > using >> >> >> >> >>> > 1.5.0.dev8106. >> >> >> >> >>> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >> >>> >>> np.logaddexp2(-0.5849625007211563, -53.584962500721154) >> >> >> >> >>> nan >> >> >> >> >>> >>> np.logaddexp2(-1.5849625007211563, -53.584962500721154) >> >> >> >> >>> -1.5849625007211561 >> >> >> >> >>> >> >> >> >> >>> >>> np.version.version >> >> >> >> >>> '1.4.0' >> >> >> >> >>> >> >> >> >> >>> WindowsXP 32 >> >> >> >> >>> >> >> >> >> >> >> >> >> >> >> What compiler? Mingw? >> >> >> >> > >> >> >> >> > yes, mingw 3.4.5. , official binaries release 1.4.0 by David >> >> >> >> >> >> >> >> sse2 Pentium M >> >> >> >> >> >> >> > >> >> >> > Can you try the exp2/log2 functions with the problem data and see >> if >> >> >> > something goes wrong? >> >> >> >> >> >> Works fine for me. >> >> >> >> >> >> If it helps clarify things, the difference between the two problem >> >> >> input values is exactly 53 (and that's what logaddexp2 does an exp2 >> >> >> of); so I can provide a simpler example: >> >> >> >> >> >> In [23]: np.logaddexp2(0, -53) >> >> >> Out[23]: nan >> >> >> >> >> >> Of course, for me it fails in both orders. >> >> >> >> >> > >> >> > Ah, that's progress then ;) The effective number of bits in a double >> is >> >> > 53 >> >> > (52 + implicit bit). That shouldn't cause problems but it sure looks >> >> > suspicious. >> >> >> >> Indeed, that's what led me to the totally wrong suspicion that >> >> denormals have something to do with the problem. More data points: >> >> >> >> In [38]: np.logaddexp2(63.999, 0) >> >> Out[38]: nan >> >> >> >> In [39]: np.logaddexp2(64, 0) >> >> Out[39]: 64.0 >> >> >> >> In [42]: np.logaddexp2(52.999, 0) >> >> Out[42]: 52.999000000000002 >> >> >> >> In [43]: np.logaddexp2(53, 0) >> >> Out[43]: nan >> >> >> >> It looks to me like perhaps the NaNs are appearing when the smaller >> >> term affects only the "extra" bits provided by the FPU's internal >> >> larger-than-double representation. Some such issue would explain why >> >> the problem seems to be hardware- and compiler-dependent. >> >> >> > >> > Hmm, that 63.999 is kinda strange. Here is a bit of code that might >> confuse >> > a compiler working with different size mantissas: >> > >> > @type@ npy_log2_1p at c@(@type@ x) >> > { >> > @type@ u = 1 + x; >> > if (u == 1) { >> > return LOG2E*x; >> > } else { >> > return npy_log2 at c@(u) * x / (u - 1); >> > } >> > } >> > >> > It might be that u != 1 does not imply u-1 != 0. >> >> That does indeed look highly suspicious. I'm not entirely sure how to >> work around it. GSL uses a volatile declaration: >> >> http://www.google.ca/codesearch/p?hl=en#p9nGS4eQGUI/gnu/gsl/gsl-1.8.tar.gz%7C8VCQSLJ5jR8/gsl-1.8/sys/log1p.c&q=log1p >> On the other hand boost declares itself defeated by optimizing >> compilers and uses a Taylor series: >> >> http://www.google.ca/codesearch/p?hl=en#sdP2GRSfgKo/dcplusplus/trunk/boost/boost/math/special_functions/log1p.hpp&q=log1p&sa=N&cd=7&ct=rc >> While R makes no mention of the corrected formula or optimizing >> compilers but takes the same approach, only with Chebyshev series: >> >> http://www.google.ca/codesearch/p?hl=en#gBBSWbwZmuk/src/base/R-2/R-2.3.1.tar.gz%7CVuh8XhRbUi8/R-2.3.1/src/nmath/log1p.c&q=log1p >> >> Since, at least on my machine, ordinary log1p appears to work fine, is >> there any reason not to have log2_1p call it and scale the result? Or >> does the compiler make a hash of our log1p too? >> >> > Calling log1p and scaling looks like the right thing to do here. And our > log1p needs improvement. > > Tinkering a bit, I think we should implement the auxiliary function f(p) = log((1+p)/(1 - p)), which is antisymmetric and has the expansion 2p*(1 + p^2/3 + p^4/5 + ...). The series in the parens is increasing, so it is easy to terminate. Note that for p = +/- 1 it goes over to the harmonic series, so convergence is slow near the ends, but they can be handled using normal logs. Given 1 + x = (1+p)/(1-p) and solving for p gives p = x/(2 + x), so when x ranges from -1/2 -> 1/2, p ranges from -1/3 -> 1/5, hence achieving double precision should involve no more than about 17 terms. I think this is better than the expansion in R. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From mtrumpis at berkeley.edu Thu Apr 1 14:30:40 2010 From: mtrumpis at berkeley.edu (M Trumpis) Date: Thu, 1 Apr 2010 11:30:40 -0700 Subject: [Numpy-discussion] ufuncs on funny strides; also isnan, isfinite, etc on a variety of dtypes Message-ID: Hi all, disclaimer: pardon my vast ignorance on the subject of ufuncs, that will explain the naivety of the following questions This morning I was looking at this line of code, which was running quite slow for me and making me think data_has_nan = numpy.isnan(data_array).any() I knew that the array was of type uint8. This raised the question of how much slower would a function like isnan() run on integer typed data, and furthermore if it's integer typed data then why bother looking at the values at all? Actually I realized later that the main slow-down comes from the fact that my array was strided in fortran order (ascending strides). But from the point of view of a ufunc that is operating independently at each value, why would it need to respect striding? And a last mini question, it doesn't appear that any() is doing short circuit evaluation. It runs in appx the same time whether an array is sparsely nonzero, fully zero, or fully nonzero. Kind regards, Mike From pav at iki.fi Thu Apr 1 14:46:52 2010 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 01 Apr 2010 20:46:52 +0200 Subject: [Numpy-discussion] Ufunc memory access optimizations (Was: ufuncs on funny strides ...) In-Reply-To: References: Message-ID: <1270147613.4025.26.camel@talisman> to, 2010-04-01 kello 11:30 -0700, M Trumpis kirjoitti: [clip] > Actually I realized later that the main slow-down comes from the fact > that my array was strided in fortran order (ascending strides). But > from the point of view of a ufunc that is operating independently at > each value, why would it need to respect striding? Correct. There has been discussion about improving ufunc performance by optimizing the memory access pattern. The main issue in your case is that the output array is in C order, so that it is *not* possible to access both the input and the output arrays in the optimal order. Fixing this issue requires allowing ufuncs to allocate arrays that are in non-C order. This needs a design decision that has not so far been made. I'd be for this, I don't think it can break anything. The second issue is that there is no universal access pattern choice for every case that is optimal on all processor cache layouts. This forces to use heuristics to determine the access pattern, which is not so simple to get right, and usually would require some information of the processor's cache architecture. (Even some code has been written along these lines, though mostly addressing the reduction: http://github.com/pv/numpy-work/tree/ticket/1143-speedup-reduce http://projects.scipy.org/numpy/ticket/1143 Not production quality so far, and the non-C-output order would definitely help also here.) -- Pauli Virtanen From warren.weckesser at enthought.com Thu Apr 1 14:53:39 2010 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Thu, 01 Apr 2010 13:53:39 -0500 Subject: [Numpy-discussion] ufuncs on funny strides; also isnan, isfinite, etc on a variety of dtypes In-Reply-To: References: Message-ID: <4BB4EBB3.4030607@enthought.com> M Trumpis wrote: > Hi all, > > > > And a last mini question, it doesn't appear that any() is doing short > circuit evaluation. It runs in appx the same time whether an array is > sparsely nonzero, fully zero, or fully nonzero. > > That's not what I see. Here's an example that shows the a linear increase in time: In [112]: x = np.zeros((1000,1000), dtype=bool) In [113]: y = np.zeros((1000,1000), dtype=bool) In [114]: z = np.zeros((1000,1000), dtype=bool) In [115]: x[0,0] = 1 In [116]: y[500,0] = 1 In [117]: z[-1,-1] = 1 In [118]: %timeit x.any() 100000 loops, best of 3: 4 us per loop In [119]: %timeit y.any() 1000 loops, best of 3: 566 us per loop In [120]: %timeit z.any() 1000 loops, best of 3: 1.13 ms per loop Warren > Kind regards, > Mike > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From mtrumpis at berkeley.edu Thu Apr 1 15:38:27 2010 From: mtrumpis at berkeley.edu (M Trumpis) Date: Thu, 1 Apr 2010 12:38:27 -0700 Subject: [Numpy-discussion] ufuncs on funny strides; also isnan, isfinite, etc on a variety of dtypes In-Reply-To: <4BB4EBB3.4030607@enthought.com> References: <4BB4EBB3.4030607@enthought.com> Message-ID: On Thu, Apr 1, 2010 at 11:53 AM, Warren Weckesser wrote: > M Trumpis wrote: >> Hi all, >> >> >> >> And a last mini question, it doesn't appear that any() is doing short >> circuit evaluation. It runs in appx the same time whether an array is >> sparsely nonzero, fully zero, or fully nonzero. >> >> > > That's not what I see. ?Here's an example that shows the a linear > increase in time: > > In [112]: x = np.zeros((1000,1000), dtype=bool) > > In [113]: y = np.zeros((1000,1000), dtype=bool) > > In [114]: z = np.zeros((1000,1000), dtype=bool) > > In [115]: x[0,0] = 1 > > In [116]: y[500,0] = 1 > > In [117]: z[-1,-1] = 1 > > In [118]: %timeit x.any() > 100000 loops, best of 3: 4 us per loop > > In [119]: %timeit y.any() > 1000 loops, best of 3: 566 us per loop > > In [120]: %timeit z.any() > 1000 loops, best of 3: 1.13 ms per loop > Yes, you're quite right. I was confused and abusing the usage a little by calling any() on non-boolean arrays. It appears in that case there's some uniform overhead that can swamp the boolean any() Mike > > Warren > >> Kind regards, >> Mike >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From friedrichromstedt at gmail.com Thu Apr 1 16:59:18 2010 From: friedrichromstedt at gmail.com (Friedrich Romstedt) Date: Thu, 1 Apr 2010 22:59:18 +0200 Subject: [Numpy-discussion] "Match" two arrays In-Reply-To: References: Message-ID: 2010/4/1 Shailendra : > Hi All, > I want to make a function which should be like this [...] We had a thread some time ago, very long and with lots of working code, which seems to me related to this: Thread title: "Iterative Matrix Multiplication" (do a search under "Bug Reports" on numpy.scipy.org) It is about 3d points and selection based on criterions and so on. Fwiw, Friedrich From peridot.faceted at gmail.com Thu Apr 1 17:23:11 2010 From: peridot.faceted at gmail.com (Anne Archibald) Date: Thu, 1 Apr 2010 17:23:11 -0400 Subject: [Numpy-discussion] ufuncs on funny strides; also isnan, isfinite, etc on a variety of dtypes In-Reply-To: References: Message-ID: On 1 April 2010 14:30, M Trumpis wrote: > Hi all, > > disclaimer: pardon my vast ignorance on the subject of ufuncs, that > will explain the naivety of the following questions > > This morning I was looking at this line of code, which was running > quite slow for me and making me think > > data_has_nan = numpy.isnan(data_array).any() > > I knew that the array was of type uint8. This raised the question of > how much slower would a function like isnan() run on integer typed > data, and furthermore if it's integer typed data then why bother > looking at the values at all? Yes, in fact there is no NaN for integer types, so this will never come out True. > Actually I realized later that the main slow-down comes from the fact > that my array was strided in fortran order (ascending strides). But > from the point of view of a ufunc that is operating independently at > each value, why would it need to respect striding? This is a subject that has come up a number of times. In an ideal world, numpy would be free to run ufuncs in a cache-optimal order. As it stands, though, numpy does some limited and possibly ill-advised reordering, and in fact the result of certain operations can depend on the order in which numpy carries out an operation, so it's kind of a pain to fix this. But we'd like to. The conclusion seems to be to do it in two steps: first, declare by fiat that all operations reading from and writing to the same memory return the same result as if the result were saved to a new array and then copied into the result. This changes the result of certain currently ill-defined operations, and requires real copies in some cases, but would remove undefined behaviour. The second step, once we have the freedom to reorder operations, would be to write some ufunc reordering code that flattens arrays as much as possible and loops over cache-coherent strides first if possible. Somebody needs to write the code, though. Anne > And a last mini question, it doesn't appear that any() is doing short > circuit evaluation. It runs in appx the same time whether an array is > sparsely nonzero, fully zero, or fully nonzero. > > Kind regards, > Mike > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From peridot.faceted at gmail.com Thu Apr 1 17:51:45 2010 From: peridot.faceted at gmail.com (Anne Archibald) Date: Thu, 1 Apr 2010 17:51:45 -0400 Subject: [Numpy-discussion] Bug in logaddexp2.reduce In-Reply-To: References: Message-ID: On 1 April 2010 13:38, Charles R Harris wrote: > > > On Thu, Apr 1, 2010 at 8:37 AM, Charles R Harris > wrote: >> >> >> On Thu, Apr 1, 2010 at 12:46 AM, Anne Archibald >> wrote: >>> >>> On 1 April 2010 02:24, Charles R Harris >>> wrote: >>> > >>> > >>> > On Thu, Apr 1, 2010 at 12:04 AM, Anne Archibald >>> > >>> > wrote: >>> >> >>> >> On 1 April 2010 01:59, Charles R Harris >>> >> wrote: >>> >> > >>> >> > >>> >> > On Wed, Mar 31, 2010 at 11:46 PM, Anne Archibald >>> >> > >>> >> > wrote: >>> >> >> >>> >> >> On 1 April 2010 01:40, Charles R Harris >>> >> >> wrote: >>> >> >> > >>> >> >> > >>> >> >> > On Wed, Mar 31, 2010 at 11:25 PM, wrote: >>> >> >> >> >>> >> >> >> On Thu, Apr 1, 2010 at 1:22 AM, ? wrote: >>> >> >> >> > On Thu, Apr 1, 2010 at 1:17 AM, Charles R Harris >>> >> >> >> > wrote: >>> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >> On Wed, Mar 31, 2010 at 6:08 PM, >>> >> >> >> >> wrote: >>> >> >> >> >>> >>> >> >> >> >>> On Wed, Mar 31, 2010 at 7:37 PM, Warren Weckesser >>> >> >> >> >>> wrote: >>> >> >> >> >>> > T J wrote: >>> >> >> >> >>> >> On Wed, Mar 31, 2010 at 1:21 PM, Charles R Harris >>> >> >> >> >>> >> wrote: >>> >> >> >> >>> >> >>> >> >> >> >>> >>> Looks like roundoff error. >>> >> >> >> >>> >>> >>> >> >> >> >>> >>> >>> >> >> >> >>> >> >>> >> >> >> >>> >> So this is "expected" behavior? >>> >> >> >> >>> >> >>> >> >> >> >>> >> In [1]: np.logaddexp2(-1.5849625007211563, >>> >> >> >> >>> >> -53.584962500721154) >>> >> >> >> >>> >> Out[1]: -1.5849625007211561 >>> >> >> >> >>> >> >>> >> >> >> >>> >> In [2]: np.logaddexp2(-0.5849625007211563, >>> >> >> >> >>> >> -53.584962500721154) >>> >> >> >> >>> >> Out[2]: nan >>> >> >> >> >>> >> >>> >> >> >> >>> > >>> >> >> >> >>> > Is any able to reproduce this? ?I don't get 'nan' in >>> >> >> >> >>> > either >>> >> >> >> >>> > 1.4.0 >>> >> >> >> >>> > or >>> >> >> >> >>> > 2.0.0.dev8313 (32 bit Mac OSX). ?In an earlier email T J >>> >> >> >> >>> > reported >>> >> >> >> >>> > using >>> >> >> >> >>> > 1.5.0.dev8106. >>> >> >> >> >>> >>> >> >> >> >>> >>> >> >> >> >>> >>> >> >> >> >>> >>> np.logaddexp2(-0.5849625007211563, -53.584962500721154) >>> >> >> >> >>> nan >>> >> >> >> >>> >>> np.logaddexp2(-1.5849625007211563, -53.584962500721154) >>> >> >> >> >>> -1.5849625007211561 >>> >> >> >> >>> >>> >> >> >> >>> >>> np.version.version >>> >> >> >> >>> '1.4.0' >>> >> >> >> >>> >>> >> >> >> >>> WindowsXP 32 >>> >> >> >> >>> >>> >> >> >> >> >>> >> >> >> >> What compiler? Mingw? >>> >> >> >> > >>> >> >> >> > yes, mingw 3.4.5. , official binaries release 1.4.0 by David >>> >> >> >> >>> >> >> >> sse2 Pentium M >>> >> >> >> >>> >> >> > >>> >> >> > Can you try the exp2/log2 functions with the problem data and see >>> >> >> > if >>> >> >> > something goes wrong? >>> >> >> >>> >> >> Works fine for me. >>> >> >> >>> >> >> If it helps clarify things, the difference between the two problem >>> >> >> input values is exactly 53 (and that's what logaddexp2 does an exp2 >>> >> >> of); so I can provide a simpler example: >>> >> >> >>> >> >> In [23]: np.logaddexp2(0, -53) >>> >> >> Out[23]: nan >>> >> >> >>> >> >> Of course, for me it fails in both orders. >>> >> >> >>> >> > >>> >> > Ah, that's progress then ;) The effective number of bits in a double >>> >> > is >>> >> > 53 >>> >> > (52 + implicit bit). That shouldn't cause problems but it sure looks >>> >> > suspicious. >>> >> >>> >> Indeed, that's what led me to the totally wrong suspicion that >>> >> denormals have something to do with the problem. More data points: >>> >> >>> >> In [38]: np.logaddexp2(63.999, 0) >>> >> Out[38]: nan >>> >> >>> >> In [39]: np.logaddexp2(64, 0) >>> >> Out[39]: 64.0 >>> >> >>> >> In [42]: np.logaddexp2(52.999, 0) >>> >> Out[42]: 52.999000000000002 >>> >> >>> >> In [43]: np.logaddexp2(53, 0) >>> >> Out[43]: nan >>> >> >>> >> It looks to me like perhaps the NaNs are appearing when the smaller >>> >> term affects only the "extra" bits provided by the FPU's internal >>> >> larger-than-double representation. Some such issue would explain why >>> >> the problem seems to be hardware- and compiler-dependent. >>> >> >>> > >>> > Hmm, that 63.999 is kinda strange. Here is a bit of code that might >>> > confuse >>> > a compiler working with different size mantissas: >>> > >>> > @type@ npy_log2_1p at c@(@type@ x) >>> > { >>> > ??? @type@ u = 1 + x; >>> > ??? if (u == 1) { >>> > ??????? return LOG2E*x; >>> > ??? } else { >>> > ??????? return npy_log2 at c@(u) * x / (u - 1); >>> > ??? } >>> > } >>> > >>> > It might be that u != 1 does not imply u-1 != 0. >>> >>> That does indeed look highly suspicious. I'm not entirely sure how to >>> work around it. GSL uses a volatile declaration: >>> >>> http://www.google.ca/codesearch/p?hl=en#p9nGS4eQGUI/gnu/gsl/gsl-1.8.tar.gz%7C8VCQSLJ5jR8/gsl-1.8/sys/log1p.c&q=log1p >>> On the other hand boost declares itself defeated by optimizing >>> compilers and uses a Taylor series: >>> >>> http://www.google.ca/codesearch/p?hl=en#sdP2GRSfgKo/dcplusplus/trunk/boost/boost/math/special_functions/log1p.hpp&q=log1p&sa=N&cd=7&ct=rc >>> While R makes no mention of the corrected formula or optimizing >>> compilers but takes the same approach, only with Chebyshev series: >>> >>> http://www.google.ca/codesearch/p?hl=en#gBBSWbwZmuk/src/base/R-2/R-2.3.1.tar.gz%7CVuh8XhRbUi8/R-2.3.1/src/nmath/log1p.c&q=log1p >>> >>> Since, at least on my machine, ordinary log1p appears to work fine, is >>> there any reason not to have log2_1p call it and scale the result? Or >>> does the compiler make a hash of our log1p too? >>> >> >> Calling log1p and scaling looks like the right thing to do here. And our >> log1p needs improvement. >> > > Tinkering a bit, I think we should implement the auxiliary function f(p) = > log((1+p)/(1 - p)), which is antisymmetric and has the expansion 2p*(1 + > p^2/3 + p^4/5 + ...). The series in the parens is increasing, so it is easy > to terminate. Note that for p = +/- 1 it goes over to the harmonic series, > so convergence is slow near the ends, but they can be handled using normal > logs. Given 1 + x = (1+p)/(1-p) and solving for p gives p = x/(2 + x), so > when x ranges from -1/2 -> 1/2, p ranges from -1/3 -> 1/5, hence achieving > double precision should involve no more than? about 17 terms. I think this > is better than the expansion in R. First I guess we should check which systems don't have log1p; if glibc has it as an intrinsic, that should cover Linux (though I suppose we should check its quality). Do Windows and Mac have log1p? For testing I suppose we should make our own implementation somehow available even on systems where it's unnecessary. Power series are certainly easy, but would some of the other available tricks - Chebyshev series or rational function approximations - be better? I notice R uses Chebyshev series, although maybe that's just because they have a good evaluator handy. Anne > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From ranavishal at gmail.com Thu Apr 1 19:55:14 2010 From: ranavishal at gmail.com (Vishal Rana) Date: Thu, 1 Apr 2010 16:55:14 -0700 Subject: [Numpy-discussion] Can this be done more efficiently using numpy? In-Reply-To: <1270110061.2845.20.camel@homer.dipole.lan> References: <1270110061.2845.20.camel@homer.dipole.lan> Message-ID: Thanks Didrik! On Thu, Apr 1, 2010 at 1:21 AM, Didrik Pinte wrote: > On Wed, 2010-03-31 at 23:13 -0700, Vishal Rana wrote: > > Hi, > > > > > > A calculation which goes like this... > > > > > > n = 5 > > a = np.arange(1000) > > b = np.arange(n - 1, 1000) > > > > > > l = [] > > for i in range(b.size): > > # Absolute difference of n a elements and nth b element > > x = np.abs(a[i:i + n] - b[i]) > > > > # Average of x > > y = x.sum() / n > > > > l.append(y) > > > > > > It takes a while if I have like 200K records! > > Is there a more efficient way to do this using numpy? > > Like this ? > > import numpy > from numpy.lib.stride_tricks import as_strided > n = 5 > a = numpy.arange(1000) > b = numpy.arange(n - 1, 1000) > > > def sum_1(n, a, b): > # original method > l = [] > for i in range(b.size): > # Absolute difference of n a elements and nth b element > x = numpy.abs(a[i:i + n] - b[i]) > > # Average of x > y = x.sum() / n > > l.append(y) > return l > > def sum_numpy(n, a, b): > ast = as_strided(a, shape=(1000-n+1, n), strides=(8,8)) > diff = numpy.abs(ast - b[:,None]) > y = diff.sum(axis=1) /n > return y > > test1 = sum_1(n,a,b) > test2 = sum_numpy(n,a,b) > > assert(numpy.alltrue(test2 == test1)) > > -- Didrik > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From d.l.goldsmith at gmail.com Thu Apr 1 21:42:23 2010 From: d.l.goldsmith at gmail.com (David Goldsmith) Date: Thu, 1 Apr 2010 18:42:23 -0700 Subject: [Numpy-discussion] Asymmetry in Chebyshev.deriv v. Chebyshev.integ Message-ID: >>> np.version.version '1.4.0' >>> c = np.polynomial.chebyshev.Chebyshev(1) >>> c.deriv(1.0) Chebyshev([ 0.], [-1., 1.]) >>> c.integ(1.0) Traceback (most recent call last): File "", line 1, in File "", line 441, in integ File "C:\Python26\lib\site-packages\numpy\polynomial\chebyshev.py", line 739, in chebint k = list(k) + [0]*(m - len(k)) TypeError: can't multiply sequence by non-int of type 'float' >>> c.integ(1) Chebyshev([ 0., 1.], [-1., 1.]) i.e., deriv accepts int_like input but integ doesn't. Given the way I just embarrassed myself on the scipy-dev list :-(, I'm confirming this is a bug before I file a ticket. DG -- Mathematician: noun, someone who disavows certainty when their uncertainty set is non-empty, even if that set has measure zero. -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Apr 1 21:56:09 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 1 Apr 2010 19:56:09 -0600 Subject: [Numpy-discussion] Bug in logaddexp2.reduce In-Reply-To: References: Message-ID: On Thu, Apr 1, 2010 at 3:51 PM, Anne Archibald wrote: > On 1 April 2010 13:38, Charles R Harris wrote: > > > > > > On Thu, Apr 1, 2010 at 8:37 AM, Charles R Harris < > charlesr.harris at gmail.com> > > wrote: > >> > >> > >> On Thu, Apr 1, 2010 at 12:46 AM, Anne Archibald > >> wrote: > >>> > >>> On 1 April 2010 02:24, Charles R Harris > >>> wrote: > >>> > > >>> > > >>> > On Thu, Apr 1, 2010 at 12:04 AM, Anne Archibald > >>> > > >>> > wrote: > >>> >> > >>> >> On 1 April 2010 01:59, Charles R Harris > >>> >> wrote: > >>> >> > > >>> >> > > >>> >> > On Wed, Mar 31, 2010 at 11:46 PM, Anne Archibald > >>> >> > > >>> >> > wrote: > >>> >> >> > >>> >> >> On 1 April 2010 01:40, Charles R Harris < > charlesr.harris at gmail.com> > >>> >> >> wrote: > >>> >> >> > > >>> >> >> > > >>> >> >> > On Wed, Mar 31, 2010 at 11:25 PM, > wrote: > >>> >> >> >> > >>> >> >> >> On Thu, Apr 1, 2010 at 1:22 AM, > wrote: > >>> >> >> >> > On Thu, Apr 1, 2010 at 1:17 AM, Charles R Harris > >>> >> >> >> > wrote: > >>> >> >> >> >> > >>> >> >> >> >> > >>> >> >> >> >> On Wed, Mar 31, 2010 at 6:08 PM, > >>> >> >> >> >> wrote: > >>> >> >> >> >>> > >>> >> >> >> >>> On Wed, Mar 31, 2010 at 7:37 PM, Warren Weckesser > >>> >> >> >> >>> wrote: > >>> >> >> >> >>> > T J wrote: > >>> >> >> >> >>> >> On Wed, Mar 31, 2010 at 1:21 PM, Charles R Harris > >>> >> >> >> >>> >> wrote: > >>> >> >> >> >>> >> > >>> >> >> >> >>> >>> Looks like roundoff error. > >>> >> >> >> >>> >>> > >>> >> >> >> >>> >>> > >>> >> >> >> >>> >> > >>> >> >> >> >>> >> So this is "expected" behavior? > >>> >> >> >> >>> >> > >>> >> >> >> >>> >> In [1]: np.logaddexp2(-1.5849625007211563, > >>> >> >> >> >>> >> -53.584962500721154) > >>> >> >> >> >>> >> Out[1]: -1.5849625007211561 > >>> >> >> >> >>> >> > >>> >> >> >> >>> >> In [2]: np.logaddexp2(-0.5849625007211563, > >>> >> >> >> >>> >> -53.584962500721154) > >>> >> >> >> >>> >> Out[2]: nan > >>> >> >> >> >>> >> > >>> >> >> >> >>> > > >>> >> >> >> >>> > Is any able to reproduce this? I don't get 'nan' in > >>> >> >> >> >>> > either > >>> >> >> >> >>> > 1.4.0 > >>> >> >> >> >>> > or > >>> >> >> >> >>> > 2.0.0.dev8313 (32 bit Mac OSX). In an earlier email T J > >>> >> >> >> >>> > reported > >>> >> >> >> >>> > using > >>> >> >> >> >>> > 1.5.0.dev8106. > >>> >> >> >> >>> > >>> >> >> >> >>> > >>> >> >> >> >>> > >>> >> >> >> >>> >>> np.logaddexp2(-0.5849625007211563, > -53.584962500721154) > >>> >> >> >> >>> nan > >>> >> >> >> >>> >>> np.logaddexp2(-1.5849625007211563, > -53.584962500721154) > >>> >> >> >> >>> -1.5849625007211561 > >>> >> >> >> >>> > >>> >> >> >> >>> >>> np.version.version > >>> >> >> >> >>> '1.4.0' > >>> >> >> >> >>> > >>> >> >> >> >>> WindowsXP 32 > >>> >> >> >> >>> > >>> >> >> >> >> > >>> >> >> >> >> What compiler? Mingw? > >>> >> >> >> > > >>> >> >> >> > yes, mingw 3.4.5. , official binaries release 1.4.0 by David > >>> >> >> >> > >>> >> >> >> sse2 Pentium M > >>> >> >> >> > >>> >> >> > > >>> >> >> > Can you try the exp2/log2 functions with the problem data and > see > >>> >> >> > if > >>> >> >> > something goes wrong? > >>> >> >> > >>> >> >> Works fine for me. > >>> >> >> > >>> >> >> If it helps clarify things, the difference between the two > problem > >>> >> >> input values is exactly 53 (and that's what logaddexp2 does an > exp2 > >>> >> >> of); so I can provide a simpler example: > >>> >> >> > >>> >> >> In [23]: np.logaddexp2(0, -53) > >>> >> >> Out[23]: nan > >>> >> >> > >>> >> >> Of course, for me it fails in both orders. > >>> >> >> > >>> >> > > >>> >> > Ah, that's progress then ;) The effective number of bits in a > double > >>> >> > is > >>> >> > 53 > >>> >> > (52 + implicit bit). That shouldn't cause problems but it sure > looks > >>> >> > suspicious. > >>> >> > >>> >> Indeed, that's what led me to the totally wrong suspicion that > >>> >> denormals have something to do with the problem. More data points: > >>> >> > >>> >> In [38]: np.logaddexp2(63.999, 0) > >>> >> Out[38]: nan > >>> >> > >>> >> In [39]: np.logaddexp2(64, 0) > >>> >> Out[39]: 64.0 > >>> >> > >>> >> In [42]: np.logaddexp2(52.999, 0) > >>> >> Out[42]: 52.999000000000002 > >>> >> > >>> >> In [43]: np.logaddexp2(53, 0) > >>> >> Out[43]: nan > >>> >> > >>> >> It looks to me like perhaps the NaNs are appearing when the smaller > >>> >> term affects only the "extra" bits provided by the FPU's internal > >>> >> larger-than-double representation. Some such issue would explain why > >>> >> the problem seems to be hardware- and compiler-dependent. > >>> >> > >>> > > >>> > Hmm, that 63.999 is kinda strange. Here is a bit of code that might > >>> > confuse > >>> > a compiler working with different size mantissas: > >>> > > >>> > @type@ npy_log2_1p at c@(@type@ x) > >>> > { > >>> > @type@ u = 1 + x; > >>> > if (u == 1) { > >>> > return LOG2E*x; > >>> > } else { > >>> > return npy_log2 at c@(u) * x / (u - 1); > >>> > } > >>> > } > >>> > > >>> > It might be that u != 1 does not imply u-1 != 0. > >>> > >>> That does indeed look highly suspicious. I'm not entirely sure how to > >>> work around it. GSL uses a volatile declaration: > >>> > >>> > http://www.google.ca/codesearch/p?hl=en#p9nGS4eQGUI/gnu/gsl/gsl-1.8.tar.gz%7C8VCQSLJ5jR8/gsl-1.8/sys/log1p.c&q=log1p > >>> On the other hand boost declares itself defeated by optimizing > >>> compilers and uses a Taylor series: > >>> > >>> > http://www.google.ca/codesearch/p?hl=en#sdP2GRSfgKo/dcplusplus/trunk/boost/boost/math/special_functions/log1p.hpp&q=log1p&sa=N&cd=7&ct=rc > >>> While R makes no mention of the corrected formula or optimizing > >>> compilers but takes the same approach, only with Chebyshev series: > >>> > >>> > http://www.google.ca/codesearch/p?hl=en#gBBSWbwZmuk/src/base/R-2/R-2.3.1.tar.gz%7CVuh8XhRbUi8/R-2.3.1/src/nmath/log1p.c&q=log1p > >>> > >>> Since, at least on my machine, ordinary log1p appears to work fine, is > >>> there any reason not to have log2_1p call it and scale the result? Or > >>> does the compiler make a hash of our log1p too? > >>> > >> > >> Calling log1p and scaling looks like the right thing to do here. And our > >> log1p needs improvement. > >> > > > > Tinkering a bit, I think we should implement the auxiliary function f(p) > = > > log((1+p)/(1 - p)), which is antisymmetric and has the expansion 2p*(1 + > > p^2/3 + p^4/5 + ...). The series in the parens is increasing, so it is > easy > > to terminate. Note that for p = +/- 1 it goes over to the harmonic > series, > > so convergence is slow near the ends, but they can be handled using > normal > > logs. Given 1 + x = (1+p)/(1-p) and solving for p gives p = x/(2 + x), so > > when x ranges from -1/2 -> 1/2, p ranges from -1/3 -> 1/5, hence > achieving > > double precision should involve no more than about 17 terms. I think > this > > is better than the expansion in R. > > First I guess we should check which systems don't have log1p; if glibc > has it as an intrinsic, that should cover Linux (though I suppose we > should check its quality). Do Windows and Mac have log1p? For testing > I suppose we should make our own implementation somehow available even > on systems where it's unnecessary. > > Power series are certainly easy, but would some of the other available > tricks - Chebyshev series or rational function approximations - be > better? I notice R uses Chebyshev series, although maybe that's just > because they have a good evaluator handy. > > The Chebyshev series for 1 + x^2/3 + ... is just as bad as the one in R, i.e., stinky. Rational approximation works well, the ratio of two tenth order polynomials is good to 1e-32 (quad precision) over the range of interest. I'd like to use continued fractions, though, so the approximation could terminate early for small values of x. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at silveregg.co.jp Thu Apr 1 21:59:11 2010 From: david at silveregg.co.jp (David Cournapeau) Date: Fri, 02 Apr 2010 10:59:11 +0900 Subject: [Numpy-discussion] Bug in logaddexp2.reduce In-Reply-To: References: Message-ID: <4BB54F6F.2070302@silveregg.co.jp> Anne Archibald wrote: > > First I guess we should check which systems don't have log1p This is already done - we do use the system log1p on linux (but note that log2_1p is not standard AFAIK). I would guess few systems outside windows don't have log1p, given that msun has an implementation, David From charlesr.harris at gmail.com Thu Apr 1 22:16:45 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 1 Apr 2010 20:16:45 -0600 Subject: [Numpy-discussion] Bug in logaddexp2.reduce In-Reply-To: <4BB54F6F.2070302@silveregg.co.jp> References: <4BB54F6F.2070302@silveregg.co.jp> Message-ID: On Thu, Apr 1, 2010 at 7:59 PM, David Cournapeau wrote: > Anne Archibald wrote: > > > > > First I guess we should check which systems don't have log1p > > This is already done - we do use the system log1p on linux (but note > that log2_1p is not standard AFAIK). I would guess few systems outside > windows don't have log1p, given that msun has an implementation, > > I see that msun uses the same series I came to. However, my rational (Pade) approximation is a lot better than their polynomial. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Apr 1 23:01:36 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 1 Apr 2010 21:01:36 -0600 Subject: [Numpy-discussion] Bug in logaddexp2.reduce In-Reply-To: References: <4BB54F6F.2070302@silveregg.co.jp> Message-ID: On Thu, Apr 1, 2010 at 8:16 PM, Charles R Harris wrote: > > > On Thu, Apr 1, 2010 at 7:59 PM, David Cournapeau wrote: > >> Anne Archibald wrote: >> >> > >> > First I guess we should check which systems don't have log1p >> >> This is already done - we do use the system log1p on linux (but note >> that log2_1p is not standard AFAIK). I would guess few systems outside >> windows don't have log1p, given that msun has an implementation, >> >> > I see that msun uses the same series I came to. However, my rational (Pade) > approximation is a lot better than their polynomial. > > In fact I get better than 119 bits using the same range as sun and the ratio of two 7'th degree polynomials. I suspect it's better than that, but I only have mpmath set to that precision. Sun got 58 bits with a 14'th degree polynomial. So we can definitely improve on sun. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From shailendra.vikas at gmail.com Fri Apr 2 00:07:46 2010 From: shailendra.vikas at gmail.com (Shailendra) Date: Fri, 2 Apr 2010 00:07:46 -0400 Subject: [Numpy-discussion] Is this odd? Message-ID: Hi All, Below is some array behaviour which i think is odd >>> a=arange(10) >>> a array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) >>> b=nonzero(a<0) >>> b (array([], dtype=int32),) >>> if not b[0]: ... print 'b[0] is false' ... b[0] is false Above case the b[0] is empty so it is fine it is considered false >>> b=nonzero(a<1) >>> b (array([0]),) >>> if not b[0]: ... print 'b[0] is false' ... b[0] is false Above case b[0] is a non-empty array. Why should this be consider false. >>> b=nonzero(a>8) >>> b (array([9]),) >>> if not b[0]: ... print 'b[0] is false' ... >>> Above case b[0] is non-empty and should be consider true.Which it does. I don't understand why non-empty array should not be considered true irrespective to what value they have. Also, please suggest the best way to differentiate between an empty array and non-empty array( irrespective to what is inside array). Thanks, Shailendra From nathanielpeterson08 at gmail.com Fri Apr 2 01:08:49 2010 From: nathanielpeterson08 at gmail.com (Nathaniel Peterson) Date: Fri, 02 Apr 2010 01:08:49 -0400 Subject: [Numpy-discussion] Possible bug in indexed masked arrays Message-ID: <87wrwqfnku.fsf@gmail.com> Is this behavior of masked arrays intended, or is it a bug? This part works as I would expected: import numpy as np a=np.ma.fix_invalid(np.array([np.nan,-1,0,1])) b=np.ma.fix_invalid(np.array([np.nan,-1,0,1])) idx=(a==b) print(a[idx][3]) # 1.0 Note that a[idx] has shape (4,). But if I change the first element of b from np.nan to 2.0 then a[idx2] has shape (3,) despite np.alltrue(idx==idx2) being True: c=np.ma.fix_invalid(np.array([2.0,-1,0,1])) idx2=(a==c) assert(np.alltrue(idx==idx2)) a[idx2][3] # Traceback (most recent call last): # File "/home/np/test.py", line 12, in # a[idx2][3] # File "/usr/lib/python2.6/dist-packages/numpy/ma/core.py", line 2578, in __getitem__ # dout = ndarray.__getitem__(ndarray.view(self, ndarray), indx) # IndexError: index out of bounds I looked at repr(idx) and repr(idx2) and they appear to have equal values in all respects. If that is true, why should a[idx] be different than a[idx2]? From nadavh at visionsense.com Fri Apr 2 08:29:18 2010 From: nadavh at visionsense.com (Nadav Horesh) Date: Fri, 2 Apr 2010 15:29:18 +0300 Subject: [Numpy-discussion] Is this odd? References: Message-ID: <710F2847B0018641891D9A21602763605AD391@ex3.envision.co.il> In python empty sequences are always equivalent to False, and non-empty to True. You can use this property or: if len(b) > 0: ..... Nadav -----Original Message----- From: numpy-discussion-bounces at scipy.org on behalf of Shailendra Sent: Fri 02-Apr-10 06:07 To: numpy-discussion at scipy.org Subject: [Numpy-discussion] Is this odd? Hi All, Below is some array behaviour which i think is odd >>> a=arange(10) >>> a array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) >>> b=nonzero(a<0) >>> b (array([], dtype=int32),) >>> if not b[0]: ... print 'b[0] is false' ... b[0] is false Above case the b[0] is empty so it is fine it is considered false >>> b=nonzero(a<1) >>> b (array([0]),) >>> if not b[0]: ... print 'b[0] is false' ... b[0] is false Above case b[0] is a non-empty array. Why should this be consider false. >>> b=nonzero(a>8) >>> b (array([9]),) >>> if not b[0]: ... print 'b[0] is false' ... >>> Above case b[0] is non-empty and should be consider true.Which it does. I don't understand why non-empty array should not be considered true irrespective to what value they have. Also, please suggest the best way to differentiate between an empty array and non-empty array( irrespective to what is inside array). Thanks, Shailendra _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 3104 bytes Desc: not available URL: From aisaac at american.edu Fri Apr 2 09:38:44 2010 From: aisaac at american.edu (Alan G Isaac) Date: Fri, 02 Apr 2010 09:38:44 -0400 Subject: [Numpy-discussion] Is this odd? In-Reply-To: <710F2847B0018641891D9A21602763605AD391@ex3.envision.co.il> References: <710F2847B0018641891D9A21602763605AD391@ex3.envision.co.il> Message-ID: <4BB5F364.2080308@american.edu> On 4/2/2010 8:29 AM, Nadav Horesh wrote: > In python empty sequences are always equivalent to False, and non-empty to True. I think that was why the OP objected to this behavior: >>> bool(np.array([0])) False Alan Isaac From robert.kern at gmail.com Fri Apr 2 10:11:45 2010 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 2 Apr 2010 08:11:45 -0600 Subject: [Numpy-discussion] Is this odd? In-Reply-To: References: Message-ID: On Thu, Apr 1, 2010 at 22:07, Shailendra wrote: > Hi All, > Below is some array behaviour which i think is odd >>>> a=arange(10) >>>> a > array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) >>>> b=nonzero(a<0) >>>> b > (array([], dtype=int32),) >>>> if not b[0]: > ... ? ? print 'b[0] is false' > ... > b[0] is false > > Above case the b[0] is empty so it is fine it is considered false > >>>> b=nonzero(a<1) >>>> b > (array([0]),) >>>> if not b[0]: > ... ? ? print 'b[0] is false' > ... > b[0] is false > > Above case b[0] is a non-empty array. Why should this be consider false. > >>>> b=nonzero(a>8) >>>> b > (array([9]),) >>>> if not b[0]: > ... ? ? print 'b[0] is false' > ... >>>> > Above case b[0] is non-empty and should be consider true.Which it does. > > I don't understand why non-empty array should not be considered true > irrespective to what value they have. We raise an exception for most arrays. We used to evaluate bool(arr) as True if any element in arr were nonzero. However, people routinely got confused and thought that the behavior was that it would return False if any element were zero. It's ambiguous, so we raise an exception to prevent errors and misconceptions. Empty arrays are very rare, so we decided that evaluating bool(arr) as True if there were any elements had very little use case and would still have the potential to confuse people. As shown below, there is a better, more explicit way to test for emptiness. However, we do allow single-element arrays to evaluate to True or False based on its single element because that is unambiguous. This is possibly a mistake, but it tends not to cause too many problems. I don't know why we also allow bool(array([])) to evaluate to False, too; we probably shouldn't. > Also, please suggest the best way to differentiate between an empty > array and non-empty array( irrespective to what is inside array). a.size > 0 -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From josef.pktd at gmail.com Fri Apr 2 10:16:42 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 2 Apr 2010 10:16:42 -0400 Subject: [Numpy-discussion] Is this odd? In-Reply-To: References: Message-ID: On Fri, Apr 2, 2010 at 10:11 AM, Robert Kern wrote: > On Thu, Apr 1, 2010 at 22:07, Shailendra wrote: >> Hi All, >> Below is some array behaviour which i think is odd >>>>> a=arange(10) >>>>> a >> array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) >>>>> b=nonzero(a<0) >>>>> b >> (array([], dtype=int32),) >>>>> if not b[0]: >> ... ? ? print 'b[0] is false' >> ... >> b[0] is false >> >> Above case the b[0] is empty so it is fine it is considered false >> >>>>> b=nonzero(a<1) >>>>> b >> (array([0]),) >>>>> if not b[0]: >> ... ? ? print 'b[0] is false' >> ... >> b[0] is false >> >> Above case b[0] is a non-empty array. Why should this be consider false. >> >>>>> b=nonzero(a>8) >>>>> b >> (array([9]),) >>>>> if not b[0]: >> ... ? ? print 'b[0] is false' >> ... >>>>> >> Above case b[0] is non-empty and should be consider true.Which it does. >> >> I don't understand why non-empty array should not be considered true >> irrespective to what value they have. > > We raise an exception for most arrays. We used to evaluate bool(arr) > as True if any element in arr were nonzero. However, people routinely > got confused and thought that the behavior was that it would return > False if any element were zero. It's ambiguous, so we raise an > exception to prevent errors and misconceptions. Empty arrays are very > rare, so we decided that evaluating bool(arr) as True if there were > any elements had very little use case and would still have the > potential to confuse people. As shown below, there is a better, more > explicit way to test for emptiness. > > However, we do allow single-element arrays to evaluate to True or > False based on its single element because that is unambiguous. This is > possibly a mistake, but it tends not to cause too many problems. I > don't know why we also allow bool(array([])) to evaluate to False, > too; we probably shouldn't. > >> Also, please suggest the best way to differentiate between an empty >> array and non-empty array( irrespective to what is inside array). > > a.size > 0 > > -- > Robert Kern with examples: >>> bool(0) False >>> bool(1) True >>> bool(np.array([0])) False >>> bool(np.array([1])) True >>> bool([0]) True >>> bool([1]) True >>> bool(np.array([[0]])) False >>> bool(np.array([[1]])) True >>> np.array([[1]]).size 1 Josef > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > ?-- Umberto Eco > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From rmay31 at gmail.com Fri Apr 2 10:28:19 2010 From: rmay31 at gmail.com (Ryan May) Date: Fri, 2 Apr 2010 08:28:19 -0600 Subject: [Numpy-discussion] Is this odd? In-Reply-To: References: Message-ID: On Thu, Apr 1, 2010 at 10:07 PM, Shailendra wrote: > Hi All, > Below is some array behaviour which i think is odd >>>> a=arange(10) >>>> a > array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) >>>> b=nonzero(a<0) >>>> b > (array([], dtype=int32),) >>>> if not b[0]: > ... ? ? print 'b[0] is false' > ... > b[0] is false > > Above case the b[0] is empty so it is fine it is considered false > >>>> b=nonzero(a<1) >>>> b > (array([0]),) >>>> if not b[0]: > ... ? ? print 'b[0] is false' > ... > b[0] is false > > Above case b[0] is a non-empty array. Why should this be consider false. > >>>> b=nonzero(a>8) >>>> b > (array([9]),) >>>> if not b[0]: > ... ? ? print 'b[0] is false' > ... >>>> > Above case b[0] is non-empty and should be consider true.Which it does. > > I don't understand why non-empty array should not be considered true > irrespective to what value they have. > Also, please suggest the best way to differentiate between an empty > array and non-empty array( irrespective to what is inside array). But by using: if not b[0]: You're not considering the array as a whole, you're looking at the first element, which is giving expected results. As I'm sure you're aware, however, you can't simply do: if not b: # Raises exception So what you need to do is: if b.any(): or: if b.all() Now for determining empty or not, you'll need to look at len(b) or b.shape Ryan -- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma From robert.kern at gmail.com Fri Apr 2 10:31:59 2010 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 2 Apr 2010 08:31:59 -0600 Subject: [Numpy-discussion] Is this odd? In-Reply-To: References: Message-ID: On Fri, Apr 2, 2010 at 08:28, Ryan May wrote: > On Thu, Apr 1, 2010 at 10:07 PM, Shailendra wrote: >> Hi All, >> Below is some array behaviour which i think is odd >>>>> a=arange(10) >>>>> a >> array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) >>>>> b=nonzero(a<0) >>>>> b >> (array([], dtype=int32),) >>>>> if not b[0]: >> ... ? ? print 'b[0] is false' >> ... >> b[0] is false >> >> Above case the b[0] is empty so it is fine it is considered false >> >>>>> b=nonzero(a<1) >>>>> b >> (array([0]),) >>>>> if not b[0]: >> ... ? ? print 'b[0] is false' >> ... >> b[0] is false >> >> Above case b[0] is a non-empty array. Why should this be consider false. >> >>>>> b=nonzero(a>8) >>>>> b >> (array([9]),) >>>>> if not b[0]: >> ... ? ? print 'b[0] is false' >> ... >>>>> >> Above case b[0] is non-empty and should be consider true.Which it does. >> >> I don't understand why non-empty array should not be considered true >> irrespective to what value they have. >> Also, please suggest the best way to differentiate between an empty >> array and non-empty array( irrespective to what is inside array). > > But by using: > > if not b[0]: > > You're not considering the array as a whole, you're looking at the > first element, which is giving expected results. No, b is a tuple containing the array. b[0] is the array itself. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From gberbeglia at gmail.com Fri Apr 2 10:35:44 2010 From: gberbeglia at gmail.com (gerardob) Date: Fri, 2 Apr 2010 07:35:44 -0700 (PDT) Subject: [Numpy-discussion] Matrix operation. Message-ID: <28119663.post@talk.nabble.com> Let A be a square matrix of 0's and 1's, and let X be a one dimesional vector. The length of X is equal to the number of 1's that A has. I would like to produce a new matrix B by traversing the matrix A row by row and: 1- whenever i find a 0, set B in that position to zero. 2- whenever i find a 1, set B in that position with the ith value of X (where i represents the ith time i found a 1). Example. input: A=[[1,1,0], [1,0,0], [0,0,1]] X=[2,9,10,3] Output: B =[[2,9,0], [10,0,0], [0,0,3]] Which is an efficient way to accomplish this using numpy? Thanks. -- View this message in context: http://old.nabble.com/Matrix-operation.-tp28119663p28119663.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From shailendra.vikas at gmail.com Fri Apr 2 10:35:51 2010 From: shailendra.vikas at gmail.com (Shailendra) Date: Fri, 2 Apr 2010 10:35:51 -0400 Subject: [Numpy-discussion] Is this odd? In-Reply-To: References: Message-ID: Thanks everyone for replies/suggestion. It is simple to avoid this problem. But my point that given the behavior of python this behavior seems inconsistent. There could other method provided which could evaluate bool value depending on values stored in the array. Thanks, Shailendra On Fri, Apr 2, 2010 at 10:28 AM, Ryan May wrote: > On Thu, Apr 1, 2010 at 10:07 PM, Shailendra wrote: >> Hi All, >> Below is some array behaviour which i think is odd >>>>> a=arange(10) >>>>> a >> array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) >>>>> b=nonzero(a<0) >>>>> b >> (array([], dtype=int32),) >>>>> if not b[0]: >> ... ? ? print 'b[0] is false' >> ... >> b[0] is false >> >> Above case the b[0] is empty so it is fine it is considered false >> >>>>> b=nonzero(a<1) >>>>> b >> (array([0]),) >>>>> if not b[0]: >> ... ? ? print 'b[0] is false' >> ... >> b[0] is false >> >> Above case b[0] is a non-empty array. Why should this be consider false. >> >>>>> b=nonzero(a>8) >>>>> b >> (array([9]),) >>>>> if not b[0]: >> ... ? ? print 'b[0] is false' >> ... >>>>> >> Above case b[0] is non-empty and should be consider true.Which it does. >> >> I don't understand why non-empty array should not be considered true >> irrespective to what value they have. >> Also, please suggest the best way to differentiate between an empty >> array and non-empty array( irrespective to what is inside array). > > But by using: > > if not b[0]: > > You're not considering the array as a whole, you're looking at the > first element, which is giving expected results. ?As I'm sure you're > aware, however, you can't simply do: > > if not b: # Raises exception > > So what you need to do is: > > if b.any(): > > or: > > if b.all() > > Now for determining empty or not, you'll need to look at len(b) or b.shape > > Ryan > > -- > Ryan May > Graduate Research Assistant > School of Meteorology > University of Oklahoma > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From shailendra.vikas at gmail.com Fri Apr 2 10:43:40 2010 From: shailendra.vikas at gmail.com (Shailendra) Date: Fri, 2 Apr 2010 10:43:40 -0400 Subject: [Numpy-discussion] Matrix operation. In-Reply-To: <28119663.post@talk.nabble.com> References: <28119663.post@talk.nabble.com> Message-ID: >>> A=[[1,1,0], ... [1,0,0], ... [0,0,1]] >>> X=[2,9,10,3] >>> import numpy >>> A=numpy.asarray(A) >>> X=numpy.asarray(X) >>> A array([[1, 1, 0], [1, 0, 0], [0, 0, 1]]) >>> X array([ 2, 9, 10, 3]) >>> non_zero=numpy.nonzero(A) >>> non_zero (array([0, 0, 1, 2]), array([0, 1, 0, 2])) >>> A[non_zero] array([1, 1, 1, 1]) >>> A[non_zero]=X >>> A array([[ 2, 9, 0], [10, 0, 0], [ 0, 0, 3]]) Thanks, Shailendra On Fri, Apr 2, 2010 at 10:35 AM, gerardob wrote: > > Let A be a square matrix of 0's and 1's, and let X be a one dimesional > vector. > The length of X is equal to the number of 1's that A has. > I would like to produce a new matrix B by traversing the matrix A row by row > and: > 1- whenever i find a 0, set B in that position to zero. > 2- whenever i find a 1, set B in that position with the ith value of X > (where i represents the ith time i found a 1). > > Example. > input: > A=[[1,1,0], > [1,0,0], > [0,0,1]] > > X=[2,9,10,3] > > Output: > B =[[2,9,0], > ? [10,0,0], > ? [0,0,3]] > > Which is an efficient way to accomplish this using numpy? > > Thanks. > -- > View this message in context: http://old.nabble.com/Matrix-operation.-tp28119663p28119663.html > Sent from the Numpy-discussion mailing list archive at Nabble.com. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From rmay31 at gmail.com Fri Apr 2 11:20:57 2010 From: rmay31 at gmail.com (Ryan May) Date: Fri, 2 Apr 2010 09:20:57 -0600 Subject: [Numpy-discussion] Is this odd? In-Reply-To: References: Message-ID: On Fri, Apr 2, 2010 at 8:31 AM, Robert Kern wrote: > On Fri, Apr 2, 2010 at 08:28, Ryan May wrote: >> On Thu, Apr 1, 2010 at 10:07 PM, Shailendra wrote: >>> Hi All, >>> Below is some array behaviour which i think is odd >>>>>> a=arange(10) >>>>>> a >>> array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) >>>>>> b=nonzero(a<0) >>>>>> b >>> (array([], dtype=int32),) >>>>>> if not b[0]: >>> ... ? ? print 'b[0] is false' >>> ... >>> b[0] is false >>> >>> Above case the b[0] is empty so it is fine it is considered false >>> >>>>>> b=nonzero(a<1) >>>>>> b >>> (array([0]),) >>>>>> if not b[0]: >>> ... ? ? print 'b[0] is false' >>> ... >>> b[0] is false >>> >>> Above case b[0] is a non-empty array. Why should this be consider false. >>> >>>>>> b=nonzero(a>8) >>>>>> b >>> (array([9]),) >>>>>> if not b[0]: >>> ... ? ? print 'b[0] is false' >>> ... >>>>>> >>> Above case b[0] is non-empty and should be consider true.Which it does. >>> >>> I don't understand why non-empty array should not be considered true >>> irrespective to what value they have. >>> Also, please suggest the best way to differentiate between an empty >>> array and non-empty array( irrespective to what is inside array). >> >> But by using: >> >> if not b[0]: >> >> You're not considering the array as a whole, you're looking at the >> first element, which is giving expected results. > > No, b is a tuple containing the array. b[0] is the array itself. Wow, that's what I get for trying to read code *before* coffee. On the plus side, I now know how nonzero() actually works. Ryan -- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma From shailendra.vikas at gmail.com Fri Apr 2 11:31:45 2010 From: shailendra.vikas at gmail.com (Shailendra) Date: Fri, 2 Apr 2010 11:31:45 -0400 Subject: [Numpy-discussion] Getting index of array after applying cond Message-ID: Hi All, I have a following model problem. Let i have a array >>> x array([[1, 2, 3, 4, 5], [6, 7, 8, 7, 6], [1, 2, 3, 4, 5]]) suppose i apply some cond on it. >>> cond= x>5 >>> x[cond] array([6, 7, 8, 7, 6]) Now, i want to argmax on this >>> max=argmax(x[cond]) >>> max 2 >>> x[cond][max] 8 Now , I want to get the index of this element in x. How to acheive this. In real situation x will be huge. Thanks, Shailendra From robert.kern at gmail.com Fri Apr 2 11:37:23 2010 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 2 Apr 2010 09:37:23 -0600 Subject: [Numpy-discussion] Getting index of array after applying cond In-Reply-To: References: Message-ID: On Fri, Apr 2, 2010 at 09:31, Shailendra wrote: > Hi All, > I have a following model problem. Let i have a array >>>> x > array([[1, 2, 3, 4, 5], > ? ? ? [6, 7, 8, 7, 6], > ? ? ? [1, 2, 3, 4, 5]]) > > suppose i apply some cond on it. >>>> cond= x>5 >>>> x[cond] > array([6, 7, 8, 7, 6]) > > Now, i want to argmax on this >>>> max=argmax(x[cond]) >>>> max > 2 >>>> x[cond][max] > 8 > Now , I want to get the index of this element in x. How to acheive > this. In real situation x will be huge. In [1]: x = array([[1,2,3,4,5], [6,7,8,7,6], [1,2,3,4,5]]) In [2]: x Out[2]: array([[1, 2, 3, 4, 5], [6, 7, 8, 7, 6], [1, 2, 3, 4, 5]]) In [3]: cond = (x > 5) In [4]: i, j = where(cond) In [5]: i Out[5]: array([1, 1, 1, 1, 1]) In [6]: j Out[6]: array([0, 1, 2, 3, 4]) In [7]: argmax(x[cond]) Out[7]: 2 In [8]: i[2], j[2] Out[8]: (1, 2) In [9]: x[i[2], j[2]] Out[9]: 8 -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From sole at esrf.fr Fri Apr 2 11:49:37 2010 From: sole at esrf.fr (Vicente Sole) Date: Fri, 02 Apr 2010 17:49:37 +0200 Subject: [Numpy-discussion] Matrix operation. In-Reply-To: <28119663.post@talk.nabble.com> References: <28119663.post@talk.nabble.com> Message-ID: <20100402174937.6hw315178k4g04gc@160.103.2.152> With A and X being arrays: B=numpy.zeros(A.shape, A.dtype) B[A>0] = X Armando Quoting gerardob : > > Let A be a square matrix of 0's and 1's, and let X be a one dimesional > vector. > The length of X is equal to the number of 1's that A has. > I would like to produce a new matrix B by traversing the matrix A row by row > and: > 1- whenever i find a 0, set B in that position to zero. > 2- whenever i find a 1, set B in that position with the ith value of X > (where i represents the ith time i found a 1). > > Example. > input: > A=[[1,1,0], > [1,0,0], > [0,0,1]] > > X=[2,9,10,3] > > Output: > B =[[2,9,0], > [10,0,0], > [0,0,3]] > > Which is an efficient way to accomplish this using numpy? > > Thanks. > -- > View this message in context: > http://old.nabble.com/Matrix-operation.-tp28119663p28119663.html > Sent from the Numpy-discussion mailing list archive at Nabble.com. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From shailendra.vikas at gmail.com Fri Apr 2 12:24:45 2010 From: shailendra.vikas at gmail.com (Shailendra) Date: Fri, 2 Apr 2010 12:24:45 -0400 Subject: [Numpy-discussion] Getting index of array after applying cond In-Reply-To: References: Message-ID: I forgot to mention that i wanted this to work for general shape. So i modified it little bit >>> x = array([[1,2,3,4,5], [6,7,8,7,6], [1,2,3,4,5]]) >>> cond = (x > 5) >>> loc= where(cond) >>> arg_max=argmax(x[cond]) >>> x[tuple([e[arg_max] for e in loc])] 8 Thanks for your solution. Shailendra On Fri, Apr 2, 2010 at 11:37 AM, Robert Kern wrote: > On Fri, Apr 2, 2010 at 09:31, Shailendra wrote: >> Hi All, >> I have a following model problem. Let i have a array >>>>> x >> array([[1, 2, 3, 4, 5], >> ? ? ? [6, 7, 8, 7, 6], >> ? ? ? [1, 2, 3, 4, 5]]) >> >> suppose i apply some cond on it. >>>>> cond= x>5 >>>>> x[cond] >> array([6, 7, 8, 7, 6]) >> >> Now, i want to argmax on this >>>>> max=argmax(x[cond]) >>>>> max >> 2 >>>>> x[cond][max] >> 8 >> Now , I want to get the index of this element in x. How to acheive >> this. In real situation x will be huge. > > In [1]: x = array([[1,2,3,4,5], [6,7,8,7,6], [1,2,3,4,5]]) > > In [2]: x > Out[2]: > array([[1, 2, 3, 4, 5], > ? ? ? [6, 7, 8, 7, 6], > ? ? ? [1, 2, 3, 4, 5]]) > > In [3]: cond = (x > 5) > > In [4]: i, j = where(cond) > > In [5]: i > Out[5]: array([1, 1, 1, 1, 1]) > > In [6]: j > Out[6]: array([0, 1, 2, 3, 4]) > > In [7]: argmax(x[cond]) > Out[7]: 2 > > In [8]: i[2], j[2] > Out[8]: (1, 2) > > In [9]: x[i[2], j[2]] > Out[9]: 8 > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > ?-- Umberto Eco > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From shailendra.vikas at gmail.com Fri Apr 2 12:34:06 2010 From: shailendra.vikas at gmail.com (Shailendra) Date: Fri, 2 Apr 2010 12:34:06 -0400 Subject: [Numpy-discussion] List of indices Message-ID: Hi All, >>> x=arange(10) >>> indices=[(1,),(2,)] >>> x[indices] Traceback (most recent call last): File "", line 1, in IndexError: unsupported iterator index But following works. >>> x=x.reshape(5,2) >>> x array([[0, 1], [2, 3], [4, 5], [6, 7], [8, 9]]) >>> indices=[(1,0),(0,1)] >>> x[indices] array([2, 1]) Should not the first one also work? How to convert [(1,),(2,)] to [1,2] efficiently. Thanks, Shailendra From d.l.goldsmith at gmail.com Fri Apr 2 13:27:32 2010 From: d.l.goldsmith at gmail.com (David Goldsmith) Date: Fri, 2 Apr 2010 10:27:32 -0700 Subject: [Numpy-discussion] Asymmetry in Chebyshev.deriv v. Chebyshev.integ In-Reply-To: References: Message-ID: On Thu, Apr 1, 2010 at 6:42 PM, David Goldsmith wrote: > >>> np.version.version > '1.4.0' > >>> c = np.polynomial.chebyshev.Chebyshev(1) > >>> c.deriv(1.0) > Chebyshev([ 0.], [-1., 1.]) > >>> c.integ(1.0) > Traceback (most recent call last): > File "", line 1, in > File "", line 441, in integ > File "C:\Python26\lib\site-packages\numpy\polynomial\chebyshev.py", line > 739, > in chebint > k = list(k) + [0]*(m - len(k)) > TypeError: can't multiply sequence by non-int of type 'float' > >>> c.integ(1) > Chebyshev([ 0., 1.], [-1., 1.]) > > i.e., deriv accepts int_like input but integ doesn't. > > Given the way I just embarrassed myself on the scipy-dev list :-(, I'm > confirming this is a bug before I file a ticket. > Also: >>> c.deriv(0) Chebyshev([ 1.], [-1., 1.]) >>> c.integ(0) Traceback (most recent call last): File "", line 1, in File "", line 441, in integ File "C:\Python26\lib\site-packages\numpy\polynomial\chebyshev.py", line 729, in chebint raise ValueError, "The order of integration must be positive" ValueError: The order of integration must be positive i.e., deriv supports zero-order differentiation, but integ doesn't support zero-order integration (though I acknowledge that this may be a feature, not a bug). -- Mathematician: noun, someone who disavows certainty when their uncertainty set is non-empty, even if that set has measure zero. -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Apr 2 13:42:34 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 2 Apr 2010 11:42:34 -0600 Subject: [Numpy-discussion] Asymmetry in Chebyshev.deriv v. Chebyshev.integ In-Reply-To: References: Message-ID: On Thu, Apr 1, 2010 at 7:42 PM, David Goldsmith wrote: > >>> np.version.version > '1.4.0' > >>> c = np.polynomial.chebyshev.Chebyshev(1) > >>> c.deriv(1.0) > Chebyshev([ 0.], [-1., 1.]) > >>> c.integ(1.0) > Traceback (most recent call last): > File "", line 1, in > File "", line 441, in integ > File "C:\Python26\lib\site-packages\numpy\polynomial\chebyshev.py", line > 739, > in chebint > k = list(k) + [0]*(m - len(k)) > TypeError: can't multiply sequence by non-int of type 'float' > >>> c.integ(1) > Chebyshev([ 0., 1.], [-1., 1.]) > > I don't think it should accept a float when an integer is needed. That said, I should either raise a more informative error or folks should convince me that floats are a reasonable input for the number of integrations. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Apr 2 13:46:24 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 2 Apr 2010 11:46:24 -0600 Subject: [Numpy-discussion] Asymmetry in Chebyshev.deriv v. Chebyshev.integ In-Reply-To: References: Message-ID: On Fri, Apr 2, 2010 at 11:27 AM, David Goldsmith wrote: > On Thu, Apr 1, 2010 at 6:42 PM, David Goldsmith wrote: > >> >>> np.version.version >> '1.4.0' >> >>> c = np.polynomial.chebyshev.Chebyshev(1) >> >>> c.deriv(1.0) >> Chebyshev([ 0.], [-1., 1.]) >> >>> c.integ(1.0) >> Traceback (most recent call last): >> File "", line 1, in >> File "", line 441, in integ >> File "C:\Python26\lib\site-packages\numpy\polynomial\chebyshev.py", line >> 739, >> in chebint >> k = list(k) + [0]*(m - len(k)) >> TypeError: can't multiply sequence by non-int of type 'float' >> >>> c.integ(1) >> Chebyshev([ 0., 1.], [-1., 1.]) >> >> i.e., deriv accepts int_like input but integ doesn't. >> >> Given the way I just embarrassed myself on the scipy-dev list :-(, I'm >> confirming this is a bug before I file a ticket. >> > > Also: > > >>> c.deriv(0) > Chebyshev([ 1.], [-1., 1.]) > >>> c.integ(0) > > Traceback (most recent call last): > File "", line 1, in > File "", line 441, in integ > File "C:\Python26\lib\site-packages\numpy\polynomial\chebyshev.py", line > 729, > in chebint > raise ValueError, "The order of integration must be positive" > ValueError: The order of integration must be positive > > i.e., deriv supports zero-order differentiation, but integ doesn't support > zero-order integration (though I acknowledge that this may be a feature, not > a bug). > > It was inherited. I have no qualms about letting it integrate zero times if folks think it should go that way. I think the reason derivatives allowed 0 for the number of derivations was for classroom instruction. Chuck > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From d.l.goldsmith at gmail.com Fri Apr 2 14:04:31 2010 From: d.l.goldsmith at gmail.com (David Goldsmith) Date: Fri, 2 Apr 2010 11:04:31 -0700 Subject: [Numpy-discussion] Asymmetry in Chebyshev.deriv v. Chebyshev.integ In-Reply-To: References: Message-ID: On Fri, Apr 2, 2010 at 10:42 AM, Charles R Harris wrote: > > On Thu, Apr 1, 2010 at 7:42 PM, David Goldsmith wrote: > >> >>> np.version.version >> '1.4.0' >> >>> c = np.polynomial.chebyshev.Chebyshev(1) >> >>> c.deriv(1.0) >> Chebyshev([ 0.], [-1., 1.]) >> >>> c.integ(1.0) >> Traceback (most recent call last): >> File "", line 1, in >> File "", line 441, in integ >> File "C:\Python26\lib\site-packages\numpy\polynomial\chebyshev.py", line >> 739, >> in chebint >> k = list(k) + [0]*(m - len(k)) >> TypeError: can't multiply sequence by non-int of type 'float' >> >>> c.integ(1) >> Chebyshev([ 0., 1.], [-1., 1.]) > > > I don't think it should accept a float when an integer is needed. That > said, I should either raise a more informative error or folks should > convince me that floats are a reasonable input for the number of > integrations. > > Chuck > My only concern is API consistency: if you want to restrict the integ input to int dtypes, that's fine, but then why allow non-int dtypes in deriv (which, BTW, accepts much more than just int_like floats - it works with one element lists containing an int_like float, and similar numpy arrays, even zero-order Polynomial objects (I didn't check tuples, but given all that, I'd be surprised if it didn't))? Superficially, this is a pretty big API discrepancy; ultimately of course it doesn't matter, but I'd like to know where we want to land so I can make sure the docstrings correctly document desired behavior. DG -------------- next part -------------- An HTML attachment was scrubbed... URL: From d.l.goldsmith at gmail.com Fri Apr 2 14:07:16 2010 From: d.l.goldsmith at gmail.com (David Goldsmith) Date: Fri, 2 Apr 2010 11:07:16 -0700 Subject: [Numpy-discussion] Asymmetry in Chebyshev.deriv v. Chebyshev.integ In-Reply-To: References: Message-ID: On Fri, Apr 2, 2010 at 10:46 AM, Charles R Harris wrote: > On Fri, Apr 2, 2010 at 11:27 AM, David Goldsmith wrote: > >> Also: >> >> >>> c.deriv(0) >> Chebyshev([ 1.], [-1., 1.]) >> >>> c.integ(0) >> >> Traceback (most recent call last): >> File "", line 1, in >> File "", line 441, in integ >> File "C:\Python26\lib\site-packages\numpy\polynomial\chebyshev.py", line >> 729, >> in chebint >> raise ValueError, "The order of integration must be positive" >> ValueError: The order of integration must be positive >> >> i.e., deriv supports zero-order differentiation, but integ doesn't support >> zero-order integration (though I acknowledge that this may be a feature, not >> a bug). > > > It was inherited. I have no qualms about letting it integrate zero times if > folks think it should go that way. I think the reason derivatives allowed 0 > for the number of derivations was for classroom instruction. > Again, my only concern is API consistency, yada, yada, yada. DG -------------- next part -------------- An HTML attachment was scrubbed... URL: From e.antero.tammi at gmail.com Fri Apr 2 15:00:26 2010 From: e.antero.tammi at gmail.com (eat) Date: Fri, 2 Apr 2010 19:00:26 +0000 (UTC) Subject: [Numpy-discussion] Getting index of array after applying cond References: Message-ID: Shailendra gmail.com> writes: > > I forgot to mention that i wanted this to work for general shape. So i > modified it little bit > > >>> x = array([[1,2,3,4,5], [6,7,8,7,6], [1,2,3,4,5]]) > >>> cond = (x > 5) > >>> loc= where(cond) > >>> arg_max=argmax(x[cond]) > >>> x[tuple([e[arg_max] for e in loc])] > 8 But what happens if your x is for example x = array([[1,2,3,4,5], [6,8,8,8,6], [1,2,3,4,5]]) x[tuple([e[arg_max] for e in loc])] would still yield to 8, which may or may not be an acceptable answer. Basically what I mean is that why to bother with the argmax at all, if your only interest is x[cond].max()? Just my 2 cents. Regards, eat From erik.tollerud at gmail.com Fri Apr 2 15:03:44 2010 From: erik.tollerud at gmail.com (Erik Tollerud) Date: Fri, 2 Apr 2010 12:03:44 -0700 Subject: [Numpy-discussion] How do I ensure numpy headers are present in setup.py? Message-ID: I am writing a setup.py file for a package that will use cython with numpy integration. This of course requires the numpy header files, which I am including by using numpy.get_includes in the setup.py file below. The problem is for users that have not installed numpy before installing this package. If they have setuptools installed, the behavior I would want would be for numpy to be downloaded and then the setup script should be able to get at the headers even if it doesn't install numpy until after this package is installed. But that doesn't work - I have to import numpy in the setup script, which fails if it is not yet installed. So How can I get the behavior I want? setup.py ------------ from distribute_setup import use_setuptools use_setuptools() from setuptools import setup from setuptools.extension import Extension from Cython.Distutils import build_ext from numpy import get_include as numpy_includes ext_modules = [Extension("myext", ["core.pyx","somecfile.c"],include_dirs=[numpy_includes()])] setup( name = 'ext name', cmdclass = {'build_ext': build_ext}, ext_modules = ext_modules, setup_requires=['numpy'], install_requires=['numpy'], ) From robert.kern at gmail.com Fri Apr 2 15:06:08 2010 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 2 Apr 2010 13:06:08 -0600 Subject: [Numpy-discussion] How do I ensure numpy headers are present in setup.py? In-Reply-To: References: Message-ID: On Fri, Apr 2, 2010 at 13:03, Erik Tollerud wrote: > I am writing a setup.py file for a package that will use cython with > numpy integration. ?This of course requires the numpy header files, > which I am including by using numpy.get_includes in the setup.py file > below. ?The problem is for users that have not installed numpy before > installing this package. ? ?If they have setuptools installed, the > behavior I would want would be for numpy to be downloaded and then the > setup script should be able to get at the headers even if it doesn't > install numpy until after this package is installed. ?But that doesn't > work - I have to import numpy in the setup script, which fails if it > is not yet installed. ?So How can I get the behavior I want? You can't, not without some hacks to distutils. This is a basic problem with the way setuptools uses arguments to setup() to get the dependencies. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From ranavishal at gmail.com Fri Apr 2 15:17:34 2010 From: ranavishal at gmail.com (Vishal Rana) Date: Fri, 2 Apr 2010 12:17:34 -0700 Subject: [Numpy-discussion] Applying formula to all in an array which hasvalue from previous In-Reply-To: <710F2847B0018641891D9A21602763605AD37B@ex3.envision.co.il> References: <710F2847B0018641891D9A21602763605AD37B@ex3.envision.co.il> Message-ID: I get AttributeError: 'NoneType' object has no attribute 'accumulate' for the call vecfun.ufunc.accumulate(np.array([0, 1, 2, 3])) It works fine if I make a dummy call to vecfun before! Any idea for this behavior? Thanks Vishal Rana On Mon, Mar 29, 2010 at 4:07 AM, Nadav Horesh wrote: > The general guideline: > > Suppose the function definition is: > > def func(x,y): > # x and y are scalars > bla bla bla ... > return z # a scalar > > So, > > import numpy as np > > vecfun = np.vectorize(func) > > vecfun.ufunc.accumulate(array((0,1,2,3,4,5,6,7,8,9)) > > > Nadav. > > > -----Original Message----- > From: numpy-discussion-bounces at scipy.org on behalf of Vishal Rana > Sent: Sun 28-Mar-10 21:19 > To: Discussion of Numerical Python > Subject: [Numpy-discussion] Applying formula to all in an array which > hasvalue from previous > > Hi, > > For a numpy array: > > array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) > > I do some calculation with 0, 1... and get a value = 2.5, now use this > value > to do the repeat the same calculation with next element for example... > 2.5, 2 and get a value = 3.1 > 3.1, 3 and get a value = 4.2 > 4.2, 4 and get a value = 5.1 > .... > .... and get a value = 8.5 > 8.5, 9 and get a value = 9.8 > > So I should be getting a new array like array([0, 2.5, 3.1, 4.2, 5.1, ..... > 8.5,9.8]) > > Is it where numpy or scipy can help? > > Thanks > Vishal > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shailendra.vikas at gmail.com Fri Apr 2 15:44:16 2010 From: shailendra.vikas at gmail.com (Shailendra) Date: Fri, 2 Apr 2010 15:44:16 -0400 Subject: [Numpy-discussion] Getting index of array after applying cond In-Reply-To: References: Message-ID: Well, this is just a toy problem. argmax represent a method which will give me a index in x[cond] . And for the case of multiple value my requirement is fine with getting any "max" index. Thanks, Shailendra On Fri, Apr 2, 2010 at 3:00 PM, eat wrote: > Shailendra gmail.com> writes: > >> >> I forgot to mention that i wanted this to work for general shape. So i >> modified it little bit >> >> >>> x = array([[1,2,3,4,5], [6,7,8,7,6], [1,2,3,4,5]]) >> >>> cond = (x > 5) >> >>> loc= where(cond) >> >>> arg_max=argmax(x[cond]) >> >>> x[tuple([e[arg_max] for e in loc])] >> 8 > > But what happens if your x is for example > x = array([[1,2,3,4,5], [6,8,8,8,6], [1,2,3,4,5]]) > > x[tuple([e[arg_max] for e in loc])] > would still yield to 8, > which may or may not be an acceptable answer. > > Basically what I mean is that why to bother with the argmax at all, > if your only interest is x[cond].max()? > > > Just my 2 cents. > > > Regards, > eat > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From e.antero.tammi at gmail.com Fri Apr 2 17:07:17 2010 From: e.antero.tammi at gmail.com (eat) Date: Fri, 2 Apr 2010 21:07:17 +0000 (UTC) Subject: [Numpy-discussion] Getting index of array after applying cond References: Message-ID: Shailendra gmail.com> writes: > > Well, this is just a toy problem. argmax represent a method which will > give me a index in x[cond] . And for the case of multiple value my > requirement is fine with getting any "max" index. OK. My concern seems to be needless then. BTW, does this current thread relate anyway to the earlier one '"Match" two arrays'? If so, would you like to elaborate more about your 'real' problem? Regards, eat > > Thanks, > Shailendra From ranavishal at gmail.com Fri Apr 2 17:41:43 2010 From: ranavishal at gmail.com (Vishal Rana) Date: Fri, 2 Apr 2010 14:41:43 -0700 Subject: [Numpy-discussion] How do I get element on axis 1 Message-ID: Hi, I know its easy, but I am not just getting it... How do I get last element on axis=1 for: a = array([[ 0, 1, 2, 3, 4], [ 5, 6, 7, 8, 9], [10, 11, 12, 13, 14], [15, 16, 17, 18, 19], [20, 21, 22, 23, 24]]) Expected: array([4, 9, 14, 19, 24]) Thanks Vishal Rana -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at enthought.com Fri Apr 2 17:46:19 2010 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Fri, 02 Apr 2010 16:46:19 -0500 Subject: [Numpy-discussion] How do I get element on axis 1 In-Reply-To: References: Message-ID: <4BB665AB.5060807@enthought.com> Vishal Rana wrote: > Hi, > > I know its easy, but I am not just getting it... > > How do I get last element on axis=1 for: > > a = array([[ 0, 1, 2, 3, 4], > [ 5, 6, 7, 8, 9], > [10, 11, 12, 13, 14], > [15, 16, 17, 18, 19], > [20, 21, 22, 23, 24]]) > > Expected: array([4, 9, 14, 19, 24]) > a[:,-1] > Thanks > Vishal Rana > ------------------------------------------------------------------------ > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From ranavishal at gmail.com Fri Apr 2 17:49:55 2010 From: ranavishal at gmail.com (Vishal Rana) Date: Fri, 2 Apr 2010 14:49:55 -0700 Subject: [Numpy-discussion] How do I get element on axis 1 In-Reply-To: <4BB665AB.5060807@enthought.com> References: <4BB665AB.5060807@enthought.com> Message-ID: Thanks Warren! On Fri, Apr 2, 2010 at 2:46 PM, Warren Weckesser < warren.weckesser at enthought.com> wrote: > Vishal Rana wrote: > > Hi, > > > > I know its easy, but I am not just getting it... > > > > How do I get last element on axis=1 for: > > > > a = array([[ 0, 1, 2, 3, 4], > > [ 5, 6, 7, 8, 9], > > [10, 11, 12, 13, 14], > > [15, 16, 17, 18, 19], > > [20, 21, 22, 23, 24]]) > > > > Expected: array([4, 9, 14, 19, 24]) > > > > a[:,-1] > > > Thanks > > Vishal Rana > > ------------------------------------------------------------------------ > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sccolbert at gmail.com Fri Apr 2 23:45:41 2010 From: sccolbert at gmail.com (Chris Colbert) Date: Fri, 2 Apr 2010 23:45:41 -0400 Subject: [Numpy-discussion] How do I ensure numpy headers are present in setup.py? In-Reply-To: References: Message-ID: On Fri, Apr 2, 2010 at 3:03 PM, Erik Tollerud wrote: you could try something like this (untested): if __name__ == '__main__': try: import numpy except ImportError: import subprocess subprocess.check_call(['easy_install', 'numpy']) # will except if call fails -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Sat Apr 3 00:17:59 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sat, 3 Apr 2010 00:17:59 -0400 Subject: [Numpy-discussion] How do I ensure numpy headers are present in setup.py? In-Reply-To: References: Message-ID: On Fri, Apr 2, 2010 at 11:45 PM, Chris Colbert wrote: > > > On Fri, Apr 2, 2010 at 3:03 PM, Erik Tollerud > wrote: > you could try something like this (untested): > if __name__ == '__main__': > ?? ?try: > ?? ? ? ?import numpy > ?? ?except ImportError: > ?? ? ? ?import subprocess > ?? ? ? ?subprocess.check_call(['easy_install', 'numpy']) ?# will except if > call fails Personally, I don't like it at all if packages automatically easy_install big dependencies like numpy especially if it is connected to a version requirement. Is it possible to kill an installation subprocess with CRTL-C ? Josef > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From yogeshkarpate at gmail.com Sat Apr 3 05:57:05 2010 From: yogeshkarpate at gmail.com (yogesh karpate) Date: Sat, 3 Apr 2010 15:27:05 +0530 Subject: [Numpy-discussion] TypeError: only integer arrays with one element can be converted to an index Message-ID: I am facing the error mentioned in subject line. error is encountered at data_temp=filtered[a:b] TypeError: only integer arrays with one element can be converted to an index what can be error The code snippet is as follows '''left=some array right =some array, both are having same length. filtered is datafile(.txt) containing 4000 sample values left=(array([ 107, 594, 1082, 1569, 2057, 2545, 3033, 3520]),) right=(array([ 133, 621, 1109, 1597, 2085, 2573, 3061, 3550]),) maxval=empty(a) minval=empty(a) maxloc=empty(a) minloc=empty(a) data_temp=empty(a) a1=len(left) for i in arange(a1): a=left[i] b=right[i] print a,b data_temp=filtered[a:b] maxloc[i]=data_temp.argmax() maxval[i]=data_temp[maxloc] minloc[i]=data_temp.argmin() minval[i]=data_temp[minloc] maxloc[i] = maxloc[i]-1+left[i]# % add offset of present location minloc[i] = minloc[i]-1+left[i]# % add offset of present location''' R_index = maxloc R_t = t[maxloc] Thanks in advance Regards Yogesh -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at enthought.com Sat Apr 3 07:40:10 2010 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Sat, 03 Apr 2010 06:40:10 -0500 Subject: [Numpy-discussion] TypeError: only integer arrays with one element can be converted to an index In-Reply-To: References: Message-ID: <4BB7291A.8060002@enthought.com> Are you sure you wanted to make `left` and `right` tuples that contain just one element (which is a numpy array)? In your code, len(a1) will be 1. Maybe I am misinterpreting your code, but I think you want this: left = array([ 107, 594, 1082, 1569, 2057, 2545, 3033, 3520]) right = array([ 133, 621, 1109, 1597, 2085, 2573, 3061, 3550]) Warren yogesh karpate wrote: > I am facing the error mentioned in subject line. > error is encountered at > data_temp=filtered[a:b] > TypeError: only integer arrays with one element can be converted to an > index > what can be error > The code snippet is as follows > > '''left=some array > right =some array, both are having same length. > filtered is datafile(.txt) containing 4000 sample values > left=(array([ 107, 594, 1082, 1569, 2057, 2545, 3033, 3520]),) > right=(array([ 133, 621, 1109, 1597, 2085, 2573, 3061, 3550]),) > > maxval=empty(a) > minval=empty(a) > maxloc=empty(a) > minloc=empty(a) > data_temp=empty(a) > a1=len(left) > for i in arange(a1): > a=left[i] > b=right[i] > print a,b > data_temp=filtered[a:b] > maxloc[i]=data_temp.argmax() > maxval[i]=data_temp[maxloc] > minloc[i]=data_temp.argmin() > minval[i]=data_temp[minloc] > maxloc[i] = maxloc[i]-1+left[i]# % add offset of present location > minloc[i] = minloc[i]-1+left[i]# % add offset of present location''' > R_index = maxloc > R_t = t[maxloc] > Thanks in advance > Regards > Yogesh > ------------------------------------------------------------------------ > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From sccolbert at gmail.com Sat Apr 3 10:17:25 2010 From: sccolbert at gmail.com (Chris Colbert) Date: Sat, 3 Apr 2010 10:17:25 -0400 Subject: [Numpy-discussion] How do I ensure numpy headers are present in setup.py? In-Reply-To: References: Message-ID: On Sat, Apr 3, 2010 at 12:17 AM, wrote: > > On Fri, Apr 2, 2010 at 11:45 PM, Chris Colbert wrote: > > > > > > On Fri, Apr 2, 2010 at 3:03 PM, Erik Tollerud > > wrote: > > you could try something like this (untested): > > if __name__ == '__main__': > > ?? ?try: > > ?? ? ? ?import numpy > > ?? ?except ImportError: > > ?? ? ? ?import subprocess > > ?? ? ? ?subprocess.check_call(['easy_install', 'numpy']) ?# will except if > > call fails > > Personally, I don't like it at all if packages automatically > easy_install big dependencies like numpy especially if it is connected > to a version requirement. > > Is it possible to kill an installation subprocess with CRTL-C ? Hmm, that's a good question. Just thinking about it, I don't think it would. From the docs: Popen.kill() Kills the child. On Posix OSs the function sends SIGKILL to the child. On Windows?kill()?is an alias for?terminate(). But, I listed the check_call() function, which waits for the subprocess to end. You never actually get the handle to the subprocess object to issue the kill() command. So, I'm not sure how you could make it work. > > Josef > > > > > > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion From lists at pairet.be Sat Apr 3 11:52:16 2010 From: lists at pairet.be (Antoine Pairet) Date: Sat, 03 Apr 2010 17:52:16 +0200 Subject: [Numpy-discussion] install problems Message-ID: <1270309936.29623.18.camel@portant> Hi all! I am trying to install numpy on a computer which runs Linux and on which I have no root access. The only possibility is therefore to compile numpy from source. With the following command, the compilation seems to work fine: python setup.py install --prefix=/home/pcpm/pairet/pythonModules/numpy However, every time I try to use numpy in a python script I have the following error: from numpy import * ImportError: No module named numpy I did add the following line to my .bashrc file: export PYTHONPATH= $PYTHONPATH:/home/pcpm/pairet/pythonModules/numpy/lib64/python2.4/site-packages/numpy I am a bit puzzled with the fact that there is no file called numpy.py. What should I do to get numpy working on this machine? Any help is appreciated, Thanks in adavance, Antoine From warren.weckesser at enthought.com Sat Apr 3 12:04:46 2010 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Sat, 03 Apr 2010 11:04:46 -0500 Subject: [Numpy-discussion] install problems In-Reply-To: <1270309936.29623.18.camel@portant> References: <1270309936.29623.18.camel@portant> Message-ID: <4BB7671E.1020706@enthought.com> Antoine Pairet wrote: > Hi all! > > I am trying to install numpy on a computer which runs Linux and on which > I have no root access. The only possibility is therefore to compile > numpy from source. With the following command, the compilation seems to > work fine: > > python setup.py install --prefix=/home/pcpm/pairet/pythonModules/numpy > > However, every time I try to use numpy in a python script I have the > following error: > from numpy import * > ImportError: No module named numpy > > I did add the following line to my .bashrc file: > export PYTHONPATH= > $PYTHONPATH:/home/pcpm/pairet/pythonModules/numpy/lib64/python2.4/site-packages/numpy > > Don't include that last "numpy" in the path. E.g. export PYTHONPATH=$PYTHONPATH:/home/pcpm/pairet/pythonModules/numpy/lib64/python2.4/site-packages > I am a bit puzzled with the fact that there is no file called numpy.py. > > numpy is installed as a "package": a directory that contains the file __init__.py along with all its other files. See http://docs.python.org/tutorial/modules.html#packages Warren > What should I do to get numpy working on this machine? > Any help is appreciated, > Thanks in adavance, > > Antoine > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From lists at pairet.be Sat Apr 3 12:52:10 2010 From: lists at pairet.be (Antoine Pairet) Date: Sat, 03 Apr 2010 18:52:10 +0200 Subject: [Numpy-discussion] install problems In-Reply-To: <4BB7671E.1020706@enthought.com> References: <1270309936.29623.18.camel@portant> <4BB7671E.1020706@enthought.com> Message-ID: <1270313530.29623.20.camel@portant> On Sat, 2010-04-03 at 11:04 -0500, Warren Weckesser wrote: > Don't include that last "numpy" in the path. E.g. > > export PYTHONPATH=$PYTHONPATH:/home/pcpm/pairet/pythonModules/numpy/lib64/python2.4/site-packages > numpy is installed as a "package": a directory that contains the file > __init__.py along with all its other files. > > See http://docs.python.org/tutorial/modules.html#packages Thanks a lot! Everything works fine now. I will go through the doc. Antoine From charlesr.harris at gmail.com Sat Apr 3 13:23:05 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 3 Apr 2010 11:23:05 -0600 Subject: [Numpy-discussion] Bug in logaddexp2.reduce In-Reply-To: References: <4BB54F6F.2070302@silveregg.co.jp> Message-ID: On Thu, Apr 1, 2010 at 9:01 PM, Charles R Harris wrote: > > > On Thu, Apr 1, 2010 at 8:16 PM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> >> >> On Thu, Apr 1, 2010 at 7:59 PM, David Cournapeau wrote: >> >>> Anne Archibald wrote: >>> >>> > >>> > First I guess we should check which systems don't have log1p >>> >>> This is already done - we do use the system log1p on linux (but note >>> that log2_1p is not standard AFAIK). I would guess few systems outside >>> windows don't have log1p, given that msun has an implementation, >>> >>> >> I see that msun uses the same series I came to. However, my rational >> (Pade) approximation is a lot better than their polynomial. >> >> > In fact I get better than 119 bits using the same range as sun and the > ratio of two 7'th degree polynomials. I suspect it's better than that, but I > only have mpmath set to that precision. Sun got 58 bits with a 14'th degree > polynomial. So we can definitely improve on sun. > > A few notes. The Sun routine is pretty good, but results of almost equal quality can be obtained using barycentric interpolation at the Chebyshev II points, i.e., one doesn't need the slight improvement that comes from the Reme(z) algorithm. To get extended precision requires going from degree 14 to degree 16, and quad precision needs degree 30. Note that the function in question is even, so the effective degree is half of those quoted. Next up: using Chebyshev points to interpolate the polynomials from the Pade representation. The Pade version doesn't actually gain any precision over the power series of equivalent degree, but it does allow one to use polynomials of half the degree in num/den. Short term, however, I'm going to take Anne's advice and simply use log1p. That should fix the problem for most users. It's amusing that the Reme[z] typo has persisted in the Sun documentation all these years ;) Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From d.p.reichert at sms.ed.ac.uk Sat Apr 3 15:13:09 2010 From: d.p.reichert at sms.ed.ac.uk (David Reichert) Date: Sat, 3 Apr 2010 20:13:09 +0100 Subject: [Numpy-discussion] defmatrix move and unpickling of old data Message-ID: Hi, After some work I got an optimized numpy compiled on a machine where I don't have root access, but I had to use numpy 1.4.0 to make it work. Now I have the problem that I cannot seem to unpickle data I had created using numpy 1.3, getting an ImportError about defmatrix not being found. I understand defmatrix was moved from core to matrixlib? Is there some workaround I could use? I might have to move my data in between machines with either versions of numpy installed in the future as well... I already tried some renaming tricks but to no avail. Thanks David -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: not available URL: From nadavh at visionsense.com Sat Apr 3 16:29:37 2010 From: nadavh at visionsense.com (Nadav Horesh) Date: Sat, 3 Apr 2010 23:29:37 +0300 Subject: [Numpy-discussion] Annoyance of memap rray with multiprocessing.Pool.applay_async Message-ID: <710F2847B0018641891D9A21602763605AD392@ex3.envision.co.il> The following script generate the following error on every loop iteration in the function average: Exception AttributeError: AttributeError("'NoneType' object has no attribute 'tell'",) in ignored where xxxxxx is a scalar (the array sum). I get this error with numpy1.4 on a linux64 (dual core) machine. A winXP/Pentium4 with 2GB Ram could not run it since it explode the memory. Any idea what is the origin on the error (my interset is the linux box)? BTW if in the function average the 2nd line is commented and the 3rd line is uncommented I get no error on linux, but the win32 problem pertains. ---------------------------------------------------------------- import numpy as N import multiprocessing as MP import sys try: count = int(sys.argv[1]) except: count = 4 filename = '%dx100x100_int32.dat' % count def average(cube): return [plane.mean() for plane in cube] # return [N.asarray(plane).mean() for plane in cube] data = N.memmap(filename, dtype=N.int32, shape=(count,100,100)) pool = MP.Pool(processes=1) job = pool.apply_async(average, [data,]) print job.get() import numpy as N import multiprocessing as MP count = 4 filename = '%dx100x100_int32.dat' % count def average(cube): return [plane.mean() for plane in cube] # return [N.asarray(plane).mean() for plane in cube] data = N.memmap(filename, dtype=N.int32, shape=(count,100,100)) pool = MP.Pool(processes=1) job = pool.apply_async(average, [data,]) print job.get() ---------------------------------------------------------------- Nadav From pjs67 at cam.ac.uk Sat Apr 3 17:28:51 2010 From: pjs67 at cam.ac.uk (Philip Sterne) Date: Sat, 03 Apr 2010 22:28:51 +0100 Subject: [Numpy-discussion] Getting data from NDarrays to Blitz++ and back again Message-ID: <4BB7B313.2060602@cam.ac.uk> Hi all, I hope this is the correct place to post my question. I'd like python to interface with c++ code that makes heavy use of Blitz++ arrays. After a day's hacking I appear to have a simple working solution which I am attaching. (It uses Boost.Python and CMake.) However I believe this solution won't work if the blitz array is not laid out contiguously in memory. I also haven't really thought about reference counting issues (although the example seems to work). I imagine that those sorts of issues will lead me to call the more complicated: PyArray_New(...) or: PyArray_NewFromDescr(...) instead of the PyArray_SimpleNewFromData(...) that I currently use. However I couldn't figure out some of the extra arguments from the API documentation. Can someone point out all the things that will break when this code actually gets used in the real world (and maybe even how to avoid them)? Thanks for your time! -Philip. -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: CMakeLists.txt URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: simple.cpp Type: text/x-c++src Size: 1553 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: simpletest.py Type: text/x-python Size: 136 bytes Desc: not available URL: From robert.kern at gmail.com Sat Apr 3 17:47:26 2010 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 3 Apr 2010 15:47:26 -0600 Subject: [Numpy-discussion] Annoyance of memap rray with multiprocessing.Pool.applay_async In-Reply-To: <710F2847B0018641891D9A21602763605AD392@ex3.envision.co.il> References: <710F2847B0018641891D9A21602763605AD392@ex3.envision.co.il> Message-ID: On Sat, Apr 3, 2010 at 14:29, Nadav Horesh wrote: > > The following script generate the following error on every loop iteration in the function average: > > Exception AttributeError: AttributeError("'NoneType' object has no attribute 'tell'",) in ignored > > where xxxxxx is a scalar (the array sum). > > I get this error with numpy1.4 on a linux64 (dual core) machine. A winXP/Pentium4 with 2GB Ram could not run it since it explode the memory. > > ?Any idea what is the origin on the error (my interset is the linux box)? > memmap instances don't pickle. Don't pass them as arguments to apply_async() or related functions. Instead, pass the filename and other arguments to memmap() and reconstruct the arrays in each process. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From sccolbert at gmail.com Sat Apr 3 17:50:32 2010 From: sccolbert at gmail.com (Chris Colbert) Date: Sat, 3 Apr 2010 17:50:32 -0400 Subject: [Numpy-discussion] install problems In-Reply-To: <1270313530.29623.20.camel@portant> References: <1270309936.29623.18.camel@portant> <4BB7671E.1020706@enthought.com> <1270313530.29623.20.camel@portant> Message-ID: On Sat, Apr 3, 2010 at 12:52 PM, Antoine Pairet wrote: > On Sat, 2010-04-03 at 11:04 -0500, Warren Weckesser wrote: >> Don't include that last "numpy" in the path. ?E.g. >> >> export PYTHONPATH=$PYTHONPATH:/home/pcpm/pairet/pythonModules/numpy/lib64/python2.4/site-packages >> numpy is installed as a "package": a directory that contains the file >> __init__.py along with all its other files. >> >> See http://docs.python.org/tutorial/modules.html#packages > Thanks a lot! Everything works fine now. I will go through the doc. > Antoine > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > you don't have to build from source btw. you can easy_install it or pip install it with user switch. From sccolbert at gmail.com Sat Apr 3 17:51:13 2010 From: sccolbert at gmail.com (Chris Colbert) Date: Sat, 3 Apr 2010 17:51:13 -0400 Subject: [Numpy-discussion] install problems In-Reply-To: References: <1270309936.29623.18.camel@portant> <4BB7671E.1020706@enthought.com> <1270313530.29623.20.camel@portant> Message-ID: On Sat, Apr 3, 2010 at 5:50 PM, Chris Colbert wrote: > On Sat, Apr 3, 2010 at 12:52 PM, Antoine Pairet wrote: >> On Sat, 2010-04-03 at 11:04 -0500, Warren Weckesser wrote: >>> Don't include that last "numpy" in the path. ?E.g. >>> >>> export PYTHONPATH=$PYTHONPATH:/home/pcpm/pairet/pythonModules/numpy/lib64/python2.4/site-packages >>> numpy is installed as a "package": a directory that contains the file >>> __init__.py along with all its other files. >>> >>> See http://docs.python.org/tutorial/modules.html#packages >> Thanks a lot! Everything works fine now. I will go through the doc. >> Antoine >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > you don't have to build from source btw. you can easy_install it or > pip install it with user switch. > but that basically gives you the same effect as setup.py --install From nadavh at visionsense.com Sun Apr 4 00:35:29 2010 From: nadavh at visionsense.com (Nadav Horesh) Date: Sun, 4 Apr 2010 07:35:29 +0300 Subject: [Numpy-discussion] Annoyance of memap rray withmultiprocessing.Pool.applay_async References: <710F2847B0018641891D9A21602763605AD392@ex3.envision.co.il> Message-ID: <710F2847B0018641891D9A21602763605AD393@ex3.envision.co.il> Got it, thank you. But why, nevertheless, the results are correct although the pickling is impossible? Nadav. -----Original Message----- From: numpy-discussion-bounces at scipy.org on behalf of Robert Kern Sent: Sat 03-Apr-10 23:47 To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] Annoyance of memap rray withmultiprocessing.Pool.applay_async On Sat, Apr 3, 2010 at 14:29, Nadav Horesh wrote: > > The following script generate the following error on every loop iteration in the function average: > > Exception AttributeError: AttributeError("'NoneType' object has no attribute 'tell'",) in ignored > > where xxxxxx is a scalar (the array sum). > > I get this error with numpy1.4 on a linux64 (dual core) machine. A winXP/Pentium4 with 2GB Ram could not run it since it explode the memory. > > ?Any idea what is the origin on the error (my interset is the linux box)? > memmap instances don't pickle. Don't pass them as arguments to apply_async() or related functions. Instead, pass the filename and other arguments to memmap() and reconstruct the arrays in each process. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 3757 bytes Desc: not available URL: From shailendra.vikas at gmail.com Sun Apr 4 10:05:03 2010 From: shailendra.vikas at gmail.com (Shailendra) Date: Sun, 4 Apr 2010 10:05:03 -0400 Subject: [Numpy-discussion] Resending: List of indices Message-ID: On Fri, Apr 2, 2010 at 12:34 PM, Shailendra wrote: > Hi All, > >>>> x=arange(10) >>>> indices=[(1,),(2,)] >>>> x[indices] > Traceback (most recent call last): > ?File "", line 1, in > IndexError: unsupported iterator index > > > But following works. >>>> x=x.reshape(5,2) >>>> x > array([[0, 1], > ? ? ? [2, 3], > ? ? ? [4, 5], > ? ? ? [6, 7], > ? ? ? [8, 9]]) >>>> indices=[(1,0),(0,1)] >>>> x[indices] > array([2, 1]) > > > > Should not the first one also work? How to convert [(1,),(2,)] to > [1,2] efficiently. > > Thanks, > Shailendra > From josef.pktd at gmail.com Sun Apr 4 10:37:30 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 4 Apr 2010 10:37:30 -0400 Subject: [Numpy-discussion] Resending: List of indices In-Reply-To: References: Message-ID: On Sun, Apr 4, 2010 at 10:05 AM, Shailendra wrote: > On Fri, Apr 2, 2010 at 12:34 PM, Shailendra wrote: >> Hi All, >> >>>>> x=arange(10) >>>>> indices=[(1,),(2,)] >>>>> x[indices] >> Traceback (most recent call last): >> ?File "", line 1, in >> IndexError: unsupported iterator index You are using two indices/coordinates in a 1d array >>> np.ravel([(1,),(2,)]) array([1, 2]) >>> np.arange(5)[np.ravel([(1,),(2,)])] array([1, 2]) >>> np.arange(5)[[1,2]] array([1, 2]) >>> np.arange(5)[1,2] Traceback (most recent call last): File "", line 1, in IndexError: invalid index >> >> >> But following works. >>>>> x=x.reshape(5,2) >>>>> x >> array([[0, 1], >> ? ? ? [2, 3], >> ? ? ? [4, 5], >> ? ? ? [6, 7], >> ? ? ? [8, 9]]) >>>>> indices=[(1,0),(0,1)] >>>>> x[indices] >> array([2, 1]) I don't know if it is clear in this example: >>> np.arange(10).reshape(5,2)[(1,0,1),(0,1,1)] array([2, 1, 3]) is equivalent to >>> [np.arange(10).reshape(5,2)[i,j] for i,j in zip((1,0,1),(0,1,1))] [2, 1, 3] Josef >> >> >> Should not the first one also work? How to convert [(1,),(2,)] to >> [1,2] efficiently. >> >> Thanks, >> Shailendra >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From robert.kern at gmail.com Sun Apr 4 12:45:16 2010 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 4 Apr 2010 11:45:16 -0500 Subject: [Numpy-discussion] Annoyance of memap rray withmultiprocessing.Pool.applay_async In-Reply-To: <710F2847B0018641891D9A21602763605AD393@ex3.envision.co.il> References: <710F2847B0018641891D9A21602763605AD392@ex3.envision.co.il> <710F2847B0018641891D9A21602763605AD393@ex3.envision.co.il> Message-ID: On Sat, Apr 3, 2010 at 22:35, Nadav Horesh wrote: > Got it, thank you. > But why, nevertheless, the results are correct although the pickling is impossible? Rather, I meant that they don't pickle correctly. They use ndarray's pickling, which will copy the data, and then reconstruct an ndarray on the other side and just change the type to memmap without actually memory-mapping the file. Thus you have a __del__ method referring to attributes that haven't been set up. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From pgmdevlist at gmail.com Mon Apr 5 03:08:05 2010 From: pgmdevlist at gmail.com (Pierre GM) Date: Mon, 5 Apr 2010 03:08:05 -0400 Subject: [Numpy-discussion] Possible bug in indexed masked arrays In-Reply-To: <87wrwqfnku.fsf@gmail.com> References: <87wrwqfnku.fsf@gmail.com> Message-ID: <4E4871E7-FB43-487D-807F-4B0022EB928A@gmail.com> On Apr 2, 2010, at 1:08 AM, Nathaniel Peterson wrote: > > Is this behavior of masked arrays intended, or is it a bug? It's not a bug, it's an unfortunate side effect of using boolean masked arrays for indices. Don't. Instead, you should fill the masked arrays with either True or False (depending on what you want). Now, for some explanations: > import numpy as np > a=np.ma.fix_invalid(np.array([np.nan,-1,0,1])) > b=np.ma.fix_invalid(np.array([np.nan,-1,0,1])) When using ma.fix_invalid, the nans and infs are masked and the corresponding set to a default (1e+20 for floats). Thus, you have: >>> print a.data [ 1.00000000e+20 -1.00000000e+00 0.00000000e+00 1.00000000e+00] > idx=(a==b) Now, you compare two masked arrays. In practice, the arrays are first filled with 0, compared, and the mask is created afterwards. In the current case, we get a new masked array, whose first entry is masked (because a[0] is masked), and because the two underlying ndarrays are identical, the underlying ndarray of the result is [True True True True]. > print(a[idx][3]) > # 1.0 The fun starts now: you are using idx, a masked array, as indices. Because the fancy indexing mechanism of numpy doesn't know how to process masked arrays, their underlying ndarray are used instead. Consider a[idx] equivalent to a[np.array(idx)]. Because np.array(idx) == idx.data == [True True True True], a[idx] returns a, hence the (4,) shape. > But if I change the first element of b from np.nan to 2.0 then > a[idx2] has shape (3,) despite np.alltrue(idx==idx2) being True: > > c=np.ma.fix_invalid(np.array([2.0,-1,0,1])) > idx2=(a==c) So, c is a masked array without any masked values. When comparing a and c, the arrays are once again filled with 0 before the comparison. The ndarray underlying idx2 is therefore [False True True True], and the first item is masked (still because a[0] is masked). If you use idx2 for indexing, it's transformed to a ndarray, and you end up with the last three items of a (hence the (3.) shape). > assert(np.alltrue(idx==idx2)) Now, you compare the two masked arrays idx and idx2. Remember the filling with 0 that happens below the hood, so you end up comparing [False True True True] and [False True True True] with np.alltrue, which of course returns True... Morale of the story: don't use masked arrays in fancy indexing, as you may not get what you expect. I hope it clarified the situation a bit, but don't hesitate to ask more questions. Cheers P. From nadavh at visionsense.com Mon Apr 5 04:08:08 2010 From: nadavh at visionsense.com (Nadav Horesh) Date: Mon, 5 Apr 2010 11:08:08 +0300 Subject: [Numpy-discussion] Annoyance of memap rraywithmultiprocessing.Pool.applay_async References: <710F2847B0018641891D9A21602763605AD392@ex3.envision.co.il> <710F2847B0018641891D9A21602763605AD393@ex3.envision.co.il> Message-ID: <710F2847B0018641891D9A21602763605AD394@ex3.envision.co.il> Is there a way to use memory mapped files as if they were shared memory? I made an application in which some (very often non contiguous) parts of a memmap array are processed by different processors. However I might use shared memory array instead. I wonder, since both types share common properties, if there a way to interchange then transparently. Nadav -----Original Message----- From: numpy-discussion-bounces at scipy.org on behalf of Robert Kern Sent: Sun 04-Apr-10 18:45 To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] Annoyance of memap rraywithmultiprocessing.Pool.applay_async On Sat, Apr 3, 2010 at 22:35, Nadav Horesh wrote: > Got it, thank you. > But why, nevertheless, the results are correct although the pickling is impossible? Rather, I meant that they don't pickle correctly. They use ndarray's pickling, which will copy the data, and then reconstruct an ndarray on the other side and just change the type to memmap without actually memory-mapping the file. Thus you have a __del__ method referring to attributes that haven't been set up. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 3824 bytes Desc: not available URL: From robert.kern at gmail.com Mon Apr 5 09:39:59 2010 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 5 Apr 2010 08:39:59 -0500 Subject: [Numpy-discussion] Annoyance of memap rraywithmultiprocessing.Pool.applay_async In-Reply-To: <710F2847B0018641891D9A21602763605AD394@ex3.envision.co.il> References: <710F2847B0018641891D9A21602763605AD392@ex3.envision.co.il> <710F2847B0018641891D9A21602763605AD393@ex3.envision.co.il> <710F2847B0018641891D9A21602763605AD394@ex3.envision.co.il> Message-ID: On Mon, Apr 5, 2010 at 03:08, Nadav Horesh wrote: > Is there a way to use memory mapped files as if they were shared memory? I made an application in which some (very often non contiguous) parts of a memmap array are processed by different processors. However I might use shared memory array instead. I wonder, since both types share common properties, if there a way to interchange then transparently. Yes, you just need to instantiate the memmap arrays separately in each process. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Mon Apr 5 11:40:38 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 5 Apr 2010 09:40:38 -0600 Subject: [Numpy-discussion] Math Library Message-ID: Hi All, David Cournapeau has mentioned that he would like to have a numpy math library that would supply missing functions and I'm wondering how we should organise the source code. Should we put a mathlib directory in numpy/core/src? Inside that directory would be functions for single/double/extended/quad precision. Should they be in separate directories? What about complex versions? I'm thinking that a good start would be to borrow the msun functions for doubles. We should also make a list of what functions would go into the library and what interface the complex functions present. Thoughts? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Mon Apr 5 11:43:43 2010 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 5 Apr 2010 10:43:43 -0500 Subject: [Numpy-discussion] Math Library In-Reply-To: References: Message-ID: On Mon, Apr 5, 2010 at 10:40, Charles R Harris wrote: > Hi All, > > David Cournapeau has mentioned that he would like to have a numpy math > library that would supply missing functions and I'm wondering how we should > organise the source code. Should we put a mathlib directory in > numpy/core/src? David already did this: numpy/core/src/npymath/ -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From kbasye1 at jhu.edu Mon Apr 5 11:44:29 2010 From: kbasye1 at jhu.edu (Ken Basye) Date: Mon, 05 Apr 2010 11:44:29 -0400 Subject: [Numpy-discussion] Extracting values from one array corresponding to argmax elements in another array Message-ID: <4BBA055D.9080607@jhu.edu> Hi Folks, I have two arrays, A and B, with the same shape. I want to find the highest values in A along some axis, then extract the corresponding values from B. I can get the highest values in A with A.max(axis=0) and the indices of these highest values with A.argmax(axis=0). I'm trying to figure out a loop-free way to extract the corresponding elements from B using these indices. Here's code with a loop that will do what I want for two-dimensional arrays: >>> a array([[ 100., 0., 0.], [ 0., 100., 100.], [ 0., 0., 0.]]) >>> a.max(axis=0) array([ 100., 100., 100.]) >>> sel = a.argmax(axis=0) >>>sel array([0, 1, 1]) >>> b = np.arange(9).reshape((3,3)) >>> b array([[0, 1, 2], [3, 4, 5], [6, 7, 8]]) >>> b_best = np.empty(3) >>> for i in xrange(3): ... b_best[i] = b[sel[i], i] ... >>> b_best array([ 0., 4., 5.]) I tried several approaches with take() but now that I understand how take() works when you give it an axis argument it seems like this isn't going to do what I want. Still, it seems like there should be some shortcut... TIA, Ken From d.l.goldsmith at gmail.com Mon Apr 5 11:45:23 2010 From: d.l.goldsmith at gmail.com (David Goldsmith) Date: Mon, 5 Apr 2010 08:45:23 -0700 Subject: [Numpy-discussion] Math Library In-Reply-To: References: Message-ID: On Mon, Apr 5, 2010 at 8:40 AM, Charles R Harris wrote: > Hi All, > > David Cournapeau has mentioned that he would like to have a numpy math > library that would supply missing functions and I'm wondering how we should > organise the source code. Should we put a mathlib directory in > numpy/core/src? Inside that directory would be functions for > single/double/extended/quad precision. Should they be in separate > directories? What about complex versions? I'm thinking that a good start > would be to borrow the msun functions for doubles. We should also make a > list of what functions would go into the library and what interface the > complex functions present. > > Thoughts? > For starters: you talking things like Airy, Bessel, Gamma, stuff like that? DG > > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -- Mathematician: noun, someone who disavows certainty when their uncertainty set is non-empty, even if that set has measure zero. -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Apr 5 11:56:08 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 5 Apr 2010 09:56:08 -0600 Subject: [Numpy-discussion] Math Library In-Reply-To: References: Message-ID: On Mon, Apr 5, 2010 at 9:43 AM, Robert Kern wrote: > On Mon, Apr 5, 2010 at 10:40, Charles R Harris > wrote: > > Hi All, > > > > David Cournapeau has mentioned that he would like to have a numpy math > > library that would supply missing functions and I'm wondering how we > should > > organise the source code. Should we put a mathlib directory in > > numpy/core/src? > > David already did this: numpy/core/src/npymath/ > > Yeah, but there isn't much low level stuff there and I don't want to toss a lot of real numerical code into it. Maybe a subdirectory? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Mon Apr 5 12:00:21 2010 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 5 Apr 2010 11:00:21 -0500 Subject: [Numpy-discussion] Math Library In-Reply-To: References: Message-ID: On Mon, Apr 5, 2010 at 10:56, Charles R Harris wrote: > > > On Mon, Apr 5, 2010 at 9:43 AM, Robert Kern wrote: >> >> On Mon, Apr 5, 2010 at 10:40, Charles R Harris >> wrote: >> > Hi All, >> > >> > David Cournapeau has mentioned that he would like to have a numpy math >> > library that would supply missing functions and I'm wondering how we >> > should >> > organise the source code. Should we put a mathlib directory in >> > numpy/core/src? >> >> David already did this: numpy/core/src/npymath/ >> > > Yeah, but there isn't much low level stuff there and I don't want to toss a > lot of real numerical code into it. Who cares? I don't. > Maybe a subdirectory? I'm guessing YAGNI. Code first. Reorganize later, if necessary. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From kwgoodman at gmail.com Mon Apr 5 12:06:29 2010 From: kwgoodman at gmail.com (Keith Goodman) Date: Mon, 5 Apr 2010 09:06:29 -0700 Subject: [Numpy-discussion] Extracting values from one array corresponding to argmax elements in another array In-Reply-To: <4BBA055D.9080607@jhu.edu> References: <4BBA055D.9080607@jhu.edu> Message-ID: On Mon, Apr 5, 2010 at 8:44 AM, Ken Basye wrote: > Hi Folks, > ?I have two arrays, A and B, with the same shape. ?I want to find the > highest values in A along some axis, then extract the corresponding > values from B. ?I can get the highest values in A with A.max(axis=0) and > the indices of these highest values with A.argmax(axis=0). ?I'm trying > to figure out a loop-free way to extract the corresponding elements from > B using these indices. ?Here's code with a loop that will do what I want > for two-dimensional arrays: > > ?>>> a > array([[ 100., ? ?0., ? ?0.], > ? ? ? [ ? 0., ?100., ?100.], > ? ? ? [ ? 0., ? ?0., ? ?0.]]) > > ?>>> a.max(axis=0) > array([ 100., ?100., ?100.]) > > ?>>> sel = a.argmax(axis=0) > ?>>>sel > array([0, 1, 1]) > > ?>>> b = np.arange(9).reshape((3,3)) > ?>>> b > array([[0, 1, 2], > ? ? ? [3, 4, 5], > ? ? ? [6, 7, 8]]) > > ?>>> b_best = np.empty(3) > ?>>> for i in xrange(3): > ... ? ?b_best[i] = b[sel[i], i] > ... > ?>>> b_best > array([ 0., ?4., ?5.]) Here's one way: >> b[a.argmax(axis=0), range(3)] array([0, 4, 5]) From charlesr.harris at gmail.com Mon Apr 5 12:11:53 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 5 Apr 2010 10:11:53 -0600 Subject: [Numpy-discussion] Math Library In-Reply-To: References: Message-ID: On Mon, Apr 5, 2010 at 10:00 AM, Robert Kern wrote: > On Mon, Apr 5, 2010 at 10:56, Charles R Harris > wrote: > > > > > > On Mon, Apr 5, 2010 at 9:43 AM, Robert Kern > wrote: > >> > >> On Mon, Apr 5, 2010 at 10:40, Charles R Harris > >> wrote: > >> > Hi All, > >> > > >> > David Cournapeau has mentioned that he would like to have a numpy math > >> > library that would supply missing functions and I'm wondering how we > >> > should > >> > organise the source code. Should we put a mathlib directory in > >> > numpy/core/src? > >> > >> David already did this: numpy/core/src/npymath/ > >> > > > > Yeah, but there isn't much low level stuff there and I don't want to toss > a > > lot of real numerical code into it. > > Who cares? I don't. > > I care. I want the code to be organized. > > Maybe a subdirectory? > > I'm guessing YAGNI. Code first. Reorganize later, if necessary. > > Would that be Are or Aren't? I'm going for the first. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From d.p.reichert at sms.ed.ac.uk Mon Apr 5 12:30:44 2010 From: d.p.reichert at sms.ed.ac.uk (David Reichert) Date: Mon, 5 Apr 2010 17:30:44 +0100 Subject: [Numpy-discussion] defmatrix move and unpickling of old data In-Reply-To: References: Message-ID: Hi, Just to let you know, I now fixed the problem using: import sys import numpy sys.modules['numpy.core.defmatrix'] = numpy.matrixlib.defmatrix The key is that the statement "import numpy.core.defmatrix" needs to work for unpickling to succeed, and just renaming things isn't enough. Cheers David On Sat, Apr 3, 2010 at 8:13 PM, David Reichert wrote: > Hi, > > After some work I got an optimized numpy compiled on a machine where I > don't > have root access, but I had to use numpy 1.4.0 to make it work. Now I have > the problem that I cannot seem to unpickle data I had created using numpy > 1.3, > getting an ImportError about defmatrix not being found. > > I understand defmatrix was moved from core to matrixlib? Is there some > workaround > I could use? I might have to move my data in between machines with either > versions of > numpy installed in the future as well... I already tried some renaming > tricks but to > no avail. > > Thanks > > David > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: not available URL: From robert.kern at gmail.com Mon Apr 5 12:43:04 2010 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 5 Apr 2010 11:43:04 -0500 Subject: [Numpy-discussion] Math Library In-Reply-To: References: Message-ID: On Mon, Apr 5, 2010 at 11:11, Charles R Harris wrote: > > > On Mon, Apr 5, 2010 at 10:00 AM, Robert Kern wrote: >> >> On Mon, Apr 5, 2010 at 10:56, Charles R Harris >> wrote: >> > >> > >> > On Mon, Apr 5, 2010 at 9:43 AM, Robert Kern >> > wrote: >> >> >> >> On Mon, Apr 5, 2010 at 10:40, Charles R Harris >> >> wrote: >> >> > Hi All, >> >> > >> >> > David Cournapeau has mentioned that he would like to have a numpy >> >> > math >> >> > library that would supply missing functions and I'm wondering how we >> >> > should >> >> > organise the source code. Should we put a mathlib directory in >> >> > numpy/core/src? >> >> >> >> David already did this: numpy/core/src/npymath/ >> >> >> > >> > Yeah, but there isn't much low level stuff there and I don't want to >> > toss a >> > lot of real numerical code into it. >> >> Who cares? I don't. > > I care. I want the code to be organized. Then do it when there is code and we can see what needs to be organized. >> > Maybe a subdirectory? >> >> I'm guessing YAGNI. Code first. Reorganize later, if necessary. > > Would that be Are or Aren't? I'm going for the first. "Ain't", actually. Show us the code first. Until you do, I can't imagine that there will be enough to add another layer of hierarchy to the four we already have. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Mon Apr 5 12:50:11 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 5 Apr 2010 10:50:11 -0600 Subject: [Numpy-discussion] Math Library In-Reply-To: References: Message-ID: On Mon, Apr 5, 2010 at 10:43 AM, Robert Kern wrote: > On Mon, Apr 5, 2010 at 11:11, Charles R Harris > wrote: > > > > > > On Mon, Apr 5, 2010 at 10:00 AM, Robert Kern > wrote: > >> > >> On Mon, Apr 5, 2010 at 10:56, Charles R Harris > >> wrote: > >> > > >> > > >> > On Mon, Apr 5, 2010 at 9:43 AM, Robert Kern > >> > wrote: > >> >> > >> >> On Mon, Apr 5, 2010 at 10:40, Charles R Harris > >> >> wrote: > >> >> > Hi All, > >> >> > > >> >> > David Cournapeau has mentioned that he would like to have a numpy > >> >> > math > >> >> > library that would supply missing functions and I'm wondering how > we > >> >> > should > >> >> > organise the source code. Should we put a mathlib directory in > >> >> > numpy/core/src? > >> >> > >> >> David already did this: numpy/core/src/npymath/ > >> >> > >> > > >> > Yeah, but there isn't much low level stuff there and I don't want to > >> > toss a > >> > lot of real numerical code into it. > >> > >> Who cares? I don't. > > > > I care. I want the code to be organized. > > Then do it when there is code and we can see what needs to be organized. > > I am writing code and I want to decide up front where to put it. I know where you stand, so you need say no more. I'm waiting to see if other folks have an opinion. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From d.l.goldsmith at gmail.com Mon Apr 5 12:56:10 2010 From: d.l.goldsmith at gmail.com (David Goldsmith) Date: Mon, 5 Apr 2010 09:56:10 -0700 Subject: [Numpy-discussion] Math Library In-Reply-To: References: Message-ID: On Mon, Apr 5, 2010 at 9:50 AM, Charles R Harris wrote: > On Mon, Apr 5, 2010 at 10:43 AM, Robert Kern wrote: > >> On Mon, Apr 5, 2010 at 11:11, Charles R Harris >> wrote: >> > >> > >> > On Mon, Apr 5, 2010 at 10:00 AM, Robert Kern >> wrote: >> >> >> >> On Mon, Apr 5, 2010 at 10:56, Charles R Harris >> >> wrote: >> >> > >> >> > >> >> > On Mon, Apr 5, 2010 at 9:43 AM, Robert Kern >> >> > wrote: >> >> >> >> >> >> On Mon, Apr 5, 2010 at 10:40, Charles R Harris >> >> >> wrote: >> >> >> > Hi All, >> >> >> > >> >> >> > David Cournapeau has mentioned that he would like to have a numpy >> >> >> > math >> >> >> > library that would supply missing functions and I'm wondering how >> we >> >> >> > should >> >> >> > organise the source code. Should we put a mathlib directory in >> >> >> > numpy/core/src? >> >> >> >> >> >> David already did this: numpy/core/src/npymath/ >> >> >> >> >> > >> >> > Yeah, but there isn't much low level stuff there and I don't want to >> >> > toss a >> >> > lot of real numerical code into it. >> >> >> >> Who cares? I don't. >> > >> > I care. I want the code to be organized. >> >> Then do it when there is code and we can see what needs to be organized. >> >> > I am writing code and I want to decide up front where to put it. I know > where you stand, so you need say no more. I'm waiting to see if other folks > have an opinion. > > Chuck > Will you be using it right away? If so, organize it locally how think it'll work best, work w/ it a little while and see if you "guessed" right or if you find yourself wanting to reorganize; then provide it to us w/ the benefit of your experience. :-) DG -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Mon Apr 5 13:10:34 2010 From: cournape at gmail.com (David Cournapeau) Date: Tue, 6 Apr 2010 02:10:34 +0900 Subject: [Numpy-discussion] Math Library In-Reply-To: References: Message-ID: On Tue, Apr 6, 2010 at 12:56 AM, Charles R Harris wrote: > >> > > Yeah, but there isn't much low level stuff there and I don't want to toss a > lot of real numerical code into it. I don't understand: there is already math code there, and you cannot be much more low level than what's there (there is already quite a bit of bit twiddling for long double). I split the code into complex and real, IEEE 754 macros/funcs in another file. I don' think we need to split into one file / function, at least not with the current size of the library. I think it is much more worthwhile to think about reorganizing the rest of numpy.core C code, the npymath library is very low hanging fruit in comparison, if only by size. David From charlesr.harris at gmail.com Mon Apr 5 13:11:09 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 5 Apr 2010 11:11:09 -0600 Subject: [Numpy-discussion] Math Library In-Reply-To: References: Message-ID: On Mon, Apr 5, 2010 at 10:56 AM, David Goldsmith wrote: > On Mon, Apr 5, 2010 at 9:50 AM, Charles R Harris < > charlesr.harris at gmail.com> wrote: > >> On Mon, Apr 5, 2010 at 10:43 AM, Robert Kern wrote: >> >>> On Mon, Apr 5, 2010 at 11:11, Charles R Harris >>> wrote: >>> > >>> > >>> > On Mon, Apr 5, 2010 at 10:00 AM, Robert Kern >>> wrote: >>> >> >>> >> On Mon, Apr 5, 2010 at 10:56, Charles R Harris >>> >> wrote: >>> >> > >>> >> > >>> >> > On Mon, Apr 5, 2010 at 9:43 AM, Robert Kern >>> >> > wrote: >>> >> >> >>> >> >> On Mon, Apr 5, 2010 at 10:40, Charles R Harris >>> >> >> wrote: >>> >> >> > Hi All, >>> >> >> > >>> >> >> > David Cournapeau has mentioned that he would like to have a numpy >>> >> >> > math >>> >> >> > library that would supply missing functions and I'm wondering how >>> we >>> >> >> > should >>> >> >> > organise the source code. Should we put a mathlib directory in >>> >> >> > numpy/core/src? >>> >> >> >>> >> >> David already did this: numpy/core/src/npymath/ >>> >> >> >>> >> > >>> >> > Yeah, but there isn't much low level stuff there and I don't want to >>> >> > toss a >>> >> > lot of real numerical code into it. >>> >> >>> >> Who cares? I don't. >>> > >>> > I care. I want the code to be organized. >>> >>> Then do it when there is code and we can see what needs to be organized. >>> >>> >> I am writing code and I want to decide up front where to put it. I know >> where you stand, so you need say no more. I'm waiting to see if other folks >> have an opinion. >> >> Chuck >> > > Will you be using it right away? If so, organize it locally how think > it'll work best, work w/ it a little while and see if you "guessed" right or > if you find yourself wanting to reorganize; then provide it to us w/ the > benefit of your experience. :-) > > No, but since at some point it will involve the numpy build I would like some feedback from David C. on how he thinks it should be organized. The first routines I want to add are for log1p. Note that BSD has both single and double versions but the single version copies the approximation coefficients from the double. BSD doesn't have extended or quad precision versions. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Apr 5 13:14:13 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 5 Apr 2010 11:14:13 -0600 Subject: [Numpy-discussion] Math Library In-Reply-To: References: Message-ID: On Mon, Apr 5, 2010 at 11:10 AM, David Cournapeau wrote: > On Tue, Apr 6, 2010 at 12:56 AM, Charles R Harris > wrote: > > > >> > > > > Yeah, but there isn't much low level stuff there and I don't want to toss > a > > lot of real numerical code into it. > > I don't understand: there is already math code there, and you cannot > be much more low level than what's there (there is already quite a bit > of bit twiddling for long double). I split the code into complex and > real, IEEE 754 macros/funcs in another file. I don' think we need to > split into one file / function, at least not with the current size of > the library. > > Yeah, but the added code in four versions with documentation for log1p alone will add substantially to the current size. What I am saying is that the current code is small because it uses current functions or falls back to double versions. It doesn't really implement the low level stuff. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From erik.tollerud at gmail.com Mon Apr 5 14:26:39 2010 From: erik.tollerud at gmail.com (Erik Tollerud) Date: Mon, 5 Apr 2010 11:26:39 -0700 Subject: [Numpy-discussion] How do I ensure numpy headers are present in setup.py? In-Reply-To: References: Message-ID: Hmm, unfortunate. So the best approach then is probably just to tell people to install numpy first, then my package? On Fri, Apr 2, 2010 at 12:06 PM, Robert Kern wrote: > On Fri, Apr 2, 2010 at 13:03, Erik Tollerud wrote: >> I am writing a setup.py file for a package that will use cython with >> numpy integration. ?This of course requires the numpy header files, >> which I am including by using numpy.get_includes in the setup.py file >> below. ?The problem is for users that have not installed numpy before >> installing this package. ? ?If they have setuptools installed, the >> behavior I would want would be for numpy to be downloaded and then the >> setup script should be able to get at the headers even if it doesn't >> install numpy until after this package is installed. ?But that doesn't >> work - I have to import numpy in the setup script, which fails if it >> is not yet installed. ?So How can I get the behavior I want? > > You can't, not without some hacks to distutils. This is a basic > problem with the way setuptools uses arguments to setup() to get the > dependencies. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > ?-- Umberto Eco > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From robert.kern at gmail.com Mon Apr 5 14:28:20 2010 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 5 Apr 2010 13:28:20 -0500 Subject: [Numpy-discussion] How do I ensure numpy headers are present in setup.py? In-Reply-To: References: Message-ID: On Mon, Apr 5, 2010 at 13:26, Erik Tollerud wrote: > Hmm, unfortunate. ?So the best approach then is probably just to tell > people to install numpy first, then my package? Yup. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From nathanielpeterson08 at gmail.com Mon Apr 5 14:36:45 2010 From: nathanielpeterson08 at gmail.com (Nathaniel Peterson) Date: Mon, 05 Apr 2010 14:36:45 -0400 Subject: [Numpy-discussion] Possible bug in indexed masked arrays In-Reply-To: <4E4871E7-FB43-487D-807F-4B0022EB928A@gmail.com> (message from Pierre GM on Mon, 5 Apr 2010 03:08:05 -0400) References: <87wrwqfnku.fsf@gmail.com> <4E4871E7-FB43-487D-807F-4B0022EB928A@gmail.com> Message-ID: <87sk79pwzm.fsf@gmail.com> Pierre, Thank you for the wonderful explanation. I get it! np.alltrue(idx.data == idx2.data) is False. PS. Thank you for closing ticket #1447; sorry for the trouble. From pgmdevlist at gmail.com Mon Apr 5 14:46:42 2010 From: pgmdevlist at gmail.com (Pierre GM) Date: Mon, 5 Apr 2010 14:46:42 -0400 Subject: [Numpy-discussion] Possible bug in indexed masked arrays In-Reply-To: <87sk79pwzm.fsf@gmail.com> References: <87wrwqfnku.fsf@gmail.com> <4E4871E7-FB43-487D-807F-4B0022EB928A@gmail.com> <87sk79pwzm.fsf@gmail.com> Message-ID: <80DE52AD-4EC3-4316-AD44-A3EEF9A075B6@gmail.com> On Apr 5, 2010, at 2:36 PM, Nathaniel Peterson wrote: > Pierre, Thank you for the wonderful explanation. > I get it! np.alltrue(idx.data == idx2.data) is False. > > PS. Thank you for closing ticket #1447; sorry for the trouble. No problem whatsoever. Thanks for your patience... From oliphant at enthought.com Mon Apr 5 18:43:41 2010 From: oliphant at enthought.com (Travis Oliphant) Date: Mon, 5 Apr 2010 17:43:41 -0500 Subject: [Numpy-discussion] Math Library In-Reply-To: References: Message-ID: On Apr 5, 2010, at 12:10 PM, David Cournapeau wrote: > On Tue, Apr 6, 2010 at 12:56 AM, Charles R Harris > wrote: >> >>> >> >> Yeah, but there isn't much low level stuff there and I don't want >> to toss a >> lot of real numerical code into it. > > I don't understand: there is already math code there, and you cannot > be much more low level than what's there (there is already quite a bit > of bit twiddling for long double). I split the code into complex and > real, IEEE 754 macros/funcs in another file. I don' think we need to > split into one file / function, at least not with the current size of > the library. > > I think it is much more worthwhile to think about reorganizing the > rest of numpy.core C code, the npymath library is very low hanging > fruit in comparison, if only by size. I should have some time over the next couple of weeks, and I am very interested in refactoring the NumPy code to separate out the Python interface layer from the "library" layer as much as possible. I had some discussions with people at PyCon about making it easier for Jython, IronPython, and perhaps even other high-level languages to utilize NumPy. Is there a willingness to consider as part of this reorganization creating a clear boundary between the NumPy library code and the Python-specific interface to it? What other re-organization thoughts are you having David? -Travis From tjhnson at gmail.com Mon Apr 5 23:14:21 2010 From: tjhnson at gmail.com (T J) Date: Mon, 5 Apr 2010 20:14:21 -0700 Subject: [Numpy-discussion] How do I ensure numpy headers are present in setup.py? In-Reply-To: References: Message-ID: On Mon, Apr 5, 2010 at 11:28 AM, Robert Kern wrote: > On Mon, Apr 5, 2010 at 13:26, Erik Tollerud wrote: >> Hmm, unfortunate. ?So the best approach then is probably just to tell >> people to install numpy first, then my package? > > Yup. > And really, this isn't that unreasonable. Not only does this make users more aware of their environment (ie, the distinction between your package and the major numerical package in Python), but its so much cleaner. With the combined approach, any NumPy installation problems would be (frequently) associated with your package. On the other hand, if there are any NumPy installation problems in the separated approach, at least your package is blameless. From gael.varoquaux at normalesup.org Tue Apr 6 01:35:07 2010 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Tue, 6 Apr 2010 07:35:07 +0200 Subject: [Numpy-discussion] Math Library In-Reply-To: References: Message-ID: <20100406053507.GB15501@phare.normalesup.org> On Mon, Apr 05, 2010 at 05:43:41PM -0500, Travis Oliphant wrote: > I should have some time over the next couple of weeks, and I am very > interested in refactoring the NumPy code to separate out the Python > interface layer from the "library" layer as much as possible. I had > some discussions with people at PyCon about making it easier for > Jython, IronPython, and perhaps even other high-level languages to > utilize NumPy. > Is there a willingness to consider as part of this reorganization > creating a clear boundary between the NumPy library code and the > Python-specific interface to it? What other re-organization thoughts > are you having David? I have been following discussion too well, so please pardon me if my answer is off topic or irrelevant... At work, we want to code in a mixture of Python and C, optimizing only the bottlenecks of the computation in C. When we want to use numerical facilities in C, we would like to benefit from the fact that numpy already went through the hard work of getting basic vectorized math compiled and running on the user's computer. Indeed, one of the issues that we have been facing lately is that deploying a Python application with some C can increase a lot the difficulty of building and installing due to the required C libraries. The reason I bring this up, is that your refactor could make it easier for C or Cython coders to use the numpy internal to do their own dirty work. If the corresponding functions are exposed in the numpy headers, it would be fairly easy to include them in a numpy.distutils-driven build, via a call to 'numpy.get_include()'. My 2 cents, Ga?l From pjs67 at cam.ac.uk Tue Apr 6 05:23:17 2010 From: pjs67 at cam.ac.uk (Philip Sterne) Date: Tue, 06 Apr 2010 10:23:17 +0100 Subject: [Numpy-discussion] Getting data from NDarrays to Blitz++ and back again In-Reply-To: <4BB7B313.2060602@cam.ac.uk> References: <4BB7B313.2060602@cam.ac.uk> Message-ID: <4BBAFD85.9010507@cam.ac.uk> If anyone was interested I found that the easiest solution involved patching up the strides after calling PyArray_SimpleNewFromData(). I still haven't gotten any sort of memory interaction so any objects created by Blitz are deleted by Blitz, while Python objects are deleted by Python. (Irrespective of any pointers held by the other side.) I find for my purposes that this is sufficient since I just want to be able to peer into the workings of a c++ program. -Philip. Philip Sterne wrote: > Hi all, > I hope this is the correct place to post my question. I'd like python > to interface with c++ code that makes heavy use of Blitz++ arrays. After > a day's hacking I appear to have a simple working solution which I am > attaching. (It uses Boost.Python and CMake.) > > However I believe this solution won't work if the blitz array is not > laid out contiguously in memory. I also haven't really thought about > reference counting issues (although the example seems to work). I > imagine that those sorts of issues will lead me to call the more > complicated: > PyArray_New(...) > or: > PyArray_NewFromDescr(...) > instead of the PyArray_SimpleNewFromData(...) that I currently use. > However I couldn't figure out some of the extra arguments from the API > documentation. > > Can someone point out all the things that will break when this code > actually gets used in the real world (and maybe even how to avoid them)? > > Thanks for your time! > -Philip. > > > ------------------------------------------------------------------------ > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: simple.cpp Type: text/x-c++src Size: 1894 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: simpletest.py Type: text/x-python Size: 160 bytes Desc: not available URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: CMakeLists.txt URL: From yogesh.tomar at gmail.com Tue Apr 6 06:46:17 2010 From: yogesh.tomar at gmail.com (Yogesh Tomar) Date: Tue, 6 Apr 2010 16:16:17 +0530 Subject: [Numpy-discussion] Numpy Segmentation Fault In-Reply-To: References: Message-ID: Hi all, I am getting numpy segmentation fault on a custom install of python to a prefix. I am running this example code. *import numpy import numpy.linalg x=numpy.eye(1000) for i in range(10): eigenvalues,eigenvectors=numpy.linalg.eig(x) eigenvalues,eigenvectors=numpy.linalg.eig(x) print str(i),'-------------------------------'* I have been trying to debug this for the last two weeks and with no success so far. The same code runs fine with Python-2.6.1 and numpy-1.2.1 but seg faults in Python-2.6.4 and numpy-1.3.0 Also, the error occurs non-deterministically in the loop and sometimes it does not even occur at all. I have tried reinstalling and rebuilding python but even that does not help. Please find attached the information which I think might be relevant for us to figure out the root cause. System info *Python 2.6.4 (r264:75706, Apr 6 2010, 04:49:11) [GCC 4.1.2 20071124 (Red Hat 4.1.2-42)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>>* strace output tail *) = 34 futex(0x1001f6e0, FUTEX_WAKE, 1) = 0 futex(0x1001f6e0, FUTEX_WAKE, 1) = 0 write(1, "here\n", 5here ) = 5 rt_sigaction(SIGINT, {SIG_DFL}, {0x2b78c55fec60, [], SA_RESTORER, 0x34d940de70}, 8) = 0 brk(0x117ff000) = 0x117ff000 brk(0x108b1000) = 0x108b1000 --- SIGSEGV (Segmentation fault) @ 0 (0) --- +++ killed by SIGSEGV +++* Regards, Yogesh Tomar -------------- next part -------------- An HTML attachment was scrubbed... URL: From yogesh.tomar at gmail.com Tue Apr 6 06:48:06 2010 From: yogesh.tomar at gmail.com (Yogesh Tomar) Date: Tue, 6 Apr 2010 16:18:06 +0530 Subject: [Numpy-discussion] Numpy linalg.eig seg faults Message-ID: Hi all, I am getting numpy segmentation fault on a custom install of python to a prefix. I am running this example code. *import numpy import numpy.linalg x=numpy.eye(1000) for i in range(10): eigenvalues,eigenvectors=numpy.linalg.eig(x) eigenvalues,eigenvectors=numpy.linalg.eig(x) print str(i),'-------------------------------'* I have been trying to debug this for the last two weeks and with no success so far. The same code runs fine with Python-2.6.1 and numpy-1.2.1 but seg faults in Python-2.6.4 and numpy-1.3.0 Also, the error occurs non-deterministically in the loop and sometimes it does not even occur at all. I have tried reinstalling and rebuilding python but even that does not help. Please find some information which I think might be relevant for us to figure out the root cause. System info *Python 2.6.4 (r264:75706, Apr 6 2010, 04:49:11) [GCC 4.1.2 20071124 (Red Hat 4.1.2-42)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>>* strace output tail *) = 34 futex(0x1001f6e0, FUTEX_WAKE, 1) = 0 futex(0x1001f6e0, FUTEX_WAKE, 1) = 0 write(1, "here\n", 5here ) = 5 rt_sigaction(SIGINT, {SIG_DFL}, {0x2b78c55fec60, [], SA_RESTORER, 0x34d940de70}, 8) = 0 brk(0x117ff000) = 0x117ff000 brk(0x108b1000) = 0x108b1000 --- SIGSEGV (Segmentation fault) @ 0 (0) --- +++ killed by SIGSEGV +++ *Regards, Yogesh Tomar -------------- next part -------------- An HTML attachment was scrubbed... URL: From schut at sarvision.nl Tue Apr 6 06:51:12 2010 From: schut at sarvision.nl (Vincent Schut) Date: Tue, 06 Apr 2010 12:51:12 +0200 Subject: [Numpy-discussion] Extracting values from one array corresponding to argmax elements in another array In-Reply-To: References: <4BBA055D.9080607@jhu.edu> Message-ID: On 04/05/2010 06:06 PM, Keith Goodman wrote: > On Mon, Apr 5, 2010 at 8:44 AM, Ken Basye wrote: >> Hi Folks, >> I have two arrays, A and B, with the same shape. I want to find the >> highest values in A along some axis, then extract the corresponding >> values from B. I can get the highest values in A with A.max(axis=0) and >> the indices of these highest values with A.argmax(axis=0). I'm trying >> to figure out a loop-free way to extract the corresponding elements from >> B using these indices. Here's code with a loop that will do what I want >> for two-dimensional arrays: >> >> >>> a >> array([[ 100., 0., 0.], >> [ 0., 100., 100.], >> [ 0., 0., 0.]]) >> >> >>> a.max(axis=0) >> array([ 100., 100., 100.]) >> >> >>> sel = a.argmax(axis=0) >> >>>sel >> array([0, 1, 1]) >> >> >>> b = np.arange(9).reshape((3,3)) >> >>> b >> array([[0, 1, 2], >> [3, 4, 5], >> [6, 7, 8]]) >> >> >>> b_best = np.empty(3) >> >>> for i in xrange(3): >> ... b_best[i] = b[sel[i], i] >> ... >> >>> b_best >> array([ 0., 4., 5.]) > > Here's one way: > >>> b[a.argmax(axis=0), range(3)] > array([0, 4, 5]) Which does not work anymore when your arrays become more-dimensional (like in my case: 4 or more) and the axis you want to select on is not the first/last one. If I recall correctly, I needed to construct the full index arrays for the other dimensions too (like with ogrid I think). So: create the ogrid, replace the one for the dimensions you want the argmax selection to take place on with the argmax parameter, and use those index arrays to index your b array. I'd need to look up my source code to be more sure/precise. If anyone would like me to, please let me know. If anyone knows a less elaborate way, also please let us know! :-) From cournape at gmail.com Tue Apr 6 06:52:48 2010 From: cournape at gmail.com (David Cournapeau) Date: Tue, 6 Apr 2010 19:52:48 +0900 Subject: [Numpy-discussion] Numpy Segmentation Fault In-Reply-To: References: Message-ID: On Tue, Apr 6, 2010 at 7:46 PM, Yogesh Tomar wrote: > > Hi all, > > I am getting numpy segmentation fault on a custom install of python to a > prefix. What happens if you build numpy 1.3.0 against python 2.6.1 (instead of your own 2.6.4) ? For numpy, a strace is not really useful (a backtrace from gdb much more), cheers, David From ytomar at gmail.com Tue Apr 6 07:28:53 2010 From: ytomar at gmail.com (Yogesh Tomar) Date: Tue, 6 Apr 2010 16:58:53 +0530 Subject: [Numpy-discussion] Numpy Segmentation Fault In-Reply-To: References: Message-ID: numpy 1.3.0 also segfaults the same way. Is it the problem with libc library? Gdb stacktrace. gdb) run /home/eqan/64bit/current/segf.py Starting program: /home/eqan/tapas/64bit/Python/2.6.4_x641/bin/python /home/eqan/64bit/current/segf.py [Thread debugging using libthread_db enabled] [New Thread 47653213733440 (LWP 24970)] 0 ------------------------------- 1 ------------------------------- 2 ------------------------------- 3 ------------------------------- 4 ------------------------------- 5 ------------------------------- 6 ------------------------------- 7 ------------------------------- 8 ------------------------------- 9 ------------------------------- here Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 47653213733440 (LWP 24970)] 0x00000034d8c71033 in _int_free () from /lib64/libc.so.6 (gdb) up #1 0x00000034d8c74c5c in free () from /lib64/libc.so.6 (gdb) up #2 0x00002b57203451db in code_dealloc (co=0xdab63f0) at Objects/codeobject.c:260 260 Py_XDECREF(co->co_code); (gdb) up #3 0x00002b5720359a73 in func_dealloc (op=0xdab67d0) at Objects/funcobject.c:454 454 Py_DECREF(op->func_code); (gdb) up #4 0x00002b57203691ed in insertdict (mp=0xda8ae90, key=0xda2f180, hash=1904558708720393281, value=0x2b572066c210) at Objects/dictobject.c:459 459 Py_DECREF(old_value); /* which **CAN** re-enter */ (gdb) up #5 0x00002b572036b153 in PyDict_SetItem (op=0xda8ae90, key=0xda2f180, value=0x2b572066c210) at Objects/dictobject.c:701 701 if (insertdict(mp, key, hash, value) != 0) (gdb) up #6 0x00002b572036d5c8 in _PyModule_Clear (m=) at Objects/moduleobject.c:138 138 PyDict_SetItem(d, key, Py_None); (gdb) up #7 0x00002b57203dfa34 in PyImport_Cleanup () at Python/import.c:474 474 _PyModule_Clear(value); (gdb) up #8 0x00002b57203ed167 in Py_Finalize () at Python/pythonrun.c:434 434 PyImport_Cleanup(); (gdb) up #9 0x00002b57203f9cdc in Py_Main (argc=, argv=0x7fff8a7bf478) at Modules/main.c:625 625 Py_Finalize(); (gdb) up #10 0x00000034d8c1d8b4 in __libc_start_main () from /lib64/libc.so.6 (gdb) up Regards, Yogesh Tomar On Tue, Apr 6, 2010 at 4:22 PM, David Cournapeau wrote: > On Tue, Apr 6, 2010 at 7:46 PM, Yogesh Tomar > wrote: > > > > Hi all, > > > > I am getting numpy segmentation fault on a custom install of python to a > > prefix. > > What happens if you build numpy 1.3.0 against python 2.6.1 (instead of > your own 2.6.4) ? For numpy, a strace is not really useful (a > backtrace from gdb much more), > > cheers, > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Tue Apr 6 07:47:15 2010 From: cournape at gmail.com (David Cournapeau) Date: Tue, 6 Apr 2010 20:47:15 +0900 Subject: [Numpy-discussion] Numpy Segmentation Fault In-Reply-To: References: Message-ID: On Tue, Apr 6, 2010 at 8:28 PM, Yogesh Tomar wrote: > numpy 1.3.0 also segfaults the same way. I mean building numpy 1.3.0 against python 2.6.1 instead of 2.6.4 - since the crash happen on a python you built by yourself, that's the first thing I would look into before looking into numpy or python bug. > Is it the problem with libc library? Very unlikely, this looks like a ref count bug, cheers, David From ytomar at gmail.com Tue Apr 6 08:42:14 2010 From: ytomar at gmail.com (Yogesh Tomar) Date: Tue, 6 Apr 2010 18:12:14 +0530 Subject: [Numpy-discussion] Numpy Segmentation Fault In-Reply-To: References: Message-ID: I also think the same. There is some problem with my python installation. Because a similar installation python-2.6.4 and numpy-1.3.0 which I did elsewhere does not seg fault for the same code. But I need your help to figure it out. Can you please elaborate on ref count bug? that might help. Regards, Yogesh Tomar On Tue, Apr 6, 2010 at 5:17 PM, David Cournapeau wrote: > On Tue, Apr 6, 2010 at 8:28 PM, Yogesh Tomar wrote: > > numpy 1.3.0 also segfaults the same way. > > I mean building numpy 1.3.0 against python 2.6.1 instead of 2.6.4 - > since the crash happen on a python you built by yourself, that's the > first thing I would look into before looking into numpy or python bug. > > > Is it the problem with libc library? > > Very unlikely, this looks like a ref count bug, > > cheers, > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aisaac at american.edu Tue Apr 6 08:42:56 2010 From: aisaac at american.edu (Alan G Isaac) Date: Tue, 06 Apr 2010 08:42:56 -0400 Subject: [Numpy-discussion] Numpy Segmentation Fault In-Reply-To: References: Message-ID: <4BBB2C50.70908@american.edu> On 4/6/2010 6:46 AM, Yogesh Tomar wrote: > import numpy > import numpy.linalg > x=numpy.eye(1000) > for i in range(10): > eigenvalues,eigenvectors=numpy.linalg.eig(x) > eigenvalues,eigenvectors=numpy.linalg.eig(x) > print str(i),'-------------------------------'* I'm not seeing any problem with 1.4.1rc. Alan Isaac (Python 2.6.5 on Vista) From kbasye1 at jhu.edu Tue Apr 6 09:22:10 2010 From: kbasye1 at jhu.edu (Ken Basye) Date: Tue, 06 Apr 2010 09:22:10 -0400 Subject: [Numpy-discussion] Extracting values from one array corresponding to argmax elements in another array In-Reply-To: References: Message-ID: <4BBB3582.5080205@jhu.edu> From: Vincent Schut > On 04/05/2010 06:06 PM, Keith Goodman wrote: > >> On Mon, Apr 5, 2010 at 8:44 AM, Ken Basye wrote: >> >>> Hi Folks, >>> I have two arrays, A and B, with the same shape. I want to find the >>> highest values in A along some axis, then extract the corresponding >>> values from B. I can get the highest values in A with A.max(axis=0) and >>> the indices of these highest values with A.argmax(axis=0). I'm trying >>> to figure out a loop-free way to extract the corresponding elements from >>> B using these indices. Here's code with a loop that will do what I want >>> for two-dimensional arrays: >>> >>> >>> a >>> array([[ 100., 0., 0.], >>> [ 0., 100., 100.], >>> [ 0., 0., 0.]]) >>> >>> >>> a.max(axis=0) >>> array([ 100., 100., 100.]) >>> >>> >>> sel = a.argmax(axis=0) >>> >>>sel >>> array([0, 1, 1]) >>> >>> >>> b = np.arange(9).reshape((3,3)) >>> >>> b >>> array([[0, 1, 2], >>> [3, 4, 5], >>> [6, 7, 8]]) >>> >>> >>> b_best = np.empty(3) >>> >>> for i in xrange(3): >>> ... b_best[i] = b[sel[i], i] >>> ... >>> >>> b_best >>> array([ 0., 4., 5.]) >>> >> Here's one way: >> >> >>>> b[a.argmax(axis=0), range(3)] >>>> >> array([0, 4, 5]) >> > > Which does not work anymore when your arrays become more-dimensional > (like in my case: 4 or more) and the axis you want to select on is not > the first/last one. If I recall correctly, I needed to construct the > full index arrays for the other dimensions too (like with ogrid I > think). So: create the ogrid, replace the one for the dimensions you > want the argmax selection to take place on with the argmax parameter, > and use those index arrays to index your b array. > I'd need to look up my source code to be more sure/precise. If anyone > would like me to, please let me know. If anyone knows a less elaborate > way, also please let us know! :-) > Hi Vincent, I'd like to see more about your solution. For my present purposes, Keith's solution was sufficient, but I'm still very interested in a solution that's independent of dimension and axis. Thanks (and thanks, Keith), Ken -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Tue Apr 6 09:33:29 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 6 Apr 2010 09:33:29 -0400 Subject: [Numpy-discussion] Extracting values from one array corresponding to argmax elements in another array In-Reply-To: <4BBB3582.5080205@jhu.edu> References: <4BBB3582.5080205@jhu.edu> Message-ID: On Tue, Apr 6, 2010 at 9:22 AM, Ken Basye wrote: > From: Vincent Schut > > On 04/05/2010 06:06 PM, Keith Goodman wrote: > > > On Mon, Apr 5, 2010 at 8:44 AM, Ken Basye wrote: > > > Hi Folks, > I have two arrays, A and B, with the same shape. I want to find the > highest values in A along some axis, then extract the corresponding > values from B. I can get the highest values in A with A.max(axis=0) and > the indices of these highest values with A.argmax(axis=0). I'm trying > to figure out a loop-free way to extract the corresponding elements from > B using these indices. Here's code with a loop that will do what I want > for two-dimensional arrays: > > >>> a > array([[ 100., 0., 0.], > [ 0., 100., 100.], > [ 0., 0., 0.]]) > > >>> a.max(axis=0) > array([ 100., 100., 100.]) > > >>> sel = a.argmax(axis=0) > >>>sel > array([0, 1, 1]) > > >>> b = np.arange(9).reshape((3,3)) > >>> b > array([[0, 1, 2], > [3, 4, 5], > [6, 7, 8]]) > > >>> b_best = np.empty(3) > >>> for i in xrange(3): > ... b_best[i] = b[sel[i], i] > ... > >>> b_best > array([ 0., 4., 5.]) > > > Here's one way: > > > > b[a.argmax(axis=0), range(3)] > > > array([0, 4, 5]) > > > Which does not work anymore when your arrays become more-dimensional > (like in my case: 4 or more) and the axis you want to select on is not > the first/last one. If I recall correctly, I needed to construct the > full index arrays for the other dimensions too (like with ogrid I > think). So: create the ogrid, replace the one for the dimensions you > want the argmax selection to take place on with the argmax parameter, > and use those index arrays to index your b array. > I'd need to look up my source code to be more sure/precise. If anyone > would like me to, please let me know. If anyone knows a less elaborate > way, also please let us know! :-) > > > Hi Vincent, > ? I'd like to see more about your solution.? For my present purposes, > Keith's solution was sufficient, but I'm still very interested in a solution > that's independent of dimension and axis. > ? Thanks (and thanks, Keith), > ???? Ken an alternative to Vincent's general solution, if you have unique max or want all argmax is using a mask >>> a array([[ 100., 0., 0.], [ 0., 100., 100.], [ 0., 0., 0.]]) >>> b array([[0, 1, 2], [3, 4, 5], [6, 7, 8]]) >>> ax=0; b[a==np.expand_dims(a.max(ax),ax)] array([0, 4, 5]) >>> ax=1; b[a==np.expand_dims(a.max(ax),ax)] array([0, 4, 5, 6, 7, 8]) >>> aa=np.eye(3) >>> ax=1; b[aa==np.expand_dims(aa.max(ax),ax)] array([0, 4, 8]) >>> ax=0; b[aa==np.expand_dims(aa.max(ax),ax)] array([0, 4, 8]) Josef > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From cournape at gmail.com Tue Apr 6 10:08:39 2010 From: cournape at gmail.com (David Cournapeau) Date: Tue, 6 Apr 2010 23:08:39 +0900 Subject: [Numpy-discussion] Math Library In-Reply-To: References: Message-ID: Hi Travis, On Tue, Apr 6, 2010 at 7:43 AM, Travis Oliphant wrote: > > > I should have some time over the next couple of weeks, and I am very > interested in refactoring the NumPy code to separate out the Python > interface layer from the "library" layer as much as possible. ? I had > some discussions with people at PyCon about making it easier for > Jython, IronPython, and perhaps even other high-level languages to > utilize NumPy. > > Is there a willingness to consider as part of this reorganization > creating a clear boundary between the NumPy library code and the > Python-specific interface to it? ? What other re-organization thoughts > are you having David? This is mainly it, reorganizing the code for clearer boundaries between boilerplate (python C API) and actual compuational code. Besides helping other python implementations, I think this would benefit NumPy itself in the long run (code maintainability), as well as scipy (and other C extensions). I think the npymath effort is a good example: albeit simple in nature (the API and boundaries are obvious), it has already helped a lot to solve numerous platform specific issues in numpy and scipy, and I think the overall code quality is better. My own goals were: - exposing core computational parts through an exported C API, so that other C extensions may use it (for example, exposing basic blas/lapack operations) - dynamic loading of the code (for example depending on the CPU capabilities - I have a git branch somewhere where I started exposing a simple C API to query cpu capabilities like cache size or SSE dynamically to that intent) - more amenable codebase: I think multiarray in particular is too big. I don't know the code well enough to know what can be split and how, but I would have hoped that the scalartypes, the type descriptor could be put out of multiarray proper. Also, exposing an API for things like fancy indexing would be very useful, but I don't know if it even makes sense - I think a pure python implementation of fancy indexing as a reference would be very useful for array-like classes (I am thinking about scipy.sparse, for example). Unfortunately, I won't be able to help much in the near future (except maybe for the fancy indexing as this could be useful for my job), David From oliphant at enthought.com Tue Apr 6 15:16:37 2010 From: oliphant at enthought.com (Travis Oliphant) Date: Tue, 6 Apr 2010 14:16:37 -0500 Subject: [Numpy-discussion] Math Library In-Reply-To: References: Message-ID: On Apr 6, 2010, at 9:08 AM, David Cournapeau wrote: > Hi Travis, > > On Tue, Apr 6, 2010 at 7:43 AM, Travis Oliphant > wrote: >> >> >> I should have some time over the next couple of weeks, and I am very >> interested in refactoring the NumPy code to separate out the Python >> interface layer from the "library" layer as much as possible. I had >> some discussions with people at PyCon about making it easier for >> Jython, IronPython, and perhaps even other high-level languages to >> utilize NumPy. >> >> Is there a willingness to consider as part of this reorganization >> creating a clear boundary between the NumPy library code and the >> Python-specific interface to it? What other re-organization >> thoughts >> are you having David? > > This is mainly it, reorganizing the code for clearer boundaries > between boilerplate (python C API) and actual compuational code. > Besides helping other python implementations, I think this would > benefit NumPy itself in the long run (code maintainability), as well > as scipy (and other C extensions). I think the npymath effort is a > good example: albeit simple in nature (the API and boundaries are > obvious), it has already helped a lot to solve numerous platform > specific issues in numpy and scipy, and I think the overall code > quality is better. > > My own goals were: > - exposing core computational parts through an exported C API, so > that other C extensions may use it (for example, exposing basic > blas/lapack operations) > - dynamic loading of the code (for example depending on the CPU > capabilities - I have a git branch somewhere where I started exposing > a simple C API to query cpu capabilities like cache size or SSE > dynamically to that intent) > - more amenable codebase: I think multiarray in particular is too > big. I don't know the code well enough to know what can be split and > how, but I would have hoped that the scalartypes, the type descriptor > could be put out of multiarray proper. Also, exposing an API for > things like fancy indexing would be very useful, but I don't know if > it even makes sense - I think a pure python implementation of fancy > indexing as a reference would be very useful for array-like classes (I > am thinking about scipy.sparse, for example). > > Unfortunately, I won't be able to help much in the near future (except > maybe for the fancy indexing as this could be useful for my job), > I understand. It just happens that there is some significant time for me to look at this over the next few weeks and I would really like to make progress on re-factoring. I think it's O.K. if you don't have time right now to help as long as you have time to offer criticism and suggestions. We could even do that over Skype with whomever else wanted to join us (we could do a GotoMeeting discussion as well) if you think it would be faster to just talk in a group setting instead of email. Of course, a summary of any off-line discussion should be sent to the list. Thanks for the input, -Travis -------------- next part -------------- An HTML attachment was scrubbed... URL: From kbasye1 at jhu.edu Tue Apr 6 15:42:41 2010 From: kbasye1 at jhu.edu (Ken Basye) Date: Tue, 06 Apr 2010 15:42:41 -0400 Subject: [Numpy-discussion] Do ufuncs returned by frompyfunc() have the out arg? Message-ID: <4BBB8EB1.9010000@jhu.edu> Folks, I hope this is a simple question. When I created a ufunc with np.frompyfunc(), I got an error when I called the result with an 'out' argument: >>> def foo(x): return x * x + 1 >>> ufoo = np.frompyfunc(foo, 1, 1) >>> arr = np.arange(9).reshape(3,3) >>> ufoo(arr, out=arr) Traceback (most recent call last): File "", line 1, in TypeError: 'out' is an invalid keyword to foo (vectorized) But I notice that if I just put the array there as a second argument, it seems to work: >>> ufoo(arr, arr) array([[2, 5, 26], [101, 290, 677], [1370, 2501, 4226]], dtype=object) # and now arr is the same as the return value Is it reasonable to conclude that there is an out-arg in the resulting ufunc and I just don't know the right name for it? I also tried putting some other right-shaped array as a second argument and it did indeed get filled in. Thanks as always, Ken From fperez.net at gmail.com Tue Apr 6 16:11:29 2010 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 6 Apr 2010 13:11:29 -0700 Subject: [Numpy-discussion] Getting data from NDarrays to Blitz++ and back again In-Reply-To: <4BB7B313.2060602@cam.ac.uk> References: <4BB7B313.2060602@cam.ac.uk> Message-ID: On Sat, Apr 3, 2010 at 2:28 PM, Philip Sterne wrote: > I hope this is the correct place to post my question. ?I'd like python to > interface with c++ code that makes heavy use of Blitz++ arrays. After a > day's hacking I appear to have a simple working solution which I am > attaching. ?(It uses Boost.Python and CMake.) You may want to have a look at the blitz support in scipy.weave: http://svn.scipy.org/svn/scipy/trunk/scipy/weave/blitz_spec.py http://svn.scipy.org/svn/scipy/trunk/scipy/weave/blitz_tools.py Cheers, f From michael.s.gilbert at gmail.com Tue Apr 6 17:07:30 2010 From: michael.s.gilbert at gmail.com (Michael Gilbert) Date: Tue, 6 Apr 2010 17:07:30 -0400 Subject: [Numpy-discussion] Possible improvement to numpy.mean() Message-ID: <20100406170730.7ba64a4d.michael.s.gilbert@gmail.com> Hi, I am applying Monte Carlo for a problem involving mixed deterministic and random values. In order to avoid a lot of special handling and corner cases, I am using using numpy arrays full of a single value to represent the deterministic quantities. Anyway, I found that the standard deviation turns out to be non-zero when these deterministic sets take on large values, which is wrong. This is due to machine precision loss. It turns out to be fairly straightforward to check for this situation upfront. See attached code. I've also shown a more accurate algorithm for computing the mean, but it adds an additional multiplication for every term in the sum, which is obviously undesirable from a performance perspective. Would it make sense to automatically detect the precision loss and use the more accurate approach when that is the case? If that seems ok, I can take a look at the numpy code, and submit a patch. Best wishes, Mike -------------- next part -------------- A non-text attachment was scrubbed... Name: mean-problem Type: application/octet-stream Size: 1223 bytes Desc: not available URL: From bsouthey at gmail.com Tue Apr 6 17:31:01 2010 From: bsouthey at gmail.com (Bruce Southey) Date: Tue, 06 Apr 2010 16:31:01 -0500 Subject: [Numpy-discussion] Possible improvement to numpy.mean() In-Reply-To: <20100406170730.7ba64a4d.michael.s.gilbert@gmail.com> References: <20100406170730.7ba64a4d.michael.s.gilbert@gmail.com> Message-ID: <4BBBA815.1020705@gmail.com> On 04/06/2010 04:07 PM, Michael Gilbert wrote: > Hi, > > I am applying Monte Carlo for a problem involving mixed deterministic > and random values. In order to avoid a lot of special handling and > corner cases, I am using using numpy arrays full of a single value to > represent the deterministic quantities. > > Anyway, I found that the standard deviation turns out to be non-zero > when these deterministic sets take on large values, which is wrong. > This is due to machine precision loss. > > It turns out to be fairly straightforward to check for this situation > upfront. See attached code. I've also shown a more accurate algorithm > for computing the mean, but it adds an additional multiplication for > every term in the sum, which is obviously undesirable from a > performance perspective. Would it make sense to automatically detect > the precision loss and use the more accurate approach when that is the > case? > > If that seems ok, I can take a look at the numpy code, and submit a > patch. > > Best wishes, > Mike > > Hi, Actually you fail to check for the reverse situation when your 'norm' value is greater than the expected eps value. But these differences are within the expected numerical precision even for float128. In the last two cases you are unfairly comparing float64 arrays with float128 arrays. It would be fairer to use float128 precision as the dtype argument in the mean function. Also any patch has to work with the axis argument. So any 'pre-division by the number of observations' has to be done across the user-selected axis. For some situations I expect that this 'pre-division' will create it's own numerical imprecision - try using the inverse of your inputs. Bruce From peridot.faceted at gmail.com Tue Apr 6 17:52:55 2010 From: peridot.faceted at gmail.com (Anne Archibald) Date: Tue, 6 Apr 2010 17:52:55 -0400 Subject: [Numpy-discussion] Do ufuncs returned by frompyfunc() have the out arg? In-Reply-To: <4BBB8EB1.9010000@jhu.edu> References: <4BBB8EB1.9010000@jhu.edu> Message-ID: On 6 April 2010 15:42, Ken Basye wrote: > Folks, > I hope this is a simple question. When I created a ufunc with > np.frompyfunc(), I got an error when I called the result with an 'out' > argument: In fact, ordinary ufuncs do not accept names for their arguments. This is annoying, but fixing it involves rooting around in the bowels of the ufunc machinery, which are not hacker-friendly. Anne > >>> def foo(x): return x * x + 1 > >>> ufoo = np.frompyfunc(foo, 1, 1) > >>> arr = np.arange(9).reshape(3,3) > >>> ufoo(arr, out=arr) > Traceback (most recent call last): > File "", line 1, in > TypeError: 'out' is an invalid keyword to foo (vectorized) > > But I notice that if I just put the array there as a second argument, it > seems to work: > >>> ufoo(arr, arr) > array([[2, 5, 26], > [101, 290, 677], > [1370, 2501, 4226]], dtype=object) > > # and now arr is the same as the return value > > > Is it reasonable to conclude that there is an out-arg in the resulting > ufunc and I just don't know the right name for it? I also tried putting > some other right-shaped array as a second argument and it did indeed get > filled in. > > Thanks as always, > Ken > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From kwmsmith at gmail.com Tue Apr 6 22:52:13 2010 From: kwmsmith at gmail.com (Kurt Smith) Date: Tue, 6 Apr 2010 21:52:13 -0500 Subject: [Numpy-discussion] numpy build system questions for use in another project (fwrap) Message-ID: Briefly, I'm encountering difficulties getting things working in numpy distutils for fwrap's build system. Here are the steps I want the build system to accomplish: 1) Compile a directory of Fortran 90 source code -- this works. - The .mod files generated by this compilation step are put in the build directory. 2) Fwrap needs to call numpy.distutils.config.try_compile with some generated fortran source code that depends on the .mod files generated in 1). For example: Suppose a fortran 90 module 'type_params.mod' is generated in step 1). Fwrap would generate this fortran source code and try to compile it: subroutine s0(a) use, intrinsic :: iso_c_binding integer(c_int) :: a interface subroutine s1(a) use type_params, only : int_param integer(int_param) :: a end subroutine s1 end interface call s1(a) end subroutine s0 If this compiles correctly with no errors about type incompatibilities, then we know that the int_param in type_params module corresponds to the C int type. If it doesn't, then we try the other C integer types ('c_long', 'c_long_long', 'c_short', 'c_char') until no error is generated (otherwise fail). This then ensures that the type mappings between Fortran and C are correct, and we can then go on to generate appropriate config files with this information. So, this generated fortran source code depends on step 1), since the type_params.mod file must exist first. Then we call config.try_compile and see if it works. My problem is in instantiating numpy.distutils.config such that it is appropriately configured with command line flags. I've tried the following with no success: ('self' is a build_ext instance) cfg = self.distribution.get_command_obj('config') cfg.initialize_options() cfg.finalize_options() # doesn't do what I hoped it would do. This creates a config object, but it doesn't use the command line flags (e.g. --fcompiler=gfortran doesn't affect the fortran compiler used). Any pointers? More generally -- seeing the above, any ideas on how to go about doing what I'm trying to do better? Thanks, Kurt From kbasye1 at jhu.edu Tue Apr 6 23:48:24 2010 From: kbasye1 at jhu.edu (Ken Basye) Date: Tue, 06 Apr 2010 23:48:24 -0400 Subject: [Numpy-discussion] Do ufuncs returned by frompyfunc(), have the out arg? In-Reply-To: References: Message-ID: <4BBC0088.6040206@jhu.edu> From: Anne Archibald On 6 April 2010 15:42, Ken Basye wrote: > > Folks, > > I hope this is a simple question. When I created a ufunc with > > np.frompyfunc(), I got an error when I called the result with an 'out' > > argument: > In fact, ordinary ufuncs do not accept names for their arguments. This is annoying, but fixing it involves rooting around in the bowels of the ufunc machinery, which are not hacker-friendly. Anne > > >>> def foo(x): return x * x + 1 > > >>> ufoo = np.frompyfunc(foo, 1, 1) > > >>> arr = np.arange(9).reshape(3,3) > > >>> ufoo(arr, out=arr) > > Traceback (most recent call last): > > File "", line 1, in > > TypeError: 'out' is an invalid keyword to foo (vectorized) > > > > But I notice that if I just put the array there as a second argument, it > > seems to work: > > >>> ufoo(arr, arr) > > array([[2, 5, 26], > > [101, 290, 677], > > [1370, 2501, 4226]], dtype=object) > > > > # and now arr is the same as the return value > > > > > > Is it reasonable to conclude that there is an out-arg in the resulting > > ufunc and I just don't know the right name for it? I also tried putting > > some other right-shaped array as a second argument and it did indeed get > > filled in. > > > > Thanks as always, > > Ken Thanks - I hadn't noticed that it's apparently only the array methods that can take keyword arguments. So I assume that if I call a '1-arg' ufunc (whether from frompyfunc or an already existing one) with a second argument, the second argument will be used as the output location. Ken -------------- next part -------------- An HTML attachment was scrubbed... URL: From schut at sarvision.nl Wed Apr 7 08:24:43 2010 From: schut at sarvision.nl (Vincent Schut) Date: Wed, 07 Apr 2010 14:24:43 +0200 Subject: [Numpy-discussion] Extracting values from one array corresponding to argmax elements in another array In-Reply-To: <4BBB3582.5080205@jhu.edu> References: <4BBB3582.5080205@jhu.edu> Message-ID: On 04/06/2010 03:22 PM, Ken Basye wrote: > From: Vincent Schut >> On 04/05/2010 06:06 PM, Keith Goodman wrote: >> >>> On Mon, Apr 5, 2010 at 8:44 AM, Ken Basye wrote: >>> snip >>>>> b[a.argmax(axis=0), range(3)] >>>>> >>> array([0, 4, 5]) >>> >> >> Which does not work anymore when your arrays become more-dimensional >> (like in my case: 4 or more) and the axis you want to select on is not >> the first/last one. If I recall correctly, I needed to construct the >> full index arrays for the other dimensions too (like with ogrid I >> think). So: create the ogrid, replace the one for the dimensions you >> want the argmax selection to take place on with the argmax parameter, >> and use those index arrays to index your b array. >> I'd need to look up my source code to be more sure/precise. If anyone >> would like me to, please let me know. If anyone knows a less elaborate >> way, also please let us know! :-) >> > Hi Vincent, > I'd like to see more about your solution. For my present purposes, > Keith's solution was sufficient, but I'm still very interested in a > solution that's independent of dimension and axis. > Thanks (and thanks, Keith), > Ken > I've tracked it down. I have created the following quick'n dirty function: def selectFromDimension(data, selection, dimension): ogridStatement = "numpy.ogrid[" slices = [] for d in data.shape: slices.append("0:" + str(d)) ogridStatement += ",".join(slices) ogridStatement += "]" grid = eval(ogridStatement) grid[dimension] = selection result = data[grid] return result which you call with the array to select from, the result of argmin/max(axis=...), and the axis to select from (usually the same as the axis argument to the argmin/max) It uses eval to be able to create the ogrid independent of dimensionality. There's probably a much cleaner way, but this worked for me :-) Now I think about it, probably another way could be to transpose your axes such that the axis to select on becomes the first axis. Then just index with the result of argmin/max, and transpose back if necessary. However, I have not tried this and I might miss something obvious here... Vincent. From ioannis87 at gmail.com Wed Apr 7 10:40:24 2010 From: ioannis87 at gmail.com (ioannis syntychakis) Date: Wed, 7 Apr 2010 16:40:24 +0200 Subject: [Numpy-discussion] get information/position of value from matrix Message-ID: Hallo Everybody, I am new in this mail list and python. But I am working at something and I need your help. I have a very big matrix. What I want is to search in that matrix for values above the (for example:) 150. If there are values above the 150, I also want to get their position and value. Is this possible? Thanks in advance!! ps. could you also tell me how i can show the whole matrix? because now i see: [[ 0. 0. 0. ..., 87. 4. 4.] [ 0. 0. 0. ..., 89. 6. 4.] [ 0. 1. 0. ..., 87. 4. 3.] ..., [ 0. 0. 0. ..., 87. 4. 4.] [ 0. 3. 0. ..., 87. 4. 4.] [ 0. 0. 4. ..., 87. 4. 4.]] -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.s.gilbert at gmail.com Wed Apr 7 11:15:50 2010 From: michael.s.gilbert at gmail.com (Michael Gilbert) Date: Wed, 7 Apr 2010 11:15:50 -0400 Subject: [Numpy-discussion] get information/position of value from matrix In-Reply-To: References: Message-ID: <20100407111550.6398c9d6.michael.s.gilbert@gmail.com> On Wed, 7 Apr 2010 16:40:24 +0200, ioannis syntychakis wrote: > Hallo Everybody, > > I am new in this mail list and python. But I am working at something and I > need your help. > > I have a very big matrix. What I want is to search in that matrix for values > above the (for example:) 150. If there are values above the 150, I also want > to get their position and value. a[a>150] > Is this possible? > Thanks in advance!! > > ps. could you also tell me how i can show the whole matrix? because now i > see: i use a small routine for this. i just walk through each element, and print it with sys.stdout.write, which doesn't automatically add end-of-lines. mike From josef.pktd at gmail.com Wed Apr 7 11:10:40 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 7 Apr 2010 11:10:40 -0400 Subject: [Numpy-discussion] get information/position of value from matrix In-Reply-To: <20100407111550.6398c9d6.michael.s.gilbert@gmail.com> References: <20100407111550.6398c9d6.michael.s.gilbert@gmail.com> Message-ID: On Wed, Apr 7, 2010 at 11:15 AM, Michael Gilbert wrote: > On Wed, 7 Apr 2010 16:40:24 +0200, ioannis syntychakis wrote: >> Hallo Everybody, >> >> I am new in this mail list and python. But I am working at something and I >> need your help. >> >> I have a very big matrix. What I want is to search in that matrix for values >> above the (for example:) 150. If there are values above the 150, I also want >> to get their position and value. > > a[a>150] > >> Is this possible? >> Thanks in advance!! >> >> ps. could you also tell me how i can show the whole matrix? because now i >> see: > > i use a small routine for this. ?i just walk through each element, and > print it with sys.stdout.write, which doesn't automatically add > end-of-lines. I think threshold keyword allows to change the default for this numpy.set_printoptions(precision=None, threshold=None, edgeitems=None, linewidth=None, suppress=None, nanstr=None, infstr=None) threshold : int, optional Total number of array elements which trigger summarization rather than full repr (default 1000). Josef > > mike > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From geometrian at gmail.com Wed Apr 7 12:01:07 2010 From: geometrian at gmail.com (Ian Mallett) Date: Wed, 7 Apr 2010 09:01:07 -0700 Subject: [Numpy-discussion] get information/position of value from matrix In-Reply-To: References: Message-ID: On Wed, Apr 7, 2010 at 7:40 AM, ioannis syntychakis wrote: > Hallo Everybody, > > I am new in this mail list and python. But I am working at something and I > need your help. > > I have a very big matrix. What I want is to search in that matrix for > values above the (for example:) 150. If there are values above the 150, I > also want to get their position and value. > I think: your_matrix = # indices = np.where(matrix>150.0) values = your_matrix[indices] should work. > Is this possible? > Thanks in advance!! > > ps. could you also tell me how i can show the whole matrix? because now i > see: > > [[ 0. 0. 0. ..., 87. 4. 4.] > [ 0. 0. 0. ..., 89. 6. 4.] > [ 0. 1. 0. ..., 87. 4. 3.] > ..., > [ 0. 0. 0. ..., 87. 4. 4.] > [ 0. 3. 0. ..., 87. 4. 4.] > [ 0. 0. 4. ..., 87. 4. 4.]] > set_printoptions(precision=None, threshold=None, edgeitems=None, linewidth=None, suppress=None, nanstr=None, infstr=None) Ian -------------- next part -------------- An HTML attachment was scrubbed... URL: From geometrian at gmail.com Wed Apr 7 22:36:03 2010 From: geometrian at gmail.com (Ian Mallett) Date: Wed, 7 Apr 2010 19:36:03 -0700 Subject: [Numpy-discussion] Physics Engine Message-ID: Hi, I'm working on various projects, I think it would be great to have a built-in physics engine to my rendering library. I would rather not rely on a large dedicated Physics library (PyOGRE, PyODE, etc.). I thought it might be interesting to try to make a simple soft-body physics engine with NumPy. To this end, over the past few hours, I've constructed this per-vertex physics object class, which plugs directly into my rendering object's class. So far, it implements global forces, collision detection with a horizontal plane, and length tensors: class glLibCPUSoftPhysMesh: def __init__(self,object): self.object = object self.original_vertices = np.array(object.raw_vertices) #object.raw_vertices is the unique list of vertices self.vertices = np.array(object.raw_vertices) self.vertex_indices = np.array(object.indexed_vertices[0]) #object.indexed_vertices[0] is a list of indices into object.raw_vertices self.vertex_speeds = np.zeros((self.vertices.shape[0],3)) self.delta_pos = np.zeros((self.vertices.shape[0],3)) v1s = self.original_vertices[self.vertex_indices[0::3]] v2s = self.original_vertices[self.vertex_indices[1::3]] v3s = self.original_vertices[self.vertex_indices[2::3]] self.target_lengths1 = np.sum((v1s-v2s)**2.0,axis=1)**0.5 self.target_lengths2 = np.sum((v2s-v3s)**2.0,axis=1)**0.5 self.target_lengths3 = np.sum((v3s-v1s)**2.0,axis=1)**0.5 self.dampening = 0.999 def add_force(self,force): self.vertex_speeds += np.array(force) def add_position(self,pos): self.delta_pos += np.array(pos) def move(self): #Length tensors v1s = self.vertices[self.vertex_indices[0::3]] v2s = self.vertices[self.vertex_indices[1::3]] v3s = self.vertices[self.vertex_indices[2::3]] side1 = v2s - v1s side2 = v3s - v2s side3 = v1s - v3s delta_side1 = np.transpose( ((np.sum(side1**2.0,axis=1)**0.5)-self.target_lengths1,)*3 ) * side1 delta_side2 = np.transpose( ((np.sum(side2**2.0,axis=1)**0.5)-self.target_lengths2,)*3 ) * side2 delta_side3 = np.transpose( ((np.sum(side3**2.0,axis=1)**0.5)-self.target_lengths3,)*3 ) * side3 tensor = 0.01 self.vertex_speeds[self.vertex_indices[0::3]] += tensor*delta_side1 self.vertex_speeds[self.vertex_indices[1::3]] -= tensor*delta_side1 self.vertex_speeds[self.vertex_indices[1::3]] += tensor*delta_side2 self.vertex_speeds[self.vertex_indices[2::3]] -= tensor*delta_side2 self.vertex_speeds[self.vertex_indices[2::3]] += tensor*delta_side3 self.vertex_speeds[self.vertex_indices[0::3]] -= tensor*delta_side3 self.delta_pos += self.vertex_speeds self.vertex_speeds *= self.dampening self.vertices += self.delta_pos self.delta_pos.fill(0.0) def collision_detect(self): indices = np.where(self.vertices[:,1]<0.0001) self.vertices[indices,1] = 0.0001 self.vertex_speeds[indices] *= -0.1 def get_data(self): return [self.vertices[self.vertex_indices].tolist()] The code works fairly well for its complexity (and runs very quickly). I believe it also needs angle tensors, which would be more complicated. There are a few strange errors I can't account for (like the object very slowly drifts, as it rests on the ground). At the moment, I'm wondering whether a physics engine has already been done with NumPy before, and if not, whether anyone has any interest in making one. Ian -------------- next part -------------- An HTML attachment was scrubbed... URL: From ioannis87 at gmail.com Thu Apr 8 03:06:16 2010 From: ioannis87 at gmail.com (ioannis syntychakis) Date: Thu, 8 Apr 2010 09:06:16 +0200 Subject: [Numpy-discussion] get information/position of value from matrix In-Reply-To: References: Message-ID: thanks for all your answers. Now i can get al the values above the 150, but i would also like to have their positions in de matrix. excample: [[1. 4. 5. 6. 7. 1 2. 5. 7. 8. 9. 3 3. 5. 7. 1. 3. 7]] so, if i now look for values above the 8. the ouput should be; 1 value on position (4,1) (beginnning from 0) is this possible? thanks in advance!!! 2010/4/7 Ian Mallett > On Wed, Apr 7, 2010 at 7:40 AM, ioannis syntychakis wrote: > >> Hallo Everybody, >> >> I am new in this mail list and python. But I am working at something and I >> need your help. >> >> I have a very big matrix. What I want is to search in that matrix for >> values above the (for example:) 150. If there are values above the 150, I >> also want to get their position and value. >> > I think: > > your_matrix = # > indices = np.where(matrix>150.0) > values = your_matrix[indices] > > should work. > >> Is this possible? >> Thanks in advance!! >> >> ps. could you also tell me how i can show the whole matrix? because now i >> see: >> >> [[ 0. 0. 0. ..., 87. 4. 4.] >> [ 0. 0. 0. ..., 89. 6. 4.] >> [ 0. 1. 0. ..., 87. 4. 3.] >> ..., >> [ 0. 0. 0. ..., 87. 4. 4.] >> [ 0. 3. 0. ..., 87. 4. 4.] >> [ 0. 0. 4. ..., 87. 4. 4.]] >> > set_printoptions(precision=None, threshold=None, edgeitems=None, linewidth=None, suppress=None, nanstr=None, infstr=None) > > Ian > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ioannis87 at gmail.com Thu Apr 8 04:29:28 2010 From: ioannis87 at gmail.com (ioannis syntychakis) Date: Thu, 8 Apr 2010 10:29:28 +0200 Subject: [Numpy-discussion] get information/position of value from matrix In-Reply-To: References: Message-ID: anybody knows how you can draw with arrays? i have two arrays, the x and the y coordinates. i do: d.ImageDraw.Draw(im) d.point(E[0],E[1]) im.show() i get this error: only length-1 arrays can be converted to python scalars 2010/4/8 ioannis syntychakis > thanks for all your answers. > Now i can get al the values above the 150, but i would also like to have > their positions in de matrix. > > excample: > > [[1. 4. 5. 6. 7. 1 > 2. 5. 7. 8. 9. 3 > 3. 5. 7. 1. 3. 7]] > > so, if i now look for values above the 8. the ouput should be; 1 value on > position (4,1) (beginnning from 0) > > is this possible? thanks in advance!!! > 2010/4/7 Ian Mallett > >> On Wed, Apr 7, 2010 at 7:40 AM, ioannis syntychakis < >> ioannis87 at gmail.com> wrote: >> >>> Hallo Everybody, >>> >>> I am new in this mail list and python. But I am working at something and >>> I need your help. >>> >>> I have a very big matrix. What I want is to search in that matrix for >>> values above the (for example:) 150. If there are values above the 150, I >>> also want to get their position and value. >>> >> I think: >> >> your_matrix = # >> indices = np.where(matrix>150.0) >> values = your_matrix[indices] >> >> should work. >> >>> Is this possible? >>> Thanks in advance!! >>> >>> ps. could you also tell me how i can show the whole matrix? because now i >>> see: >>> >>> [[ 0. 0. 0. ..., 87. 4. 4.] >>> [ 0. 0. 0. ..., 89. 6. 4.] >>> [ 0. 1. 0. ..., 87. 4. 3.] >>> ..., >>> [ 0. 0. 0. ..., 87. 4. 4.] >>> [ 0. 3. 0. ..., 87. 4. 4.] >>> [ 0. 0. 4. ..., 87. 4. 4.]] >>> >> set_printoptions(precision=None, threshold=None, edgeitems=None, linewidth=None, suppress=None, nanstr=None, infstr=None) >> >> Ian >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.h.jaffe at gmail.com Thu Apr 8 10:42:02 2010 From: a.h.jaffe at gmail.com (Andrew Jaffe) Date: Thu, 08 Apr 2010 15:42:02 +0100 Subject: [Numpy-discussion] long integers in loadtxt -- bad parsing? Message-ID: Hi all, I am trying to read some 19-digit integers using loadtxt (or genfromtxt -- same problem). The numbers are smaller than the max int64 (and the max uint64 -- same problem with either one). Below, Out[184] shows that python has no problem with the conversion, but loadtxt gets the last few digits wrong in Out[185]. Am I doing something stupid? Yours, Andrew In [175]: import numpy as np In [176]: np.__version__ Out[176]: '1.4.0' In [177]: from StringIO import StringIO In [178]: fc = """1621174818000457763 1621209600996363377 1621258907994644735 1621296000994995765 1621374194996298305 """ In [184]: long(fc[:19]) Out[184]: 1621174818000457763L In [185]: np.loadtxt(StringIO(fc), dtype=np.int64) Out[185]: array([1621174818000457728, 1621209600996363264, 1621258907994644736, 1621296000994995712, 1621374194996298240], dtype=int64) From michael.s.gilbert at gmail.com Thu Apr 8 11:04:40 2010 From: michael.s.gilbert at gmail.com (Michael Gilbert) Date: Thu, 8 Apr 2010 11:04:40 -0400 Subject: [Numpy-discussion] get information/position of value from matrix In-Reply-To: References: Message-ID: <20100408110440.746341d8.michael.s.gilbert@gmail.com> On Thu, 8 Apr 2010 09:06:16 +0200, ioannis syntychakis wrote: > thanks for all your answers. > Now i can get al the values above the 150, but i would also like to have > their positions in de matrix. > > excample: > > [[1. 4. 5. 6. 7. 1 > 2. 5. 7. 8. 9. 3 > 3. 5. 7. 1. 3. 7]] > > so, if i now look for values above the 8. the ouput should be; 1 value on > position (4,1) (beginnning from 0) a == 8 From ytomar at gmail.com Thu Apr 8 11:05:59 2010 From: ytomar at gmail.com (Yogesh Tomar) Date: Thu, 8 Apr 2010 20:35:59 +0530 Subject: [Numpy-discussion] Numpy Segmentation Fault In-Reply-To: References: Message-ID: I have confirmed it, It seems like this is a garbage collection issue and very likely a ref count one. How can these kind of things can be fixed? Regards, Yogesh Tomar On Tue, Apr 6, 2010 at 6:12 PM, Yogesh Tomar wrote: > I also think the same. There is some problem with my python installation. > Because a similar installation python-2.6.4 and numpy-1.3.0 which I did > elsewhere does not seg fault for the same code. > > But I need your help to figure it out. > Can you please elaborate on ref count bug? that might help. > > Regards, > Yogesh Tomar > > > > On Tue, Apr 6, 2010 at 5:17 PM, David Cournapeau wrote: > >> On Tue, Apr 6, 2010 at 8:28 PM, Yogesh Tomar wrote: >> > numpy 1.3.0 also segfaults the same way. >> >> I mean building numpy 1.3.0 against python 2.6.1 instead of 2.6.4 - >> since the crash happen on a python you built by yourself, that's the >> first thing I would look into before looking into numpy or python bug. >> >> > Is it the problem with libc library? >> >> Very unlikely, this looks like a ref count bug, >> >> cheers, >> >> David >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsouthey at gmail.com Thu Apr 8 12:04:04 2010 From: bsouthey at gmail.com (Bruce Southey) Date: Thu, 08 Apr 2010 11:04:04 -0500 Subject: [Numpy-discussion] Numpy Segmentation Fault In-Reply-To: References: Message-ID: <4BBDFE74.5030504@gmail.com> On 04/08/2010 10:05 AM, Yogesh Tomar wrote: > I have confirmed it, > It seems like this is a garbage collection issue and very likely a ref > count one. > > How can these kind of things can be fixed? > > Regards, > Yogesh Tomar > > > On Tue, Apr 6, 2010 at 6:12 PM, Yogesh Tomar > wrote: > > I also think the same. There is some problem with my python > installation. > Because a similar installation python-2.6.4 and numpy-1.3.0 which > I did elsewhere does not seg fault for the same code. > > But I need your help to figure it out. > Can you please elaborate on ref count bug? that might help. > > Regards, > Yogesh Tomar > > > > On Tue, Apr 6, 2010 at 5:17 PM, David Cournapeau > > wrote: > > On Tue, Apr 6, 2010 at 8:28 PM, Yogesh Tomar > wrote: > > numpy 1.3.0 also segfaults the same way. > > I mean building numpy 1.3.0 against python 2.6.1 instead of > 2.6.4 - > since the crash happen on a python you built by yourself, > that's the > first thing I would look into before looking into numpy or > python bug. > > > Is it the problem with libc library? > > Very unlikely, this looks like a ref count bug, > > cheers, > > David > Hi, Based on the list, I always suspect lapack/atlas issues when I see problems related to eigenvalues. What is configuration including lapack/atlas ie the output from 'numpy.show_config()'? Bruce -------------- next part -------------- An HTML attachment was scrubbed... URL: From ytomar at gmail.com Thu Apr 8 12:17:02 2010 From: ytomar at gmail.com (Yogesh Tomar) Date: Thu, 8 Apr 2010 21:47:02 +0530 Subject: [Numpy-discussion] Numpy Segmentation Fault In-Reply-To: <4BBDFE74.5030504@gmail.com> References: <4BBDFE74.5030504@gmail.com> Message-ID: Here it is atlas_threads_info: libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/home/eqan/tapas/install/atlas/64/lib'] language = f77 blas_opt_info: libraries = ['ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/home/eqan/tapas/install/atlas/64/lib'] define_macros = [('ATLAS_INFO', '"\\"3.8.3\\""')] language = c atlas_blas_threads_info: libraries = ['ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/home/eqan/tapas/install/atlas/64/lib'] language = c lapack_opt_info: libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/home/eqan/tapas/install/atlas/64/lib'] define_macros = [('ATLAS_INFO', '"\\"3.8.3\\""')] language = f77 lapack_mkl_info: NOT AVAILABLE blas_mkl_info: NOT AVAILABLE mkl_info: NOT AVAILABLE Regards, Yogesh Tomar On Thu, Apr 8, 2010 at 9:34 PM, Bruce Southey wrote: > On 04/08/2010 10:05 AM, Yogesh Tomar wrote: > > I have confirmed it, > It seems like this is a garbage collection issue and very likely a ref > count one. > > How can these kind of things can be fixed? > > Regards, > Yogesh Tomar > > > On Tue, Apr 6, 2010 at 6:12 PM, Yogesh Tomar wrote: > >> I also think the same. There is some problem with my python installation. >> Because a similar installation python-2.6.4 and numpy-1.3.0 which I did >> elsewhere does not seg fault for the same code. >> >> But I need your help to figure it out. >> Can you please elaborate on ref count bug? that might help. >> >> Regards, >> Yogesh Tomar >> >> >> >> On Tue, Apr 6, 2010 at 5:17 PM, David Cournapeau wrote: >> >>> On Tue, Apr 6, 2010 at 8:28 PM, Yogesh Tomar wrote: >>> > numpy 1.3.0 also segfaults the same way. >>> >>> I mean building numpy 1.3.0 against python 2.6.1 instead of 2.6.4 - >>> since the crash happen on a python you built by yourself, that's the >>> first thing I would look into before looking into numpy or python bug. >>> >>> > Is it the problem with libc library? >>> >>> Very unlikely, this looks like a ref count bug, >>> >>> cheers, >>> >>> David >>> >> Hi, > Based on the list, I always suspect lapack/atlas issues when I see problems > related to eigenvalues. > > What is configuration including lapack/atlas ie the output from > 'numpy.show_config()'? > > > > Bruce > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From a.h.jaffe at gmail.com Thu Apr 8 13:24:25 2010 From: a.h.jaffe at gmail.com (Andrew Jaffe) Date: Thu, 08 Apr 2010 18:24:25 +0100 Subject: [Numpy-discussion] long integers in loadtxt -- bad parsing? In-Reply-To: References: Message-ID: <4BBE1149.1030700@gmail.com> Hi, > I am trying to read some 19-digit integers using loadtxt (or genfromtxt > -- same problem). The numbers are smaller than the max int64 (and the > max uint64 -- same problem with either one). > > Below, Out[184] shows that python has no problem with the conversion, > but loadtxt gets the last few digits wrong in Out[185]. > > Am I doing something stupid? > > Yours, > > Andrew > > > > In [175]: import numpy as np > > In [176]: np.__version__ > Out[176]: '1.4.0' > > In [177]: from StringIO import StringIO > > In [178]: fc = """1621174818000457763 > 1621209600996363377 > 1621258907994644735 > 1621296000994995765 > 1621374194996298305 > """ > > In [184]: long(fc[:19]) > Out[184]: 1621174818000457763L > > In [185]: np.loadtxt(StringIO(fc), dtype=np.int64) > Out[185]: > array([1621174818000457728, 1621209600996363264, 1621258907994644736, > 1621296000994995712, 1621374194996298240], dtype=int64) The slightly hack-ish solution is to explicitly use the python long() function as a converter: In [215]: np.loadtxt(StringIO(fc), dtype=np.int64, converters={0:long}) Out[215]: array([1621174818000457763, 1621209600996363377, 1621258907994644735, 1621296000994995765, 1621374194996298305], dtype=int64) From a.h.jaffe at gmail.com Thu Apr 8 13:24:25 2010 From: a.h.jaffe at gmail.com (Andrew Jaffe) Date: Thu, 08 Apr 2010 18:24:25 +0100 Subject: [Numpy-discussion] long integers in loadtxt -- bad parsing? In-Reply-To: References: Message-ID: <4BBE1149.1030700@gmail.com> Hi, > I am trying to read some 19-digit integers using loadtxt (or genfromtxt > -- same problem). The numbers are smaller than the max int64 (and the > max uint64 -- same problem with either one). > > Below, Out[184] shows that python has no problem with the conversion, > but loadtxt gets the last few digits wrong in Out[185]. > > Am I doing something stupid? > > Yours, > > Andrew > > > > In [175]: import numpy as np > > In [176]: np.__version__ > Out[176]: '1.4.0' > > In [177]: from StringIO import StringIO > > In [178]: fc = """1621174818000457763 > 1621209600996363377 > 1621258907994644735 > 1621296000994995765 > 1621374194996298305 > """ > > In [184]: long(fc[:19]) > Out[184]: 1621174818000457763L > > In [185]: np.loadtxt(StringIO(fc), dtype=np.int64) > Out[185]: > array([1621174818000457728, 1621209600996363264, 1621258907994644736, > 1621296000994995712, 1621374194996298240], dtype=int64) The slightly hack-ish solution is to explicitly use the python long() function as a converter: In [215]: np.loadtxt(StringIO(fc), dtype=np.int64, converters={0:long}) Out[215]: array([1621174818000457763, 1621209600996363377, 1621258907994644735, 1621296000994995765, 1621374194996298305], dtype=int64) From andrea.gavana at gmail.com Thu Apr 8 16:08:44 2010 From: andrea.gavana at gmail.com (Andrea Gavana) Date: Thu, 8 Apr 2010 21:08:44 +0100 Subject: [Numpy-discussion] Interpolation question In-Reply-To: References: <3d375d731003281634wa6d2cc2g4ee1106f0dbac708@mail.gmail.com> Message-ID: (Resending as numpy-discussion has a 40 Kb message limit) On 30 March 2010 22:44, Friedrich Romstedt wrote: > 2010/3/30 Andrea Gavana : >> However, from the first 100 or so interpolated simulations, I could >> gather these findings: >> >> 1) Interpolations on *cumulative* productions on oil and gas are >> extremely good, with a maximum range of relative error of -3% / +2%: >> most of them (95% more or less) show errors < 1%, but for few of them >> I get the aforementioned range of errors in the interpolations; >> 2) Interpolations on oil and gas *rates* (time dependent), I usually >> get a -5% / +3% error per timestep, which is very good for my >> purposes. I still need to check these values but the first set of >> results were very promising; >> 3) Interpolations on gas injection (cumulative and rate) are a bit >> more shaky (15% error more or less), but this is essentially due to a >> particular complex behaviour of the reservoir simulator when it needs >> to decide (based on user input) if the gas is going to be re-injected, >> sold, used as excess gas and a few more; I am not that worried about >> this issue for the moment. > > Have a nice time in Greece, and what you write makes me laughing. :-) > When you are back, you should maybe elaborate a bit on what gas > injections, wells, re-injected gas and so on is, I don't know about > it. Just as a little follow-up, I managed to obtain a more extended set of interpolated simulations yesterday. As a little recap, I took two thirds of the simulations database to use them as interpolation base and tried to reproduce the rest using the interpolation. I have a couple of plots to demonstrate how good/bad the interpolation was. 1) The first one (http://img404.yfrog.com/img404/9885/doterrors.png) has 6 subplots: the first 3 at the top show relative errors (in percentage) for cumulative productions for oil and gas and cumulative gas injection, with: relative_error = 100 * (real - interpolated) / real The second set of 3 subplots at the bottom show median errors on oil/gas production rates and gas injection rates: these are series of time-dependent values (in contrast to the cumulative, which are a single floating point value), so I took the median error over all the simulated timesteps. As you can see, about 95-96% of the interpolated results have a relative error between -5% / +5%, with some outliers (which don't bother me that much anyway :-D ). 2) The second figure (http://img338.imageshack.us/img338/7527/boxploterrors.png) has been liberally adapted from a matplotlib demo (boxplot_demo2), and show a series of error box plots for all the variables I have used in the interpolation. Overall, I can say I am pretty satisfied with the results :-D . If I get a chance to test Kevin's and Robert's ideas before getting sent to Kazakhstan I'll post another follow-up, unless you got bored by now and you tell me to shut up. Thank you again to the list for the wonderful help. Andrea. "Imagination Is The Only Weapon In The War Against Reality." http://xoomer.alice.it/infinity77/ ==> Never *EVER* use RemovalGroup for your house removal. You'll regret it forever. http://thedoomedcity.blogspot.com/2010/03/removal-group-nightmare.html <== From friedrichromstedt at gmail.com Thu Apr 8 17:44:33 2010 From: friedrichromstedt at gmail.com (Friedrich Romstedt) Date: Thu, 8 Apr 2010 23:44:33 +0200 Subject: [Numpy-discussion] List of indices In-Reply-To: References: Message-ID: 2010/4/2 Shailendra : >>>> x=arange(10) >>>> indices=[(1,),(2,)] >>>> x[indices] > Traceback (most recent call last): > ?File "", line 1, in > IndexError: unsupported iterator index I have more odd things to report (numpy 1.4.0): >>> b = numpy.asarray([[1, 2], [3, 4]]) >>> b[[(0, 0)]] array([[1, 2], [1, 2]]) How does the advanced integer indicing mode work precisely? The docu (http://docs.scipy.org/doc/numpy/reference/arrays.indexing.html#integer) says: "All sequences and scalars in the selection tuple are converted to intp indexing arrays." How can then numpy distinguish between lists and tuples? My thought is, that maybe only the lists are taken into account to calculate the broadcasting target, and tuples are handed over to the indicing step, as described in the docu. But then, it seems, that length-1 items are "flattened"? Friedrich From fperez.net at gmail.com Thu Apr 8 18:53:03 2010 From: fperez.net at gmail.com (Fernando Perez) Date: Thu, 8 Apr 2010 15:53:03 -0700 Subject: [Numpy-discussion] Scipy.org down? Message-ID: Howdy, [ Robert, sorry to CC you directly, but I'm doing it in case the entire server including mailing lists is fully out and this message doesn't reach anyone via the list. I don't know who the current contact at Enthought for this should be. ] the scipy.org server has been very flaky for the past couple of days (down right now), we've actually lost several times updates to wiki pages on the ipython site due to this. Is there some quick fix available, or is there a more serious problem with the server? I ssh'd in and I don't see any runaway cpu processes, load average looks fine: top - 17:51:58 up 48 days, 6:51, 1 user, load average: 0.05, 0.01, 0.00 Tasks: 369 total, 1 running, 367 sleeping, 0 stopped, 1 zombie Cpu(s): 0.0% us, 0.2% sy, 0.0% ni, 99.7% id, 0.0% wa, 0.0% hi, 0.2% si Mem: 4090796k total, 3673924k used, 416872k free, 49208k buffers Swap: 4192944k total, 1452k used, 4191492k free, 1172792k cached Cheers, f From geometrian at gmail.com Fri Apr 9 00:19:50 2010 From: geometrian at gmail.com (Ian Mallett) Date: Thu, 8 Apr 2010 21:19:50 -0700 Subject: [Numpy-discussion] Smoothing a Grid Message-ID: Hi, So I have a 4D grid (x,y,z,vec3), where the vec3 represents a position. What I want to do is to move vec3 elements of the grid based on surrounding vec3 elements so that the grid's values overall are more orthogonal. For example, consider the following 3 x 3 x 3 x vec3 grid: [[[[-1.0,-1.0,-1.0],[ 0.0,-1.0,-1.0],[ 1.0,-1.0,-1.0]], [[-1.0,-1.0, 0.0],[ 0.0,-1.0, 0.0],[ 1.0,-1.0, 0.0]], [[-1.0,-1.0, 1.0],[ 0.0,-1.0, 1.0],[ 1.0,-1.0, 1.0]]], [[[-1.0, 0.0,-1.0],[ 0.0, 0.0,-1.0],[ 1.0, 0.0,-1.0]], [[-1.0, 0.0, 0.0],[ 0.2, 0.0, 0.0],[ 1.0, 0.0, 0.0]], [[-1.0, 0.0, 1.0],[ 0.0, 0.0, 1.0],[ 1.0, 0.0, 1.0]]], [[[-1.0, 1.0,-1.0],[ 0.0, 1.0,-1.0],[ 1.0, 1.0,-1.0]], [[-1.0, 1.0, 0.0],[ 0.0, 1.0, 0.0],[ 1.0, 1.0, 0.0]], [[-1.0, 1.0, 1.0],[ 0.0, 1.0, 1.0],[ 1.0, 1.0, 1.0]]]] Notice that, with the exception of the middle value, [0.2,0.0,0.0], the vec3 elements in the grid define a 3 x 3 x 3 grid. The middle value should thus be moved perhaps 10% of the distance to the target location [0.0,0.0,0.0], which can be inferred from the surrounding vec3 elements. After this movement, the value is [0.18,0.00,0.00]. To be more precise, the target position of each vec3 is determined from the surrounding values. Every vec3 moves towards being a certain x_distance from adjacent vec3s in the x direction. Every vec3 moves towards being a certain y_distance from adjacent vec3s in the y direction. Every vec3 moves towards being a certain z_distance from adjacent vec3s in the z direction. Each vec3 also moves towards a certain diagonal_distance from all vec3s along a major diagonal. I am trying to accomplish this. Any ideas? Thanks, Ian -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at silveregg.co.jp Fri Apr 9 03:25:14 2010 From: david at silveregg.co.jp (David) Date: Fri, 09 Apr 2010 16:25:14 +0900 Subject: [Numpy-discussion] numpy build system questions for use in another project (fwrap) In-Reply-To: References: Message-ID: <4BBED65A.209@silveregg.co.jp> On 04/07/2010 11:52 AM, Kurt Smith wrote: > Briefly, I'm encountering difficulties getting things working in numpy > distutils for fwrap's build system. > > Here are the steps I want the build system to accomplish: > > 1) Compile a directory of Fortran 90 source code -- this works. > - The .mod files generated by this compilation step are put in the > build directory. This is difficult - fortran modules are a PITA from a build perspective. Many compilers don't seem to have a way to control exactly where to put the generated .mod, so the only way I am aware of to control this is to cwd the process into the build directory... This was also a problem when I worked on fortran support for waf (see http://groups.google.com/group/waf-users/browse_thread/thread/889e2a5e5256e420/84ee939e93c9e30f?lnk=gst&q=fortran+modules#84ee939e93c9e30f ) > > My problem is in instantiating numpy.distutils.config such that it is > appropriately configured with command line flags. > > I've tried the following with no success: > > ('self' is a build_ext instance) > cfg = self.distribution.get_command_obj('config') > cfg.initialize_options() > cfg.finalize_options() # doesn't do what I hoped it would do. > > This creates a config object, but it doesn't use the command line > flags (e.g. --fcompiler=gfortran doesn't affect the fortran compiler > used). Why don't you do the testing in config ? That's how things are done normally, unless you have a reason to do otherwise. Concerning the --fcompiler option, how do you pass --fcompiler (what is the exact list of commands you used to call distutils here) ? Generally, given the general command/option screw up, the only real solution really is to pass the options to each command, and hope each one is the same. > Any pointers? More generally -- seeing the above, any ideas on how to > go about doing what I'm trying to do better? Not really, that's how you are supposed to do things with distutils, David From cournape at gmail.com Fri Apr 9 03:34:42 2010 From: cournape at gmail.com (David Cournapeau) Date: Fri, 9 Apr 2010 16:34:42 +0900 Subject: [Numpy-discussion] Numpy Segmentation Fault In-Reply-To: References: Message-ID: On Fri, Apr 9, 2010 at 12:05 AM, Yogesh Tomar wrote: > I have confirmed it, > It seems like this is a garbage collection issue and very likely a ref count > one. > > How can these kind of things can be fixed? No - as you changed both numpy and python versions, we have to isolate the problem. So I would prefer to see what happens with the same numpy version built against the Red Hat python (2.6.1) before looking into numpy proper. Given how simple your example is, it is quite unlikely that there is a ref count bug that nobody encountered before, David From ioannis87 at gmail.com Fri Apr 9 04:47:45 2010 From: ioannis87 at gmail.com (ioannis syntychakis) Date: Fri, 9 Apr 2010 10:47:45 +0200 Subject: [Numpy-discussion] get information/position of value from matrix In-Reply-To: <20100408110440.746341d8.michael.s.gilbert@gmail.com> References: <20100408110440.746341d8.michael.s.gilbert@gmail.com> Message-ID: hallo, can anybody help me with this; I think i have to post in this mail list because i think i have to use numpy to do it. I have a grayscale (L mode) image. On some places on the image i want to calculate the average pixel value (i know how to do that). But i want to let the user of the program choose the place! So i want that the user get a square on his mouseclick en be able to select the square where python has to calculate the averagepigelvalue. any ideas how i would do this? thanks in advance everybody 2010/4/8 Michael Gilbert > On Thu, 8 Apr 2010 09:06:16 +0200, ioannis syntychakis wrote: > > thanks for all your answers. > > Now i can get al the values above the 150, but i would also like to have > > their positions in de matrix. > > > > excample: > > > > [[1. 4. 5. 6. 7. 1 > > 2. 5. 7. 8. 9. 3 > > 3. 5. 7. 1. 3. 7]] > > > > so, if i now look for values above the 8. the ouput should be; 1 value on > > position (4,1) (beginnning from 0) > > a == 8 > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ytomar at gmail.com Fri Apr 9 05:33:41 2010 From: ytomar at gmail.com (Yogesh Tomar) Date: Fri, 9 Apr 2010 15:03:41 +0530 Subject: [Numpy-discussion] Numpy Segmentation Fault In-Reply-To: References: Message-ID: Hi David, Many thanks for pointing this out, it turns out that it is indeed an ATLAS issues and now I am able to run the test without any seg fault, after re-installing numpy without ATLAS A small question, if I use numpy without ATLAS will there be any functions that will not be available? Is there any major performance hit that I need to be aware of? Regards, Yogesh Tomar On Fri, Apr 9, 2010 at 1:04 PM, David Cournapeau wrote: > On Fri, Apr 9, 2010 at 12:05 AM, Yogesh Tomar wrote: > > I have confirmed it, > > It seems like this is a garbage collection issue and very likely a ref > count > > one. > > > > How can these kind of things can be fixed? > > No - as you changed both numpy and python versions, we have to isolate > the problem. So I would prefer to see what happens with the same numpy > version built against the Red Hat python (2.6.1) before looking into > numpy proper. Given how simple your example is, it is quite unlikely > that there is a ref count bug that nobody encountered before, > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Fri Apr 9 09:44:26 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 9 Apr 2010 07:44:26 -0600 Subject: [Numpy-discussion] Numpy Segmentation Fault In-Reply-To: References: Message-ID: On Fri, Apr 9, 2010 at 3:33 AM, Yogesh Tomar wrote: > Hi David, > > Many thanks for pointing this out, it turns out that it is indeed an ATLAS > issues and now I am able to run the test without any seg fault, after > re-installing numpy without ATLAS > > A small question, if I use numpy without ATLAS will there be any functions > that will not be available? > Is there any major performance hit that I need to be aware of? > > The performance hit is likely to come with large arrays. For small arrays the call overhead will dominate and most of the data will reside in cache anyway. It is large arrays where the cache awareness of ATLAS makes a difference in performance and the difference can be a substantial. Where did you get ATLAS from? Did you build it yourself? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From ytomar at gmail.com Fri Apr 9 09:53:08 2010 From: ytomar at gmail.com (Yogesh Tomar) Date: Fri, 9 Apr 2010 19:23:08 +0530 Subject: [Numpy-discussion] Numpy Segmentation Fault In-Reply-To: References: Message-ID: Yes I built it myself, some time back. I am using version 3.8.3 Regards, Yogesh Tomar On Fri, Apr 9, 2010 at 7:14 PM, Charles R Harris wrote: > > > On Fri, Apr 9, 2010 at 3:33 AM, Yogesh Tomar wrote: > >> Hi David, >> >> Many thanks for pointing this out, it turns out that it is indeed an ATLAS >> issues and now I am able to run the test without any seg fault, after >> re-installing numpy without ATLAS >> >> A small question, if I use numpy without ATLAS will there be any functions >> that will not be available? >> Is there any major performance hit that I need to be aware of? >> >> > The performance hit is likely to come with large arrays. For small arrays > the call overhead will dominate and most of the data will reside in cache > anyway. It is large arrays where the cache awareness of ATLAS makes a > difference in performance and the difference can be a substantial. Where did > you get ATLAS from? Did you build it yourself? > > > > Chuck > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kwmsmith at gmail.com Fri Apr 9 14:02:33 2010 From: kwmsmith at gmail.com (Kurt Smith) Date: Fri, 9 Apr 2010 13:02:33 -0500 Subject: [Numpy-discussion] numpy build system questions for use in another project (fwrap) In-Reply-To: <4BBED65A.209@silveregg.co.jp> References: <4BBED65A.209@silveregg.co.jp> Message-ID: On Fri, Apr 9, 2010 at 2:25 AM, David wrote: > On 04/07/2010 11:52 AM, Kurt Smith wrote: >> Briefly, I'm encountering difficulties getting things working in numpy >> distutils for fwrap's build system. >> >> Here are the steps I want the build system to accomplish: >> >> 1) Compile a directory of Fortran 90 source code -- this works. >> ? ? ?- The .mod files generated by this compilation step are put in the >> build directory. > > This is difficult - fortran modules are a PITA from a build perspective. > Many compilers don't seem to have a way to control exactly where to put > the generated .mod, so the only way I am aware of to control this is to > cwd the process into the build directory... >From what I can tell, numpy's distutils already takes care of putting the .mod files in the build directory (at least for gfortran and ifort) by manually moving them there -- see numpy.distutils.command.build_ext.build_ext.build_extension, after the line "if fmodule_source". This is fine for my purposes. All I need is the project's .mod files put in one place before the next step. > > This was also a problem when I worked on fortran support for waf (see > http://groups.google.com/group/waf-users/browse_thread/thread/889e2a5e5256e420/84ee939e93c9e30f?lnk=gst&q=fortran+modules#84ee939e93c9e30f > ) Yes, I saw that thread and it dissuaded me from pursuing waf for now. I'd like to take a stab at it sometime down the road, though. > >> >> My problem is in instantiating numpy.distutils.config such that it is >> appropriately configured with command line flags. >> >> I've tried the following with no success: >> >> ('self' is a build_ext instance) >> cfg = self.distribution.get_command_obj('config') >> cfg.initialize_options() >> cfg.finalize_options() ?# doesn't do what I hoped it would do. >> >> This creates a config object, but it doesn't use the command line >> flags (e.g. --fcompiler=gfortran doesn't affect the fortran compiler >> used). > > Why don't you do the testing in config ? That's how things are done Fortran's .mod files are essentially compiler-generated header files; fwrap needs to use these 'headers' to get type information so it can figure out how the C types and Fortran types match up. Fwrap then generates the config files with this information and compiles the wrappers with the other source files. I could make it a requirement that the user supply the .mod files to fwrap before calling 'python setup.py build_ext', but that might be a big mess when you take into account compiler flags that modify default type sizes, etc. Maybe it would work fine; I'm not sure whether different compiler flags used by distutils would change the .mod files and break the type info, though. And the source code would be compiled twice -- once to get the .mod files, and again when distutils compiles everything with the right flags which would be suboptimal for large projects. (Note: I've looked into this a bit, and it seems like it might work alright -- you can use '-fsyntax-only' in gfortran and g95, and '-syntax-only' in ifort, for example, and that generates the .mod files. I'll look into it some more...) It would be a PITA to test for every kind-type-parameter that someone might use in the config stage. As you probably know, Fortran 9x allows types like this: real(kind=selected_real_kind(10,100)) :: ffed_real Which means I'd have to do exhaustive testing for every combination of arguments to selected_real_kind() (up to some limit). This would be *extremely* slow. I could do a binary search, but it's not something I'd like to pursue if it's avoidable. There are some other issues that I'm not mentioning that make this approach a pain, too. > normally, unless you have a reason to do otherwise. Concerning the > --fcompiler option, how do you pass --fcompiler (what is the exact list > of commands you used to call distutils here) ? Generally, given the > general command/option screw up, the only real solution really is to > pass the options to each command, and hope each one is the same. > The commandline is 'python setup.py build_ext --inplace --fcompiler=gfortran' (or --fcompiler=intelem). When I pass --fcompiler=gfortran, the try_compile() command ignores it, searches for a fortran compiler and finds ifort instead and uses it, which breaks everything. Is there an example in numpy.distutils that shows how to pass the commandline options to each command? I'm thinking of rolling my own 'try_compile()' function as part of a custom build_ext command class that suits my purposes. We'll see how far that gets me. >> Any pointers? ?More generally -- seeing the above, any ideas on how to >> go about doing what I'm trying to do better? > > Not really, that's how you are supposed to do things with distutils, > Then I can see why you'd like to use scons & waf instead. Thanks for the comments, Kurt From friedrichromstedt at gmail.com Fri Apr 9 15:19:13 2010 From: friedrichromstedt at gmail.com (Friedrich Romstedt) Date: Fri, 9 Apr 2010 21:19:13 +0200 Subject: [Numpy-discussion] Extracting values from one array corresponding to argmax elements in another array In-Reply-To: <4BBA055D.9080607@jhu.edu> References: <4BBA055D.9080607@jhu.edu> Message-ID: 2010/4/5 Ken Basye : > ?I have two arrays, A and B, with the same shape. ?I want to find the > highest values in A along some axis, then extract the corresponding > values from B. Maybe: def select(A, B, axis): # Extract incomplete index tuples: argmax = a.argmax(axis = axis) # Create the selection list to be handed over to B.__getitem__() ... advanced_index = [] # Decompose the index tuples. for dimi in xrange(argmax.shape[1]): advanced_index.append(argmax[:, dimi]) # Insert the missing dimension. advanced_index.insert(axis, numpy.arange(0, B.shape[axis])) # Perform advanced (integer) selection ... return B[advanced_index] >>> a array([[[0, 1], [2, 3]], [[4, 5], [6, 7]]]) >>> select(a, a, 0) array([3, 7]) >>> select(a, a, 1) array([5, 7]) >>> select(a, a, 2) array([6, 7]) It seems to work. Friedrich From lane at brooks.nu Sat Apr 10 03:49:03 2010 From: lane at brooks.nu (Lane Brooks) Date: Sat, 10 Apr 2010 01:49:03 -0600 Subject: [Numpy-discussion] Masked Array Usage Problems Message-ID: <4BC02D6F.1070408@brooks.nu> I am trying out masked arrays for the first time and having some problems. I have a 2-D image as dtype=numpy.int16 I create a mask of all False to not mask out any pixels. I calculate the mean of the image original image and it comes out ~597. I calculate the mean of the masked array and it comes out differently around -179. It produces the same negative mean value no matter what masks I try, e.g. (all True, all False, etc). Furthermore there are no negative samples in the entire array. Any ideas on what am I doing wrong? Here is some sample code showing the behavior: In [1]: img.dtype, img.shape Out[1]: (dtype('int16'), (3200, 3456)) In [2]: mask = numpy.zeros(img.shape, dtype=bool) In [3]: imgma = ma.masked_array(img, mask) In [4]: img.mean() Out[4]: 597.15437617549185 In [5]: imgma.mean() Out[5]: -179.56858678747108 In [6]: imgma.min() Out[6]: 25 In [7]: numpy.__version__ Out[7]: '1.3.0' In [8]: numpy.ma.__version__ Out[8]: '1.0' From josef.pktd at gmail.com Sat Apr 10 08:17:31 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sat, 10 Apr 2010 08:17:31 -0400 Subject: [Numpy-discussion] Masked Array Usage Problems In-Reply-To: <4BC02D6F.1070408@brooks.nu> References: <4BC02D6F.1070408@brooks.nu> Message-ID: On Sat, Apr 10, 2010 at 3:49 AM, Lane Brooks wrote: > I am trying out masked arrays for the first time and having some > problems. I have a 2-D image as dtype=numpy.int16 > > I create a mask of all False to not mask out any pixels. > > I calculate the mean of the image original image and it comes out ~597. > I calculate the mean of the masked array and it comes out differently > around -179. ?It produces the same negative mean value no matter what > masks I try, e.g. (all True, all False, etc). ?Furthermore there are no > negative samples in the entire array. > > Any ideas on what am I doing wrong? > > Here is some sample code showing the behavior: > > In [1]: img.dtype, img.shape > Out[1]: (dtype('int16'), (3200, 3456)) > > In [2]: mask = numpy.zeros(img.shape, dtype=bool) > > In [3]: imgma = ma.masked_array(img, mask) > > In [4]: img.mean() > Out[4]: 597.15437617549185 > > In [5]: imgma.mean() > Out[5]: -179.56858678747108 > > In [6]: imgma.min() > Out[6]: 25 > > In [7]: numpy.__version__ > Out[7]: '1.3.0' > > In [8]: numpy.ma.__version__ > Out[8]: '1.0' Just a guess untill Pierre replies: It looks to me like an integer overflow bug. Can you try imgma.mean(dtype=float) to do the accumulation with floating points? Josef > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From dagss at student.matnat.uio.no Sat Apr 10 10:53:48 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 10 Apr 2010 16:53:48 +0200 Subject: [Numpy-discussion] numpy build system questions for use in another project (fwrap) In-Reply-To: References: <4BBED65A.209@silveregg.co.jp> Message-ID: <4BC090FC.9070001@student.matnat.uio.no> Kurt Smith wrote: > On Fri, Apr 9, 2010 at 2:25 AM, David wrote: >> On 04/07/2010 11:52 AM, Kurt Smith wrote: >>> Briefly, I'm encountering difficulties getting things working in numpy >>> distutils for fwrap's build system. >>> >>> Here are the steps I want the build system to accomplish: >>> >>> 1) Compile a directory of Fortran 90 source code -- this works. >>> - The .mod files generated by this compilation step are put in the >>> build directory. >> This is difficult - fortran modules are a PITA from a build perspective. >> Many compilers don't seem to have a way to control exactly where to put >> the generated .mod, so the only way I am aware of to control this is to >> cwd the process into the build directory... > >>From what I can tell, numpy's distutils already takes care of putting > the .mod files in the build directory (at least for gfortran and > ifort) by manually moving them there -- see > numpy.distutils.command.build_ext.build_ext.build_extension, after the > line "if fmodule_source". This is fine for my purposes. All I need > is the project's .mod files put in one place before the next step. > >> This was also a problem when I worked on fortran support for waf (see >> http://groups.google.com/group/waf-users/browse_thread/thread/889e2a5e5256e420/84ee939e93c9e30f?lnk=gst&q=fortran+modules#84ee939e93c9e30f >> ) > > Yes, I saw that thread and it dissuaded me from pursuing waf for now. > I'd like to take a stab at it sometime down the road, though. > >>> My problem is in instantiating numpy.distutils.config such that it is >>> appropriately configured with command line flags. >>> >>> I've tried the following with no success: >>> >>> ('self' is a build_ext instance) >>> cfg = self.distribution.get_command_obj('config') >>> cfg.initialize_options() >>> cfg.finalize_options() # doesn't do what I hoped it would do. >>> >>> This creates a config object, but it doesn't use the command line >>> flags (e.g. --fcompiler=gfortran doesn't affect the fortran compiler >>> used). >> Why don't you do the testing in config ? That's how things are done > > Fortran's .mod files are essentially compiler-generated header files; > fwrap needs to use these 'headers' to get type information so it can > figure out how the C types and Fortran types match up. Fwrap then > generates the config files with this information and compiles the > wrappers with the other source files. I hope there will at least be an option to supply mod files instead of using this mechanism. For "wrapper" projects which already (presumably) has a build system for the Fortran code set up it seems more reasonable to me to just refer to the output of the existing build. In particular I don't like the situation today with Python wrappers around C code, where the C code files are often copied into the Python wrapper project. I hope the same won't happen with fwrap, i.e. that people don't copy the Fortran sources to the fwrap wrapper just to make things easy to build (but end up forking the project and not keep it up to date). Of course, if one is not wrapping an existing module but rather developing an application with fwrap then the situation is different, I suppose. -- Dag Sverre From oliphant at enthought.com Sat Apr 10 13:23:03 2010 From: oliphant at enthought.com (Travis Oliphant) Date: Sat, 10 Apr 2010 12:23:03 -0500 Subject: [Numpy-discussion] Proposal for new ufunc functionality Message-ID: <3BB52545-B5B9-4B9C-AD9A-0D69432A6B2D@enthought.com> Hi, I've been mulling over a couple of ideas for new ufunc methods plus a couple of numpy functions that I think will help implement group-by operations with NumPy arrays. I wanted to discuss them on this list before putting forward an actual proposal or patch to get input from others. The group-by operation is very common in relational algebra and NumPy arrays (especially structured arrays) can often be seen as a database table. There are common and easy-to implement approaches for select and other relational algebra concepts, but group-by basically has to be implemented yourself. Here are my suggested additions to NumPy: ufunc methods: * reduceby (array, by, sorted=1, axis=0) array is the array to reduce by is the array to provide the grouping (can be a structured array or a list of arrays) if sorted is 1, then possibly a faster algorithm can be used. * reducein (array, indices, axis=0) similar to reduce-at, but the indices provide both the start and end points (rather than being fence-posts like reduceat). numpy functions (or methods): * segment(array) (produce an array of integers from an array producing the different "regions" of an array: segment([10,20,10,20,30,30,10]) would produce ([0,1,0,1,2,2,0]) * edges(array, at=True) produce an index array providing the edges (with either fence-post like syntax for reduce-at or both boundaries like reducein. Thoughts? -Travis Thoughts on the general idea? -- Travis Oliphant Enthought Inc. 1-512-536-1057 http://www.enthought.com oliphant at enthought.com From pav at iki.fi Sat Apr 10 13:45:12 2010 From: pav at iki.fi (Pauli Virtanen) Date: Sat, 10 Apr 2010 19:45:12 +0200 Subject: [Numpy-discussion] Proposal for new ufunc functionality In-Reply-To: <3BB52545-B5B9-4B9C-AD9A-0D69432A6B2D@enthought.com> References: <3BB52545-B5B9-4B9C-AD9A-0D69432A6B2D@enthought.com> Message-ID: <1270921512.11355.10.camel@talisman> la, 2010-04-10 kello 12:23 -0500, Travis Oliphant kirjoitti: [clip] > Here are my suggested additions to NumPy: > ufunc methods: [clip] > * reducein (array, indices, axis=0) > similar to reduce-at, but the indices provide both the > start and end points (rather than being fence-posts like reduceat). Is the `reducein` important to have, as compared to `reduceat`? [clip] > numpy functions (or methods): I'd prefer functions here. ndarray already has a huge number of methods. > * segment(array) > > (produce an array of integers from an array producing the > different "regions" of an array: > > segment([10,20,10,20,30,30,10]) would produce ([0,1,0,1,2,2,0]) Sounds like `np.digitize(x, bins=np.unique(x))-1`. What would the behavior be with structured arrays? > * edges(array, at=True) > > produce an index array providing the edges (with either fence-post > like syntax for reduce-at or both boundaries like reducein. This can probably easily be based on segment(). > Thoughts on the general idea? One question is whether these methods should be stuffed to the main namespace, or under numpy.rec. Another addition to ufuncs that should be though about is specifying the Python-side interface to generalized ufuncs. -- Pauli Virtanen From josef.pktd at gmail.com Sat Apr 10 13:49:37 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sat, 10 Apr 2010 13:49:37 -0400 Subject: [Numpy-discussion] Proposal for new ufunc functionality In-Reply-To: <3BB52545-B5B9-4B9C-AD9A-0D69432A6B2D@enthought.com> References: <3BB52545-B5B9-4B9C-AD9A-0D69432A6B2D@enthought.com> Message-ID: On Sat, Apr 10, 2010 at 1:23 PM, Travis Oliphant wrote: > > Hi, > > I've been mulling over a couple of ideas for new ufunc methods plus a > couple of numpy functions that I think will help implement group-by > operations with NumPy arrays. > > I wanted to discuss them on this list before putting forward an actual > proposal or patch to get input from others. > > The group-by operation is very common in relational algebra and NumPy > arrays (especially structured arrays) can often be seen as a database > table. ? ?There are common and easy-to implement approaches for select > and other relational algebra concepts, but group-by basically has to > be implemented yourself. > > Here are my suggested additions to NumPy: > > ufunc methods: > ? ? ? ?* reduceby (array, by, sorted=1, axis=0) > > ? ? ? ? ? ? ?array is the array to reduce > ? ? ? ? ? ? by is the array to provide the grouping (can be a structured > array or a list of arrays) > > ? ? ? ? ? ? ?if sorted is 1, then possibly a faster algorithm can be > used. how is the grouping in "by" specified? These functions would be very useful for statistics. One problem with the current bincount is that it doesn't allow multi-dimensional weight arrays (with axis argument). Josef > > ? ? ? ?* reducein (array, indices, axis=0) > > ? ? ? ? ? ? ? similar to reduce-at, but the indices provide both the > start and end points (rather than being fence-posts like reduceat). > > numpy functions (or methods): > > ? ? ? ? * segment(array) > > ? ? ? ? ? (produce an array of integers from an array producing the > different "regions" of an array: > > ? ? ? ? ? ?segment([10,20,10,20,30,30,10]) ?would produce ([0,1,0,1,2,2,0]) > > > ? ? ? ? * edges(array, at=True) > > ? ? ? ? ? produce an index array providing the edges (with either fence-post > like syntax for reduce-at or both boundaries like reducein. > > > Thoughts? > > -Travis > > > > > > > Thoughts on the general idea? > > > -- > Travis Oliphant > Enthought Inc. > 1-512-536-1057 > http://www.enthought.com > oliphant at enthought.com > > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From stefan at sun.ac.za Sat Apr 10 17:47:25 2010 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Sat, 10 Apr 2010 23:47:25 +0200 Subject: [Numpy-discussion] Proposal for new ufunc functionality In-Reply-To: <1270921512.11355.10.camel@talisman> References: <3BB52545-B5B9-4B9C-AD9A-0D69432A6B2D@enthought.com> <1270921512.11355.10.camel@talisman> Message-ID: On 10 April 2010 19:45, Pauli Virtanen wrote: > Another addition to ufuncs that should be though about is specifying the > Python-side interface to generalized ufuncs. This is an interesting idea; what do you have in mind? Regards St?fan From akikumar1983 at gmail.com Sat Apr 10 18:44:55 2010 From: akikumar1983 at gmail.com (AKI) Date: Sat, 10 Apr 2010 16:44:55 -0600 Subject: [Numpy-discussion] INSTALLATION ON CYGWIN Message-ID: There is too much out there which is making me confuse, I want to install Numpy and Scipy on cygwin....Can some body give me steps...There is different version of Numpy ...which one i need to download.....and how to check it after installing...... I already have cygwin full version on my pc... Aki -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sat Apr 10 18:59:21 2010 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 10 Apr 2010 17:59:21 -0500 Subject: [Numpy-discussion] Proposal for new ufunc functionality In-Reply-To: <1270921512.11355.10.camel@talisman> References: <3BB52545-B5B9-4B9C-AD9A-0D69432A6B2D@enthought.com> <1270921512.11355.10.camel@talisman> Message-ID: On Sat, Apr 10, 2010 at 12:45, Pauli Virtanen wrote: > la, 2010-04-10 kello 12:23 -0500, Travis Oliphant kirjoitti: > [clip] >> Here are my suggested additions to NumPy: >> ufunc methods: > [clip] >> ? ? ? * reducein (array, indices, axis=0) >> ? ? ? ? ? ? ? ?similar to reduce-at, but the indices provide both the >> start and end points (rather than being fence-posts like reduceat). > > Is the `reducein` important to have, as compared to `reduceat`? Yes, I think so. If there are some areas you want to ignore, that's difficult to do with reduceat(). -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From gokhansever at gmail.com Sat Apr 10 20:17:26 2010 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=) Date: Sat, 10 Apr 2010 19:17:26 -0500 Subject: [Numpy-discussion] Simple way to shift array elements Message-ID: Hello, Is there a simpler way to get "c" from "a" I[1]: a = np.arange(10) I[2]: b = a[3:] I[3]: b O[3]: array([3, 4, 5, 6, 7, 8, 9]) I[4]: c = np.insert(b, [7]*3, 0) O[5]: array([3, 4, 5, 6, 7, 8, 9, 0, 0, 0]) a and c have to be same in length and the left shift must be balanced with equal number of 0's Thanks. -- G?khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From kwgoodman at gmail.com Sat Apr 10 20:30:24 2010 From: kwgoodman at gmail.com (Keith Goodman) Date: Sat, 10 Apr 2010 17:30:24 -0700 Subject: [Numpy-discussion] Simple way to shift array elements In-Reply-To: References: Message-ID: On Sat, Apr 10, 2010 at 5:17 PM, G?khan Sever wrote: > Hello, > > Is there a simpler way to get "c" from "a" > > I[1]: a = np.arange(10) > > I[2]: b = a[3:] > > I[3]: b > O[3]: array([3, 4, 5, 6, 7, 8, 9]) > > I[4]: c = np.insert(b, [7]*3, 0) > O[5]: array([3, 4, 5, 6, 7, 8, 9, 0, 0, 0]) > > a and c have to be same in length and the left shift must be balanced with > equal number of 0's Does this count as simpler? >> c = np.zeros_like(a) >> c[:-3] = a[3:] From charlesr.harris at gmail.com Sat Apr 10 20:31:59 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 10 Apr 2010 18:31:59 -0600 Subject: [Numpy-discussion] Simple way to shift array elements In-Reply-To: References: Message-ID: On Sat, Apr 10, 2010 at 6:17 PM, G?khan Sever wrote: > Hello, > > Is there a simpler way to get "c" from "a" > > I[1]: a = np.arange(10) > > I[2]: b = a[3:] > > I[3]: b > O[3]: array([3, 4, 5, 6, 7, 8, 9]) > > I[4]: c = np.insert(b, [7]*3, 0) > O[5]: array([3, 4, 5, 6, 7, 8, 9, 0, 0, 0]) > > a and c have to be same in length and the left shift must be balanced with > equal number of 0's > > Maybe something like In [1]: a = np.arange(10) In [2]: b = zeros_like(a) In [3]: b[:-3] = a[3:] In [4]: b Out[4]: array([3, 4, 5, 6, 7, 8, 9, 0, 0, 0]) Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From gokhansever at gmail.com Sat Apr 10 20:49:38 2010 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=) Date: Sat, 10 Apr 2010 19:49:38 -0500 Subject: [Numpy-discussion] Simple way to shift array elements In-Reply-To: References: Message-ID: On Sat, Apr 10, 2010 at 7:31 PM, Charles R Harris wrote: > > > On Sat, Apr 10, 2010 at 6:17 PM, G?khan Sever wrote: > >> Hello, >> >> Is there a simpler way to get "c" from "a" >> >> I[1]: a = np.arange(10) >> >> I[2]: b = a[3:] >> >> I[3]: b >> O[3]: array([3, 4, 5, 6, 7, 8, 9]) >> >> I[4]: c = np.insert(b, [7]*3, 0) >> O[5]: array([3, 4, 5, 6, 7, 8, 9, 0, 0, 0]) >> >> a and c have to be same in length and the left shift must be balanced with >> equal number of 0's >> >> > Maybe something like > > In [1]: a = np.arange(10) > > In [2]: b = zeros_like(a) > > In [3]: b[:-3] = a[3:] > > In [4]: b > Out[4]: array([3, 4, 5, 6, 7, 8, 9, 0, 0, 0]) > > Chuck > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > Thanks, Your ways are more obvious than my first approach. With a bit more playing I get a one-liner: c=np.append(a[3:], [0]*3) -- G?khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From kwmsmith at gmail.com Sat Apr 10 22:21:03 2010 From: kwmsmith at gmail.com (Kurt Smith) Date: Sat, 10 Apr 2010 21:21:03 -0500 Subject: [Numpy-discussion] numpy build system questions for use in another project (fwrap) In-Reply-To: <4BC090FC.9070001@student.matnat.uio.no> References: <4BBED65A.209@silveregg.co.jp> <4BC090FC.9070001@student.matnat.uio.no> Message-ID: On Sat, Apr 10, 2010 at 9:53 AM, Dag Sverre Seljebotn wrote: > Kurt Smith wrote: >> On Fri, Apr 9, 2010 at 2:25 AM, David wrote: >>> On 04/07/2010 11:52 AM, Kurt Smith wrote: >> Fortran's .mod files are essentially compiler-generated header files; >> fwrap needs to use these 'headers' to get type information so it can >> figure out how the C types and Fortran types match up. ?Fwrap then >> generates the config files with this information and compiles the >> wrappers with the other source files. > > I hope there will at least be an option to supply mod files instead of > using this mechanism. For "wrapper" projects which already (presumably) > has a build system for the Fortran code set up it seems more reasonable > to me to just refer to the output of the existing build. Yes -- I've been thinking about this usecase recently, and I was thinking along the same lines. Fwrap shouldn't have to have a fully general Fortran build system, that obviously isn't its intended purpose. (Build systems for mixed Fortran/C/C++/Whatever are becoming quite an albatross[0], IMHO, and fwrap would do well to avoid the whole issue). I've been assuming up until now that since f2py takes care of it, so should fwrap, but that's foolish. The scipy sources that f2py wraps are all fairly simple, and they're F77 source, which have no concept of interfaces, etc. F9X source is a different beast. I'm realizing the many conflicting usecases that are out there, and the impossibility for fwrap to handle them all well. The simple cases (you mention below) with a few fortran source files are fine, but for a big project with a complicated & long build process, it would be foolish for fwrap to try and become a fully general Fortran build system in addition to its intended purpose. It isn't always clear what compiler flags one needs to use to ensure that the fortran code is compiled suitably for a Python extension module. Here's what I'll probably do: Python has a 'python-config' command that indicates the different flags, include and link libraries to use to compile extension modules. Fwrap could do something similar, and the user is responsible for supplying the compiled .o and .mod files, using the configuration flags supplied by fwrap for compilation. So you could do something like the following to get the compiler flags: $ fwrap --get-cflags --fcompiler=intelem And it would spit out the flags necessary to include when compiling the fortran source files. The user is responsible for handing off the .o and .mod files to fwrap, and fwrap then does the rest. This is good -- I think we're converging on a solution. And it keeps fwrap focused on what it's supposed to do. > > In particular I don't like the situation today with Python wrappers > around C code, where the C code files are often copied into the Python > wrapper project. I hope the same won't happen with fwrap, i.e. that > people don't copy the Fortran sources to the fwrap wrapper just to make > things easy to build (but end up forking the project and not keep it up > to date). No -- I'll do everything in my power to keep anything like this from being necessary :-) Thanks for the input. Kurt [0] http://en.wikipedia.org/wiki/Albatross_%28metaphor%29 > > Of course, if one is not wrapping an existing module but rather > developing an application with fwrap then the situation is different, I > suppose. > > -- > Dag Sverre > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From lane at brooks.nu Sun Apr 11 01:00:10 2010 From: lane at brooks.nu (Lane Brooks) Date: Sat, 10 Apr 2010 22:00:10 -0700 Subject: [Numpy-discussion] Masked Array Usage Problems In-Reply-To: References: <4BC02D6F.1070408@brooks.nu> Message-ID: <3C81114B-6697-484E-8C91-9EC720089808@brooks.nu> On Apr 10, 2010, at 5:17 AM, josef.pktd at gmail.com wrote: > On Sat, Apr 10, 2010 at 3:49 AM, Lane Brooks wrote: >> I am trying out masked arrays for the first time and having some >> problems. I have a 2-D image as dtype=numpy.int16 >> >> I create a mask of all False to not mask out any pixels. >> >> I calculate the mean of the image original image and it comes out >> ~597. >> I calculate the mean of the masked array and it comes out differently >> around -179. It produces the same negative mean value no matter what >> masks I try, e.g. (all True, all False, etc). Furthermore there >> are no >> negative samples in the entire array. >> >> Any ideas on what am I doing wrong? >> >> Here is some sample code showing the behavior: >> >> In [1]: img.dtype, img.shape >> Out[1]: (dtype('int16'), (3200, 3456)) >> >> In [2]: mask = numpy.zeros(img.shape, dtype=bool) >> >> In [3]: imgma = ma.masked_array(img, mask) >> >> In [4]: img.mean() >> Out[4]: 597.15437617549185 >> >> In [5]: imgma.mean() >> Out[5]: -179.56858678747108 >> >> In [6]: imgma.min() >> Out[6]: 25 >> >> In [7]: numpy.__version__ >> Out[7]: '1.3.0' >> >> In [8]: numpy.ma.__version__ >> Out[8]: '1.0' > > Just a guess untill Pierre replies: > > It looks to me like an integer overflow bug. Can you try > > imgma.mean(dtype=float) > > to do the accumulation with floating points? > > Josef >> Indeed using dtype=float solved the problem. The numpy.mean doc string says the default accumulator type for all int types is a float. Why is ma.mean different, especially since the ma.mean doc string says to see the numpy.mean doc string? Thanks Lane From josef.pktd at gmail.com Sun Apr 11 01:10:28 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 11 Apr 2010 01:10:28 -0400 Subject: [Numpy-discussion] Masked Array Usage Problems In-Reply-To: <3C81114B-6697-484E-8C91-9EC720089808@brooks.nu> References: <4BC02D6F.1070408@brooks.nu> <3C81114B-6697-484E-8C91-9EC720089808@brooks.nu> Message-ID: On Sun, Apr 11, 2010 at 1:00 AM, Lane Brooks wrote: > > On Apr 10, 2010, at 5:17 AM, josef.pktd at gmail.com wrote: > >> On Sat, Apr 10, 2010 at 3:49 AM, Lane Brooks wrote: >>> I am trying out masked arrays for the first time and having some >>> problems. I have a 2-D image as dtype=numpy.int16 >>> >>> I create a mask of all False to not mask out any pixels. >>> >>> I calculate the mean of the image original image and it comes out >>> ~597. >>> I calculate the mean of the masked array and it comes out differently >>> around -179. ?It produces the same negative mean value no matter what >>> masks I try, e.g. (all True, all False, etc). ?Furthermore there >>> are no >>> negative samples in the entire array. >>> >>> Any ideas on what am I doing wrong? >>> >>> Here is some sample code showing the behavior: >>> >>> In [1]: img.dtype, img.shape >>> Out[1]: (dtype('int16'), (3200, 3456)) >>> >>> In [2]: mask = numpy.zeros(img.shape, dtype=bool) >>> >>> In [3]: imgma = ma.masked_array(img, mask) >>> >>> In [4]: img.mean() >>> Out[4]: 597.15437617549185 >>> >>> In [5]: imgma.mean() >>> Out[5]: -179.56858678747108 >>> >>> In [6]: imgma.min() >>> Out[6]: 25 >>> >>> In [7]: numpy.__version__ >>> Out[7]: '1.3.0' >>> >>> In [8]: numpy.ma.__version__ >>> Out[8]: '1.0' >> >> Just a guess untill Pierre replies: >> >> It looks to me like an integer overflow bug. Can you try >> >> imgma.mean(dtype=float) >> >> to do the accumulation with floating points? >> >> Josef >>> > > Indeed using dtype=float solved the problem. The numpy.mean doc string > says the default accumulator type for all int types is a float. Why is > ma.mean different, especially since the ? ?ma.mean doc string says to > see the numpy.mean doc string? I think it is a bug in ma.mean, since the docstring clearly specifies the casting to float. Can you file a ticket? I think Pierre will look at it when he finds time. Thanks, Josef > > Thanks > Lane > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From ralf.gommers at googlemail.com Sun Apr 11 05:09:53 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Sun, 11 Apr 2010 17:09:53 +0800 Subject: [Numpy-discussion] rc2 for NumPy 1.4.1 and Scipy 0.7.2 Message-ID: Hi, I am pleased to announce the second release candidate of both Scipy 0.7.2 and NumPy 1.4.1, please test them. The issues reported with rc1 should be fixed, and for NumPy there are now Python 2.5 binaries as well. For SciPy there will be no 2.5 binaries - because 0.7.x is built against NumPy 1.2 it does not work on OS X 10.6, and on Windows I see some incomprehensible build failures. Binaries, sources and release notes can be found at http://www.filefactory.com/f/4452c50056df8bba/ (apologies for the flashy ads again). Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebastian.walter at gmail.com Sun Apr 11 06:59:45 2010 From: sebastian.walter at gmail.com (Sebastian Walter) Date: Sun, 11 Apr 2010 12:59:45 +0200 Subject: [Numpy-discussion] Math Library In-Reply-To: References: Message-ID: On Tue, Apr 6, 2010 at 9:16 PM, Travis Oliphant wrote: > > On Apr 6, 2010, at 9:08 AM, David Cournapeau wrote: > > Hi Travis, > > On Tue, Apr 6, 2010 at 7:43 AM, Travis Oliphant > wrote: > > > I should have some time over the next couple of weeks, and I am very > > interested in refactoring the NumPy code to separate out the Python > > interface layer from the "library" layer as much as possible. ? I had > > some discussions with people at PyCon about making it easier for > > Jython, IronPython, and perhaps even other high-level languages to > > utilize NumPy. > > Is there a willingness to consider as part of this reorganization > > creating a clear boundary between the NumPy library code and the > > Python-specific interface to it? ? What other re-organization thoughts > > are you having David? > > This is mainly it, reorganizing the code for clearer boundaries > between boilerplate (python C API) and actual compuational code. > Besides helping other python implementations, I think this would > benefit NumPy itself in the long run (code maintainability), as well > as scipy (and other C extensions). I think the npymath effort is a > good example: albeit simple in nature (the API and boundaries are > obvious), it has already helped a lot to solve numerous platform > specific issues in numpy and scipy, and I think the overall code > quality is better. > > My own goals were: > - exposing core computational parts through an exported C API, so > that other C extensions may use it (for example, exposing basic > blas/lapack operations) > - dynamic loading of the code (for example depending on the CPU > capabilities - I have a git branch somewhere where I started exposing > a simple C API to query cpu capabilities like cache size or SSE > dynamically to that intent) > - more amenable codebase: I think multiarray in particular is too > big. I don't know the code well enough to know what can be split and > how, but I would have hoped that the scalartypes, the type descriptor > could be put out of multiarray proper. Also, exposing an API for > things like fancy indexing would be very useful, but I don't know if > it even makes sense - I think a pure python implementation of fancy > indexing as a reference would be very useful for array-like classes (I > am thinking about scipy.sparse, for example). > > Unfortunately, I won't be able to help much in the near future (except > maybe for the fancy indexing as this could be useful for my job), > > > I understand. ? It just happens that there is some significant time for me > to look at this over the next few weeks and I would really like to make > progress on re-factoring. ? I think it's O.K. if you don't have time right > now to help as long as you have time to offer criticism and suggestions. > We could even do that over Skype with whomever else wanted to join us (we > could do a GotoMeeting discussion as well) if you think it would be faster > to just talk in a group setting instead of email. ? ??Of course, a summary > of any off-line discussion should be sent to the list. I'm not sure how much I could contribute to the discussion since I have only quite hazy knowledge of the numpy core. However, I'm interested in the outcome of the refactoring since I'm facing a "similar" problem in http://github.com/b45ch1/taylorpoly where I've implemented core algorithms in C for formal power series over the reals. Basically the formal powerseries are a new dtype and the goal is to be able to do computations like x = numpy.zeros((2,3),dtype='name of the formal powerseries') y = numpy.sin(x) Sebastian > Thanks for the input, > -Travis > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From seb.haase at gmail.com Sun Apr 11 10:58:11 2010 From: seb.haase at gmail.com (Sebastian Haase) Date: Sun, 11 Apr 2010 16:58:11 +0200 Subject: [Numpy-discussion] INSTALLATION ON CYGWIN In-Reply-To: References: Message-ID: Hi Aki, what version of python of are you using ? Are you using the standard python.org Windows build ? Most people here use cygwin only to build python extensions (from source) - and use the numpy (currently version 1.3) and scipy from the sourceforge download page. Look for "superpack" to get somthing that is automatically optimized for your CPU. (You are using 32 bit XP or Vista or 7, right ?) Regards, Sebastian Haase On Sun, Apr 11, 2010 at 12:44 AM, AKI wrote: > There is too much out there which is making me confuse, I want to install > Numpy and Scipy on cygwin....Can some body give me steps...There is > different version > of Numpy ...which one i need to download.....and how to check it after > installing...... > I already have cygwin full version on my pc... > Aki > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From seb.haase at gmail.com Sun Apr 11 11:08:17 2010 From: seb.haase at gmail.com (Sebastian Haase) Date: Sun, 11 Apr 2010 17:08:17 +0200 Subject: [Numpy-discussion] rc2 for NumPy 1.4.1 and Scipy 0.7.2 In-Reply-To: References: Message-ID: Hi Ralf, congratulation to the new RC. Just for the record, where did you announce RC1 ? Only on numpy-dev ? On Sun, Apr 11, 2010 at 11:09 AM, Ralf Gommers wrote: > For SciPy there will be no 2.5 binaries - > because 0.7.x is built against NumPy 1.2 Could you elaborate on this ? I thought the new numpy should build with the new scipy ... ? > it does not work on OS X 10.6, and > on Windows I see some incomprehensible build failures. > > Binaries, sources and release notes can be found at > http://www.filefactory.com/f/4452c50056df8bba/ (apologies for the flashy ads > again). Why not put these on the sourceforge page ? (I don't know anything about the technical issues, only a thought...) Regards, -Sebastian From akikumar1983 at gmail.com Sun Apr 11 13:06:11 2010 From: akikumar1983 at gmail.com (AKI) Date: Sun, 11 Apr 2010 11:06:11 -0600 Subject: [Numpy-discussion] INSTALLATION ON CYGWIN In-Reply-To: References: Message-ID: > Thanks Sebastian :).;But I think i will install Ubuntu on my system > :)..now........Window 7 is giving me kind of trouble ......i wasted whole > night trying to make MiniGW 64 work for me........too much too do too little > time... > > Hope installing Scipy And Numpy will be easy on Ubuntu > > Thanks Again! > > Aki > > > On Sun, Apr 11, 2010 at 8:58 AM, Sebastian Haase wrote: > >> Hi Aki, >> >> what version of python of are you using ? Are you using the standard >> python.org Windows build ? >> Most people here use cygwin only to build python extensions (from >> source) - and use the numpy (currently version 1.3) and scipy from the >> sourceforge download page. Look for "superpack" to get somthing that >> is automatically optimized for your CPU. >> (You are using 32 bit XP or Vista or 7, right ?) >> >> Regards, >> Sebastian Haase >> >> >> >> On Sun, Apr 11, 2010 at 12:44 AM, AKI wrote: >> > There is too much out there which is making me confuse, I want to >> install >> > Numpy and Scipy on cygwin....Can some body give me steps...There is >> > different version >> > of Numpy ...which one i need to download.....and how to check it after >> > installing...... >> > I already have cygwin full version on my pc... >> > Aki >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at scipy.org >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >> > >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dagss at student.matnat.uio.no Sun Apr 11 13:54:04 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sun, 11 Apr 2010 19:54:04 +0200 Subject: [Numpy-discussion] numpy build system questions for use in another project (fwrap) In-Reply-To: References: <4BBED65A.209@silveregg.co.jp> <4BC090FC.9070001@student.matnat.uio.no> Message-ID: <4BC20CBC.3040903@student.matnat.uio.no> Kurt Smith wrote: > On Sat, Apr 10, 2010 at 9:53 AM, Dag Sverre Seljebotn > wrote: >> Kurt Smith wrote: >>> On Fri, Apr 9, 2010 at 2:25 AM, David wrote: >>>> On 04/07/2010 11:52 AM, Kurt Smith wrote: > >>> Fortran's .mod files are essentially compiler-generated header files; >>> fwrap needs to use these 'headers' to get type information so it can >>> figure out how the C types and Fortran types match up. Fwrap then >>> generates the config files with this information and compiles the >>> wrappers with the other source files. >> I hope there will at least be an option to supply mod files instead of >> using this mechanism. For "wrapper" projects which already (presumably) >> has a build system for the Fortran code set up it seems more reasonable >> to me to just refer to the output of the existing build. > > Yes -- I've been thinking about this usecase recently, and I was > thinking along the same lines. Fwrap shouldn't have to have a fully > general Fortran build system, that obviously isn't its intended > purpose. (Build systems for mixed Fortran/C/C++/Whatever are becoming > quite an albatross[0], IMHO, and fwrap would do well to avoid the > whole issue). I've been assuming up until now that since f2py takes > care of it, so should fwrap, but that's foolish. The scipy sources > that f2py wraps are all fairly simple, and they're F77 source, which > have no concept of interfaces, etc. F9X source is a different beast. > > I'm realizing the many conflicting usecases that are out there, and > the impossibility for fwrap to handle them all well. The simple cases > (you mention below) with a few fortran source files are fine, but for > a big project with a complicated & long build process, it would be > foolish for fwrap to try and become a fully general Fortran build > system in addition to its intended purpose. > > It isn't always clear what compiler flags one needs to use to ensure > that the fortran code is compiled suitably for a Python extension > module. > > Here's what I'll probably do: > > Python has a 'python-config' command that indicates the different > flags, include and link libraries to use to compile extension modules. > Fwrap could do something similar, and the user is responsible for > supplying the compiled .o and .mod files, using the configuration > flags supplied by fwrap for compilation. > > So you could do something like the following to get the compiler flags: > > $ fwrap --get-cflags --fcompiler=intelem > > And it would spit out the flags necessary to include when compiling > the fortran source files. Hmm. Would --get-cflags ever return anything but (the equivalent of) -fPIC? For C source which includes "Python.h" there are more flags (-fno-strict-aliasing for instance) because Python.h requires it. BUT, note that this is only for "Python.h"-including C source. For just linking in C source without a Python.h dependency, the compiler flags don't matter at all, just as long as one uses -fPIC (IIUC). Fortran code is also in this latter category, so I wonder whether what you mention above is really necesarry. To me it seems like one can just document: a) Use the same compiler for both the Fortran project and the fwrap wrapper. b) Use -fPIC. c) Feed the build system of the fwrap wrapper the path to the mod files. > > The user is responsible for handing off the .o and .mod files to > fwrap, and fwrap then does the rest. > > This is good -- I think we're converging on a solution. And it keeps > fwrap focused on what it's supposed to do. Yes, apart from my nitpicking questions above, this sounds good to me at least. Dag Sverre > >> In particular I don't like the situation today with Python wrappers >> around C code, where the C code files are often copied into the Python >> wrapper project. I hope the same won't happen with fwrap, i.e. that >> people don't copy the Fortran sources to the fwrap wrapper just to make >> things easy to build (but end up forking the project and not keep it up >> to date). > > No -- I'll do everything in my power to keep anything like this from > being necessary :-) > > Thanks for the input. > > Kurt > > [0] http://en.wikipedia.org/wiki/Albatross_%28metaphor%29 > >> Of course, if one is not wrapping an existing module but rather >> developing an application with fwrap then the situation is different, I >> suppose. >> >> -- >> Dag Sverre >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -- Dag Sverre From russel at appliedminds.com Sun Apr 11 14:43:27 2010 From: russel at appliedminds.com (russel) Date: Sun, 11 Apr 2010 11:43:27 -0700 Subject: [Numpy-discussion] rc2 for NumPy 1.4.1 and Scipy 0.7.2 In-Reply-To: References: Message-ID: <4BC2184F.2070208@appliedminds.com> Any chance of getting this into the next release? http://projects.scipy.org/scipy/ticket/1122 I marked it 0.8 because that was the default, but it is a very simple fix. Russel Sebastian Haase wrote: > Hi Ralf, > congratulation to the new RC. > Just for the record, where did you announce RC1 ? Only on numpy-dev ? > > On Sun, Apr 11, 2010 at 11:09 AM, Ralf Gommers > wrote: > > >> For SciPy there will be no 2.5 binaries - >> because 0.7.x is built against NumPy 1.2 >> > > Could you elaborate on this ? I thought the new numpy should build > with the new scipy ... ? > > >> it does not work on OS X 10.6, and >> on Windows I see some incomprehensible build failures. >> >> Binaries, sources and release notes can be found at >> http://www.filefactory.com/f/4452c50056df8bba/ (apologies for the flashy ads >> again). >> > > Why not put these on the sourceforge page ? (I don't know anything > about the technical issues, only a thought...) > > Regards, > -Sebastian > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From peridot.faceted at gmail.com Sun Apr 11 15:56:24 2010 From: peridot.faceted at gmail.com (Anne Archibald) Date: Sun, 11 Apr 2010 15:56:24 -0400 Subject: [Numpy-discussion] Proposal for new ufunc functionality In-Reply-To: References: <3BB52545-B5B9-4B9C-AD9A-0D69432A6B2D@enthought.com> <1270921512.11355.10.camel@talisman> Message-ID: 2010/4/10 St?fan van der Walt : > On 10 April 2010 19:45, Pauli Virtanen wrote: >> Another addition to ufuncs that should be though about is specifying the >> Python-side interface to generalized ufuncs. > > This is an interesting idea; what do you have in mind? I can see two different kinds of answer to this question: one is a tool like vectorize/frompyfunc that allows construction of generalized ufuncs from python functions, and the other is thinking out what methods and support functions generalized ufuncs need. The former would be very handy for prototyping gufunc-based libraries before delving into the templated C required to make them actually efficient. The latter is more essential in the long run: it'd be nice to have a reduce-like function, but obviously only when the arity and dimensions work out right (which I think means (shape1,shape2)->(shape2) ). This could be applied along an axis or over a whole array. reduceat and the other, more sophisticated, schemes might also be worth supporting. At a more elementary level, gufunc objects should have good introspection - docstrings, shape specification accessible from python, named formal arguments, et cetera. (So should ufuncs, for that matter.) Anne > Regards > St?fan > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From sebastian.walter at gmail.com Sun Apr 11 17:17:58 2010 From: sebastian.walter at gmail.com (Sebastian Walter) Date: Sun, 11 Apr 2010 23:17:58 +0200 Subject: [Numpy-discussion] Math Library In-Reply-To: References: Message-ID: On Sun, Apr 11, 2010 at 12:59 PM, Sebastian Walter wrote: > On Tue, Apr 6, 2010 at 9:16 PM, Travis Oliphant wrote: >> >> On Apr 6, 2010, at 9:08 AM, David Cournapeau wrote: >> >> Hi Travis, >> >> On Tue, Apr 6, 2010 at 7:43 AM, Travis Oliphant >> wrote: >> >> >> I should have some time over the next couple of weeks, and I am very >> >> interested in refactoring the NumPy code to separate out the Python >> >> interface layer from the "library" layer as much as possible. ? I had >> >> some discussions with people at PyCon about making it easier for >> >> Jython, IronPython, and perhaps even other high-level languages to >> >> utilize NumPy. >> >> Is there a willingness to consider as part of this reorganization >> >> creating a clear boundary between the NumPy library code and the >> >> Python-specific interface to it? ? What other re-organization thoughts >> >> are you having David? >> >> This is mainly it, reorganizing the code for clearer boundaries >> between boilerplate (python C API) and actual compuational code. >> Besides helping other python implementations, I think this would >> benefit NumPy itself in the long run (code maintainability), as well >> as scipy (and other C extensions). I think the npymath effort is a >> good example: albeit simple in nature (the API and boundaries are >> obvious), it has already helped a lot to solve numerous platform >> specific issues in numpy and scipy, and I think the overall code >> quality is better. >> >> My own goals were: >> - exposing core computational parts through an exported C API, so >> that other C extensions may use it (for example, exposing basic >> blas/lapack operations) >> - dynamic loading of the code (for example depending on the CPU >> capabilities - I have a git branch somewhere where I started exposing >> a simple C API to query cpu capabilities like cache size or SSE >> dynamically to that intent) >> - more amenable codebase: I think multiarray in particular is too >> big. I don't know the code well enough to know what can be split and >> how, but I would have hoped that the scalartypes, the type descriptor >> could be put out of multiarray proper. Also, exposing an API for >> things like fancy indexing would be very useful, but I don't know if >> it even makes sense - I think a pure python implementation of fancy >> indexing as a reference would be very useful for array-like classes (I >> am thinking about scipy.sparse, for example). >> >> Unfortunately, I won't be able to help much in the near future (except >> maybe for the fancy indexing as this could be useful for my job), >> >> >> I understand. ? It just happens that there is some significant time for me >> to look at this over the next few weeks and I would really like to make >> progress on re-factoring. ? I think it's O.K. if you don't have time right >> now to help as long as you have time to offer criticism and suggestions. >> We could even do that over Skype with whomever else wanted to join us (we >> could do a GotoMeeting discussion as well) if you think it would be faster >> to just talk in a group setting instead of email. ? ??Of course, a summary >> of any off-line discussion should be sent to the list. > > I'm not sure how much I could contribute to the discussion since I > have only quite hazy > knowledge of the numpy core. However, I'm interested in the outcome of > the refactoring > since I'm facing a "similar" problem in > http://github.com/b45ch1/taylorpoly where I've implemented core > algorithms in C > for formal power series over the reals. Basically the formal > powerseries are a new dtype and the goal is to be able to > do computations like > x = numpy.zeros((2,3),dtype='name of the formal powerseries') > y = numpy.sin(x) Ermm, the reply above is quite poor, sorry about that. What I meant to say is the following: If there is going to be a discussion about creating a pure C numpy library I'd like to join ;) > > Sebastian > > >> Thanks for the input, >> -Travis >> >> >> >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > From ralf.gommers at googlemail.com Sun Apr 11 20:17:03 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Mon, 12 Apr 2010 08:17:03 +0800 Subject: [Numpy-discussion] rc2 for NumPy 1.4.1 and Scipy 0.7.2 In-Reply-To: References: Message-ID: On Sun, Apr 11, 2010 at 11:08 PM, Sebastian Haase wrote: > Hi Ralf, > congratulation to the new RC. > Just for the record, where did you announce RC1 ? Only on numpy-dev ? > Since I'm new to this and wasn't sure it was all going to work right, I announced rc1 only on scipy-dev (which in my mind is roughly a subset of numpy-dev). This one I should probably also announce on scipy/MPL/pygame lists to get more feedback. Will do that today, with the below explanation. > On Sun, Apr 11, 2010 at 11:09 AM, Ralf Gommers > wrote: > > > For SciPy there will be no 2.5 binaries - > > because 0.7.x is built against NumPy 1.2 > > Could you elaborate on this ? I thought the new numpy should build > with the new scipy ... ? > This "new" scipy works with the new numpy, but for backwards compatibility is built against older versions. For python 2.6 scipy is built against numpy 1.3, for 2.5 it should be built against 1.2. I'm not so sure about the reason for that difference, but that's how it was done for 0.7.0 and 0.7.1. Note that these releases are based on the 1.4.x and 0.7.x branches, which are already quite old. Especially scipy, that branch was created in Jan 2009. Releases based on currents trunks will be numpy 2.0 and scipy 0.8, which are the next target. These releases are maintenance releases to recover from the binary compatibility issues that plagued numpy 1.4.0. > > it does not work on OS X 10.6, and > > on Windows I see some incomprehensible build failures. > > > > Binaries, sources and release notes can be found at > > http://www.filefactory.com/f/4452c50056df8bba/ (apologies for the flashy > ads > > again). > > Why not put these on the sourceforge page ? (I don't know anything > about the technical issues, only a thought...) > Will do that the moment I get the password. Cheers, Ralf > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Sun Apr 11 20:20:52 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Mon, 12 Apr 2010 08:20:52 +0800 Subject: [Numpy-discussion] rc2 for NumPy 1.4.1 and Scipy 0.7.2 In-Reply-To: <4BC2184F.2070208@appliedminds.com> References: <4BC2184F.2070208@appliedminds.com> Message-ID: On Mon, Apr 12, 2010 at 2:43 AM, russel wrote: > Any chance of getting this into the next release? > > http://projects.scipy.org/scipy/ticket/1122 > > I marked it 0.8 because that was the default, but it is a very simple fix. > > Don't think so, sorry. If everything is OK then the final 0.7.2 will be the same as rc2. The scipy tickets are a bit out of control, for 0.8 (which should not be too far into the future) these kind of tickets should be addressed. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From kwmsmith at gmail.com Sun Apr 11 21:23:15 2010 From: kwmsmith at gmail.com (Kurt Smith) Date: Sun, 11 Apr 2010 20:23:15 -0500 Subject: [Numpy-discussion] numpy build system questions for use in another project (fwrap) In-Reply-To: <4BC20CBC.3040903@student.matnat.uio.no> References: <4BBED65A.209@silveregg.co.jp> <4BC090FC.9070001@student.matnat.uio.no> <4BC20CBC.3040903@student.matnat.uio.no> Message-ID: On Sun, Apr 11, 2010 at 12:54 PM, Dag Sverre Seljebotn wrote: > Kurt Smith wrote: ... >> >> So you could do something like the following to get the compiler flags: >> >> $ fwrap --get-cflags --fcompiler=intelem >> >> And it would spit out the flags necessary to include when compiling >> the fortran source files. > > Hmm. Would --get-cflags ever return anything but (the equivalent of) -fPIC? > > For C source which includes "Python.h" there are more flags > (-fno-strict-aliasing for instance) because Python.h requires it. > > BUT, note that this is only for "Python.h"-including C source. For just > linking in C source without a Python.h dependency, the compiler flags > don't matter at all, just as long as one uses -fPIC (IIUC). > > Fortran code is also in this latter category, so I wonder whether what > you mention above is really necesarry. To me it seems like one can just > document: > > ?a) Use the same compiler for both the Fortran project and the fwrap > wrapper. > ?b) Use -fPIC. > ?c) Feed the build system of the fwrap wrapper the path to the mod files. Yes, (b) is a possibility (if indeed -fPIC (or it's equivalent) is all you need -- are you sure this is the case?), but if it's easy for fwrap to query numpy.distutils.fcompiler for the compiler and platform specific flags, then why not? IOW, the work has already been done in the fcompiler package, and it should be simple to query those modules. One fewer barrier for the user, as I see it. Kurt From david at silveregg.co.jp Sun Apr 11 21:45:11 2010 From: david at silveregg.co.jp (David) Date: Mon, 12 Apr 2010 10:45:11 +0900 Subject: [Numpy-discussion] numpy build system questions for use in another project (fwrap) In-Reply-To: References: <4BBED65A.209@silveregg.co.jp> Message-ID: <4BC27B27.6020008@silveregg.co.jp> On 04/10/2010 03:02 AM, Kurt Smith wrote: > On Fri, Apr 9, 2010 at 2:25 AM, David wrote: >> On 04/07/2010 11:52 AM, Kurt Smith wrote: >>> Briefly, I'm encountering difficulties getting things working in numpy >>> distutils for fwrap's build system. >>> >>> Here are the steps I want the build system to accomplish: >>> >>> 1) Compile a directory of Fortran 90 source code -- this works. >>> - The .mod files generated by this compilation step are put in the >>> build directory. >> >> This is difficult - fortran modules are a PITA from a build perspective. >> Many compilers don't seem to have a way to control exactly where to put >> the generated .mod, so the only way I am aware of to control this is to >> cwd the process into the build directory... > >> From what I can tell, numpy's distutils already takes care of putting > the .mod files in the build directory (at least for gfortran and > ifort) by manually moving them there -- see > numpy.distutils.command.build_ext.build_ext.build_extension, after the > line "if fmodule_source". This is fine for my purposes. All I need > is the project's .mod files put in one place before the next step. Moving files is not good for various reasons - I think it contribute to making the build more fragile, and it may cause race conditions for // builds (I am not sure about that last point, I think it is also OS dependent). I agree that it is not the main worry at that point, though. > > I could make it a requirement that the user supply the .mod files to > fwrap before calling 'python setup.py build_ext', but that might be a > big mess when you take into account compiler flags that modify default > type sizes, etc. Maybe it would work fine; I'm not sure whether > different compiler flags used by distutils would change the .mod files > and break the type info, though. And the source code would be compiled > twice -- once to get the .mod files, and again when distutils compiles > everything with the right flags which would be suboptimal for large > projects. > > (Note: I've looked into this a bit, and it seems like it might work > alright -- you can use '-fsyntax-only' in gfortran and g95, and > '-syntax-only' in ifort, for example, and that generates the .mod > files. I'll look into it some more...) > > It would be a PITA to test for every kind-type-parameter that someone > might use in the config stage. As you probably know, Fortran 9x > allows types like this: > > real(kind=selected_real_kind(10,100)) :: ffed_real > > Which means I'd have to do exhaustive testing for every combination of > arguments to selected_real_kind() (up to some limit). This would be > *extremely* slow. I did not know about this feature, but I think it can be solved relatively easily with type checking, and it would not be that slow if you allow "expected size". Also, the code for this is already there in numpy/distutils/commands/config.py, and is relatively independent of the rest of distutils (you only need to give _compile). > >> normally, unless you have a reason to do otherwise. Concerning the >> --fcompiler option, how do you pass --fcompiler (what is the exact list >> of commands you used to call distutils here) ? Generally, given the >> general command/option screw up, the only real solution really is to >> pass the options to each command, and hope each one is the same. >> > > The commandline is 'python setup.py build_ext --inplace > --fcompiler=gfortran' (or --fcompiler=intelem). When I pass > --fcompiler=gfortran, the try_compile() command ignores it, searches > for a fortran compiler and finds ifort instead and uses it, which > breaks everything. Yes, only build_ext knows about --fcompiler, and try_compile comes from configure command. That's one aspect of the utterly broken command design of distutils, that's why you need to say something like python setup.py config --fcompiler=gfortran build_ext --fcompiler=gfortran. > > Is there an example in numpy.distutils that shows how to pass the > commandline options to each command? It is not possible AFAIK - you only know about --fcompiler in build_ext.finalize_options call, and at that point, config may already have run. > > I'm thinking of rolling my own 'try_compile()' function as part of a > custom build_ext command class that suits my purposes. We'll see how > far that gets me. That's another solution. cheers, David From nadavh at visionsense.com Mon Apr 12 05:03:04 2010 From: nadavh at visionsense.com (Nadav Horesh) Date: Mon, 12 Apr 2010 12:03:04 +0300 Subject: [Numpy-discussion] Name of the file associated with a memmap Message-ID: <710F2847B0018641891D9A21602763605AD3A8@ex3.envision.co.il> Is there a way to get the file-name given a memmap array object? Nadav From nadavh at visionsense.com Mon Apr 12 05:03:36 2010 From: nadavh at visionsense.com (Nadav Horesh) Date: Mon, 12 Apr 2010 12:03:36 +0300 Subject: [Numpy-discussion] rc2 for NumPy 1.4.1 and Scipy 0.7.2 References: Message-ID: <710F2847B0018641891D9A21602763605AD3A9@ex3.envision.co.il> Tried of install numy-1.4.1-rc2 on python-2.7b1 and got an error: (64 bit linux on core2, gcc4.4.3) compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/include -I/usr/local/include/python2.7 -c' gcc: _configtest.c _configtest.c:1: warning: conflicting types for built-in function ?exp? gcc -pthread _configtest.o -o _configtest _configtest.o: In function `main': /dev/shm/numpy-1.4.1rc2/_configtest.c:6: undefined reference to `exp' collect2: ld returned 1 exit status _configtest.o: In function `main': /dev/shm/numpy-1.4.1rc2/_configtest.c:6: undefined reference to `exp' collect2: ld returned 1 exit status Traceback (most recent call last): File "setup.py", line 187, in setup_package() File "setup.py", line 180, in setup_package configuration=configuration ) File "/dev/shm/numpy-1.4.1rc2/numpy/distutils/core.py", line 186, in setup return old_setup(**new_attr) File "/usr/local/lib/python2.7/distutils/core.py", line 152, in setup dist.run_commands() File "/usr/local/lib/python2.7/distutils/dist.py", line 953, in run_commands self.run_command(cmd) File "/usr/local/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "/dev/shm/numpy-1.4.1rc2/numpy/distutils/command/build.py", line 37, in run old_build.run(self) File "/usr/local/lib/python2.7/distutils/command/build.py", line 127, in run self.run_command(cmd_name) File "/usr/local/lib/python2.7/distutils/cmd.py", line 326, in run_command self.distribution.run_command(command) File "/usr/local/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "/dev/shm/numpy-1.4.1rc2/numpy/distutils/command/build_src.py", line 152, in run self.build_sources() File "/dev/shm/numpy-1.4.1rc2/numpy/distutils/command/build_src.py", line 163, in build_sources self.build_library_sources(*libname_info) File "/dev/shm/numpy-1.4.1rc2/numpy/distutils/command/build_src.py", line 298, in build_library_sources sources = self.generate_sources(sources, (lib_name, build_info)) File "/dev/shm/numpy-1.4.1rc2/numpy/distutils/command/build_src.py", line 385, in generate_sources source = func(extension, build_dir) File "numpy/core/setup.py", line 658, in get_mathlib_info mlibs = check_mathlib(config_cmd) File "numpy/core/setup.py", line 328, in check_mathlib if config_cmd.check_func("exp", libraries=libs, decl=True, call=True): File "/dev/shm/numpy-1.4.1rc2/numpy/distutils/command/config.py", line 310, in check_func libraries, library_dirs) File "/usr/local/lib/python2.7/distutils/command/config.py", line 251, in try_link libraries, library_dirs, lang) File "/dev/shm/numpy-1.4.1rc2/numpy/distutils/command/config.py", line 146, in _link libraries, library_dirs, lang)) File "/dev/shm/numpy-1.4.1rc2/numpy/distutils/command/config.py", line 87, in _wrap_method ret = mth(*((self,)+args)) File "/usr/local/lib/python2.7/distutils/command/config.py", line 148, in _link target_lang=lang) File "/usr/local/lib/python2.7/distutils/ccompiler.py", line 750, in link_executable debug, extra_preargs, extra_postargs, None, target_lang) File "/usr/local/lib/python2.7/distutils/unixccompiler.py", line 256, in link self.spawn(linker + ld_args) File "/dev/shm/numpy-1.4.1rc2/numpy/distutils/ccompiler.py", line 64, in CCompiler_spawn raise DistutilsExecError,\ Nadav -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 3903 bytes Desc: not available URL: From david at silveregg.co.jp Mon Apr 12 05:25:54 2010 From: david at silveregg.co.jp (David) Date: Mon, 12 Apr 2010 18:25:54 +0900 Subject: [Numpy-discussion] rc2 for NumPy 1.4.1 and Scipy 0.7.2 In-Reply-To: <710F2847B0018641891D9A21602763605AD3A9@ex3.envision.co.il> References: <710F2847B0018641891D9A21602763605AD3A9@ex3.envision.co.il> Message-ID: <4BC2E722.3070805@silveregg.co.jp> On 04/12/2010 06:03 PM, Nadav Horesh wrote: > > Tried of install numy-1.4.1-rc2 on python-2.7b1 and got an error: > > (64 bit linux on core2, gcc4.4.3) > > > compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/include -I/usr/local/include/python2.7 -c' > gcc: _configtest.c > _configtest.c:1: warning: conflicting types for built-in function ?exp? > gcc -pthread _configtest.o -o _configtest > _configtest.o: In function `main': > /dev/shm/numpy-1.4.1rc2/_configtest.c:6: undefined reference to `exp' > collect2: ld returned 1 exit status > _configtest.o: In function `main': > /dev/shm/numpy-1.4.1rc2/_configtest.c:6: undefined reference to `exp' > collect2: ld returned 1 exit status > Traceback (most recent call last): > File "setup.py", line 187, in > setup_package() > File "setup.py", line 180, in setup_package > configuration=configuration ) > File "/dev/shm/numpy-1.4.1rc2/numpy/distutils/core.py", line 186, in setup > return old_setup(**new_attr) > File "/usr/local/lib/python2.7/distutils/core.py", line 152, in setup > dist.run_commands() > File "/usr/local/lib/python2.7/distutils/dist.py", line 953, in run_commands > self.run_command(cmd) > File "/usr/local/lib/python2.7/distutils/dist.py", line 972, in run_command > cmd_obj.run() > File "/dev/shm/numpy-1.4.1rc2/numpy/distutils/command/build.py", line 37, in run > old_build.run(self) > File "/usr/local/lib/python2.7/distutils/command/build.py", line 127, in run > self.run_command(cmd_name) > File "/usr/local/lib/python2.7/distutils/cmd.py", line 326, in run_command > self.distribution.run_command(command) > File "/usr/local/lib/python2.7/distutils/dist.py", line 972, in run_command > cmd_obj.run() > File "/dev/shm/numpy-1.4.1rc2/numpy/distutils/command/build_src.py", line 152, in run > self.build_sources() > File "/dev/shm/numpy-1.4.1rc2/numpy/distutils/command/build_src.py", line 163, in build_sources > self.build_library_sources(*libname_info) > File "/dev/shm/numpy-1.4.1rc2/numpy/distutils/command/build_src.py", line 298, in build_library_sources > sources = self.generate_sources(sources, (lib_name, build_info)) > File "/dev/shm/numpy-1.4.1rc2/numpy/distutils/command/build_src.py", line 385, in generate_sources > source = func(extension, build_dir) > File "numpy/core/setup.py", line 658, in get_mathlib_info > mlibs = check_mathlib(config_cmd) > File "numpy/core/setup.py", line 328, in check_mathlib > if config_cmd.check_func("exp", libraries=libs, decl=True, call=True): > File "/dev/shm/numpy-1.4.1rc2/numpy/distutils/command/config.py", line 310, in check_func > libraries, library_dirs) > File "/usr/local/lib/python2.7/distutils/command/config.py", line 251, in try_link > libraries, library_dirs, lang) Looks like another distutils regression in 2.7 to me - try_link should never cause an error if it fails linking, that's the whole point of the function. cheers, David From bsouthey at gmail.com Mon Apr 12 09:53:05 2010 From: bsouthey at gmail.com (Bruce Southey) Date: Mon, 12 Apr 2010 08:53:05 -0500 Subject: [Numpy-discussion] rc2 for NumPy 1.4.1 and Scipy 0.7.2 In-Reply-To: <4BC2E722.3070805@silveregg.co.jp> References: <710F2847B0018641891D9A21602763605AD3A9@ex3.envision.co.il> <4BC2E722.3070805@silveregg.co.jp> Message-ID: <4BC325C1.4010003@gmail.com> On 04/12/2010 04:25 AM, David wrote: > On 04/12/2010 06:03 PM, Nadav Horesh wrote: > >> Tried of install numy-1.4.1-rc2 on python-2.7b1 and got an error: >> >> (64 bit linux on core2, gcc4.4.3) >> >> >> compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/include -I/usr/local/include/python2.7 -c' >> gcc: _configtest.c >> _configtest.c:1: warning: conflicting types for built-in function ?exp? >> gcc -pthread _configtest.o -o _configtest >> _configtest.o: In function `main': >> /dev/shm/numpy-1.4.1rc2/_configtest.c:6: undefined reference to `exp' >> collect2: ld returned 1 exit status >> _configtest.o: In function `main': >> /dev/shm/numpy-1.4.1rc2/_configtest.c:6: undefined reference to `exp' >> collect2: ld returned 1 exit status >> Traceback (most recent call last): >> File "setup.py", line 187, in >> setup_package() >> File "setup.py", line 180, in setup_package >> configuration=configuration ) >> File "/dev/shm/numpy-1.4.1rc2/numpy/distutils/core.py", line 186, in setup >> return old_setup(**new_attr) >> File "/usr/local/lib/python2.7/distutils/core.py", line 152, in setup >> dist.run_commands() >> File "/usr/local/lib/python2.7/distutils/dist.py", line 953, in run_commands >> self.run_command(cmd) >> File "/usr/local/lib/python2.7/distutils/dist.py", line 972, in run_command >> cmd_obj.run() >> File "/dev/shm/numpy-1.4.1rc2/numpy/distutils/command/build.py", line 37, in run >> old_build.run(self) >> File "/usr/local/lib/python2.7/distutils/command/build.py", line 127, in run >> self.run_command(cmd_name) >> File "/usr/local/lib/python2.7/distutils/cmd.py", line 326, in run_command >> self.distribution.run_command(command) >> File "/usr/local/lib/python2.7/distutils/dist.py", line 972, in run_command >> cmd_obj.run() >> File "/dev/shm/numpy-1.4.1rc2/numpy/distutils/command/build_src.py", line 152, in run >> self.build_sources() >> File "/dev/shm/numpy-1.4.1rc2/numpy/distutils/command/build_src.py", line 163, in build_sources >> self.build_library_sources(*libname_info) >> File "/dev/shm/numpy-1.4.1rc2/numpy/distutils/command/build_src.py", line 298, in build_library_sources >> sources = self.generate_sources(sources, (lib_name, build_info)) >> File "/dev/shm/numpy-1.4.1rc2/numpy/distutils/command/build_src.py", line 385, in generate_sources >> source = func(extension, build_dir) >> File "numpy/core/setup.py", line 658, in get_mathlib_info >> mlibs = check_mathlib(config_cmd) >> File "numpy/core/setup.py", line 328, in check_mathlib >> if config_cmd.check_func("exp", libraries=libs, decl=True, call=True): >> File "/dev/shm/numpy-1.4.1rc2/numpy/distutils/command/config.py", line 310, in check_func >> libraries, library_dirs) >> File "/usr/local/lib/python2.7/distutils/command/config.py", line 251, in try_link >> libraries, library_dirs, lang) >> > Looks like another distutils regression in 2.7 to me - try_link should > never cause an error if it fails linking, that's the whole point of the > function. > > cheers, > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > Hi, I think this is due to the error: 'NameError: global name 'DistutilsExecError' is not defined' from: File "/home/bsouthey/python/numpy-1.4.1rc2/numpy/distutils/ccompiler.py", line 64, in CCompiler_spawn raise DistutilsExecError,\ This is a bug in numpy that was fixed in SVN - scipy.org is slow (or down) so I can not find the actual ticket. The following lines *must* be added to the numpy 1.4.1 version of "numpy/distutils/ccompiler.py": from distutils.errors import DistutilsExecError, DistutilsModuleError, \ DistutilsPlatformError Bruce From ncreati at inogs.it Mon Apr 12 10:21:45 2010 From: ncreati at inogs.it (Nicola Creati) Date: Mon, 12 Apr 2010 16:21:45 +0200 Subject: [Numpy-discussion] Sum over array elements Message-ID: <4BC32C79.4080304@inogs.it> Hello, I want to calculate, given a one dimension array, the sum over every two elements of the array. I found this working solution: a = N.arange(10) b = a.reshape(a, (5, 2)) c = b.sum(axis=1) Is there any better solution? Thanks, Nicola Creati From gokhansever at gmail.com Mon Apr 12 10:26:19 2010 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=) Date: Mon, 12 Apr 2010 09:26:19 -0500 Subject: [Numpy-discussion] Sum over array elements In-Reply-To: <4BC32C79.4080304@inogs.it> References: <4BC32C79.4080304@inogs.it> Message-ID: On Mon, Apr 12, 2010 at 9:21 AM, Nicola Creati wrote: > Hello, > I want to calculate, given a one dimension array, the sum over every two > elements of the array. > I found this working solution: > > a = N.arange(10) > b = a.reshape(a, (5, 2)) > c = b.sum(axis=1) > > Is there any better solution? > > Thanks, > > Nicola Creati > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > Sum odds and evens: I[3]: d = a[1::2] + a[::2] I[4]: d O[4]: array([ 1, 5, 9, 13, 17]) -- G?khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From patrickmarshwx at gmail.com Mon Apr 12 10:29:51 2010 From: patrickmarshwx at gmail.com (Patrick Marsh) Date: Mon, 12 Apr 2010 09:29:51 -0500 Subject: [Numpy-discussion] Sum over array elements In-Reply-To: <4BC32C79.4080304@inogs.it> References: <4BC32C79.4080304@inogs.it> Message-ID: a = np.arange(10) b = (a[::2]).sum() Patrick On Mon, Apr 12, 2010 at 9:21 AM, Nicola Creati wrote: > Hello, > I want to calculate, given a one dimension array, the sum over every two > elements of the array. > I found this working solution: > > a = N.arange(10) > b = a.reshape(a, (5, 2)) > c = b.sum(axis=1) > > Is there any better solution? > > Thanks, > > Nicola Creati > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Patrick Marsh Ph.D. Student / NSSL Liaison to the HWT School of Meteorology / University of Oklahoma Cooperative Institute for Mesoscale Meteorological Studies National Severe Storms Laboratory http://www.patricktmarsh.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Mon Apr 12 10:47:30 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Mon, 12 Apr 2010 22:47:30 +0800 Subject: [Numpy-discussion] rc2 for NumPy 1.4.1 and Scipy 0.7.2 In-Reply-To: <4BC325C1.4010003@gmail.com> References: <710F2847B0018641891D9A21602763605AD3A9@ex3.envision.co.il> <4BC2E722.3070805@silveregg.co.jp> <4BC325C1.4010003@gmail.com> Message-ID: On Mon, Apr 12, 2010 at 9:53 PM, Bruce Southey wrote: > > Hi, > I think this is due to the error: 'NameError: global name > 'DistutilsExecError' is not defined' > from: > File > "/home/bsouthey/python/numpy-1.4.1rc2/numpy/distutils/ccompiler.py", > line 64, in CCompiler_spawn > raise DistutilsExecError,\ > > This is a bug in numpy that was fixed in SVN - scipy.org is slow (or > down) so I can not find the actual ticket. > > http://projects.scipy.org/numpy/ticket/1355 > The following lines *must* be added to the numpy 1.4.1 version of > "numpy/distutils/ccompiler.py": > from distutils.errors import DistutilsExecError, DistutilsModuleError, \ > DistutilsPlatformError > Sure. Do you know if this is the only 2.7 related fix? Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From bsouthey at gmail.com Mon Apr 12 10:58:39 2010 From: bsouthey at gmail.com (Bruce Southey) Date: Mon, 12 Apr 2010 09:58:39 -0500 Subject: [Numpy-discussion] rc2 for NumPy 1.4.1 and Scipy 0.7.2 In-Reply-To: References: <710F2847B0018641891D9A21602763605AD3A9@ex3.envision.co.il> <4BC2E722.3070805@silveregg.co.jp> <4BC325C1.4010003@gmail.com> Message-ID: <4BC3351F.5080006@gmail.com> On 04/12/2010 09:47 AM, Ralf Gommers wrote: > > > On Mon, Apr 12, 2010 at 9:53 PM, Bruce Southey > wrote: > > > Hi, > I think this is due to the error: 'NameError: global name > 'DistutilsExecError' is not defined' > from: > File > "/home/bsouthey/python/numpy-1.4.1rc2/numpy/distutils/ccompiler.py", > line 64, in CCompiler_spawn > raise DistutilsExecError,\ > > This is a bug in numpy that was fixed in SVN - scipy.org > is slow (or > down) so I can not find the actual ticket. > > http://projects.scipy.org/numpy/ticket/1355 > > The following lines *must* be added to the numpy 1.4.1 version of > "numpy/distutils/ccompiler.py": > from distutils.errors import DistutilsExecError, > DistutilsModuleError, \ > DistutilsPlatformError > > > Sure. Do you know if this is the only 2.7 related fix? > > Ralf > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > The only other one that I recall is 1409: http://projects.scipy.org/numpy/ticket/1409 However, Python2.7b1 doesn't appear to trigger it unlike the alpha series. As Robert indicates, '_new_init_posix' is no longer used by numpy so it can be removed. Bruce -------------- next part -------------- An HTML attachment was scrubbed... URL: From ogdude at googlemail.com Mon Apr 12 11:50:19 2010 From: ogdude at googlemail.com (David Koch) Date: Mon, 12 Apr 2010 17:50:19 +0200 Subject: [Numpy-discussion] sparse.hstack with empty sparse matrices Message-ID: Hello, For scipy.sparse.csc_matrices, is there an equivalent for the following dense array operation? Example: tempA = scipy.empty((10,0)) tempB = scipy.random.random((10,1)) scipy.hstack((tempA, tempB)) i.e - you concatenate an empty n x 0 sparse matrix with a n x 1 sparse matrix to yield a n x 1 output I need to do this so I can iteratively build a matrix by adding new columns. The problem is that sparse matrix constructors don't seem expect "0" as input for a dimension. Thank you, /David From itsmilesdavis at gmail.com Mon Apr 12 11:59:46 2010 From: itsmilesdavis at gmail.com (John Jack) Date: Mon, 12 Apr 2010 11:59:46 -0400 Subject: [Numpy-discussion] Best dtype for Boolean values Message-ID: Hello all. I am (relatively) new to python, and 100% new to numpy. I need a way to store arrays of booleans and compare the arrays for equality. I assume I want arrays of dtype Boolean, and I should compare the arrays with array_equal >>> tmp.all_states array([False, True, False], dtype=bool) >>> tmp1.all_states array([False, False, False], dtype=bool) >>> tmp1.all_states[1]=True >>> tmp1.all_states array([False, True, False], dtype=bool) >>> array_equal(tmp.all_states,tmp1.all_states) True >>> any(tmp.all_states) True Would this be (a) the cheapest way (w.r.t. memory) to store Booleans and (b) the most efficient way to compare two lists of Booleans? Thanks for your advice. -JJ. -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Mon Apr 12 12:49:48 2010 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 12 Apr 2010 11:49:48 -0500 Subject: [Numpy-discussion] Name of the file associated with a memmap In-Reply-To: <710F2847B0018641891D9A21602763605AD3A8@ex3.envision.co.il> References: <710F2847B0018641891D9A21602763605AD3A8@ex3.envision.co.il> Message-ID: On Mon, Apr 12, 2010 at 04:03, Nadav Horesh wrote: > > Is there a way to get the file-name given a memmap array object? Not at this time. This would be very useful, though, so patches are welcome. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Mon Apr 12 12:53:10 2010 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 12 Apr 2010 11:53:10 -0500 Subject: [Numpy-discussion] Best dtype for Boolean values In-Reply-To: References: Message-ID: On Mon, Apr 12, 2010 at 10:59, John Jack wrote: > Hello all. > I am (relatively) new to python, and 100% new to numpy. > I need a way to store arrays of booleans and compare the arrays for > equality. > I assume I want arrays of dtype Boolean, and I should compare the arrays > with array_equal >>>> tmp.all_states > array([False, ?True, False], dtype=bool) >>>> tmp1.all_states > array([False, False, False], dtype=bool) >>>> tmp1.all_states[1]=True >>>> tmp1.all_states > array([False, ?True, False], dtype=bool) >>>> array_equal(tmp.all_states,tmp1.all_states) > True >>>> any(tmp.all_states) > True > Would this be (a) the cheapest way (w.r.t. memory) to store Booleans and Yes. > (b) > the most efficient way to compare two lists of Booleans? Yes. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From bpederse at gmail.com Mon Apr 12 15:55:43 2010 From: bpederse at gmail.com (Brent Pedersen) Date: Mon, 12 Apr 2010 12:55:43 -0700 Subject: [Numpy-discussion] Name of the file associated with a memmap In-Reply-To: References: <710F2847B0018641891D9A21602763605AD3A8@ex3.envision.co.il> Message-ID: On Mon, Apr 12, 2010 at 9:49 AM, Robert Kern wrote: > On Mon, Apr 12, 2010 at 04:03, Nadav Horesh wrote: >> >> Is there a way to get the file-name given a memmap array object? > > Not at this time. This would be very useful, though, so patches are welcome. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > ?-- Umberto Eco > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > this sounded easy, so i gave it a shot: http://projects.scipy.org/numpy/ticket/1451 i think that works. -b From charlesr.harris at gmail.com Mon Apr 12 16:37:36 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 12 Apr 2010 14:37:36 -0600 Subject: [Numpy-discussion] Name of the file associated with a memmap In-Reply-To: References: <710F2847B0018641891D9A21602763605AD3A8@ex3.envision.co.il> Message-ID: On Mon, Apr 12, 2010 at 1:55 PM, Brent Pedersen wrote: > On Mon, Apr 12, 2010 at 9:49 AM, Robert Kern > wrote: > > On Mon, Apr 12, 2010 at 04:03, Nadav Horesh > wrote: > >> > >> Is there a way to get the file-name given a memmap array object? > > > > Not at this time. This would be very useful, though, so patches are > welcome. > > > > -- > > Robert Kern > > > > "I have come to believe that the whole world is an enigma, a harmless > > enigma that is made terrible by our own mad attempt to interpret it as > > though it had an underlying truth." > > -- Umberto Eco > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > this sounded easy, so i gave it a shot: > http://projects.scipy.org/numpy/ticket/1451 > > i think that works. > Looks OK, I applied it. Could you add some documentation? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Apr 12 16:52:55 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 12 Apr 2010 14:52:55 -0600 Subject: [Numpy-discussion] Name of the file associated with a memmap In-Reply-To: References: <710F2847B0018641891D9A21602763605AD3A8@ex3.envision.co.il> Message-ID: On Mon, Apr 12, 2010 at 2:37 PM, Charles R Harris wrote: > > > On Mon, Apr 12, 2010 at 1:55 PM, Brent Pedersen wrote: > >> On Mon, Apr 12, 2010 at 9:49 AM, Robert Kern >> wrote: >> > On Mon, Apr 12, 2010 at 04:03, Nadav Horesh >> wrote: >> >> >> >> Is there a way to get the file-name given a memmap array object? >> > >> > Not at this time. This would be very useful, though, so patches are >> welcome. >> > >> > -- >> > Robert Kern >> > >> > "I have come to believe that the whole world is an enigma, a harmless >> > enigma that is made terrible by our own mad attempt to interpret it as >> > though it had an underlying truth." >> > -- Umberto Eco >> > _______________________________________________ >> > NumPy-Discussion mailing list >> > NumPy-Discussion at scipy.org >> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > >> >> this sounded easy, so i gave it a shot: >> http://projects.scipy.org/numpy/ticket/1451 >> >> i think that works. >> > > Looks OK, I applied it. Could you add some documentation? > > And maybe the filename should be the whole path? Thoughts? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From peridot.faceted at gmail.com Mon Apr 12 16:52:55 2010 From: peridot.faceted at gmail.com (Anne Archibald) Date: Mon, 12 Apr 2010 16:52:55 -0400 Subject: [Numpy-discussion] Best dtype for Boolean values In-Reply-To: References: Message-ID: On 12 April 2010 11:59, John Jack wrote: > Hello all. > I am (relatively) new to python, and 100% new to numpy. > I need a way to store arrays of booleans and compare the arrays for > equality. > I assume I want arrays of dtype Boolean, and I should compare the arrays > with array_equal >>>> tmp.all_states > array([False, True, False], dtype=bool) >>>> tmp1.all_states > array([False, False, False], dtype=bool) >>>> tmp1.all_states[1]=True >>>> tmp1.all_states > array([False, True, False], dtype=bool) >>>> array_equal(tmp.all_states,tmp1.all_states) > True >>>> any(tmp.all_states) > True > Would this be (a) the cheapest way (w.r.t. memory) to store Booleans and (b) > the most efficient way to compare two lists of Booleans? The short answer is yes and yes. The longer answer is, that uses one byte per Boolean, which is a tradeoff. In some sense, modern machines are happier working with 32- or 64-bit quantities, so loading a one-byte Boolean requires a small amount of byte-shuffling. On the other hand, if you're really short of memory, 8 bits for a Boolean is really wasteful. In fact, since modern machines are almost always limited by memory bandwidth, a packed Boolean data structure would probably be much faster for almost all operations in spite of the bit-fiddling required. But such a representation is incompatible with the whole infrastructure of numpy and so would require a great deal of messy code to support. So yes, it's the best representation of Booleans available, unless you're dealing with mind-bogglingly large arrays of them, in which case some sort of packed-Boolean representation would be better. This can even be partially supported by numpy, using uint8s, bitwise operations, and manually-specified bitmasks. There are probably not many applications for which this is worth the pain. Anne P.S. There's actually at least one python package for bit vectors, outside numpy; I can't speak for how good it is, though. -A > Thanks for your advice. > -JJ. > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From robert.kern at gmail.com Mon Apr 12 16:59:25 2010 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 12 Apr 2010 15:59:25 -0500 Subject: [Numpy-discussion] Name of the file associated with a memmap In-Reply-To: References: <710F2847B0018641891D9A21602763605AD3A8@ex3.envision.co.il> Message-ID: On Mon, Apr 12, 2010 at 15:52, Charles R Harris wrote: > > On Mon, Apr 12, 2010 at 2:37 PM, Charles R Harris > wrote: >> >> On Mon, Apr 12, 2010 at 1:55 PM, Brent Pedersen >> wrote: >>> >>> On Mon, Apr 12, 2010 at 9:49 AM, Robert Kern >>> wrote: >>> > On Mon, Apr 12, 2010 at 04:03, Nadav Horesh >>> > wrote: >>> >> >>> >> Is there a way to get the file-name given a memmap array object? >>> > >>> > Not at this time. This would be very useful, though, so patches are >>> > welcome. >>> > >>> > -- >>> > Robert Kern >>> > >>> > "I have come to believe that the whole world is an enigma, a harmless >>> > enigma that is made terrible by our own mad attempt to interpret it as >>> > though it had an underlying truth." >>> > ?-- Umberto Eco >>> > _______________________________________________ >>> > NumPy-Discussion mailing list >>> > NumPy-Discussion at scipy.org >>> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> > >>> >>> this sounded easy, so i gave it a shot: >>> http://projects.scipy.org/numpy/ticket/1451 >>> >>> i think that works. >> >> Looks OK, I applied it. Could you add some documentation? >> > > And maybe the filename should be the whole path? Thoughts? Yes, that would help. While you are looking at it, you may want to consider recording some of the other information that is computed in or provided to __new__, like offset. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From bpederse at gmail.com Mon Apr 12 17:14:47 2010 From: bpederse at gmail.com (Brent Pedersen) Date: Mon, 12 Apr 2010 14:14:47 -0700 Subject: [Numpy-discussion] Name of the file associated with a memmap In-Reply-To: References: <710F2847B0018641891D9A21602763605AD3A8@ex3.envision.co.il> Message-ID: On Mon, Apr 12, 2010 at 1:59 PM, Robert Kern wrote: > On Mon, Apr 12, 2010 at 15:52, Charles R Harris > wrote: >> >> On Mon, Apr 12, 2010 at 2:37 PM, Charles R Harris >> wrote: >>> >>> On Mon, Apr 12, 2010 at 1:55 PM, Brent Pedersen >>> wrote: >>>> >>>> On Mon, Apr 12, 2010 at 9:49 AM, Robert Kern >>>> wrote: >>>> > On Mon, Apr 12, 2010 at 04:03, Nadav Horesh >>>> > wrote: >>>> >> >>>> >> Is there a way to get the file-name given a memmap array object? >>>> > >>>> > Not at this time. This would be very useful, though, so patches are >>>> > welcome. >>>> > >>>> > -- >>>> > Robert Kern >>>> > >>>> > "I have come to believe that the whole world is an enigma, a harmless >>>> > enigma that is made terrible by our own mad attempt to interpret it as >>>> > though it had an underlying truth." >>>> > ?-- Umberto Eco >>>> > _______________________________________________ >>>> > NumPy-Discussion mailing list >>>> > NumPy-Discussion at scipy.org >>>> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>> > >>>> >>>> this sounded easy, so i gave it a shot: >>>> http://projects.scipy.org/numpy/ticket/1451 >>>> >>>> i think that works. >>> >>> Looks OK, I applied it. Could you add some documentation? >>> >> >> And maybe the filename should be the whole path? Thoughts? > > Yes, that would help. While you are looking at it, you may want to > consider recording some of the other information that is computed in > or provided to __new__, like offset. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > ?-- Umberto Eco > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > copying what i asked in the ticket: where should i write the docs? in the file itself or through the doc editor? also re path, since it can be a file-like, that would have to be something like: if isinstance(filename, basestring): filename = os.path.abspath(filename) self.filename = filename ok with that? From numpy-discussion at maubp.freeserve.co.uk Mon Apr 12 17:15:08 2010 From: numpy-discussion at maubp.freeserve.co.uk (Peter) Date: Mon, 12 Apr 2010 22:15:08 +0100 Subject: [Numpy-discussion] Name of the file associated with a memmap In-Reply-To: References: <710F2847B0018641891D9A21602763605AD3A8@ex3.envision.co.il> Message-ID: On Mon, Apr 12, 2010 at 9:52 PM, Charles R Harris wrote: > >> >> Looks OK, I applied it. Could you add some documentation? >> > > And maybe the filename should be the whole path? Thoughts? > > Chuck Why is the new property "filename" and not "name" to match the way the Python handle work? e.g. >>> handle = open("Documents/review.pdf") >>> handle.name 'Documents/review.pdf' I concede that "filename" is more explicit, but there is something to be said for consistency as well. Peter From robert.kern at gmail.com Mon Apr 12 17:22:28 2010 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 12 Apr 2010 16:22:28 -0500 Subject: [Numpy-discussion] Name of the file associated with a memmap In-Reply-To: References: <710F2847B0018641891D9A21602763605AD3A8@ex3.envision.co.il> Message-ID: On Mon, Apr 12, 2010 at 16:15, Peter wrote: > On Mon, Apr 12, 2010 at 9:52 PM, Charles R Harris > wrote: >> >>> >>> Looks OK, I applied it. Could you add some documentation? >>> >> >> And maybe the filename should be the whole path? Thoughts? >> >> Chuck > > Why is the new property "filename" and not "name" to match > the way the Python handle work? e.g. > >>>> handle = open("Documents/review.pdf") >>>> handle.name > 'Documents/review.pdf' > > I concede that "filename" is more explicit, but there is something > to be said for consistency as well. While you would expect the .name attribute of a file object to be a file name, the .name attribute an another type of object may be something different. memmap objects are quite different from file objects; we gain no particular consistency by using .name instead of .filename. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Mon Apr 12 17:30:05 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 12 Apr 2010 15:30:05 -0600 Subject: [Numpy-discussion] Name of the file associated with a memmap In-Reply-To: References: <710F2847B0018641891D9A21602763605AD3A8@ex3.envision.co.il> Message-ID: On Mon, Apr 12, 2010 at 3:14 PM, Brent Pedersen wrote: > On Mon, Apr 12, 2010 at 1:59 PM, Robert Kern > wrote: > > On Mon, Apr 12, 2010 at 15:52, Charles R Harris > > wrote: > >> > >> On Mon, Apr 12, 2010 at 2:37 PM, Charles R Harris > >> wrote: > >>> > >>> On Mon, Apr 12, 2010 at 1:55 PM, Brent Pedersen > >>> wrote: > >>>> > >>>> On Mon, Apr 12, 2010 at 9:49 AM, Robert Kern > >>>> wrote: > >>>> > On Mon, Apr 12, 2010 at 04:03, Nadav Horesh > > >>>> > wrote: > >>>> >> > >>>> >> Is there a way to get the file-name given a memmap array object? > >>>> > > >>>> > Not at this time. This would be very useful, though, so patches are > >>>> > welcome. > >>>> > > >>>> > -- > >>>> > Robert Kern > >>>> > > >>>> > "I have come to believe that the whole world is an enigma, a > harmless > >>>> > enigma that is made terrible by our own mad attempt to interpret it > as > >>>> > though it had an underlying truth." > >>>> > -- Umberto Eco > >>>> > _______________________________________________ > >>>> > NumPy-Discussion mailing list > >>>> > NumPy-Discussion at scipy.org > >>>> > http://mail.scipy.org/mailman/listinfo/numpy-discussion > >>>> > > >>>> > >>>> this sounded easy, so i gave it a shot: > >>>> http://projects.scipy.org/numpy/ticket/1451 > >>>> > >>>> i think that works. > >>> > >>> Looks OK, I applied it. Could you add some documentation? > >>> > >> > >> And maybe the filename should be the whole path? Thoughts? > > > > Yes, that would help. While you are looking at it, you may want to > > consider recording some of the other information that is computed in > > or provided to __new__, like offset. > > > > -- > > Robert Kern > > > > "I have come to believe that the whole world is an enigma, a harmless > > enigma that is made terrible by our own mad attempt to interpret it as > > though it had an underlying truth." > > -- Umberto Eco > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > copying what i asked in the ticket: > where should i write the docs? in the file itself or through the doc > editor? also re path, since it can be a file-like, that would have to > be something like: > > if isinstance(filename, basestring): > filename = os.path.abspath(filename) > self.filename = filename > > Robert probably has a better grasp of these issues than I, but it looks OK to me. As to the documentation, I not quite sure if we have an Attributes heading for classes, but I think so. I would just put it in svn because it's easy for me to do, but I'm not up to speed on the documentation work flow these days. Add a note as to the numpy version, i.e., .. versionadded:: 2.0.0 . Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From numpy-discussion at maubp.freeserve.co.uk Mon Apr 12 17:30:53 2010 From: numpy-discussion at maubp.freeserve.co.uk (Peter) Date: Mon, 12 Apr 2010 22:30:53 +0100 Subject: [Numpy-discussion] Name of the file associated with a memmap In-Reply-To: References: <710F2847B0018641891D9A21602763605AD3A8@ex3.envision.co.il> Message-ID: On Mon, Apr 12, 2010 at 10:22 PM, Robert Kern wrote: > >Peter wrote: >> >> Why is the new property "filename" and not "name" to match >> the way the Python handle work? e.g. >> >>>>> handle = open("Documents/review.pdf") >>>>> handle.name >> 'Documents/review.pdf' >> >> I concede that "filename" is more explicit, but there is something >> to be said for consistency as well. > > While you would expect the .name attribute of a file object to be a > file name, the .name attribute an another type of object may be > something different. Of course. > memmap objects are quite different from file > objects; we gain no particular consistency by using .name instead of > .filename. OK - I'm quite happy to accept that ruling, I just want to make sure someone asked the question. Thanks, Peter From robert.kern at gmail.com Mon Apr 12 17:39:23 2010 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 12 Apr 2010 16:39:23 -0500 Subject: [Numpy-discussion] Name of the file associated with a memmap In-Reply-To: References: <710F2847B0018641891D9A21602763605AD3A8@ex3.envision.co.il> Message-ID: On Mon, Apr 12, 2010 at 16:14, Brent Pedersen wrote: > On Mon, Apr 12, 2010 at 1:59 PM, Robert Kern wrote: >> On Mon, Apr 12, 2010 at 15:52, Charles R Harris >> wrote: >>> >>> On Mon, Apr 12, 2010 at 2:37 PM, Charles R Harris >>> wrote: >>>> >>>> On Mon, Apr 12, 2010 at 1:55 PM, Brent Pedersen >>>> wrote: >>>>> >>>>> On Mon, Apr 12, 2010 at 9:49 AM, Robert Kern >>>>> wrote: >>>>> > On Mon, Apr 12, 2010 at 04:03, Nadav Horesh >>>>> > wrote: >>>>> >> >>>>> >> Is there a way to get the file-name given a memmap array object? >>>>> > >>>>> > Not at this time. This would be very useful, though, so patches are >>>>> > welcome. >>>>> > >>>>> > -- >>>>> > Robert Kern >>>>> > >>>>> > "I have come to believe that the whole world is an enigma, a harmless >>>>> > enigma that is made terrible by our own mad attempt to interpret it as >>>>> > though it had an underlying truth." >>>>> > ?-- Umberto Eco >>>>> > _______________________________________________ >>>>> > NumPy-Discussion mailing list >>>>> > NumPy-Discussion at scipy.org >>>>> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >>>>> > >>>>> >>>>> this sounded easy, so i gave it a shot: >>>>> http://projects.scipy.org/numpy/ticket/1451 >>>>> >>>>> i think that works. >>>> >>>> Looks OK, I applied it. Could you add some documentation? >>>> >>> >>> And maybe the filename should be the whole path? Thoughts? >> >> Yes, that would help. While you are looking at it, you may want to >> consider recording some of the other information that is computed in >> or provided to __new__, like offset. >> >> -- >> Robert Kern >> >> "I have come to believe that the whole world is an enigma, a harmless >> enigma that is made terrible by our own mad attempt to interpret it as >> though it had an underlying truth." >> ?-- Umberto Eco >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > copying what i asked in the ticket: > where should i write the docs? in the file itself or through the doc > editor? also re path, since it can be a file-like, that would have to > be something like: > > if isinstance(filename, basestring): > ? ?filename = os.path.abspath(filename) > self.filename = filename > > ok with that? In the case of file object, we should grab the filename from it. Whether the filename argument to the constructor was a file name or a file object, self.filename should always be the file name, IMO. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From gael.varoquaux at normalesup.org Mon Apr 12 17:43:34 2010 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Mon, 12 Apr 2010 23:43:34 +0200 Subject: [Numpy-discussion] Name of the file associated with a memmap In-Reply-To: References: <710F2847B0018641891D9A21602763605AD3A8@ex3.envision.co.il> Message-ID: <20100412214334.GA27566@phare.normalesup.org> On Mon, Apr 12, 2010 at 04:39:23PM -0500, Robert Kern wrote: > > where should i write the docs? in the file itself or through the doc > > editor? also re path, since it can be a file-like, that would have to > > be something like: > > if isinstance(filename, basestring): > > ? ?filename = os.path.abspath(filename) > > self.filename = filename > > ok with that? > In the case of file object, we should grab the filename from it. > Whether the filename argument to the constructor was a file name or a > file object, self.filename should always be the file name, IMO. +1. Once this is in, would it make it possible/desirable to have memmapped arrays pickle (I know that it would require work, I am just asking). Ga?l From robert.kern at gmail.com Mon Apr 12 17:46:52 2010 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 12 Apr 2010 16:46:52 -0500 Subject: [Numpy-discussion] Name of the file associated with a memmap In-Reply-To: <20100412214334.GA27566@phare.normalesup.org> References: <710F2847B0018641891D9A21602763605AD3A8@ex3.envision.co.il> <20100412214334.GA27566@phare.normalesup.org> Message-ID: On Mon, Apr 12, 2010 at 16:43, Gael Varoquaux wrote: > On Mon, Apr 12, 2010 at 04:39:23PM -0500, Robert Kern wrote: >> > where should i write the docs? in the file itself or through the doc >> > editor? also re path, since it can be a file-like, that would have to >> > be something like: > >> > if isinstance(filename, basestring): >> > ? ?filename = os.path.abspath(filename) >> > self.filename = filename > >> > ok with that? > >> In the case of file object, we should grab the filename from it. >> Whether the filename argument to the constructor was a file name or a >> file object, self.filename should always be the file name, IMO. > > +1. > > Once this is in, would it make it possible/desirable to have memmapped > arrays pickle (I know that it would require work, I am just asking). You need some more information from the constructor arguments, but yes. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From bpederse at gmail.com Mon Apr 12 18:00:29 2010 From: bpederse at gmail.com (Brent Pedersen) Date: Mon, 12 Apr 2010 15:00:29 -0700 Subject: [Numpy-discussion] Name of the file associated with a memmap In-Reply-To: References: <710F2847B0018641891D9A21602763605AD3A8@ex3.envision.co.il> <20100412214334.GA27566@phare.normalesup.org> Message-ID: On Mon, Apr 12, 2010 at 2:46 PM, Robert Kern wrote: > On Mon, Apr 12, 2010 at 16:43, Gael Varoquaux > wrote: >> On Mon, Apr 12, 2010 at 04:39:23PM -0500, Robert Kern wrote: >>> > where should i write the docs? in the file itself or through the doc >>> > editor? also re path, since it can be a file-like, that would have to >>> > be something like: >> >>> > if isinstance(filename, basestring): >>> > ? ?filename = os.path.abspath(filename) >>> > self.filename = filename >> >>> > ok with that? >> >>> In the case of file object, we should grab the filename from it. >>> Whether the filename argument to the constructor was a file name or a >>> file object, self.filename should always be the file name, IMO. >> >> +1. >> >> Once this is in, would it make it possible/desirable to have memmapped >> arrays pickle (I know that it would require work, I am just asking). > > You need some more information from the constructor arguments, but yes. anything other than offset and mode? > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > ?-- Umberto Eco > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From robert.kern at gmail.com Mon Apr 12 18:08:53 2010 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 12 Apr 2010 17:08:53 -0500 Subject: [Numpy-discussion] Name of the file associated with a memmap In-Reply-To: References: <710F2847B0018641891D9A21602763605AD3A8@ex3.envision.co.il> <20100412214334.GA27566@phare.normalesup.org> Message-ID: On Mon, Apr 12, 2010 at 17:00, Brent Pedersen wrote: > On Mon, Apr 12, 2010 at 2:46 PM, Robert Kern wrote: >> On Mon, Apr 12, 2010 at 16:43, Gael Varoquaux >> wrote: >>> On Mon, Apr 12, 2010 at 04:39:23PM -0500, Robert Kern wrote: >>>> > where should i write the docs? in the file itself or through the doc >>>> > editor? also re path, since it can be a file-like, that would have to >>>> > be something like: >>> >>>> > if isinstance(filename, basestring): >>>> > ? ?filename = os.path.abspath(filename) >>>> > self.filename = filename >>> >>>> > ok with that? >>> >>>> In the case of file object, we should grab the filename from it. >>>> Whether the filename argument to the constructor was a file name or a >>>> file object, self.filename should always be the file name, IMO. >>> >>> +1. >>> >>> Once this is in, would it make it possible/desirable to have memmapped >>> arrays pickle (I know that it would require work, I am just asking). >> >> You need some more information from the constructor arguments, but yes. > > > anything other than offset and mode? I think that's about it. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From oliphant at enthought.com Mon Apr 12 18:19:11 2010 From: oliphant at enthought.com (Travis Oliphant) Date: Mon, 12 Apr 2010 17:19:11 -0500 Subject: [Numpy-discussion] Math Library In-Reply-To: References: Message-ID: <6DD1061A-60A3-4B84-99B0-793C0532321D@enthought.com> On Apr 11, 2010, at 4:17 PM, Sebastian Walter wrote: > > Ermm, the reply above is quite poor, sorry about that. > What I meant to say is the following: > > If there is going to be a discussion about creating a pure C numpy > library I'd like to join ;) Great. I would really like to get the discussion going. In an ideal world we can finish any kind of significant re-factoring in time for SciPy this year. It actually feels like the kind of thing that can motivate NumPy 2.0 a bit better. It sounds to me like nobody will be opposed as long as there is continuity to the project and current code still works without disruption (i.e. the current C-API for Python extensions is available). I am interested in re-factoring in such a way to create minimal impact on current NumPy C-API users, but improve maintainability going forward and the ability for other projects to use NumPy. -Travis -- Travis Oliphant Enthought Inc. 1-512-536-1057 http://www.enthought.com oliphant at enthought.com From oliphant at enthought.com Mon Apr 12 18:26:52 2010 From: oliphant at enthought.com (Travis Oliphant) Date: Mon, 12 Apr 2010 17:26:52 -0500 Subject: [Numpy-discussion] Proposal for new ufunc functionality In-Reply-To: References: <3BB52545-B5B9-4B9C-AD9A-0D69432A6B2D@enthought.com> <1270921512.11355.10.camel@talisman> Message-ID: <52AA8493-1BE5-46F9-9CDD-7081DF465978@enthought.com> On Apr 11, 2010, at 2:56 PM, Anne Archibald wrote: > 2010/4/10 St?fan van der Walt : >> On 10 April 2010 19:45, Pauli Virtanen wrote: >>> Another addition to ufuncs that should be though about is >>> specifying the >>> Python-side interface to generalized ufuncs. >> >> This is an interesting idea; what do you have in mind? > > I can see two different kinds of answer to this question: one is a > tool like vectorize/frompyfunc that allows construction of generalized > ufuncs from python functions, and the other is thinking out what > methods and support functions generalized ufuncs need. > > The former would be very handy for prototyping gufunc-based libraries > before delving into the templated C required to make them actually > efficient. > > The latter is more essential in the long run: it'd be nice to have a > reduce-like function, but obviously only when the arity and dimensions > work out right (which I think means (shape1,shape2)->(shape2) ). This > could be applied along an axis or over a whole array. reduceat and the > other, more sophisticated, schemes might also be worth supporting. At > a more elementary level, gufunc objects should have good introspection > - docstrings, shape specification accessible from python, named formal > arguments, et cetera. (So should ufuncs, for that matter.) We should collect all of these proposals into a NEP. To clarify what I mean by "group-by" behavior. Suppose I have an array of floats and an array of integers. Each element in the array of integers represents a region in the float array of a certain "kind". The reduction should take place over like- kind values: Example: add.reduceby(array=[1,2,3,4,5,6,7,8,9], by=[0,1,0,1,2,0,0,2,2]) results in the calculations: 1 + 3 + 6 + 7 2 + 4 5 + 8 + 9 and therefore the output (notice the two arrays --- perhaps a structured array should be returned instead...) [0,1,2], [17, 6, 22] The real value is when you have tabular data and you want to do reductions in one field based on values in another field. This happens all the time in relational algebra and would be a relatively straightforward thing to support in ufuncs. -Travis -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Apr 12 18:29:57 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 12 Apr 2010 16:29:57 -0600 Subject: [Numpy-discussion] Math Library In-Reply-To: <6DD1061A-60A3-4B84-99B0-793C0532321D@enthought.com> References: <6DD1061A-60A3-4B84-99B0-793C0532321D@enthought.com> Message-ID: On Mon, Apr 12, 2010 at 4:19 PM, Travis Oliphant wrote: > > On Apr 11, 2010, at 4:17 PM, Sebastian Walter wrote: > > > > > Ermm, the reply above is quite poor, sorry about that. > > What I meant to say is the following: > > > > If there is going to be a discussion about creating a pure C numpy > > library I'd like to join ;) > > Great. I would really like to get the discussion going. In an > ideal world we can finish any kind of significant re-factoring in time > for SciPy this year. It actually feels like the kind of thing that > can motivate NumPy 2.0 a bit better. > > It sounds to me like nobody will be opposed as long as there is > continuity to the project and current code still works without > disruption (i.e. the current C-API for Python extensions is available). > > I am interested in re-factoring in such a way to create minimal impact > on current NumPy C-API users, but improve maintainability going > forward and the ability for other projects to use NumPy. > > My own thoughts were to have a lowlevel 'loop' library that worked with strided memory, and an intermediate level above that for buffer objects. Numpy ufuncs would be a level above that and implement policy type things like casting, kinds, etc. Then there is the lowlevel c-library for the functions. I don't think we should aim at duplicating commonly available functions like sin and exp, but rather that subset that are sometimes unavailable. In particular, I would like to get away from having to use double versions of functions instead of type specific versions. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From bpederse at gmail.com Mon Apr 12 18:31:33 2010 From: bpederse at gmail.com (Brent Pedersen) Date: Mon, 12 Apr 2010 15:31:33 -0700 Subject: [Numpy-discussion] Name of the file associated with a memmap In-Reply-To: References: <710F2847B0018641891D9A21602763605AD3A8@ex3.envision.co.il> <20100412214334.GA27566@phare.normalesup.org> Message-ID: On Mon, Apr 12, 2010 at 3:08 PM, Robert Kern wrote: > On Mon, Apr 12, 2010 at 17:00, Brent Pedersen wrote: >> On Mon, Apr 12, 2010 at 2:46 PM, Robert Kern wrote: >>> On Mon, Apr 12, 2010 at 16:43, Gael Varoquaux >>> wrote: >>>> On Mon, Apr 12, 2010 at 04:39:23PM -0500, Robert Kern wrote: >>>>> > where should i write the docs? in the file itself or through the doc >>>>> > editor? also re path, since it can be a file-like, that would have to >>>>> > be something like: >>>> >>>>> > if isinstance(filename, basestring): >>>>> > ? ?filename = os.path.abspath(filename) >>>>> > self.filename = filename >>>> >>>>> > ok with that? >>>> >>>>> In the case of file object, we should grab the filename from it. >>>>> Whether the filename argument to the constructor was a file name or a >>>>> file object, self.filename should always be the file name, IMO. >>>> >>>> +1. >>>> >>>> Once this is in, would it make it possible/desirable to have memmapped >>>> arrays pickle (I know that it would require work, I am just asking). >>> >>> You need some more information from the constructor arguments, but yes. >> >> >> anything other than offset and mode? > > I think that's about it. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > ?-- Umberto Eco > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > i added a new patch to the ticket and updated the docs here: http://docs.scipy.org/numpy/docs/numpy.core.memmap.memmap/ the preview is somehow rendering the sections in a different order than they appear in the RST, not sure what's going on there. -b From robert.kern at gmail.com Mon Apr 12 18:31:16 2010 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 12 Apr 2010 17:31:16 -0500 Subject: [Numpy-discussion] Proposal for new ufunc functionality In-Reply-To: <52AA8493-1BE5-46F9-9CDD-7081DF465978@enthought.com> References: <3BB52545-B5B9-4B9C-AD9A-0D69432A6B2D@enthought.com> <1270921512.11355.10.camel@talisman> <52AA8493-1BE5-46F9-9CDD-7081DF465978@enthought.com> Message-ID: On Mon, Apr 12, 2010 at 17:26, Travis Oliphant wrote: > > On Apr 11, 2010, at 2:56 PM, Anne Archibald wrote: > > 2010/4/10 St?fan van der Walt : > > On 10 April 2010 19:45, Pauli Virtanen wrote: > > Another addition to ufuncs that should be though about is specifying the > > Python-side interface to generalized ufuncs. > > This is an interesting idea; what do you have in mind? > > I can see two different kinds of answer to this question: one is a > tool like vectorize/frompyfunc that allows construction of generalized > ufuncs from python functions, and the other is thinking out what > methods and support functions generalized ufuncs need. > > The former would be very handy for prototyping gufunc-based libraries > before delving into the templated C required to make them actually > efficient. > > The latter is more essential in the long run: it'd be nice to have a > reduce-like function, but obviously only when the arity and dimensions > work out right (which I think means (shape1,shape2)->(shape2) ). This > could be applied along an axis or over a whole array. reduceat and the > other, more sophisticated, schemes might also be worth supporting. At > a more elementary level, gufunc objects should have good introspection > - docstrings, shape specification accessible from python, named formal > arguments, et cetera. (So should ufuncs, for that matter.) > > We should collect all of these proposals into a NEP. ? ? ?To clarify what I > mean by "group-by" behavior. > Suppose I have an array of floats and an array of integers. ? Each element > in the array of integers represents a region in the float array of a certain > "kind". ? The reduction should take place over like-kind values: > Example: > add.reduceby(array=[1,2,3,4,5,6,7,8,9], by=[0,1,0,1,2,0,0,2,2]) > results in the calculations: > 1 + 3 + 6 + 7 > 2 + 4 > 5 + 8 + 9 > and therefore the output (notice the two arrays --- perhaps a structured > array should be returned instead...) > [0,1,2], > [17, 6, 22] > > The real value is when you have tabular data and you want to do reductions > in one field based on values in another field. ? This happens all the time > in relational algebra and would be a relatively straightforward thing to > support in ufuncs. I might suggest a simplification where the by array must be an array of non-negative ints such that they are indices into the output. For example (note that I replace 2 with 3 and have no 2s in the by array): add.reduceby(array=[1,2,3,4,5,6,7,8,9], by=[0,1,0,1,3,0,0,3,3]) == [17, 6, 0, 22] This basically generalizes bincount() to other binary ufuncs. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From peridot.faceted at gmail.com Mon Apr 12 18:35:48 2010 From: peridot.faceted at gmail.com (Anne Archibald) Date: Mon, 12 Apr 2010 18:35:48 -0400 Subject: [Numpy-discussion] Proposal for new ufunc functionality In-Reply-To: <52AA8493-1BE5-46F9-9CDD-7081DF465978@enthought.com> References: <3BB52545-B5B9-4B9C-AD9A-0D69432A6B2D@enthought.com> <1270921512.11355.10.camel@talisman> <52AA8493-1BE5-46F9-9CDD-7081DF465978@enthought.com> Message-ID: On 12 April 2010 18:26, Travis Oliphant wrote: > We should collect all of these proposals into a NEP. Or several NEPs, since I think they are quasi-orthogonal. >?To clarify what I > mean by "group-by" behavior. > Suppose I have an array of floats and an array of integers. ? Each element > in the array of integers represents a region in the float array of a certain > "kind". ? The reduction should take place over like-kind values: > Example: > add.reduceby(array=[1,2,3,4,5,6,7,8,9], by=[0,1,0,1,2,0,0,2,2]) > results in the calculations: > 1 + 3 + 6 + 7 > 2 + 4 > 5 + 8 + 9 > and therefore the output (notice the two arrays --- perhaps a structured > array should be returned instead...) > [0,1,2], > [17, 6, 22] > > The real value is when you have tabular data and you want to do reductions > in one field based on values in another field. ? This happens all the time > in relational algebra and would be a relatively straightforward thing to > support in ufuncs. As an example, if I understand correctly, this would allow the "histogram" functions to be replaced by a one-liner, e.g.: add.reduceby(array=1, by=((A-min)*n/(max-min)).astype(int)) It would also be valuable to support output arguments of some sort, so that, for example, reduceby could be used to accumulate values into an output array at supplied indices. (I leave the value of using this on matrix multiplication or arctan2 to the reader's imagination.) Anne From charlesr.harris at gmail.com Mon Apr 12 18:52:04 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 12 Apr 2010 16:52:04 -0600 Subject: [Numpy-discussion] Duplicate functions Message-ID: Hi All, What do we do about duplicate functions? There are currently three log2's in numpy, one ufunc, one ufunclike, and one in emath. The emath version isn't automatically imported so I don't think we need to worry about it, but the ufunclike version is imported after the ufunc version and shadows it. As both of those functions look to behave the same I think we only need the one. I see two alternatives: remove the ufunclike version or move the ufunclike import * before the numpy.core import *. Thoughts? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at enthought.com Mon Apr 12 18:54:52 2010 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Mon, 12 Apr 2010 17:54:52 -0500 Subject: [Numpy-discussion] Proposal for new ufunc functionality In-Reply-To: References: <3BB52545-B5B9-4B9C-AD9A-0D69432A6B2D@enthought.com> <1270921512.11355.10.camel@talisman> <52AA8493-1BE5-46F9-9CDD-7081DF465978@enthought.com> Message-ID: <4BC3A4BC.6060906@enthought.com> Robert Kern wrote: > On Mon, Apr 12, 2010 at 17:26, Travis Oliphant wrote: > >> On Apr 11, 2010, at 2:56 PM, Anne Archibald wrote: >> >> 2010/4/10 St?fan van der Walt : >> >> On 10 April 2010 19:45, Pauli Virtanen wrote: >> >> Another addition to ufuncs that should be though about is specifying the >> >> Python-side interface to generalized ufuncs. >> >> This is an interesting idea; what do you have in mind? >> >> I can see two different kinds of answer to this question: one is a >> tool like vectorize/frompyfunc that allows construction of generalized >> ufuncs from python functions, and the other is thinking out what >> methods and support functions generalized ufuncs need. >> >> The former would be very handy for prototyping gufunc-based libraries >> before delving into the templated C required to make them actually >> efficient. >> >> The latter is more essential in the long run: it'd be nice to have a >> reduce-like function, but obviously only when the arity and dimensions >> work out right (which I think means (shape1,shape2)->(shape2) ). This >> could be applied along an axis or over a whole array. reduceat and the >> other, more sophisticated, schemes might also be worth supporting. At >> a more elementary level, gufunc objects should have good introspection >> - docstrings, shape specification accessible from python, named formal >> arguments, et cetera. (So should ufuncs, for that matter.) >> >> We should collect all of these proposals into a NEP. To clarify what I >> mean by "group-by" behavior. >> Suppose I have an array of floats and an array of integers. Each element >> in the array of integers represents a region in the float array of a certain >> "kind". The reduction should take place over like-kind values: >> Example: >> add.reduceby(array=[1,2,3,4,5,6,7,8,9], by=[0,1,0,1,2,0,0,2,2]) >> results in the calculations: >> 1 + 3 + 6 + 7 >> 2 + 4 >> 5 + 8 + 9 >> and therefore the output (notice the two arrays --- perhaps a structured >> array should be returned instead...) >> [0,1,2], >> [17, 6, 22] >> >> The real value is when you have tabular data and you want to do reductions >> in one field based on values in another field. This happens all the time >> in relational algebra and would be a relatively straightforward thing to >> support in ufuncs. >> > > I might suggest a simplification where the by array must be an array > of non-negative ints such that they are indices into the output. For > example (note that I replace 2 with 3 and have no 2s in the by array): > > add.reduceby(array=[1,2,3,4,5,6,7,8,9], by=[0,1,0,1,3,0,0,3,3]) == > [17, 6, 0, 22] > > This basically generalizes bincount() to other binary ufuncs. > > A bit more generalization of `by` gives behavior like matlab's accumarray (http://www.mathworks.com/access/helpdesk/help/techdoc/ref/accumarray.html), which I partly cloned here: [This would be a link to the scipy cookbook, but scipy.org is not responding.] Warren From robert.kern at gmail.com Mon Apr 12 19:09:50 2010 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 12 Apr 2010 18:09:50 -0500 Subject: [Numpy-discussion] Proposal for new ufunc functionality In-Reply-To: <4BC3A4BC.6060906@enthought.com> References: <3BB52545-B5B9-4B9C-AD9A-0D69432A6B2D@enthought.com> <1270921512.11355.10.camel@talisman> <52AA8493-1BE5-46F9-9CDD-7081DF465978@enthought.com> <4BC3A4BC.6060906@enthought.com> Message-ID: On Mon, Apr 12, 2010 at 17:54, Warren Weckesser wrote: > A bit more generalization of `by` gives behavior like matlab's accumarray > (http://www.mathworks.com/access/helpdesk/help/techdoc/ref/accumarray.html), > which I partly cloned here: > [This would be a link to the scipy cookbook, but scipy.org is not > responding.] I've bounced the server. Try again. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From bpederse at gmail.com Mon Apr 12 20:06:22 2010 From: bpederse at gmail.com (Brent Pedersen) Date: Mon, 12 Apr 2010 17:06:22 -0700 Subject: [Numpy-discussion] Name of the file associated with a memmap In-Reply-To: References: <710F2847B0018641891D9A21602763605AD3A8@ex3.envision.co.il> <20100412214334.GA27566@phare.normalesup.org> Message-ID: On Mon, Apr 12, 2010 at 3:31 PM, Brent Pedersen wrote: > On Mon, Apr 12, 2010 at 3:08 PM, Robert Kern wrote: >> On Mon, Apr 12, 2010 at 17:00, Brent Pedersen wrote: >>> On Mon, Apr 12, 2010 at 2:46 PM, Robert Kern wrote: >>>> On Mon, Apr 12, 2010 at 16:43, Gael Varoquaux >>>> wrote: >>>>> On Mon, Apr 12, 2010 at 04:39:23PM -0500, Robert Kern wrote: >>>>>> > where should i write the docs? in the file itself or through the doc >>>>>> > editor? also re path, since it can be a file-like, that would have to >>>>>> > be something like: >>>>> >>>>>> > if isinstance(filename, basestring): >>>>>> > ? ?filename = os.path.abspath(filename) >>>>>> > self.filename = filename >>>>> >>>>>> > ok with that? >>>>> >>>>>> In the case of file object, we should grab the filename from it. >>>>>> Whether the filename argument to the constructor was a file name or a >>>>>> file object, self.filename should always be the file name, IMO. >>>>> >>>>> +1. >>>>> >>>>> Once this is in, would it make it possible/desirable to have memmapped >>>>> arrays pickle (I know that it would require work, I am just asking). >>>> >>>> You need some more information from the constructor arguments, but yes. >>> >>> >>> anything other than offset and mode? >> >> I think that's about it. >> >> -- >> Robert Kern >> >> "I have come to believe that the whole world is an enigma, a harmless >> enigma that is made terrible by our own mad attempt to interpret it as >> though it had an underlying truth." >> ?-- Umberto Eco >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > i added a new patch to the ticket and updated the docs here: > http://docs.scipy.org/numpy/docs/numpy.core.memmap.memmap/ > the preview is somehow rendering the sections in a different order > than they appear in the RST, not sure what's going on there. > -b > Charles, thanks for committing, i just added another patch for just the tests which i forgot to include when i diffed last time. From amcmorl at gmail.com Mon Apr 12 22:41:14 2010 From: amcmorl at gmail.com (Angus McMorland) Date: Tue, 13 Apr 2010 03:41:14 +0100 Subject: [Numpy-discussion] indexing to sort with argsort(..., axis=1) Message-ID: Hi all, I want to sort a 2d array along one dimension, with the indices returned by argsort, but the subsequent indexing syntax to get the sorted array is not obvious. The following works, but I wonder if there is a simpler way: a = np.random.random(size=(5,3)) s = np.argsort(a, axis=1) sorted = a[:,s][np.eye(5,5, dtype=bool)] # it looks like this line could be simpler What's the correct, concise way to do this? Thanks, Angus. -- AJC McMorland Post-doctoral research fellow Neurobiology, University of Pittsburgh -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Mon Apr 12 22:50:07 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 12 Apr 2010 22:50:07 -0400 Subject: [Numpy-discussion] indexing to sort with argsort(..., axis=1) In-Reply-To: References: Message-ID: On Mon, Apr 12, 2010 at 10:41 PM, Angus McMorland wrote: > Hi all, > I want to sort a 2d array along one dimension, with the indices returned by > argsort, but the subsequent indexing syntax to get the sorted array is not > obvious. > The following works, but I wonder if there is a simpler way: > a = np.random.random(size=(5,3)) > s = np.argsort(a, axis=1) > sorted = a[:,s][np.eye(5,5, dtype=bool)] # it looks like this line could be Note: sorted is a python function >>> b = a[:,s][np.eye(5,5, dtype=bool)] >>> a[np.arange(a.shape[0])[:,None],s] == b array([[ True, True, True], [ True, True, True], [ True, True, True], [ True, True, True], [ True, True, True]], dtype=bool) Josef > simpler > What's the correct, concise way to do this? > Thanks, > Angus. > -- > AJC McMorland > Post-doctoral research fellow > Neurobiology, University of Pittsburgh > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From gokhansever at gmail.com Mon Apr 12 23:01:48 2010 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=) Date: Mon, 12 Apr 2010 22:01:48 -0500 Subject: [Numpy-discussion] indexing to sort with argsort(..., axis=1) In-Reply-To: References: Message-ID: On Mon, Apr 12, 2010 at 9:41 PM, Angus McMorland wrote: > Hi all, > > I want to sort a 2d array along one dimension, with the indices returned by > argsort, but the subsequent indexing syntax to get the sorted array is not > obvious. > > The following works, but I wonder if there is a simpler way: > > a = np.random.random(size=(5,3)) > s = np.argsort(a, axis=1) > sorted = a[:,s][np.eye(5,5, dtype=bool)] # it looks like this line could be > simpler > > What's the correct, concise way to do this? > Why not just: b = np.sort(a) What advantage does argsort provides in this case? -- G?khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From amcmorl at gmail.com Tue Apr 13 08:17:39 2010 From: amcmorl at gmail.com (Angus McMorland) Date: Tue, 13 Apr 2010 13:17:39 +0100 Subject: [Numpy-discussion] indexing to sort with argsort(..., axis=1) In-Reply-To: References: Message-ID: On 13 April 2010 04:01, G?khan Sever wrote: > > > On Mon, Apr 12, 2010 at 9:41 PM, Angus McMorland wrote: > >> Hi all, >> >> I want to sort a 2d array along one dimension, with the indices returned >> by argsort, but the subsequent indexing syntax to get the sorted array is >> not obvious. >> >> The following works, but I wonder if there is a simpler way: >> >> a = np.random.random(size=(5,3)) >> s = np.argsort(a, axis=1) >> sorted = a[:,s][np.eye(5,5, dtype=bool)] # it looks like this line could >> be simpler >> >> What's the correct, concise way to do this? >> > > Why not just: > > b = np.sort(a) > > What advantage does argsort provides in this case? > I want to be able to sort another array the same way; calculating b is really just a check that I was doing the sort correctly. Thanks Josef, for the reminder about using arange. I realise I've seen it before, but haven't got it intuitive in my head yet. A. -- AJC McMorland Post-doctoral research fellow Neurobiology, University of Pittsburgh -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Tue Apr 13 08:45:48 2010 From: cournape at gmail.com (David Cournapeau) Date: Tue, 13 Apr 2010 21:45:48 +0900 Subject: [Numpy-discussion] [ANN] toydist version 0.0.2 Message-ID: Hi, I am glad to announce the first release of toydist, the 0.0.2 release: http://github.com/cournape/toydist/downloads Toydist is a pythonic, no-nonsense packaging solution for python softwares. The main highlights of this first public release are: * Package description grammar formalized through PLY, the python Lex-yacc package (included in toydist for the time being) * Toymaker, the command line interface to toydist, can install itself Although still very much in infancy, I have succesfully used the conversion procedure on sphinx, jinja and a few python-only Enthought packages. How to help ? ------------- Toydist is experimental, and should not be used for anything production-worthy at this point. There are two main reasons why I have released such a preliminary version: * To get suggestions/comments on the current file format. I believe the current syntax strikes a good balance for human readers/writers, but it cannot be parsed with just regular expressions, which means syntax highlithing may be difficult (the difficulty is to be able to parse arbitary rest for some metadata). The easiest way to get a feeling of the format is to convert your favorite python package, assuming it is simple enough. * The API for package description is still regularly changed, and the command-line interface has no sane API yet, but I certainly welcome suggestions from people interested in writing debian or other builders. I have started documenting toydist, an up-to-date version can be found on http://cournape.github.com/toydist. Example of simple commands may be found in toydist/commands (sdist and install are particularly simple). The focus for 0.0.3 is hook support - the goal is to have an API robust enough so that future toydist code will be based on hooks, as well a basic support for waf or scons-based C builds. cheers, David From oliphant at enthought.com Tue Apr 13 10:03:51 2010 From: oliphant at enthought.com (Travis Oliphant) Date: Tue, 13 Apr 2010 09:03:51 -0500 Subject: [Numpy-discussion] Proposal for new ufunc functionality In-Reply-To: References: <3BB52545-B5B9-4B9C-AD9A-0D69432A6B2D@enthought.com> <1270921512.11355.10.camel@talisman> <52AA8493-1BE5-46F9-9CDD-7081DF465978@enthought.com> Message-ID: <7EAFB322-B48A-4D04-90CE-A3793E6C5CF3@enthought.com> On Apr 12, 2010, at 5:31 PM, Robert Kern wrote: >> >> We should collect all of these proposals into a NEP. To >> clarify what I >> mean by "group-by" behavior. >> Suppose I have an array of floats and an array of integers. Each >> element >> in the array of integers represents a region in the float array of >> a certain >> "kind". The reduction should take place over like-kind values: >> Example: >> add.reduceby(array=[1,2,3,4,5,6,7,8,9], by=[0,1,0,1,2,0,0,2,2]) >> results in the calculations: >> 1 + 3 + 6 + 7 >> 2 + 4 >> 5 + 8 + 9 >> and therefore the output (notice the two arrays --- perhaps a >> structured >> array should be returned instead...) >> [0,1,2], >> [17, 6, 22] >> >> The real value is when you have tabular data and you want to do >> reductions >> in one field based on values in another field. This happens all >> the time >> in relational algebra and would be a relatively straightforward >> thing to >> support in ufuncs. > > I might suggest a simplification where the by array must be an array > of non-negative ints such that they are indices into the output. For > example (note that I replace 2 with 3 and have no 2s in the by array): > > add.reduceby(array=[1,2,3,4,5,6,7,8,9], by=[0,1,0,1,3,0,0,3,3]) == > [17, 6, 0, 22] > > This basically generalizes bincount() to other binary ufuncs. > Interesting proposal. I do like the having only one output. I'm particularly interested in reductions with "by" arrays of strings. i.e. something like: add.reduceby([10,11,12,13,14,15,16], by=['red','green','red','green','red','blue', 'blue']). resulting in: 10+12+14 11+13 15+16 In practice, these would have to be essentially mapped to the kind of integer array I used in the original example, and so I suppose if we couple your proposal with the segment function from the rest of my original proposal, then the same resulting functionality is available (with perhaps the extra intermediate integer array that may not be strictly necessary). But, having simple building blocks is usually better in the long run (and typically leads to better optimizations by human programmers). Thanks, -Travis -- Travis Oliphant Enthought Inc. 1-512-536-1057 http://www.enthought.com oliphant at enthought.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From oliphant at enthought.com Tue Apr 13 10:13:10 2010 From: oliphant at enthought.com (Travis Oliphant) Date: Tue, 13 Apr 2010 09:13:10 -0500 Subject: [Numpy-discussion] Proposal for new ufunc functionality In-Reply-To: <4BC3A4BC.6060906@enthought.com> References: <3BB52545-B5B9-4B9C-AD9A-0D69432A6B2D@enthought.com> <1270921512.11355.10.camel@talisman> <52AA8493-1BE5-46F9-9CDD-7081DF465978@enthought.com> <4BC3A4BC.6060906@enthought.com> Message-ID: <92C8D156-3582-4549-8B8A-0954C223431C@enthought.com> On Apr 12, 2010, at 5:54 PM, Warren Weckesser wrote: > > A bit more generalization of `by` gives behavior like matlab's > accumarray > (http://www.mathworks.com/access/helpdesk/help/techdoc/ref/accumarray.html > ), > which I partly cloned here: > [This would be a link to the scipy cookbook, but scipy.org is not > responding.] Reading the accumarray docstring, it does seem related, but they use the "subs" array as an index into the original array (instead of an index into the "output" array like Robert's simplification). I do like the Matlab functionality, but would propose a different reduction function: reduceover to implement it. It also feels like we should figure out different kinds of reductions for generalized ufuncs as well. If anyone has a primer on generalized ufuncs, I would love to see it. Isn't a reduction on a generalized ufunc just another generalized ufunc? Perhaps we could automatically create these "reduced generalized ufuncs" I would love to explore just how general these generalized ufuncs are and what can be subsumed by them. -Travis -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Tue Apr 13 10:17:00 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 13 Apr 2010 10:17:00 -0400 Subject: [Numpy-discussion] Proposal for new ufunc functionality In-Reply-To: <7EAFB322-B48A-4D04-90CE-A3793E6C5CF3@enthought.com> References: <3BB52545-B5B9-4B9C-AD9A-0D69432A6B2D@enthought.com> <1270921512.11355.10.camel@talisman> <52AA8493-1BE5-46F9-9CDD-7081DF465978@enthought.com> <7EAFB322-B48A-4D04-90CE-A3793E6C5CF3@enthought.com> Message-ID: On Tue, Apr 13, 2010 at 10:03 AM, Travis Oliphant wrote: > > On Apr 12, 2010, at 5:31 PM, Robert Kern wrote: > > We should collect all of these proposals into a NEP. ? ? ?To clarify what I > > mean by "group-by" behavior. > > Suppose I have an array of floats and an array of integers. ? Each element > > in the array of integers represents a region in the float array of a certain > > "kind". ? The reduction should take place over like-kind values: > > Example: > > add.reduceby(array=[1,2,3,4,5,6,7,8,9], by=[0,1,0,1,2,0,0,2,2]) > > results in the calculations: > > 1 + 3 + 6 + 7 > > 2 + 4 > > 5 + 8 + 9 > > and therefore the output (notice the two arrays --- perhaps a structured > > array should be returned instead...) > > [0,1,2], > > [17, 6, 22] > > The real value is when you have tabular data and you want to do reductions > > in one field based on values in another field. ? This happens all the time > > in relational algebra and would be a relatively straightforward thing to > > support in ufuncs. > > I might suggest a simplification where the by array must be an array > of non-negative ints such that they are indices into the output. For > example (note that I replace 2 with 3 and have no 2s in the by array): > > add.reduceby(array=[1,2,3,4,5,6,7,8,9], by=[0,1,0,1,3,0,0,3,3]) == > [17, 6, 0, 22] > > This basically generalizes bincount() to other binary ufuncs. > > > Interesting proposal. ? I do like the having only one output. > I'm particularly interested in reductions with "by" arrays of strings. ?i.e. > ?something like: > add.reduceby([10,11,12,13,14,15,16], > by=['red','green','red','green','red','blue', 'blue']). > resulting in: > 10+12+14 > 11+13 > 15+16 > In practice, these would have to be essentially mapped to the kind of > integer array I used in the original example, and so?I suppose if we couple > your proposal with the segment function from the rest of my original > proposal, then the same resulting functionality is available (with perhaps > the extra intermediate integer array that may not be strictly necessary). > But, having simple building blocks is usually better in the long run (and > typically leads to better optimizations by human programmers). Currently I'm using unique return_inverse to do the recoding into integers >>> np.unique(['red','green','red','green','red','blue', 'blue'],return_inverse=True) (array(['blue', 'green', 'red'], dtype='|S5'), array([2, 1, 2, 1, 2, 0, 0])) and then feed into bincount. Your plans are a good generalization and speedup. Josef > Thanks, > -Travis > > -- > Travis Oliphant > Enthought Inc. > 1-512-536-1057 > http://www.enthought.com > oliphant at enthought.com > > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From amenity at enthought.com Tue Apr 13 16:30:46 2010 From: amenity at enthought.com (Amenity Applewhite) Date: Tue, 13 Apr 2010 15:30:46 -0500 Subject: [Numpy-discussion] SciPy 2010 News: Specialized track deadline extended Message-ID: <6AB8EBE6-52E1-4EC3-A9FE-76446829ABB0@enthought.com> Have you been meaning to prepare an abstract to submit for a SciPy 2010 specialized track (http://conference.scipy.org/scipy2010/papers.html#tracks )? Didn't find the time? Well you're in luck. This weekend, we had technical issues with the email submissions for the specialized tracks. In light of the inconvenience, we've decided to extend the deadline an additional two weeks until Sunday, April 25th. If you have an abstract ready for one of the four specialized tracks, please use the links below to submit it to the program chair. If you previously submitted one and didn't receive confirmation that we received it, it would be a great idea to submit it again to ensure we get it. * Biomedical/bioinformatics chaired by Glen Otero, Dell submit/contact: 2010bioinformatics at scipy.org * Financial analysis chaired by Wes McKinney, AQR Capital Management submit/contact: 2010finance at scipy.org * Geophysics chaired by Alan Jackson, Shell submit/contact: 2010geophysics at scipy.org * Parallel processing & cloud computing co-chaired by Ken Elkabany, PiCloud & Brian Granger, CalPoly submit/contact: 2010parallel at scipy.org Main Conference Submissions Submissions for the main SciPy 2010 conference closed Sunday. Thanks to everyone who submitted. We'll announce the accepted talks Tuesday April 20th. Student Sponsorships If you're an academic and contribute to SciPy or related projects, make sure to apply for one of our student sponsorships. The deadline to apply is April 18th. We are also accepting nominations. http://conference.scipy.org/scipy2010/student.html Don't forget to register... Registrations are coming in pretty steadily now. Remember that to get early registration prices you need to [8]register before May 10th! https://conference.scipy.org/scipy2010/registration.html The SciPy 2010 Team @SciPy2010 on Twitter -- Amenity Applewhite Enthought, Inc. Scientific Computing Solutions www.enthought.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Tue Apr 13 16:34:38 2010 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 13 Apr 2010 15:34:38 -0500 Subject: [Numpy-discussion] Proposal for new ufunc functionality In-Reply-To: References: <3BB52545-B5B9-4B9C-AD9A-0D69432A6B2D@enthought.com> <1270921512.11355.10.camel@talisman> Message-ID: On Sat, Apr 10, 2010 at 17:59, Robert Kern wrote: > On Sat, Apr 10, 2010 at 12:45, Pauli Virtanen wrote: >> la, 2010-04-10 kello 12:23 -0500, Travis Oliphant kirjoitti: >> [clip] >>> Here are my suggested additions to NumPy: >>> ufunc methods: >> [clip] >>> ? ? ? * reducein (array, indices, axis=0) >>> ? ? ? ? ? ? ? ?similar to reduce-at, but the indices provide both the >>> start and end points (rather than being fence-posts like reduceat). >> >> Is the `reducein` important to have, as compared to `reduceat`? > > Yes, I think so. If there are some areas you want to ignore, that's > difficult to do with reduceat(). And conversely overlapping areas are highly useful but completely impossible to do with reduceat. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From jah.mailinglist at gmail.com Tue Apr 13 23:16:37 2010 From: jah.mailinglist at gmail.com (jah) Date: Tue, 13 Apr 2010 20:16:37 -0700 Subject: [Numpy-discussion] binomial coefficient, factorial Message-ID: Is there any chance that a binomial coefficent and factorial function can make their way into NumPy? I know these exist in Scipy, but I don't want to have to install SciPy just to have something so "basic". -------------- next part -------------- An HTML attachment was scrubbed... URL: From bpederse at gmail.com Tue Apr 13 23:52:38 2010 From: bpederse at gmail.com (Brent Pedersen) Date: Tue, 13 Apr 2010 20:52:38 -0700 Subject: [Numpy-discussion] patch to pickle np.memmap Message-ID: hi, i posted a patch to allow pickling of np.memmap objects. http://projects.scipy.org/numpy/ticket/1452 currently, it always returns 'r' for the mode. is that the best thing to do there? any other changes? -brent From bpederse at gmail.com Tue Apr 13 23:59:49 2010 From: bpederse at gmail.com (Brent Pedersen) Date: Tue, 13 Apr 2010 20:59:49 -0700 Subject: [Numpy-discussion] patch to pickle np.memmap In-Reply-To: References: Message-ID: On Tue, Apr 13, 2010 at 8:52 PM, Brent Pedersen wrote: > hi, i posted a patch to allow pickling of np.memmap objects. > http://projects.scipy.org/numpy/ticket/1452 > > currently, it always returns 'r' for the mode. > is that the best thing to do there? > any other changes? > -brent > and i guess it should (but does not with that patch) correctly handle: >>> a = np.memmap(...) >>> b = a[2:] >>> cPickle.dumps(b) From pete at shinners.org Wed Apr 14 02:10:59 2010 From: pete at shinners.org (Peter Shinners) Date: Tue, 13 Apr 2010 23:10:59 -0700 Subject: [Numpy-discussion] how to tally the values seen Message-ID: <4BC55C73.8080407@shinners.org> I have an array that represents the number of times a value has been given. I'm trying to find a direct numpy way to add into these sums without requiring a Python loop. For example, say there are 10 possible values. I start with an array of zeros. >>> counts = numpy.zeros(10, numpy.int) Now I get an array with several values in them, I want to add into counts. All I can think of is a for loop that will give my the results I want. >>> values = numpy.array((2, 8, 1)) >>> for v in values: ... counts[v] += 1 >>> print counts [0 1 1 0 0 0 0 0 1 0] I also need to handle the case where a value is listed more than once. So if values is (2, 8, 1, 2) then count[2] would equal 2. What is the most efficient way to do this? From gokhansever at gmail.com Wed Apr 14 02:27:02 2010 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=) Date: Wed, 14 Apr 2010 01:27:02 -0500 Subject: [Numpy-discussion] how to tally the values seen In-Reply-To: <4BC55C73.8080407@shinners.org> References: <4BC55C73.8080407@shinners.org> Message-ID: On Wed, Apr 14, 2010 at 1:10 AM, Peter Shinners wrote: > I have an array that represents the number of times a value has been > given. I'm trying to find a direct numpy way to add into these sums > without requiring a Python loop. > > For example, say there are 10 possible values. I start with an array of > zeros. > > >>> counts = numpy.zeros(10, numpy.int) > > Now I get an array with several values in them, I want to add into > counts. All I can think of is a for loop that will give my the results I > want. > > > >>> values = numpy.array((2, 8, 1)) > >>> for v in values: > ... counts[v] += 1 > >>> print counts > [0 1 1 0 0 0 0 0 1 0] > This is easy: I[3]: a O[3]: array([ 0., 1., 1., 0., 0., 0., 0., 0., 1., 0.]) I[4]: a = np.zeros(10) I[5]: b = np.array((2,8,1)) I[6]: a[b] = 1 I[7]: a O[7]: array([ 0., 1., 1., 0., 0., 0., 0., 0., 1., 0.]) Let me think about the other case :) > I also need to handle the case where a value is listed more than once. > So if values is (2, 8, 1, 2) then count[2] would equal 2. > > What is the most efficient way to do this? > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- G?khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From warren.weckesser at enthought.com Wed Apr 14 02:34:17 2010 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Wed, 14 Apr 2010 01:34:17 -0500 Subject: [Numpy-discussion] how to tally the values seen In-Reply-To: References: <4BC55C73.8080407@shinners.org> Message-ID: <4BC561E9.5060103@enthought.com> G?khan Sever wrote: > > > On Wed, Apr 14, 2010 at 1:10 AM, Peter Shinners > wrote: > > I have an array that represents the number of times a value has been > given. I'm trying to find a direct numpy way to add into these sums > without requiring a Python loop. > > For example, say there are 10 possible values. I start with an > array of > zeros. > > >>> counts = numpy.zeros(10, numpy.int ) > > Now I get an array with several values in them, I want to add into > counts. All I can think of is a for loop that will give my the > results I > want. > > > >>> values = numpy.array((2, 8, 1)) > >>> for v in values: > ... counts[v] += 1 > >>> print counts > [0 1 1 0 0 0 0 0 1 0] > > > This is easy: > > I[3]: a > O[3]: array([ 0., 1., 1., 0., 0., 0., 0., 0., 1., 0.]) > > I[4]: a = np.zeros(10) > > I[5]: b = np.array((2,8,1)) > > I[6]: a[b] = 1 > > I[7]: a > O[7]: array([ 0., 1., 1., 0., 0., 0., 0., 0., 1., 0.]) > > Let me think about the other case :) > > > I also need to handle the case where a value is listed more than once. > So if values is (2, 8, 1, 2) then count[2] would equal 2. > numpy.bincount(): In [1]: import numpy as np In [2]: x = np.array([2,8,1,2,7,7,2,7,0,2]) In [3]: np.bincount(x) Out[3]: array([1, 1, 4, 0, 0, 0, 0, 3, 1]) Warren > What is the most efficient way to do this? > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > -- > G?khan > ------------------------------------------------------------------------ > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From gokhansever at gmail.com Wed Apr 14 02:44:18 2010 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=) Date: Wed, 14 Apr 2010 01:44:18 -0500 Subject: [Numpy-discussion] how to tally the values seen In-Reply-To: <4BC561E9.5060103@enthought.com> References: <4BC55C73.8080407@shinners.org> <4BC561E9.5060103@enthought.com> Message-ID: On Wed, Apr 14, 2010 at 1:34 AM, Warren Weckesser < warren.weckesser at enthought.com> wrote: > G?khan Sever wrote: > > > > > > On Wed, Apr 14, 2010 at 1:10 AM, Peter Shinners > > wrote: > > > > I have an array that represents the number of times a value has been > > given. I'm trying to find a direct numpy way to add into these sums > > without requiring a Python loop. > > > > For example, say there are 10 possible values. I start with an > > array of > > zeros. > > > > >>> counts = numpy.zeros(10, numpy.int ) > > > > Now I get an array with several values in them, I want to add into > > counts. All I can think of is a for loop that will give my the > > results I > > want. > > > > > > >>> values = numpy.array((2, 8, 1)) > > >>> for v in values: > > ... counts[v] += 1 > > >>> print counts > > [0 1 1 0 0 0 0 0 1 0] > > > > > > This is easy: > > > > I[3]: a > > O[3]: array([ 0., 1., 1., 0., 0., 0., 0., 0., 1., 0.]) > > > > I[4]: a = np.zeros(10) > > > > I[5]: b = np.array((2,8,1)) > > > > I[6]: a[b] = 1 > > > > I[7]: a > > O[7]: array([ 0., 1., 1., 0., 0., 0., 0., 0., 1., 0.]) > > > > Let me think about the other case :) > > > > > > I also need to handle the case where a value is listed more than > once. > > So if values is (2, 8, 1, 2) then count[2] would equal 2. > > > > > numpy.bincount(): > > > In [1]: import numpy as np > > In [2]: x = np.array([2,8,1,2,7,7,2,7,0,2]) > > In [3]: np.bincount(x) > Out[3]: array([1, 1, 4, 0, 0, 0, 0, 3, 1]) > > > I knew a function exists in numpy for this case too :) This is also safer way to handle the given situation to prevent index out of bounds errors. > > Warren > > > What is the most efficient way to do this? > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > > > > > > -- > > G?khan > > ------------------------------------------------------------------------ > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- G?khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From pete at shinners.org Wed Apr 14 03:05:09 2010 From: pete at shinners.org (Peter Shinners) Date: Wed, 14 Apr 2010 00:05:09 -0700 Subject: [Numpy-discussion] how to tally the values seen In-Reply-To: References: <4BC55C73.8080407@shinners.org> <4BC561E9.5060103@enthought.com> Message-ID: <4BC56925.90903@shinners.org> On 04/13/2010 11:44 PM, G?khan Sever wrote: > > > On Wed, Apr 14, 2010 at 1:34 AM, Warren Weckesser > > wrote: > > G?khan Sever wrote: > > > > > > On Wed, Apr 14, 2010 at 1:10 AM, Peter Shinners > > > >> wrote: > > > > I have an array that represents the number of times a value > has been > > given. I'm trying to find a direct numpy way to add into > these sums > > without requiring a Python loop. > > > > For example, say there are 10 possible values. I start with an > > array of > > zeros. > > > > >>> counts = numpy.zeros(10, numpy.int > ) > > > > Now I get an array with several values in them, I want to > add into > > counts. All I can think of is a for loop that will give my the > > results I > > want. > > > > > > >>> values = numpy.array((2, 8, 1)) > > >>> for v in values: > > ... counts[v] += 1 > > >>> print counts > > [0 1 1 0 0 0 0 0 1 0] > > > > > > This is easy: > > > > I[3]: a > > O[3]: array([ 0., 1., 1., 0., 0., 0., 0., 0., 1., 0.]) > > > > I[4]: a = np.zeros(10) > > > > I[5]: b = np.array((2,8,1)) > > > > I[6]: a[b] = 1 > > > > I[7]: a > > O[7]: array([ 0., 1., 1., 0., 0., 0., 0., 0., 1., 0.]) > > > > Let me think about the other case :) > > > > > > I also need to handle the case where a value is listed more > than once. > > So if values is (2, 8, 1, 2) then count[2] would equal 2. > > > > > numpy.bincount(): > > > In [1]: import numpy as np > > In [2]: x = np.array([2,8,1,2,7,7,2,7,0,2]) > > In [3]: np.bincount(x) > Out[3]: array([1, 1, 4, 0, 0, 0, 0, 3, 1]) > > > > I knew a function exists in numpy for this case too :) > > This is also safer way to handle the given situation to prevent index > out of bounds errors. Thanks guys. Numpy always surprises me. It's like you guys have borrowed Guido's time machine. This is running far faster than I ever hoped. My next problem involves a bit of indexing and reordering. But I'm gonna spend a night of my own time to see where I can get with it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ioannis87 at gmail.com Wed Apr 14 04:11:38 2010 From: ioannis87 at gmail.com (ioannis syntychakis) Date: Wed, 14 Apr 2010 10:11:38 +0200 Subject: [Numpy-discussion] look for value, depending to y position Message-ID: Hallo everybody maybe somebody can help with the following: i'm using numpy and pil to find objects in a grayscale image. I make an array of the image and then i look for pixels with the value above the 230. Then i convert the array to image and i see my objects. What i want is to make the grayscale depented to the place on th image. the image is 500 to 500 pixels. and for example i want that the pixelvalue the program is looking for decreases in the y direction. on position y= 0 to 100 the programm is looking for pixelvalues above the 250 on position y= 100 to 200 the programm is looking for pixelvalues above the 230 on position y= 200 to 300 the programm is looking for pixelvalues above the 210 on position y= 400 to 500the programm is looking for pixelvalues above the 190 is this possible? thanks in advance. greetings, Jannis -------------- next part -------------- An HTML attachment was scrubbed... URL: From fperez.net at gmail.com Wed Apr 14 04:17:37 2010 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 14 Apr 2010 01:17:37 -0700 Subject: [Numpy-discussion] Is this a small bug in numpydoc? Message-ID: Howdy, in ipython we use numpydoc, and as I was just trying to build the docs from a clean checkout of ipython's trunk, I kept getting errors that I was able to fix with this patch: amirbar[sphinxext]> svn diff Index: numpydoc.py =================================================================== --- numpydoc.py (revision 8332) +++ numpydoc.py (working copy) @@ -73,7 +73,8 @@ def mangle_signature(app, what, name, obj, options, sig, retann): # Do not try to inspect classes that don't define `__init__` if (inspect.isclass(obj) and - 'initializes x; see ' in pydoc.getdoc(obj.__init__)): + (not hasattr(obj, '__init__') or + 'initializes x; see ' in pydoc.getdoc(obj.__init__))): return '', '' if not (callable(obj) or hasattr(obj, '__argspec_is_invalid_')): return The errors were always of the type: Exception occurred: File "/home/fperez/ipython/repo/trunk-lp/docs/sphinxext/numpydoc.py", line 71, in mangle_signature 'initializes x; see ' in pydoc.getdoc(obj.__init__)): AttributeError: class Bunch has no attribute '__init__' The full traceback has been saved in /tmp/sphinx-err-H1VlaY.log, if you want to report the issue to the developers. so it seems indeed that accessing __init__ without checking it's there isn't a very good idea. But I didn't write numpydoc and I'm tired, so I don't want to commit this without a second pair of eyes... Cheers, f From nadavh at visionsense.com Wed Apr 14 05:13:21 2010 From: nadavh at visionsense.com (Nadav Horesh) Date: Wed, 14 Apr 2010 12:13:21 +0300 Subject: [Numpy-discussion] look for value, depending to y position References: Message-ID: <710F2847B0018641891D9A21602763605AD3AE@ex3.envision.co.il> I assume that you forgot to specify the range between 300 and 400. But anyway this piece of code may give you a direction: ---------------------------------- import numpy as np ythreshold = np.repeat(np.arange(4,-1,-1), 100) * 20 +190 bin_image = image > ythreshold[:,None] ---------------------------------- Anyway I advice you to look at image morphology operations in scipy.ndimage Nadav -----Original Message----- From: numpy-discussion-bounces at scipy.org on behalf of ioannis syntychakis Sent: Wed 14-Apr-10 10:11 To: Discussion of Numerical Python Subject: [Numpy-discussion] look for value, depending to y position Hallo everybody maybe somebody can help with the following: i'm using numpy and pil to find objects in a grayscale image. I make an array of the image and then i look for pixels with the value above the 230. Then i convert the array to image and i see my objects. What i want is to make the grayscale depented to the place on th image. the image is 500 to 500 pixels. and for example i want that the pixelvalue the program is looking for decreases in the y direction. on position y= 0 to 100 the programm is looking for pixelvalues above the 250 on position y= 100 to 200 the programm is looking for pixelvalues above the 230 on position y= 200 to 300 the programm is looking for pixelvalues above the 210 on position y= 400 to 500the programm is looking for pixelvalues above the 190 is this possible? thanks in advance. greetings, Jannis -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 3355 bytes Desc: not available URL: From ralf.gommers at googlemail.com Wed Apr 14 05:57:03 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 14 Apr 2010 17:57:03 +0800 Subject: [Numpy-discussion] stdlib docstring format under discussion Message-ID: Hi all, On doc-sig there is a discussion going on about adopting a standard docstring format for the stdlib. The suggested format is epydoc. If you care about readability in a terminal then it may be good to join this discussion. http://mail.python.org/pipermail/doc-sig/2010-April/003819.html Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav+sp at iki.fi Wed Apr 14 06:21:31 2010 From: pav+sp at iki.fi (Pauli Virtanen) Date: Wed, 14 Apr 2010 10:21:31 +0000 (UTC) Subject: [Numpy-discussion] Is this a small bug in numpydoc? References: Message-ID: Wed, 14 Apr 2010 01:17:37 -0700, Fernando Perez wrote: [clip] > Exception occurred: > File "/home/fperez/ipython/repo/trunk-lp/docs/sphinxext/numpydoc.py", > line 71, in mangle_signature > 'initializes x; see ' in pydoc.getdoc(obj.__init__)): > AttributeError: class Bunch has no attribute '__init__' The full > traceback has been saved in /tmp/sphinx-err-H1VlaY.log, if you want to > report the issue to the developers. > > so it seems indeed that accessing __init__ without checking it's there > isn't a very good idea. > > But I didn't write numpydoc and I'm tired, so I don't want to commit > this without a second pair of eyes... Yeah, it's a bug, I think. -- Pauli Virtanen From faltet at pytables.org Wed Apr 14 09:08:37 2010 From: faltet at pytables.org (Francesc Alted) Date: Wed, 14 Apr 2010 15:08:37 +0200 Subject: [Numpy-discussion] rc2 for NumPy 1.4.1 and Scipy 0.7.2 In-Reply-To: References: Message-ID: <201004141508.37938.faltet@pytables.org> A Sunday 11 April 2010 11:09:53 Ralf Gommers escrigu?: > Hi, > > I am pleased to announce the second release candidate of both Scipy 0.7.2 > and NumPy 1.4.1, please test them. > > The issues reported with rc1 should be fixed, and for NumPy there are now > Python 2.5 binaries as well. For SciPy there will be no 2.5 binaries - > because 0.7.x is built against NumPy 1.2 it does not work on OS X 10.6, and > on Windows I see some incomprehensible build failures. > > Binaries, sources and release notes can be found at > http://www.filefactory.com/f/4452c50056df8bba/ (apologies for the flashy > ads again). I've had a try at the new 1.4.1rc2 and my existing binaries that were linked with 1.3.0 runs all the tests without difficulty (so ABI compatibility seems good), but every time that a Cython extension is loaded, I get the next warning: /home/faltet/PyTables/pytables/trunk/tables/__init__.py:56: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility from tables.utilsExtension import getPyTablesVersion, getHDF5Version /home/faltet/PyTables/pytables/trunk/tables/file.py:43: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility from tables import hdf5Extension /home/faltet/PyTables/pytables/trunk/tables/link.py:32: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility from tables import linkExtension /home/faltet/PyTables/pytables/trunk/tables/table.py:38: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility I'm using current stable Cython 12.1. Is the warning above intended or I'm doing something wrong? Thanks, -- Francesc Alted From aisaac at american.edu Wed Apr 14 09:09:40 2010 From: aisaac at american.edu (Alan G Isaac) Date: Wed, 14 Apr 2010 09:09:40 -0400 Subject: [Numpy-discussion] binomial coefficient, factorial In-Reply-To: References: Message-ID: <4BC5BE94.60107@american.edu> On 4/13/2010 11:16 PM, jah wrote: > binomial coefficent and factorial function http://code.google.com/p/econpy/source/browse/trunk/pytrix/pytrix.py fwiw, Alan Isaac From charlesr.harris at gmail.com Wed Apr 14 09:36:00 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 14 Apr 2010 07:36:00 -0600 Subject: [Numpy-discussion] rc2 for NumPy 1.4.1 and Scipy 0.7.2 In-Reply-To: <201004141508.37938.faltet@pytables.org> References: <201004141508.37938.faltet@pytables.org> Message-ID: On Wed, Apr 14, 2010 at 7:08 AM, Francesc Alted wrote: > A Sunday 11 April 2010 11:09:53 Ralf Gommers escrigu?: > > Hi, > > > > I am pleased to announce the second release candidate of both Scipy 0.7.2 > > and NumPy 1.4.1, please test them. > > > > The issues reported with rc1 should be fixed, and for NumPy there are now > > Python 2.5 binaries as well. For SciPy there will be no 2.5 binaries - > > because 0.7.x is built against NumPy 1.2 it does not work on OS X 10.6, > and > > on Windows I see some incomprehensible build failures. > > > > Binaries, sources and release notes can be found at > > http://www.filefactory.com/f/4452c50056df8bba/ (apologies for the flashy > > ads again). > > I've had a try at the new 1.4.1rc2 and my existing binaries that were > linked > with 1.3.0 runs all the tests without difficulty (so ABI compatibility > seems > good), but every time that a Cython extension is loaded, I get the next > warning: > > /home/faltet/PyTables/pytables/trunk/tables/__init__.py:56: RuntimeWarning: > numpy.dtype size changed, may indicate binary incompatibility > from tables.utilsExtension import getPyTablesVersion, getHDF5Version > /home/faltet/PyTables/pytables/trunk/tables/file.py:43: RuntimeWarning: > numpy.dtype size changed, may indicate binary incompatibility > from tables import hdf5Extension > /home/faltet/PyTables/pytables/trunk/tables/link.py:32: RuntimeWarning: > numpy.dtype size changed, may indicate binary incompatibility > from tables import linkExtension > /home/faltet/PyTables/pytables/trunk/tables/table.py:38: RuntimeWarning: > numpy.dtype size changed, may indicate binary incompatibility > > I'm using current stable Cython 12.1. Is the warning above intended or I'm > doing something wrong? > > I believe Cython decided to issue a warning instead of bailing, which makes it usable but constantly annoying. The warning can probably be caught in the __init__ but it would be nice to have some sort of cython flag that would disable it completely. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From faltet at pytables.org Wed Apr 14 10:20:00 2010 From: faltet at pytables.org (Francesc Alted) Date: Wed, 14 Apr 2010 16:20:00 +0200 Subject: [Numpy-discussion] rc2 for NumPy 1.4.1 and Scipy 0.7.2 In-Reply-To: References: <201004141508.37938.faltet@pytables.org> Message-ID: <201004141620.00836.faltet@pytables.org> A Wednesday 14 April 2010 15:36:00 Charles R Harris escrigu?: > > /home/faltet/PyTables/pytables/trunk/tables/table.py:38: RuntimeWarning: > > numpy.dtype size changed, may indicate binary incompatibility > > > > I'm using current stable Cython 12.1. Is the warning above intended or > > I'm doing something wrong? > > I believe Cython decided to issue a warning instead of bailing, which makes > it usable but constantly annoying. The warning can probably be caught in > the __init__ but it would be nice to have some sort of cython flag that > would disable it completely. Yeah, a flag would be nice. Ok, I'll have to manage with that then. Thanks, -- Francesc Alted From pete at shinners.org Wed Apr 14 11:25:53 2010 From: pete at shinners.org (Peter Shinners) Date: Wed, 14 Apr 2010 08:25:53 -0700 Subject: [Numpy-discussion] How to combine a pair of 1D arrays? Message-ID: <4BC5DE81.9040003@shinners.org> Is there a way to combine two 1D arrays with the same size into a 2D array? It seems like the internal pointers and strides could be combined. My primary goal is to not make any copies of the data. It might be doable with a bit of ctypes if there is not a native numpy call. >>> import numpy as np >>> a = np.array((1,2,3,4)) >>> b = np.array((11,12,13,14)) >>> c = np.magical_fuse(a, b) # what goes here? >>> print c.shape (2, 4) >>> a == c[0] True >>> a[1] = -2 >>> a == c[0] True From robert.kern at gmail.com Wed Apr 14 11:34:18 2010 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 14 Apr 2010 10:34:18 -0500 Subject: [Numpy-discussion] How to combine a pair of 1D arrays? In-Reply-To: <4BC5DE81.9040003@shinners.org> References: <4BC5DE81.9040003@shinners.org> Message-ID: On Wed, Apr 14, 2010 at 10:25, Peter Shinners wrote: > Is there a way to combine two 1D arrays with the same size into a 2D > array? It seems like the internal pointers and strides could be > combined. My primary goal is to not make any copies of the data. There is absolutely no way to get around that, I am afraid. > It > might be doable with a bit of ctypes if there is not a native numpy call. > > ?>>> import numpy as np > ?>>> a = np.array((1,2,3,4)) > ?>>> b = np.array((11,12,13,14)) > ?>>> c = np.magical_fuse(a, b) ? # what goes here? c = np.vstack([a, b]) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From Nikolaus at rath.org Wed Apr 14 11:16:02 2010 From: Nikolaus at rath.org (Nikolaus Rath) Date: Wed, 14 Apr 2010 11:16:02 -0400 Subject: [Numpy-discussion] Find indices of largest elements Message-ID: <874ojedpzh.fsf@inspiron.ap.columbia.edu> Hello, How do I best find out the indices of the largest x elements in an array? Example: a = [ [1,8,2], [2,1,3] ] magic_function(a, 2) == [ (0,1), (1,2) ] Since the largest 2 elements are at positions (0,1) and (1,2). Best, -Niko -- ?Time flies like an arrow, fruit flies like a Banana.? PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C From ralf.gommers at googlemail.com Wed Apr 14 11:47:27 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 14 Apr 2010 23:47:27 +0800 Subject: [Numpy-discussion] Release candidate 2 for NumPy 1.4.1 and SciPy 0.7.2 Message-ID: Hi, I am pleased to announce the second release candidate of both Scipy 0.7.2 and NumPy 1.4.1. Please test, and report any problems on the NumPy or SciPy list. I also want to specifically ask you to report success/failure with other libraries (Matplotlib, Pygame, ) based on NumPy or SciPy. Binaries, sources and release notes can be found at https://sourceforge.net/projects/numpy/files/ https://sourceforge.net/projects/scipy/files/ NumPy 1.4.1 ========== The main change over 1.4.0 is that datetime support has been removed. This fixes the binary incompatibility issues between NumPy and other libraries like SciPy and Matplotlib. There are also a number of other bug fixes, and no new features. Binaries for Python 2.5 and 2.6 are available for both Windows and OS X. SciPy 0.7.2 ========= The only change compared to 0.7.1 is that the C sources for Cython code have been regenerated with Cython 0.12.1. This ensures that SciPy 0.7.2 will work with NumPy 1.4.1, while also retaining backwards compatibility with NumPy 1.3.0. Note that the 0.7.x branch was created in January 2009, so a lot of fixes and new functionality in current trunk is not present in this release. Binaries for Python 2.6 are available for both Windows and OS X. Due to the age of the code no binaries for Python 2.5 are available. On behalf of the NumPy and SciPy developers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsseabold at gmail.com Wed Apr 14 11:48:26 2010 From: jsseabold at gmail.com (Skipper Seabold) Date: Wed, 14 Apr 2010 11:48:26 -0400 Subject: [Numpy-discussion] Find indices of largest elements In-Reply-To: <874ojedpzh.fsf@inspiron.ap.columbia.edu> References: <874ojedpzh.fsf@inspiron.ap.columbia.edu> Message-ID: On Wed, Apr 14, 2010 at 11:16 AM, Nikolaus Rath wrote: > Hello, > > How do I best find out the indices of the largest x elements in an > array? > > Example: > > a = [ [1,8,2], [2,1,3] ] > magic_function(a, 2) == [ (0,1), (1,2) ] > > Since the largest 2 elements are at positions (0,1) and (1,2). Something like this might be made to work, if you want the max elements along all the rows. In [3]: a = np.asarray(a) In [4]: a[range(len(a)),np.argmax(a, axis=1)] Out[4]: array([8, 3]) Skipper From kwgoodman at gmail.com Wed Apr 14 11:49:58 2010 From: kwgoodman at gmail.com (Keith Goodman) Date: Wed, 14 Apr 2010 08:49:58 -0700 Subject: [Numpy-discussion] Find indices of largest elements In-Reply-To: <874ojedpzh.fsf@inspiron.ap.columbia.edu> References: <874ojedpzh.fsf@inspiron.ap.columbia.edu> Message-ID: On Wed, Apr 14, 2010 at 8:16 AM, Nikolaus Rath wrote: > Hello, > > How do I best find out the indices of the largest x elements in an > array? > > Example: > > a = [ [1,8,2], [2,1,3] ] > magic_function(a, 2) == [ (0,1), (1,2) ] > > Since the largest 2 elements are at positions (0,1) and (1,2). Here's a quick way to rank the data if there are no ties and no NaNs: >> shape = (3,2) >> x = np.random.rand(*shape) >> x array([[ 0.83424288, 0.17821326], [ 0.62718311, 0.63514286], [ 0.18373934, 0.90634162]]) >> r = x.reshape(-1).argsort().argsort().reshape(shape) >> r array([[4, 0], [2, 3], [1, 5]]) To find the indices you can use where: >> r < 2 array([[False, True], [False, False], [ True, False]], dtype=bool) >> np.where(r < 2) (array([0, 2]), array([1, 0])) ...but the indices will not be in order. From gokhansever at gmail.com Wed Apr 14 11:56:56 2010 From: gokhansever at gmail.com (=?UTF-8?Q?G=C3=B6khan_Sever?=) Date: Wed, 14 Apr 2010 10:56:56 -0500 Subject: [Numpy-discussion] Find indices of largest elements In-Reply-To: <874ojedpzh.fsf@inspiron.ap.columbia.edu> References: <874ojedpzh.fsf@inspiron.ap.columbia.edu> Message-ID: On Wed, Apr 14, 2010 at 10:16 AM, Nikolaus Rath wrote: > Hello, > > How do I best find out the indices of the largest x elements in an > array? > > Example: > > a = [ [1,8,2], [2,1,3] ] > magic_function(a, 2) == [ (0,1), (1,2) ] > > Since the largest 2 elements are at positions (0,1) and (1,2). > > I[1]: a = np.array([ [1,8,2], [2,1,3] ]) I[2]: b = a.max(axis=1)[:,np.newaxis] I[3]: b O[3]: array([[8], [3]]) I[4]: np.where(a==b) O[4]: (array([0, 1]), array([1, 2])) -- G?khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From kwgoodman at gmail.com Wed Apr 14 12:01:02 2010 From: kwgoodman at gmail.com (Keith Goodman) Date: Wed, 14 Apr 2010 09:01:02 -0700 Subject: [Numpy-discussion] Find indices of largest elements In-Reply-To: References: <874ojedpzh.fsf@inspiron.ap.columbia.edu> Message-ID: On Wed, Apr 14, 2010 at 8:49 AM, Keith Goodman wrote: > On Wed, Apr 14, 2010 at 8:16 AM, Nikolaus Rath wrote: >> Hello, >> >> How do I best find out the indices of the largest x elements in an >> array? >> >> Example: >> >> a = [ [1,8,2], [2,1,3] ] >> magic_function(a, 2) == [ (0,1), (1,2) ] >> >> Since the largest 2 elements are at positions (0,1) and (1,2). > > Here's a quick way to rank the data if there are no ties and no NaNs: > >>> shape = (3,2) >>> x = np.random.rand(*shape) >>> x > array([[ 0.83424288, ?0.17821326], > ? ? ? [ 0.62718311, ?0.63514286], > ? ? ? [ 0.18373934, ?0.90634162]]) >>> r = x.reshape(-1).argsort().argsort().reshape(shape) >>> r > array([[4, 0], > ? ? ? [2, 3], > ? ? ? [1, 5]]) > > To find the indices you can use where: > >>> r < 2 > array([[False, ?True], > ? ? ? [False, False], > ? ? ? [ True, False]], dtype=bool) >>> np.where(r < 2) > ? (array([0, 2]), array([1, 0])) > > ...but the indices will not be in order. ...or if you need the indices in order: >> shape = (3,2) >> x = np.random.rand(*shape) >> x array([[ 0.52420123, 0.43231286], [ 0.97995333, 0.87416228], [ 0.71604075, 0.66018382]]) >> r = x.reshape(-1).argsort().argsort() >> np.unravel_index(r[0], shape) (0, 1) >> np.unravel_index(r[1], shape) (0, 0) >> np.unravel_index(r[2], shape) (2, 1) >> np.unravel_index(r[-1], shape) (1, 0) From fperez.net at gmail.com Wed Apr 14 13:18:11 2010 From: fperez.net at gmail.com (Fernando Perez) Date: Wed, 14 Apr 2010 10:18:11 -0700 Subject: [Numpy-discussion] Is this a small bug in numpydoc? In-Reply-To: References: Message-ID: On Wed, Apr 14, 2010 at 3:21 AM, Pauli Virtanen wrote: >> But I didn't write numpydoc and I'm tired, so I don't want to commit >> this without a second pair of eyes... > > Yeah, it's a bug, I think. > Thanks, fixed in r8333. Cheers, f From zachary.pincus at yale.edu Wed Apr 14 13:20:08 2010 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Wed, 14 Apr 2010 13:20:08 -0400 Subject: [Numpy-discussion] How to combine a pair of 1D arrays? In-Reply-To: References: <4BC5DE81.9040003@shinners.org> Message-ID: <5BCCD841-C476-4112-9630-38FFD6FEB497@yale.edu> > On Wed, Apr 14, 2010 at 10:25, Peter Shinners > wrote: >> Is there a way to combine two 1D arrays with the same size into a 2D >> array? It seems like the internal pointers and strides could be >> combined. My primary goal is to not make any copies of the data. > > There is absolutely no way to get around that, I am afraid. You could start with the 2d array... instead of: > >>> a = np.array((1,2,3,4)) > >>> b = np.array((11,12,13,14)) > >>> c = np.magical_fuse(a, b) # what goes here? perhaps something like: >>> c = np.empty((2,4), int) >>> a = c[0] >>> b = c[1] Now a and b are views on c. So if you (say) pass them to, say, some (strides-aware) C routine that fills them in element-by-element, c will get filled in at the same time. Zach From peridot.faceted at gmail.com Wed Apr 14 14:01:16 2010 From: peridot.faceted at gmail.com (Anne Archibald) Date: Wed, 14 Apr 2010 14:01:16 -0400 Subject: [Numpy-discussion] How to combine a pair of 1D arrays? In-Reply-To: References: <4BC5DE81.9040003@shinners.org> Message-ID: On 14 April 2010 11:34, Robert Kern wrote: > On Wed, Apr 14, 2010 at 10:25, Peter Shinners wrote: >> Is there a way to combine two 1D arrays with the same size into a 2D >> array? It seems like the internal pointers and strides could be >> combined. My primary goal is to not make any copies of the data. > > There is absolutely no way to get around that, I am afraid. Well, I'm not quite sure I agree with this. The best way, of course, is to allocate the original two arrays as subarrays of one larger array, that is, start with the fused array and select your two 1D arrays as subarrays. Of course, this depends on how you're generating the 1D arrays; if they're simply being returned to you from a black-box function, you're stuck with them. But if it's a ufunc-like operation, it may have an output argument that lets you write to a supplied array rather than allocating a new one. If they're coming from a file, you may be able to map the whole file (or a large chunk) as an array and select them as subarrays (even if alignment and type are issues). You should also keep in mind that allocating arrays and copying data really isn't very expensive - malloc() is extremely fast, especially for large arrays which are just grabbed as blocks from the OS - and copying arrays is also cheap, and can be used to reorder data into a more cache-friendly order. If the problem is that your arrays almost fill available memory, you will already have noticed that using numpy is kind of a pain, because many operations involve copies. But if you really have to do this, it may be possible. numpy arrays are specified by giving a data area, and strides into that data area. The steps along each axis must be uniform, but if you really have two arrays with the same stride, you may be able to use a gruesome hack to make it work. Each of your two arrays has a data pointer, which essentially points to the first element. So if you make up your two-row array using the same data pointer as your first array, and give it a stride along the second dimension equal to the difference between pointers, this should work. Of course, you have to make sure python doesn't deallocate the second array out from under you, and you may have to defeat some error checking, but in principle it should be possible. Anne From doutriaux1 at llnl.gov Wed Apr 14 14:28:44 2010 From: doutriaux1 at llnl.gov (=?utf-8?Q?Charles_=D8=B3=D9=85=D9=8A=D8=B1_Doutriaux?=) Date: Wed, 14 Apr 2010 11:28:44 -0700 Subject: [Numpy-discussion] rc2 for NumPy 1.4.1 and Scipy 0.7.2 In-Reply-To: References: Message-ID: <336574F6-63FD-497F-BB78-5EB7BEC4D62B@llnl.gov> Just downloaded this. On my mac 10.6, using python 2.6.5 i get: Running from numpy source directory. non-existing path in 'numpy/distutils': 'site.cfg' F2PY Version 2 blas_opt_info: FOUND: extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] define_macros = [('NO_ATLAS_INFO', 3)] extra_compile_args = ['-faltivec', '-I/System/Library/Frameworks/vecLib.framework/Headers'] lapack_opt_info: FOUND: extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] define_macros = [('NO_ATLAS_INFO', 3)] extra_compile_args = ['-faltivec'] running build running config_cc unifing config_cc, config, build_clib, build_ext, build commands --compiler options running config_fc unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options running build_src build_src building py_modules sources creating build creating build/src.macosx-10.3-fat-2.6 creating build/src.macosx-10.3-fat-2.6/numpy creating build/src.macosx-10.3-fat-2.6/numpy/distutils building library "npymath" sources customize NAGFCompiler Could not locate executable f95 customize AbsoftFCompiler Could not locate executable f90 Could not locate executable f77 customize IBMFCompiler Could not locate executable xlf90 Could not locate executable xlf customize IntelFCompiler Could not locate executable ifort Could not locate executable ifc customize GnuFCompiler Could not locate executable g77 customize Gnu95FCompiler Found executable /usr/local/bin/gfortran /Users/doutriaux1/Desktop/numpy-1.4.1rc2/numpy/distutils/fcompiler/gnu.py:125: UserWarning: Env. variable MACOSX_DEPLOYMENT_TARGET set to 10.3 warnings.warn(s) customize Gnu95FCompiler customize Gnu95FCompiler using config C compiler: gcc -arch ppc -arch i386 -isysroot / -fno-strict-aliasing -DNDEBUG compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/include -I/lgm/cdat/trunk/include/python2.6 -c' gcc: _configtest.c gcc _configtest.o -o _configtest ld: warning: in _configtest.o, missing required architecture x86_64 in file Undefined symbols: "_main", referenced from: __start in crt1.o ld: symbol(s) not found collect2: ld returned 1 exit status ld: warning: in _configtest.o, missing required architecture x86_64 in file Undefined symbols: "_main", referenced from: __start in crt1.o ld: symbol(s) not found collect2: ld returned 1 exit status failure. removing: _configtest.c _configtest.o Traceback (most recent call last): File "setup.py", line 187, in setup_package() File "setup.py", line 180, in setup_package configuration=configuration ) File "/Users/doutriaux1/Desktop/numpy-1.4.1rc2/numpy/distutils/core.py", line 186, in setup return old_setup(**new_attr) File "/lgm/cdat/trunk/lib/python2.6/distutils/core.py", line 152, in setup dist.run_commands() File "/lgm/cdat/trunk/lib/python2.6/distutils/dist.py", line 975, in run_commands self.run_command(cmd) File "/lgm/cdat/trunk/lib/python2.6/distutils/dist.py", line 995, in run_command cmd_obj.run() File "/Users/doutriaux1/Desktop/numpy-1.4.1rc2/numpy/distutils/command/build.py", line 37, in run old_build.run(self) File "/lgm/cdat/trunk/lib/python2.6/distutils/command/build.py", line 134, in run self.run_command(cmd_name) File "/lgm/cdat/trunk/lib/python2.6/distutils/cmd.py", line 333, in run_command self.distribution.run_command(command) File "/lgm/cdat/trunk/lib/python2.6/distutils/dist.py", line 995, in run_command cmd_obj.run() File "/Users/doutriaux1/Desktop/numpy-1.4.1rc2/numpy/distutils/command/build_src.py", line 152, in run self.build_sources() File "/Users/doutriaux1/Desktop/numpy-1.4.1rc2/numpy/distutils/command/build_src.py", line 163, in build_sources self.build_library_sources(*libname_info) File "/Users/doutriaux1/Desktop/numpy-1.4.1rc2/numpy/distutils/command/build_src.py", line 298, in build_library_sources sources = self.generate_sources(sources, (lib_name, build_info)) File "/Users/doutriaux1/Desktop/numpy-1.4.1rc2/numpy/distutils/command/build_src.py", line 385, in generate_sources source = func(extension, build_dir) File "numpy/core/setup.py", line 657, in get_mathlib_info raise RuntimeError("Broken toolchain: cannot link a simple C program") RuntimeError: Broken toolchain: cannot link a simple C program On Apr 11, 2010, at 2:09 AM, Ralf Gommers wrote: > Hi, > > I am pleased to announce the second release candidate of both Scipy 0.7.2 and NumPy 1.4.1, please test them. > > The issues reported with rc1 should be fixed, and for NumPy there are now Python 2.5 binaries as well. For SciPy there will be no 2.5 binaries - because 0.7.x is built against NumPy 1.2 it does not work on OS X 10.6, and on Windows I see some incomprehensible build failures. > > Binaries, sources and release notes can be found at http://*www.*filefactory.com/f/4452c50056df8bba/ (apologies for the flashy ads again). > > Cheers, > Ralf > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://*mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From Nikolaus at rath.org Wed Apr 14 15:39:40 2010 From: Nikolaus at rath.org (Nikolaus Rath) Date: Wed, 14 Apr 2010 15:39:40 -0400 Subject: [Numpy-discussion] Find indices of largest elements References: <874ojedpzh.fsf@inspiron.ap.columbia.edu> Message-ID: <871vehescj.fsf@inspiron.ap.columbia.edu> Keith Goodman writes: > On Wed, Apr 14, 2010 at 8:49 AM, Keith Goodman wrote: >> On Wed, Apr 14, 2010 at 8:16 AM, Nikolaus Rath wrote: >>> Hello, >>> >>> How do I best find out the indices of the largest x elements in an >>> array? >>> >>> Example: >>> >>> a = [ [1,8,2], [2,1,3] ] >>> magic_function(a, 2) == [ (0,1), (1,2) ] >>> >>> Since the largest 2 elements are at positions (0,1) and (1,2). >> >> Here's a quick way to rank the data if there are no ties and no NaNs: > > ...or if you need the indices in order: > >>> shape = (3,2) >>> x = np.random.rand(*shape) >>> x > array([[ 0.52420123, 0.43231286], > [ 0.97995333, 0.87416228], > [ 0.71604075, 0.66018382]]) >>> r = x.reshape(-1).argsort().argsort() I don't understand why this works. Why do you call argsort() twice? Doesn't that give you the indices of the sorted indices? Thanks, -Nikolaus -- ?Time flies like an arrow, fruit flies like a Banana.? PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C From kwgoodman at gmail.com Wed Apr 14 16:56:43 2010 From: kwgoodman at gmail.com (Keith Goodman) Date: Wed, 14 Apr 2010 13:56:43 -0700 Subject: [Numpy-discussion] Find indices of largest elements In-Reply-To: <871vehescj.fsf@inspiron.ap.columbia.edu> References: <874ojedpzh.fsf@inspiron.ap.columbia.edu> <871vehescj.fsf@inspiron.ap.columbia.edu> Message-ID: On Wed, Apr 14, 2010 at 12:39 PM, Nikolaus Rath wrote: > Keith Goodman writes: >> On Wed, Apr 14, 2010 at 8:49 AM, Keith Goodman wrote: >>> On Wed, Apr 14, 2010 at 8:16 AM, Nikolaus Rath wrote: >>>> Hello, >>>> >>>> How do I best find out the indices of the largest x elements in an >>>> array? >>>> >>>> Example: >>>> >>>> a = [ [1,8,2], [2,1,3] ] >>>> magic_function(a, 2) == [ (0,1), (1,2) ] >>>> >>>> Since the largest 2 elements are at positions (0,1) and (1,2). >>> >>> Here's a quick way to rank the data if there are no ties and no NaNs: >> >> ...or if you need the indices in order: >> >>>> shape = (3,2) >>>> x = np.random.rand(*shape) >>>> x >> array([[ 0.52420123, ?0.43231286], >> ? ? ? ?[ 0.97995333, ?0.87416228], >> ? ? ? ?[ 0.71604075, ?0.66018382]]) >>>> r = x.reshape(-1).argsort().argsort() > > I don't understand why this works. Why do you call argsort() twice? > Doesn't that give you the indices of the sorted indices? It is confusing. Let's look at an example: >> x = np.random.rand(4) >> x array([ 0.37412289, 0.68248559, 0.12935131, 0.42510212]) If we call argsort once we get the index that will sort x: >> idx = x.argsort() >> idx array([2, 0, 3, 1]) >> x[idx] array([ 0.12935131, 0.37412289, 0.42510212, 0.68248559]) Notice that the first element of idx is 2. That's because element x[2] is the min of x. But that's not what we want. We want the first element to be the rank of the first element of x. So we need to shuffle idx around so that the order aligns with x. How do we do that? We sort it! >> idx.argsort() array([1, 3, 0, 2]) The min value of x is x[2], that's why 2 is the first element of idx which means that we want ranked(x) to contain a 0 at position 2 which it does. Bah, it's all magic. From kwgoodman at gmail.com Wed Apr 14 17:03:16 2010 From: kwgoodman at gmail.com (Keith Goodman) Date: Wed, 14 Apr 2010 14:03:16 -0700 Subject: [Numpy-discussion] Find indices of largest elements In-Reply-To: References: <874ojedpzh.fsf@inspiron.ap.columbia.edu> <871vehescj.fsf@inspiron.ap.columbia.edu> Message-ID: On Wed, Apr 14, 2010 at 1:56 PM, Keith Goodman wrote: > On Wed, Apr 14, 2010 at 12:39 PM, Nikolaus Rath wrote: >> Keith Goodman writes: >>> On Wed, Apr 14, 2010 at 8:49 AM, Keith Goodman wrote: >>>> On Wed, Apr 14, 2010 at 8:16 AM, Nikolaus Rath wrote: >>>>> Hello, >>>>> >>>>> How do I best find out the indices of the largest x elements in an >>>>> array? >>>>> >>>>> Example: >>>>> >>>>> a = [ [1,8,2], [2,1,3] ] >>>>> magic_function(a, 2) == [ (0,1), (1,2) ] >>>>> >>>>> Since the largest 2 elements are at positions (0,1) and (1,2). >>>> >>>> Here's a quick way to rank the data if there are no ties and no NaNs: >>> >>> ...or if you need the indices in order: >>> >>>>> shape = (3,2) >>>>> x = np.random.rand(*shape) >>>>> x >>> array([[ 0.52420123, ?0.43231286], >>> ? ? ? ?[ 0.97995333, ?0.87416228], >>> ? ? ? ?[ 0.71604075, ?0.66018382]]) >>>>> r = x.reshape(-1).argsort().argsort() >> >> I don't understand why this works. Why do you call argsort() twice? >> Doesn't that give you the indices of the sorted indices? > > It is confusing. Let's look at an example: > >>> x = np.random.rand(4) >>> x > ? array([ 0.37412289, ?0.68248559, ?0.12935131, ?0.42510212]) > > If we call argsort once we get the index that will sort x: > >>> idx = x.argsort() >>> idx > ? array([2, 0, 3, 1]) >>> x[idx] > ? array([ 0.12935131, ?0.37412289, ?0.42510212, ?0.68248559]) > > Notice that the first element of idx is 2. That's because element x[2] > is the min of x. But that's not what we want. We want the first > element to be the rank of the first element of x. So we need to > shuffle idx around so that the order aligns with x. How do we do that? > We sort it! > >>> idx.argsort() > ? array([1, 3, 0, 2]) > > The min value of x is x[2], that's why 2 is the first element of idx > which means that we want ranked(x) to contain a 0 at position 2 which > it does. > > Bah, it's all magic. You can also use rankdata from scipy: >> from scipy.stats import rankdata >> rankdata(x) array([ 2., 4., 1., 3.]) Note the the smallest rank is 1. >> rankdata(x) - 1 array([ 1., 3., 0., 2.]) From e.antero.tammi at gmail.com Wed Apr 14 17:35:37 2010 From: e.antero.tammi at gmail.com (eat) Date: Wed, 14 Apr 2010 21:35:37 +0000 (UTC) Subject: [Numpy-discussion] Find indices of largest elements References: <874ojedpzh.fsf@inspiron.ap.columbia.edu> Message-ID: Nikolaus Rath rath.org> writes: > > Hello, > > How do I best find out the indices of the largest x elements in an > array? > > Example: > > a = [ [1,8,2], [2,1,3] ] > magic_function(a, 2) == [ (0,1), (1,2) ] > > Since the largest 2 elements are at positions (0,1) and (1,2). > > Best, > > -Niko > Hi, Just a= np.asarray([[1, 8, 2], [2, 1, 3]]) print np.where((a.T== a.max(axis= 1)).T) However, if any row contains more than 1 max entity, above will fail. Please let me to know if that's relevant for you. -eat From Chris.Barker at noaa.gov Wed Apr 14 17:37:35 2010 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 14 Apr 2010 14:37:35 -0700 Subject: [Numpy-discussion] binomial coefficient, factorial In-Reply-To: References: Message-ID: <4BC6359F.50507@noaa.gov> jah wrote: > Is there any chance that a binomial coefficent and factorial function > can make their way into NumPy? probably not -- numpy is over-populated already > I know these exist in Scipy, but I don't > want to have to install SciPy just to have something so "basic". The problem is that everyone has a different "basic". What we really need is an easier way for folks to use sub-packages of scipy. I've found myself hand-extracting just what I need, too. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From mail at stevesimmons.com Wed Apr 14 17:48:47 2010 From: mail at stevesimmons.com (Stephen Simmons) Date: Wed, 14 Apr 2010 23:48:47 +0200 Subject: [Numpy-discussion] Proposal for new ufunc functionality In-Reply-To: References: <3BB52545-B5B9-4B9C-AD9A-0D69432A6B2D@enthought.com> <1270921512.11355.10.camel@talisman> Message-ID: <4BC6383F.8070309@stevesimmons.com> I would really like to see this become a core part of numpy... For groupby-like summing over arrays, I use a modified version of numpy.bincount() which has optional arguments that greatly enhance its flexibility: bincount(bin, weights=, max_bins=. out=) where: * bins - numpy array of bin numbers (uint8, int16 or int32). [1] * Negative bins numbers indicate weights to be ignored * weights - (opt) numpy array of weights (float or double) [2] * max_bin - (opt) bin numbers greater than this are ignored when counting * out - (opt) numpy output array (int32 or double) [1] This is how I support Robert Kern's comment below "If there are some areas you want to ignore, that's difficult to do with reduceat()." [2] Specifying the number of bins up front has two benefits: (i) saves scanning the bins array to see how big the output needs to be; and (ii) allows you to control the size of the output array, as you may want it bigger than the number of bins would suggest. I look forward to the draft NEP! Best regards Stephen Simmons On 13/04/2010 10:34 PM, Robert Kern wrote: > On Sat, Apr 10, 2010 at 17:59, Robert Kern wrote: > >> On Sat, Apr 10, 2010 at 12:45, Pauli Virtanen wrote: >> >>> la, 2010-04-10 kello 12:23 -0500, Travis Oliphant kirjoitti: >>> [clip] >>> >>>> Here are my suggested additions to NumPy: >>>> ufunc methods: >>>> >>> [clip] >>> >>>> * reducein (array, indices, axis=0) >>>> similar to reduce-at, but the indices provide both the >>>> start and end points (rather than being fence-posts like reduceat). >>>> >>> Is the `reducein` important to have, as compared to `reduceat`? >>> >> Yes, I think so. If there are some areas you want to ignore, that's >> difficult to do with reduceat(). >> > And conversely overlapping areas are highly useful but completely > impossible to do with reduceat. > > From peridot.faceted at gmail.com Wed Apr 14 18:12:49 2010 From: peridot.faceted at gmail.com (Anne Archibald) Date: Wed, 14 Apr 2010 18:12:49 -0400 Subject: [Numpy-discussion] Find indices of largest elements In-Reply-To: References: <874ojedpzh.fsf@inspiron.ap.columbia.edu> <871vehescj.fsf@inspiron.ap.columbia.edu> Message-ID: On 14 April 2010 16:56, Keith Goodman wrote: > On Wed, Apr 14, 2010 at 12:39 PM, Nikolaus Rath wrote: >> Keith Goodman writes: >>> On Wed, Apr 14, 2010 at 8:49 AM, Keith Goodman wrote: >>>> On Wed, Apr 14, 2010 at 8:16 AM, Nikolaus Rath wrote: >>>>> Hello, >>>>> >>>>> How do I best find out the indices of the largest x elements in an >>>>> array? >>>>> >>>>> Example: >>>>> >>>>> a = [ [1,8,2], [2,1,3] ] >>>>> magic_function(a, 2) == [ (0,1), (1,2) ] >>>>> >>>>> Since the largest 2 elements are at positions (0,1) and (1,2). >>>> >>>> Here's a quick way to rank the data if there are no ties and no NaNs: >>> >>> ...or if you need the indices in order: >>> >>>>> shape = (3,2) >>>>> x = np.random.rand(*shape) >>>>> x >>> array([[ 0.52420123, 0.43231286], >>> [ 0.97995333, 0.87416228], >>> [ 0.71604075, 0.66018382]]) >>>>> r = x.reshape(-1).argsort().argsort() >> >> I don't understand why this works. Why do you call argsort() twice? >> Doesn't that give you the indices of the sorted indices? > > It is confusing. Let's look at an example: > >>> x = np.random.rand(4) >>> x > array([ 0.37412289, 0.68248559, 0.12935131, 0.42510212]) > > If we call argsort once we get the index that will sort x: > >>> idx = x.argsort() >>> idx > array([2, 0, 3, 1]) >>> x[idx] > array([ 0.12935131, 0.37412289, 0.42510212, 0.68248559]) > > Notice that the first element of idx is 2. That's because element x[2] > is the min of x. But that's not what we want. We want the first > element to be the rank of the first element of x. So we need to > shuffle idx around so that the order aligns with x. How do we do that? > We sort it! Unless I am mistaken, what you are doing here is inverting the permutation returned by the first argsort. The second argsort is an n log n method, though, and permutations can be inverted in linear time: ix = np.argsort(X) ixinv = np.zeros_like(ix) ixinv[ix] = np.arange(len(ix)) This works because if ix is a permutation and ixinv is its inverse, A = B[ix] is the same as A[ixinv] = B This also means that you can often do without the actual inverse by moving the indexing operation to the other side of the equal sign. (Not in the OP's case, though.) I'll also add that if the OP needs the top m for 1>> idx.argsort() > array([1, 3, 0, 2]) > > The min value of x is x[2], that's why 2 is the first element of idx > which means that we want ranked(x) to contain a 0 at position 2 which > it does. > > Bah, it's all magic. > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From peridot.faceted at gmail.com Wed Apr 14 18:17:05 2010 From: peridot.faceted at gmail.com (Anne Archibald) Date: Wed, 14 Apr 2010 18:17:05 -0400 Subject: [Numpy-discussion] binomial coefficient, factorial In-Reply-To: <4BC6359F.50507@noaa.gov> References: <4BC6359F.50507@noaa.gov> Message-ID: On 14 April 2010 17:37, Christopher Barker wrote: > jah wrote: >> Is there any chance that a binomial coefficent and factorial function >> can make their way into NumPy? > > probably not -- numpy is over-populated already > >> I know these exist in Scipy, but I don't >> want to have to install SciPy just to have something so "basic". > > The problem is that everyone has a different "basic". What we really > need is an easier way for folks to use sub-packages of scipy. I've found > myself hand-extracting just what I need, too. Maybe I've been spoiled by using Linux, but my answer to this sort of thing is always "just install scipy already". If it's not already packaged for your operating system (e.g. Linux, OSX, or Windows), is it really difficult to compile it yourself? (If so, perhaps post a report to scipy-user and see if we can fix it so it isn't.) Anne > -Chris > > > > > -- > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From kwgoodman at gmail.com Wed Apr 14 18:39:20 2010 From: kwgoodman at gmail.com (Keith Goodman) Date: Wed, 14 Apr 2010 15:39:20 -0700 Subject: [Numpy-discussion] Find indices of largest elements In-Reply-To: References: <874ojedpzh.fsf@inspiron.ap.columbia.edu> <871vehescj.fsf@inspiron.ap.columbia.edu> Message-ID: On Wed, Apr 14, 2010 at 3:12 PM, Anne Archibald wrote: > On 14 April 2010 16:56, Keith Goodman wrote: >> On Wed, Apr 14, 2010 at 12:39 PM, Nikolaus Rath wrote: >>> Keith Goodman writes: >>>> On Wed, Apr 14, 2010 at 8:49 AM, Keith Goodman wrote: >>>>> On Wed, Apr 14, 2010 at 8:16 AM, Nikolaus Rath wrote: >>>>>> Hello, >>>>>> >>>>>> How do I best find out the indices of the largest x elements in an >>>>>> array? >>>>>> >>>>>> Example: >>>>>> >>>>>> a = [ [1,8,2], [2,1,3] ] >>>>>> magic_function(a, 2) == [ (0,1), (1,2) ] >>>>>> >>>>>> Since the largest 2 elements are at positions (0,1) and (1,2). >>>>> >>>>> Here's a quick way to rank the data if there are no ties and no NaNs: >>>> >>>> ...or if you need the indices in order: >>>> >>>>>> shape = (3,2) >>>>>> x = np.random.rand(*shape) >>>>>> x >>>> array([[ 0.52420123, ?0.43231286], >>>> ? ? ? ?[ 0.97995333, ?0.87416228], >>>> ? ? ? ?[ 0.71604075, ?0.66018382]]) >>>>>> r = x.reshape(-1).argsort().argsort() >>> >>> I don't understand why this works. Why do you call argsort() twice? >>> Doesn't that give you the indices of the sorted indices? >> >> It is confusing. Let's look at an example: >> >>>> x = np.random.rand(4) >>>> x >> ? array([ 0.37412289, ?0.68248559, ?0.12935131, ?0.42510212]) >> >> If we call argsort once we get the index that will sort x: >> >>>> idx = x.argsort() >>>> idx >> ? array([2, 0, 3, 1]) >>>> x[idx] >> ? array([ 0.12935131, ?0.37412289, ?0.42510212, ?0.68248559]) >> >> Notice that the first element of idx is 2. That's because element x[2] >> is the min of x. But that's not what we want. We want the first >> element to be the rank of the first element of x. So we need to >> shuffle idx around so that the order aligns with x. How do we do that? >> We sort it! > > Unless I am mistaken, what you are doing here is inverting the > permutation returned by the first argsort. The second argsort is an n > log n method, though, and permutations can be inverted in linear time: > > ix = np.argsort(X) > ixinv = np.zeros_like(ix) > ixinv[ix] = np.arange(len(ix)) > > This works because if ix is a permutation and ixinv is its inverse, > A = B[ix] > is the same as > A[ixinv] = B > This also means that you can often do without the actual inverse by > moving the indexing operation to the other side of the equal sign. > (Not in the OP's case, though.) That is very nice. And very fast for large arrays: >> x = np.random.rand(4) >> timeit idx = x.argsort().argsort() 1000000 loops, best of 3: 1.45 us per loop >> timeit idx = x.argsort(); idxinv = np.zeros_like(idx); idxinv[idx] = np.arange(len(idx)) 100000 loops, best of 3: 9.52 us per loop >> x = np.random.rand(1000) >> timeit idx = x.argsort().argsort() 10000 loops, best of 3: 112 us per loop >> timeit idx = x.argsort(); idxinv = np.zeros_like(idx); idxinv[idx] = np.arange(len(idx)) 10000 loops, best of 3: 82.9 us per loop >> x = np.random.rand(100000) >> timeit idx = x.argsort().argsort() 10 loops, best of 3: 20.4 ms per loop >> timeit idx = x.argsort(); idxinv = np.zeros_like(idx); idxinv[idx] = np.arange(len(idx)) 100 loops, best of 3: 13.2 ms per loop > I'll also add that if the OP needs the top m for 1 whole input array is not the most efficient algorithm; there are > priority-queue-based schemes that are asymptotically more efficient, > but none exists in numpy. Since numpy's sorting is quite fast, I > personally would just use the sorting. Partial sorting would find a lot of uses in the numpy community (like in median). From Chris.Barker at noaa.gov Wed Apr 14 18:50:30 2010 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 14 Apr 2010 15:50:30 -0700 Subject: [Numpy-discussion] binomial coefficient, factorial In-Reply-To: References: <4BC6359F.50507@noaa.gov> Message-ID: <4BC646B6.5000107@noaa.gov> Anne Archibald wrote: >> The problem is that everyone has a different "basic". What we really >> need is an easier way for folks to use sub-packages of scipy. I've found >> myself hand-extracting just what I need, too. > > Maybe I've been spoiled by using Linux, but my answer to this sort of > thing is always "just install scipy already". If it's not already > packaged for your operating system (e.g. Linux, OSX, or Windows), is > it really difficult to compile it yourself? yes, it is. I think the Windows builds are pretty good, but with the MS compiler mess, that can be a pain, too (64 bit issues have not been resolved, though I guess not for numpy either...). And it is certainly not trivial to do yourself. I suffer on OS-X: what with Apple's Python, python.org's one, fink, macports, etc, then you have OS-X version 10.4 - 10.6 to support, and both PPC and Intel, and all of this with Fortran code, it really is a mess. Also, I deliver stand-alone tools, built with py2exe and py2app, and really don't like to be delivering the entire pile of scipy when I do. So yes, it's still an issue. > (If so, perhaps post a > report to scipy-user and see if we can fix it so it isn't.) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From ralf.gommers at googlemail.com Wed Apr 14 20:48:01 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 15 Apr 2010 08:48:01 +0800 Subject: [Numpy-discussion] rc2 for NumPy 1.4.1 and Scipy 0.7.2 In-Reply-To: <336574F6-63FD-497F-BB78-5EB7BEC4D62B@llnl.gov> References: <336574F6-63FD-497F-BB78-5EB7BEC4D62B@llnl.gov> Message-ID: 2010/4/15 Charles ???? Doutriaux > Just downloaded this. > > On my mac 10.6, using python 2.6.5 i get: > Which download, what build command and what python/gcc versions? Looks like you're trying to build a 64-bit binary without passing in -arch x86_64 flags? Ralf > > Running from numpy source directory. > non-existing path in 'numpy/distutils': 'site.cfg' > F2PY Version 2 > blas_opt_info: > FOUND: > extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] > define_macros = [('NO_ATLAS_INFO', 3)] > extra_compile_args = ['-faltivec', > '-I/System/Library/Frameworks/vecLib.framework/Headers'] > > lapack_opt_info: > FOUND: > extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] > define_macros = [('NO_ATLAS_INFO', 3)] > extra_compile_args = ['-faltivec'] > > running build > running config_cc > unifing config_cc, config, build_clib, build_ext, build commands --compiler > options > running config_fc > unifing config_fc, config, build_clib, build_ext, build commands > --fcompiler options > running build_src > build_src > building py_modules sources > creating build > creating build/src.macosx-10.3-fat-2.6 > creating build/src.macosx-10.3-fat-2.6/numpy > creating build/src.macosx-10.3-fat-2.6/numpy/distutils > building library "npymath" sources > customize NAGFCompiler > Could not locate executable f95 > customize AbsoftFCompiler > Could not locate executable f90 > Could not locate executable f77 > customize IBMFCompiler > Could not locate executable xlf90 > Could not locate executable xlf > customize IntelFCompiler > Could not locate executable ifort > Could not locate executable ifc > customize GnuFCompiler > Could not locate executable g77 > customize Gnu95FCompiler > Found executable /usr/local/bin/gfortran > /Users/doutriaux1/Desktop/numpy-1.4.1rc2/numpy/distutils/fcompiler/gnu.py:125: > UserWarning: Env. variable MACOSX_DEPLOYMENT_TARGET set to 10.3 > warnings.warn(s) > customize Gnu95FCompiler > customize Gnu95FCompiler using config > C compiler: gcc -arch ppc -arch i386 -isysroot / -fno-strict-aliasing > -DNDEBUG > > compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core > -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath > -Inumpy/core/include -I/lgm/cdat/trunk/include/python2.6 -c' > gcc: _configtest.c > gcc _configtest.o -o _configtest > ld: warning: in _configtest.o, missing required architecture x86_64 in file > Undefined symbols: > "_main", referenced from: > __start in crt1.o > ld: symbol(s) not found > collect2: ld returned 1 exit status > ld: warning: in _configtest.o, missing required architecture x86_64 in file > Undefined symbols: > "_main", referenced from: > __start in crt1.o > ld: symbol(s) not found > collect2: ld returned 1 exit status > failure. > removing: _configtest.c _configtest.o > Traceback (most recent call last): > File "setup.py", line 187, in > setup_package() > File "setup.py", line 180, in setup_package > configuration=configuration ) > File "/Users/doutriaux1/Desktop/numpy-1.4.1rc2/numpy/distutils/core.py", > line 186, in setup > return old_setup(**new_attr) > File "/lgm/cdat/trunk/lib/python2.6/distutils/core.py", line 152, in > setup > dist.run_commands() > File "/lgm/cdat/trunk/lib/python2.6/distutils/dist.py", line 975, in > run_commands > self.run_command(cmd) > File "/lgm/cdat/trunk/lib/python2.6/distutils/dist.py", line 995, in > run_command > cmd_obj.run() > File > "/Users/doutriaux1/Desktop/numpy-1.4.1rc2/numpy/distutils/command/build.py", > line 37, in run > old_build.run(self) > File "/lgm/cdat/trunk/lib/python2.6/distutils/command/build.py", line > 134, in run > self.run_command(cmd_name) > File "/lgm/cdat/trunk/lib/python2.6/distutils/cmd.py", line 333, in > run_command > self.distribution.run_command(command) > File "/lgm/cdat/trunk/lib/python2.6/distutils/dist.py", line 995, in > run_command > cmd_obj.run() > File > "/Users/doutriaux1/Desktop/numpy-1.4.1rc2/numpy/distutils/command/build_src.py", > line 152, in run > self.build_sources() > File > "/Users/doutriaux1/Desktop/numpy-1.4.1rc2/numpy/distutils/command/build_src.py", > line 163, in build_sources > self.build_library_sources(*libname_info) > File > "/Users/doutriaux1/Desktop/numpy-1.4.1rc2/numpy/distutils/command/build_src.py", > line 298, in build_library_sources > sources = self.generate_sources(sources, (lib_name, build_info)) > File > "/Users/doutriaux1/Desktop/numpy-1.4.1rc2/numpy/distutils/command/build_src.py", > line 385, in generate_sources > source = func(extension, build_dir) > File "numpy/core/setup.py", line 657, in get_mathlib_info > raise RuntimeError("Broken toolchain: cannot link a simple C program") > RuntimeError: Broken toolchain: cannot link a simple C program > > On Apr 11, 2010, at 2:09 AM, Ralf Gommers wrote: > > Hi, > > I am pleased to announce the second release candidate of both Scipy 0.7.2 > and NumPy 1.4.1, please test them. > > The issues reported with rc1 should be fixed, and for NumPy there are now > Python 2.5 binaries as well. For SciPy there will be no 2.5 binaries - > because 0.7.x is built against NumPy 1.2 it does not work on OS X 10.6, and > on Windows I see some incomprehensible build failures. > > Binaries, sources and release notes can be found at > http://*www.*filefactory.com/f/4452c50056df8bba/ (apologies for the flashy > ads again). > > Cheers, > Ralf > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://*mail.scipy.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Apr 14 23:55:16 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 14 Apr 2010 21:55:16 -0600 Subject: [Numpy-discussion] Find indices of largest elements In-Reply-To: References: <874ojedpzh.fsf@inspiron.ap.columbia.edu> <871vehescj.fsf@inspiron.ap.columbia.edu> Message-ID: On Wed, Apr 14, 2010 at 4:39 PM, Keith Goodman wrote: > On Wed, Apr 14, 2010 at 3:12 PM, Anne Archibald > wrote: > > On 14 April 2010 16:56, Keith Goodman wrote: > >> On Wed, Apr 14, 2010 at 12:39 PM, Nikolaus Rath > wrote: > >>> Keith Goodman writes: > >>>> On Wed, Apr 14, 2010 at 8:49 AM, Keith Goodman > wrote: > >>>>> On Wed, Apr 14, 2010 at 8:16 AM, Nikolaus Rath > wrote: > >>>>>> Hello, > >>>>>> > >>>>>> How do I best find out the indices of the largest x elements in an > >>>>>> array? > >>>>>> > >>>>>> Example: > >>>>>> > >>>>>> a = [ [1,8,2], [2,1,3] ] > >>>>>> magic_function(a, 2) == [ (0,1), (1,2) ] > >>>>>> > >>>>>> Since the largest 2 elements are at positions (0,1) and (1,2). > >>>>> > >>>>> Here's a quick way to rank the data if there are no ties and no NaNs: > >>>> > >>>> ...or if you need the indices in order: > >>>> > >>>>>> shape = (3,2) > >>>>>> x = np.random.rand(*shape) > >>>>>> x > >>>> array([[ 0.52420123, 0.43231286], > >>>> [ 0.97995333, 0.87416228], > >>>> [ 0.71604075, 0.66018382]]) > >>>>>> r = x.reshape(-1).argsort().argsort() > >>> > >>> I don't understand why this works. Why do you call argsort() twice? > >>> Doesn't that give you the indices of the sorted indices? > >> > >> It is confusing. Let's look at an example: > >> > >>>> x = np.random.rand(4) > >>>> x > >> array([ 0.37412289, 0.68248559, 0.12935131, 0.42510212]) > >> > >> If we call argsort once we get the index that will sort x: > >> > >>>> idx = x.argsort() > >>>> idx > >> array([2, 0, 3, 1]) > >>>> x[idx] > >> array([ 0.12935131, 0.37412289, 0.42510212, 0.68248559]) > >> > >> Notice that the first element of idx is 2. That's because element x[2] > >> is the min of x. But that's not what we want. We want the first > >> element to be the rank of the first element of x. So we need to > >> shuffle idx around so that the order aligns with x. How do we do that? > >> We sort it! > > > > Unless I am mistaken, what you are doing here is inverting the > > permutation returned by the first argsort. The second argsort is an n > > log n method, though, and permutations can be inverted in linear time: > > > > ix = np.argsort(X) > > ixinv = np.zeros_like(ix) > > ixinv[ix] = np.arange(len(ix)) > > > > This works because if ix is a permutation and ixinv is its inverse, > > A = B[ix] > > is the same as > > A[ixinv] = B > > This also means that you can often do without the actual inverse by > > moving the indexing operation to the other side of the equal sign. > > (Not in the OP's case, though.) > > That is very nice. And very fast for large arrays: > > >> x = np.random.rand(4) > >> timeit idx = x.argsort().argsort() > 1000000 loops, best of 3: 1.45 us per loop > >> timeit idx = x.argsort(); idxinv = np.zeros_like(idx); idxinv[idx] = > np.arange(len(idx)) > 100000 loops, best of 3: 9.52 us per loop > > >> x = np.random.rand(1000) > >> timeit idx = x.argsort().argsort() > 10000 loops, best of 3: 112 us per loop > >> timeit idx = x.argsort(); idxinv = np.zeros_like(idx); idxinv[idx] = > np.arange(len(idx)) > 10000 loops, best of 3: 82.9 us per loop > > >> x = np.random.rand(100000) > >> timeit idx = x.argsort().argsort() > 10 loops, best of 3: 20.4 ms per loop > >> timeit idx = x.argsort(); idxinv = np.zeros_like(idx); idxinv[idx] = > np.arange(len(idx)) > 100 loops, best of 3: 13.2 ms per loop > > > I'll also add that if the OP needs the top m for 1 > whole input array is not the most efficient algorithm; there are > > priority-queue-based schemes that are asymptotically more efficient, > > but none exists in numpy. Since numpy's sorting is quite fast, I > > personally would just use the sorting. > > Partial sorting would find a lot of uses in the numpy community (like > in median). > Thinking about it... along with a lot of other things. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From bergstrj at iro.umontreal.ca Thu Apr 15 00:32:11 2010 From: bergstrj at iro.umontreal.ca (James Bergstra) Date: Thu, 15 Apr 2010 00:32:11 -0400 Subject: [Numpy-discussion] Math Library In-Reply-To: References: Message-ID: On Tue, Apr 6, 2010 at 10:08 AM, David Cournapeau wrote: > could be put out of multiarray proper. Also, exposing an API for > things like fancy indexing would be very useful, but I don't know if > it even makes sense - I think a pure python implementation of fancy > indexing as a reference would be very useful for array-like classes (I > am thinking about scipy.sparse, for example). > > Unfortunately, I won't be able to help much in the near future (except > maybe for the fancy indexing as this could be useful for my job), > > David Hi, I am also interested in a pure python implementation of fancy indexing... at least the array-type fancy indexing, if not the boolean kind. If someone knows of an implementation please let me know. I'll email the list again if I make any serious progress on it. James -- http://www-etud.iro.umontreal.ca/~bergstrj From pete at shinners.org Thu Apr 15 02:30:11 2010 From: pete at shinners.org (Peter Shinners) Date: Wed, 14 Apr 2010 23:30:11 -0700 Subject: [Numpy-discussion] Need faster equivalent to digitize Message-ID: <4BC6B273.9070407@shinners.org> I am using digitize to create a list of indices. This is giving me exactly what I want, but it's terribly slow. Digitize is obviously not the tool I want for this case, but what numpy alternative do I have? I have an array like np.array((4, 3, 3)). I need to create an index array with each index repeated by the its value: np.array((0, 0, 0, 0, 1, 1, 1, 2, 2, 2)). >>> a = np.array((4, 3, 3)) >>> b = np.arange(np.sum(a)) >>> c = np.digitize(b, a) >>> print c [0 0 0 0 1 1 1 2 2 2] On an array where a.size==65536 and sum(a)==65536 this is taking over 6 seconds to compute. As a comparison, using a Python list solution runs in 0.08 seconds. That is plenty fast, but I would guess there is a faster Numpy solution that does not require a dynamically growing container of PyObjects ? >>> a = np.array((4, 3, 3)) >>> c = [] >>> for i, v in enumerate(a): ... c.extend([i] * v) From nadavh at visionsense.com Thu Apr 15 02:34:36 2010 From: nadavh at visionsense.com (Nadav Horesh) Date: Thu, 15 Apr 2010 09:34:36 +0300 Subject: [Numpy-discussion] Need faster equivalent to digitize References: <4BC6B273.9070407@shinners.org> Message-ID: <710F2847B0018641891D9A21602763605AD3B2@ex3.envision.co.il> import numpy as N N.repeat(N.arange(len(a)), a) Nadav -----Original Message----- From: numpy-discussion-bounces at scipy.org on behalf of Peter Shinners Sent: Thu 15-Apr-10 08:30 To: Discussion of Numerical Python Subject: [Numpy-discussion] Need faster equivalent to digitize I am using digitize to create a list of indices. This is giving me exactly what I want, but it's terribly slow. Digitize is obviously not the tool I want for this case, but what numpy alternative do I have? I have an array like np.array((4, 3, 3)). I need to create an index array with each index repeated by the its value: np.array((0, 0, 0, 0, 1, 1, 1, 2, 2, 2)). >>> a = np.array((4, 3, 3)) >>> b = np.arange(np.sum(a)) >>> c = np.digitize(b, a) >>> print c [0 0 0 0 1 1 1 2 2 2] On an array where a.size==65536 and sum(a)==65536 this is taking over 6 seconds to compute. As a comparison, using a Python list solution runs in 0.08 seconds. That is plenty fast, but I would guess there is a faster Numpy solution that does not require a dynamically growing container of PyObjects ? >>> a = np.array((4, 3, 3)) >>> c = [] >>> for i, v in enumerate(a): ... c.extend([i] * v) _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 3282 bytes Desc: not available URL: From pete at shinners.org Thu Apr 15 02:45:31 2010 From: pete at shinners.org (Peter Shinners) Date: Wed, 14 Apr 2010 23:45:31 -0700 Subject: [Numpy-discussion] Need faster equivalent to digitize In-Reply-To: <710F2847B0018641891D9A21602763605AD3B2@ex3.envision.co.il> References: <4BC6B273.9070407@shinners.org> <710F2847B0018641891D9A21602763605AD3B2@ex3.envision.co.il> Message-ID: <4BC6B60B.9090004@shinners.org> On 04/14/2010 11:34 PM, Nadav Horesh wrote: > import numpy as N > N.repeat(N.arange(len(a)), a) > > Nadav > > -----Original Message----- > From: numpy-discussion-bounces at scipy.org on behalf of Peter Shinners > Sent: Thu 15-Apr-10 08:30 > To: Discussion of Numerical Python > Subject: [Numpy-discussion] Need faster equivalent to digitize > > I am using digitize to create a list of indices. This is giving me > exactly what I want, but it's terribly slow. Digitize is obviously not > the tool I want for this case, but what numpy alternative do I have? > > I have an array like np.array((4, 3, 3)). I need to create an index > array with each index repeated by the its value: np.array((0, 0, 0, 0, > 1, 1, 1, 2, 2, 2)). > > >>> a = np.array((4, 3, 3)) > >>> b = np.arange(np.sum(a)) > >>> c = np.digitize(b, a) > >>> print c > [0 0 0 0 1 1 1 2 2 2] > > On an array where a.size==65536 and sum(a)==65536 this is taking over 6 > seconds to compute. As a comparison, using a Python list solution runs > in 0.08 seconds. That is plenty fast, but I would guess there is a > faster Numpy solution that does not require a dynamically growing > container of PyObjects ? > > >>> a = np.array((4, 3, 3)) > >>> c = [] > >>> for i, v in enumerate(a): > ... c.extend([i] * v) > Excellent. The Numpy version is a bit faster, and I prefer having an ndarray as the end result. From michel.dupront at hotmail.fr Thu Apr 15 05:53:44 2010 From: michel.dupront at hotmail.fr (Michel Dupront) Date: Thu, 15 Apr 2010 11:53:44 +0200 Subject: [Numpy-discussion] use of %extend Message-ID: Hello, With the following example, given in the documentation: struct Vector { double x,y,z; }; %extend Vector { Vector __add__(Vector *other) { Vector v; v.x = $self->x + other->x; v.y = $self->y + other->y; v.z = $self->z + other->z; return v; } }; one can do in python: >>> import example >>> v = example.Vector(2,3,4) >>> w = example.Vector(10,11,12) >>> print v+w Vector(12,14,16) >>> There is something more that I would like to do but I don't know how. I would like to be able to do: >>>> v1 = v + 2 >>>> v2 = 2 + v where v is a vector and 2 a number that would be added to each component of the vector. I would really appreciate any help on that. Many thanks in advance. _________________________________________________________________ Hotmail arrive sur votre t?l?phone ! Compatible Iphone, Windows Phone, Blackberry, ? http://www.messengersurvotremobile.com/?d=Hotmail -------------- next part -------------- An HTML attachment was scrubbed... URL: From seb.haase at gmail.com Thu Apr 15 07:24:10 2010 From: seb.haase at gmail.com (Sebastian Haase) Date: Thu, 15 Apr 2010 13:24:10 +0200 Subject: [Numpy-discussion] use of %extend In-Reply-To: References: Message-ID: did you mean to send this to the SWIG list !? -S. On Thu, Apr 15, 2010 at 11:53 AM, Michel Dupront wrote: > Hello, > > With the following example, given in the documentation: > > struct Vector { > ?? double x,y,z; > }; > %extend Vector { > ??? Vector __add__(Vector *other) { > ???????? Vector v; > ???????? v.x = $self->x + other->x; > ???????? v.y = $self->y + other->y; > ???????? v.z = $self->z + other->z; > ???????? return v; > ??? } > }; > > one can do in python: >>>> import example >>>> v = example.Vector(2,3,4) >>>> w = example.Vector(10,11,12) >>>> print v+w > Vector(12,14,16) >>>> > > There is something more that I would like to do but I don't > know how. I would like to be able to do: >>>>> v1 = v + 2 >>>>> v2 = 2 + v > where v is a vector and 2 a number that would be added to each component > of the vector. > I would really appreciate any help on that. > Many thanks in advance. > > > ________________________________ > Acheter en ligne en toute s?curit? ? Internet Explorer 8 vous prot?ge > gratuitement ! > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From michel.dupront at hotmail.fr Thu Apr 15 07:26:51 2010 From: michel.dupront at hotmail.fr (Michel Dupront) Date: Thu, 15 Apr 2010 13:26:51 +0200 Subject: [Numpy-discussion] use of %extend In-Reply-To: References: , Message-ID: Yes. I am really sorry about that. > From: seb.haase at gmail.com > Date: Thu, 15 Apr 2010 13:24:10 +0200 > To: numpy-discussion at scipy.org > Subject: Re: [Numpy-discussion] use of %extend > > did you mean to send this to the SWIG list !? > -S. > > > On Thu, Apr 15, 2010 at 11:53 AM, Michel Dupront > wrote: > > Hello, > > > > With the following example, given in the documentation: > > > > struct Vector { > > double x,y,z; > > }; > > %extend Vector { > > Vector __add__(Vector *other) { > > Vector v; > > v.x = $self->x + other->x; > > v.y = $self->y + other->y; > > v.z = $self->z + other->z; > > return v; > > } > > }; > > > > one can do in python: > >>>> import example > >>>> v = example.Vector(2,3,4) > >>>> w = example.Vector(10,11,12) > >>>> print v+w > > Vector(12,14,16) > >>>> > > > > There is something more that I would like to do but I don't > > know how. I would like to be able to do: > >>>>> v1 = v + 2 > >>>>> v2 = 2 + v > > where v is a vector and 2 a number that would be added to each component > > of the vector. > > I would really appreciate any help on that. > > Many thanks in advance. > > > > > > ________________________________ > > Acheter en ligne en toute s?curit? ? Internet Explorer 8 vous prot?ge > > gratuitement ! > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion _________________________________________________________________ Hotmail et MSN dans la poche? HOTMAIL et MSN sont dispo gratuitement sur votre t?l?phone! http://www.messengersurvotremobile.com/?d=Hotmail -------------- next part -------------- An HTML attachment was scrubbed... URL: From kwgoodman at gmail.com Thu Apr 15 11:02:00 2010 From: kwgoodman at gmail.com (Keith Goodman) Date: Thu, 15 Apr 2010 08:02:00 -0700 Subject: [Numpy-discussion] Math Library In-Reply-To: References: Message-ID: On Wed, Apr 14, 2010 at 9:32 PM, James Bergstra wrote: > On Tue, Apr 6, 2010 at 10:08 AM, David Cournapeau wrote: >> could be put out of multiarray proper. Also, exposing an API for >> things like fancy indexing would be very useful, but I don't know if >> it even makes sense - I think a pure python implementation of fancy >> indexing as a reference would be very useful for array-like classes (I >> am thinking about scipy.sparse, for example). >> >> Unfortunately, I won't be able to help much in the near future (except >> maybe for the fancy indexing as this could be useful for my job), >> >> David > > Hi, I am also interested in a pure python implementation of fancy > indexing... at least the array-type fancy indexing, if not the boolean > kind. ?If someone knows of an implementation please let me know. ?I'll > email the list again if I make any serious progress on it. Until then, I'm interested in a function that takes an index as input and returns True if fancy indexing is requested, False otherwise. That might be useful for your function as well. I'd use it so that my __getitem__ method can at least say that fancy indexing is not supported. From doutriaux1 at llnl.gov Thu Apr 15 14:12:15 2010 From: doutriaux1 at llnl.gov (=?utf-8?Q?Charles_=D8=B3=D9=85=D9=8A=D8=B1_Doutriaux?=) Date: Thu, 15 Apr 2010 11:12:15 -0700 Subject: [Numpy-discussion] rc2 for NumPy 1.4.1 and Scipy 0.7.2 In-Reply-To: References: <336574F6-63FD-497F-BB78-5EB7BEC4D62B@llnl.gov> Message-ID: Hi There, 1.4.1rc2 gcc 4.2.1 And yes I'm trying to build a 32bit python because a lot of externals seem to be breaking with the 64bit stuff... But I'm giving it another shot building 64bit C. On Apr 14, 2010, at 5:48 PM, Ralf Gommers wrote: > > > 2010/4/15 Charles ???? Doutriaux > Just downloaded this. > > On my mac 10.6, using python 2.6.5 i get: > > Which download, what build command and what python/gcc versions? > > Looks like you're trying to build a 64-bit binary without passing in -arch x86_64 flags? > > Ralf > > > > > Running from numpy source directory. > non-existing path in 'numpy/distutils': 'site.cfg' > F2PY Version 2 > blas_opt_info: > FOUND: > extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] > define_macros = [('NO_ATLAS_INFO', 3)] > extra_compile_args = ['-faltivec', '-I/System/Library/Frameworks/vecLib.framework/Headers'] > > lapack_opt_info: > FOUND: > extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] > define_macros = [('NO_ATLAS_INFO', 3)] > extra_compile_args = ['-faltivec'] > > running build > running config_cc > unifing config_cc, config, build_clib, build_ext, build commands --compiler options > running config_fc > unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options > running build_src > build_src > building py_modules sources > creating build > creating build/src.macosx-10.3-fat-2.6 > creating build/src.macosx-10.3-fat-2.6/numpy > creating build/src.macosx-10.3-fat-2.6/numpy/distutils > building library "npymath" sources > customize NAGFCompiler > Could not locate executable f95 > customize AbsoftFCompiler > Could not locate executable f90 > Could not locate executable f77 > customize IBMFCompiler > Could not locate executable xlf90 > Could not locate executable xlf > customize IntelFCompiler > Could not locate executable ifort > Could not locate executable ifc > customize GnuFCompiler > Could not locate executable g77 > customize Gnu95FCompiler > Found executable /usr/local/bin/gfortran > /Users/doutriaux1/Desktop/numpy-1.4.1rc2/numpy/distutils/fcompiler/gnu.py:125: UserWarning: Env. variable MACOSX_DEPLOYMENT_TARGET set to 10.3 > warnings.warn(s) > customize Gnu95FCompiler > customize Gnu95FCompiler using config > C compiler: gcc -arch ppc -arch i386 -isysroot / -fno-strict-aliasing -DNDEBUG > > compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/include -I/lgm/cdat/trunk/include/python2.6 -c' > gcc: _configtest.c > gcc _configtest.o -o _configtest > ld: warning: in _configtest.o, missing required architecture x86_64 in file > Undefined symbols: > "_main", referenced from: > __start in crt1.o > ld: symbol(s) not found > collect2: ld returned 1 exit status > ld: warning: in _configtest.o, missing required architecture x86_64 in file > Undefined symbols: > "_main", referenced from: > __start in crt1.o > ld: symbol(s) not found > collect2: ld returned 1 exit status > failure. > removing: _configtest.c _configtest.o > Traceback (most recent call last): > File "setup.py", line 187, in > setup_package() > File "setup.py", line 180, in setup_package > configuration=configuration ) > File "/Users/doutriaux1/Desktop/numpy-1.4.1rc2/numpy/distutils/core.py", line 186, in setup > return old_setup(**new_attr) > File "/lgm/cdat/trunk/lib/python2.6/distutils/core.py", line 152, in setup > dist.run_commands() > File "/lgm/cdat/trunk/lib/python2.6/distutils/dist.py", line 975, in run_commands > self.run_command(cmd) > File "/lgm/cdat/trunk/lib/python2.6/distutils/dist.py", line 995, in run_command > cmd_obj.run() > File "/Users/doutriaux1/Desktop/numpy-1.4.1rc2/numpy/distutils/command/build.py", line 37, in run > old_build.run(self) > File "/lgm/cdat/trunk/lib/python2.6/distutils/command/build.py", line 134, in run > self.run_command(cmd_name) > File "/lgm/cdat/trunk/lib/python2.6/distutils/cmd.py", line 333, in run_command > self.distribution.run_command(command) > File "/lgm/cdat/trunk/lib/python2.6/distutils/dist.py", line 995, in run_command > cmd_obj.run() > File "/Users/doutriaux1/Desktop/numpy-1.4.1rc2/numpy/distutils/command/build_src.py", line 152, in run > self.build_sources() > File "/Users/doutriaux1/Desktop/numpy-1.4.1rc2/numpy/distutils/command/build_src.py", line 163, in build_sources > self.build_library_sources(*libname_info) > File "/Users/doutriaux1/Desktop/numpy-1.4.1rc2/numpy/distutils/command/build_src.py", line 298, in build_library_sources > sources = self.generate_sources(sources, (lib_name, build_info)) > File "/Users/doutriaux1/Desktop/numpy-1.4.1rc2/numpy/distutils/command/build_src.py", line 385, in generate_sources > source = func(extension, build_dir) > File "numpy/core/setup.py", line 657, in get_mathlib_info > raise RuntimeError("Broken toolchain: cannot link a simple C program") > RuntimeError: Broken toolchain: cannot link a simple C program > > On Apr 11, 2010, at 2:09 AM, Ralf Gommers wrote: > >> Hi, >> >> I am pleased to announce the second release candidate of both Scipy 0.7.2 and NumPy 1.4.1, please test them. >> >> The issues reported with rc1 should be fixed, and for NumPy there are now Python 2.5 binaries as well. For SciPy there will be no 2.5 binaries - because 0.7.x is built against NumPy 1.2 it does not work on OS X 10.6, and on Windows I see some incomprehensible build failures. >> >> Binaries, sources and release notes can be found at http://**www.**filefactory.com/f/4452c50056df8bba/ (apologies for the flashy ads again). >> >> Cheers, >> Ralf >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://**mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://*mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://*mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndbecker2 at gmail.com Thu Apr 15 14:42:46 2010 From: ndbecker2 at gmail.com (Neal Becker) Date: Thu, 15 Apr 2010 14:42:46 -0400 Subject: [Numpy-discussion] simple rectangular grid Message-ID: Is there a simple way to create a rectangular grid, like this loopy code: (This is for a QAM256 constellation, btw): import numpy as np u = np.arange (-7.5, 8) const = np.empty ((16,16), dtype=complex) for ix in range (16): for iy in range (16): const[ix, iy] = u[ix] + 1j*u[iy] From josef.pktd at gmail.com Thu Apr 15 14:48:05 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 15 Apr 2010 14:48:05 -0400 Subject: [Numpy-discussion] simple rectangular grid In-Reply-To: References: Message-ID: On Thu, Apr 15, 2010 at 2:42 PM, Neal Becker wrote: > Is there a simple way to create a rectangular grid, like this loopy code: > (This is for a QAM256 constellation, btw): > > import numpy as np > > u = np.arange (-7.5, 8) > > const = np.empty ((16,16), dtype=complex) > > for ix in range (16): > ? ?for iy in range (16): > ? ? ? ?const[ix, iy] = u[ix] + 1j*u[iy] >>> const2 = u[:,None] + 1j*u >>> (const == const2).all() True Josef > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From robert.kern at gmail.com Thu Apr 15 14:49:09 2010 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 15 Apr 2010 13:49:09 -0500 Subject: [Numpy-discussion] simple rectangular grid In-Reply-To: References: Message-ID: On Thu, Apr 15, 2010 at 13:42, Neal Becker wrote: > Is there a simple way to create a rectangular grid, like this loopy code: > (This is for a QAM256 constellation, btw): > > import numpy as np > > u = np.arange (-7.5, 8) > > const = np.empty ((16,16), dtype=complex) > > for ix in range (16): > ? ?for iy in range (16): > ? ? ? ?const[ix, iy] = u[ix] + 1j*u[iy] In [7]: real, imag = np.mgrid[-7.5:7.5:16j, -7.5:7.5:16j] In [8]: const = real + 1j * imag -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From Nikolaus at rath.org Thu Apr 15 15:45:28 2010 From: Nikolaus at rath.org (Nikolaus Rath) Date: Thu, 15 Apr 2010 15:45:28 -0400 Subject: [Numpy-discussion] Find indices of largest elements References: <874ojedpzh.fsf@inspiron.ap.columbia.edu> Message-ID: <8739yw7b53.fsf@inspiron.ap.columbia.edu> eat writes: >> How do I best find out the indices of the largest x elements in an >> array? > > Just > a= np.asarray([[1, 8, 2], [2, 1, 3]]) > print np.where((a.T== a.max(axis= 1)).T) > > However, if any row contains more than 1 max entity, above will fail. Please > let me to know if that's relevant for you. Not quite, because I'm interested in the n largest values over all elements, not the largest element in each row or column. But Keith's solution seems to work fine, even though I'm still struggling to understand what's going on there :-). Best, -Nikolaus -- ?Time flies like an arrow, fruit flies like a Banana.? PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C From Nikolaus at rath.org Thu Apr 15 15:41:29 2010 From: Nikolaus at rath.org (Nikolaus Rath) Date: Thu, 15 Apr 2010 15:41:29 -0400 Subject: [Numpy-discussion] Find indices of largest elements References: <874ojedpzh.fsf@inspiron.ap.columbia.edu> <871vehescj.fsf@inspiron.ap.columbia.edu> Message-ID: <87633s7bbq.fsf@inspiron.ap.columbia.edu> Keith Goodman writes: > On Wed, Apr 14, 2010 at 12:39 PM, Nikolaus Rath wrote: >> Keith Goodman writes: >>> On Wed, Apr 14, 2010 at 8:49 AM, Keith Goodman wrote: >>>> On Wed, Apr 14, 2010 at 8:16 AM, Nikolaus Rath wrote: >>>>> Hello, >>>>> >>>>> How do I best find out the indices of the largest x elements in an >>>>> array? >>>>> >>>>> Example: >>>>> >>>>> a = [ [1,8,2], [2,1,3] ] >>>>> magic_function(a, 2) == [ (0,1), (1,2) ] >>>>> >>>>> Since the largest 2 elements are at positions (0,1) and (1,2). >>>> >>>> Here's a quick way to rank the data if there are no ties and no NaNs: >>> >>> ...or if you need the indices in order: >>> >>>>> shape = (3,2) >>>>> x = np.random.rand(*shape) >>>>> x >>> array([[ 0.52420123, ?0.43231286], >>> ? ? ? ?[ 0.97995333, ?0.87416228], >>> ? ? ? ?[ 0.71604075, ?0.66018382]]) >>>>> r = x.reshape(-1).argsort().argsort() >> >> I don't understand why this works. Why do you call argsort() twice? >> Doesn't that give you the indices of the sorted indices? > > It is confusing. Let's look at an example: > >>> x = np.random.rand(4) >>> x > array([ 0.37412289, 0.68248559, 0.12935131, 0.42510212]) > > If we call argsort once we get the index that will sort x: > >>> idx = x.argsort() >>> idx > array([2, 0, 3, 1]) >>> x[idx] > array([ 0.12935131, 0.37412289, 0.42510212, 0.68248559]) > > Notice that the first element of idx is 2. That's because element x[2] > is the min of x. But that's not what we want. I think that's exactly what I want, the index of the smallest element. It also seems to work: In [3]: x = np.random.rand(3,3) In [4]: x Out[4]: array([[ 0.49064281, 0.54989584, 0.05319183], [ 0.50510206, 0.39683101, 0.22801874], [ 0.04595144, 0.3329171 , 0.61156205]]) In [5]: idx = x.reshape(-1).argsort() In [6]: [ np.unravel_index(i, x.shape) for i in idx[-3:] ] Out[6]: [(1, 0), (0, 1), (2, 2)] So why the additional complication with the second argsort? I just don't get it... > We want the first > element to be the rank of the first element of x. I'm not quite sure why we want that...? Best, -Nikolaus -- ?Time flies like an arrow, fruit flies like a Banana.? PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C From kwgoodman at gmail.com Thu Apr 15 16:48:32 2010 From: kwgoodman at gmail.com (Keith Goodman) Date: Thu, 15 Apr 2010 13:48:32 -0700 Subject: [Numpy-discussion] Find indices of largest elements In-Reply-To: <87633s7bbq.fsf@inspiron.ap.columbia.edu> References: <874ojedpzh.fsf@inspiron.ap.columbia.edu> <871vehescj.fsf@inspiron.ap.columbia.edu> <87633s7bbq.fsf@inspiron.ap.columbia.edu> Message-ID: On Thu, Apr 15, 2010 at 12:41 PM, Nikolaus Rath wrote: > Keith Goodman writes: >> On Wed, Apr 14, 2010 at 12:39 PM, Nikolaus Rath wrote: >>> Keith Goodman writes: >>>> On Wed, Apr 14, 2010 at 8:49 AM, Keith Goodman wrote: >>>>> On Wed, Apr 14, 2010 at 8:16 AM, Nikolaus Rath wrote: >>>>>> Hello, >>>>>> >>>>>> How do I best find out the indices of the largest x elements in an >>>>>> array? >>>>>> >>>>>> Example: >>>>>> >>>>>> a = [ [1,8,2], [2,1,3] ] >>>>>> magic_function(a, 2) == [ (0,1), (1,2) ] >>>>>> >>>>>> Since the largest 2 elements are at positions (0,1) and (1,2). >>>>> >>>>> Here's a quick way to rank the data if there are no ties and no NaNs: >>>> >>>> ...or if you need the indices in order: >>>> >>>>>> shape = (3,2) >>>>>> x = np.random.rand(*shape) >>>>>> x >>>> array([[ 0.52420123, ?0.43231286], >>>> ? ? ? ?[ 0.97995333, ?0.87416228], >>>> ? ? ? ?[ 0.71604075, ?0.66018382]]) >>>>>> r = x.reshape(-1).argsort().argsort() >>> >>> I don't understand why this works. Why do you call argsort() twice? >>> Doesn't that give you the indices of the sorted indices? >> >> It is confusing. Let's look at an example: >> >>>> x = np.random.rand(4) >>>> x >> ? ?array([ 0.37412289, ?0.68248559, ?0.12935131, ?0.42510212]) >> >> If we call argsort once we get the index that will sort x: >> >>>> idx = x.argsort() >>>> idx >> ? ?array([2, 0, 3, 1]) >>>> x[idx] >> ? ?array([ 0.12935131, ?0.37412289, ?0.42510212, ?0.68248559]) >> >> Notice that the first element of idx is 2. That's because element x[2] >> is the min of x. But that's not what we want. > > I think that's exactly what I want, the index of the smallest element. > It also seems to work: > > In [3]: x = np.random.rand(3,3) > In [4]: x > Out[4]: > array([[ 0.49064281, ?0.54989584, ?0.05319183], > ? ? ? [ 0.50510206, ?0.39683101, ?0.22801874], > ? ? ? [ 0.04595144, ?0.3329171 , ?0.61156205]]) > In [5]: idx = x.reshape(-1).argsort() > In [6]: [ np.unravel_index(i, x.shape) for i in idx[-3:] ] > Out[6]: [(1, 0), (0, 1), (2, 2)] Yes, you are right. My first thought was to approach the problem by ranking the data. But that is not needed here since the position in the argsorted index tells us the rank. I guess my approach was to rank first and then ask questions later. Well, at least we got to see Anne's fast ranking method. > > So why the additional complication with the second argsort? I just don't > get it... > >> We want the first >> element to be the rank of the first element of x. > > I'm not quite sure why we want that...? > > > Best, > > ? -Nikolaus > > -- > ??Time flies like an arrow, fruit flies like a Banana.? > > ?PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 ?02CF A9AD B7F8 AE4E 425C > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From e.antero.tammi at gmail.com Thu Apr 15 18:44:06 2010 From: e.antero.tammi at gmail.com (eat) Date: Thu, 15 Apr 2010 22:44:06 +0000 (UTC) Subject: [Numpy-discussion] Find indices of largest elements References: <874ojedpzh.fsf@inspiron.ap.columbia.edu> <8739yw7b53.fsf@inspiron.ap.columbia.edu> Message-ID: Nikolaus Rath rath.org> writes: [snip] > Not quite, because I'm interested in the n largest values over all > elements, not the largest element in each row or column. But Keith's > solution seems to work fine, even though I'm still struggling to > understand what's going on there . My bad. I just concentrated on your example, not the actual question. However, what's wrong with your above approach "[ np.unravel_index(i, x.shape) for i in idx[-3:] ]" ? Especially if your n largest elements are just a small fraction of all elements. # Note also the differencies a= np.asarray([[1, 8, 2], [2, 3, 3], [3, 4, 1]]) n= 3 # between print [np.unravel_index(ind, a.shape) for ind in np.argsort(a.ravel())[-n:]] # and print [np.where(val== a) for val in np.sort(a.ravel())[-n:]] Regards, eat > > Best, > > -Nikolaus > From antoinedelmotte at hotmail.com Thu Apr 15 19:33:33 2010 From: antoinedelmotte at hotmail.com (Antoine Delmotte) Date: Thu, 15 Apr 2010 23:33:33 +0000 Subject: [Numpy-discussion] Numpy+SWIG with arrays in input and output of a function Message-ID: Dear Numpy and SWIG users, I am currently trying to use SWIG and Numpy to launch C++ codes from python. My C++ code takes an array as an input (as well as integers, but this will be my next problem...) and returns a different array (with different dimensions). I have managed to make very simple codes taking an array in input OR returning an array in output. However, I have not managed to develop a code with an array in input and returning a different array in output. Here is one of my attempts, which simply copies a vector (I know I could use inplace for this example, but this is not what I will need later since I want to return a different kind of array than what I have as an input): /* matrix.c */ #include #include void vecOut(double* vec,int m,double* vec2,int n){ int i,j; double sum =0.0; for(i=0;i From ralf.gommers at googlemail.com Thu Apr 15 19:53:45 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Fri, 16 Apr 2010 07:53:45 +0800 Subject: [Numpy-discussion] rc2 for NumPy 1.4.1 and Scipy 0.7.2 In-Reply-To: References: <336574F6-63FD-497F-BB78-5EB7BEC4D62B@llnl.gov> Message-ID: 2010/4/16 Charles ???? Doutriaux > Hi There, > > 1.4.1rc2 > gcc 4.2.1 > > And yes I'm trying to build a 32bit python because a lot of externals seem > to be breaking with the 64bit stuff... > > But I'm giving it another shot building 64bit > For 64-bit the command is: $ LDFLAGS="-arch x86_64" FFLAGS="-arch x86_64" python setup.py install or equivalent with numscons. For 32-bit $ python setup.py install should work, but the compiler should be gcc-4.0 (when building against python.org python this happens automatically), even though your default compiler is gcc-4.2. Also, you need to have the 10.4SDK installed from XCode optional installs. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From kwgoodman at gmail.com Thu Apr 15 20:22:46 2010 From: kwgoodman at gmail.com (Keith Goodman) Date: Thu, 15 Apr 2010 17:22:46 -0700 Subject: [Numpy-discussion] Find indices of largest elements In-Reply-To: References: <874ojedpzh.fsf@inspiron.ap.columbia.edu> <871vehescj.fsf@inspiron.ap.columbia.edu> <87633s7bbq.fsf@inspiron.ap.columbia.edu> Message-ID: On Thu, Apr 15, 2010 at 1:48 PM, Keith Goodman wrote: > On Thu, Apr 15, 2010 at 12:41 PM, Nikolaus Rath wrote: >> Keith Goodman writes: >>> On Wed, Apr 14, 2010 at 12:39 PM, Nikolaus Rath wrote: >>>> Keith Goodman writes: >>>>> On Wed, Apr 14, 2010 at 8:49 AM, Keith Goodman wrote: >>>>>> On Wed, Apr 14, 2010 at 8:16 AM, Nikolaus Rath wrote: >>>>>>> Hello, >>>>>>> >>>>>>> How do I best find out the indices of the largest x elements in an >>>>>>> array? >>>>>>> >>>>>>> Example: >>>>>>> >>>>>>> a = [ [1,8,2], [2,1,3] ] >>>>>>> magic_function(a, 2) == [ (0,1), (1,2) ] >>>>>>> >>>>>>> Since the largest 2 elements are at positions (0,1) and (1,2). >>>>>> >>>>>> Here's a quick way to rank the data if there are no ties and no NaNs: >>>>> >>>>> ...or if you need the indices in order: >>>>> >>>>>>> shape = (3,2) >>>>>>> x = np.random.rand(*shape) >>>>>>> x >>>>> array([[ 0.52420123, ?0.43231286], >>>>> ? ? ? ?[ 0.97995333, ?0.87416228], >>>>> ? ? ? ?[ 0.71604075, ?0.66018382]]) >>>>>>> r = x.reshape(-1).argsort().argsort() >>>> >>>> I don't understand why this works. Why do you call argsort() twice? >>>> Doesn't that give you the indices of the sorted indices? >>> >>> It is confusing. Let's look at an example: >>> >>>>> x = np.random.rand(4) >>>>> x >>> ? ?array([ 0.37412289, ?0.68248559, ?0.12935131, ?0.42510212]) >>> >>> If we call argsort once we get the index that will sort x: >>> >>>>> idx = x.argsort() >>>>> idx >>> ? ?array([2, 0, 3, 1]) >>>>> x[idx] >>> ? ?array([ 0.12935131, ?0.37412289, ?0.42510212, ?0.68248559]) >>> >>> Notice that the first element of idx is 2. That's because element x[2] >>> is the min of x. But that's not what we want. >> >> I think that's exactly what I want, the index of the smallest element. >> It also seems to work: >> >> In [3]: x = np.random.rand(3,3) >> In [4]: x >> Out[4]: >> array([[ 0.49064281, ?0.54989584, ?0.05319183], >> ? ? ? [ 0.50510206, ?0.39683101, ?0.22801874], >> ? ? ? [ 0.04595144, ?0.3329171 , ?0.61156205]]) >> In [5]: idx = x.reshape(-1).argsort() >> In [6]: [ np.unravel_index(i, x.shape) for i in idx[-3:] ] >> Out[6]: [(1, 0), (0, 1), (2, 2)] > > Yes, you are right. My first thought was to approach the problem by > ranking the data. But that is not needed here since the position in > the argsorted index tells us the rank. I guess my approach was to rank > first and then ask questions later. Well, at least we got to see > Anne's fast ranking method. I see now that the first method I tried in this thread requires ranking. But the second method, the one that uses unravel_index, doesn't. From doutriaux1 at llnl.gov Thu Apr 15 20:39:39 2010 From: doutriaux1 at llnl.gov (=?utf-8?Q?Charles_=D8=B3=D9=85=D9=8A=D8=B1_Doutriaux?=) Date: Thu, 15 Apr 2010 17:39:39 -0700 Subject: [Numpy-discussion] rc2 for NumPy 1.4.1 and Scipy 0.7.2 In-Reply-To: References: <336574F6-63FD-497F-BB78-5EB7BEC4D62B@llnl.gov> Message-ID: Thx Ralf, I was able to get the 64bit version going. But as I mentioned earlier this doesn't help me since I have other externals dying... I'll try the 32bit one again with these hints. Thanks again, C. On Apr 15, 2010, at 4:53 PM, Ralf Gommers wrote: > > 2010/4/16 Charles ???? Doutriaux > Hi There, > > 1.4.1rc2 > gcc 4.2.1 > > And yes I'm trying to build a 32bit python because a lot of externals seem to be breaking with the 64bit stuff... > > But I'm giving it another shot building 64bit > > For 64-bit the command is: > $ LDFLAGS="-arch x86_64" FFLAGS="-arch x86_64" python setup.py install > or equivalent with numscons. > > For 32-bit > $ python setup.py install > should work, but the compiler should be gcc-4.0 (when building against python.org python this happens automatically), even though your default compiler is gcc-4.2. Also, you need to have the 10.4SDK installed from XCode optional installs. > > Cheers, > Ralf > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://*mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From wfspotz at sandia.gov Thu Apr 15 21:07:27 2010 From: wfspotz at sandia.gov (Bill Spotz) Date: Thu, 15 Apr 2010 21:07:27 -0400 Subject: [Numpy-discussion] Numpy+SWIG with arrays in input and output of a function In-Reply-To: References: Message-ID: Antoine, You want a python function with two python array arguments, therefore you are dealing with two different typemaps, each of which requires its own %apply directive: %apply (double* IN_ARRAY1 , int DIM1) {(double* vec , int m)}; %apply (double* ARGOUT_ARRAY1, int DIM1) {(double* vec2, int n)}; -Bill On Apr 15, 2010, at 7:33 PM, Antoine Delmotte wrote: > Dear Numpy and SWIG users, > > I am currently trying to use SWIG and Numpy to launch C++ codes from > python. > > My C++ code takes an array as an input (as well as integers, but > this will be my next problem...) and returns a different array (with > different dimensions). > > I have managed to make very simple codes taking an array in input OR > returning an array in output. However, I have not managed to develop > a code with an array in input and returning a different array in > output. Here is one of my attempts, which simply copies a vector (I > know I could use inplace for this example, but this is not what I > will need later since I want to return a different kind of array > than what I have as an input): > > /* matrix.c */ > > #include > #include > void vecOut(double* vec,int m,double* vec2,int n){ > int i,j; > double sum =0.0; > > for(i=0;i vec2[i] = 2*vec[i]; > } > > } > > > /* matrix.h */ > > void vecOut(double* vec,int m,double* vec2,int n); > > > /* matrix.i */ > > %module matrix > %{ > #define SWIG_FILE_WITH_INIT > #include "matrix.h" > %} > > > %include "numpy.i" > > %init %{ > import_array(); > %} > > %apply (double* IN_ARRAY1, int DIM1, int* ARGOUT_ARRAY1, int DIM1) > {(double* vec, int m, double* vec2, int n)}; > %include "matrix.h" > > > /* matrixSetup.py */ > > from distutils.core import setup, Extension > import numpy > > setup(name='matrix', version='1.0', ext_modules =[Extension('_matrix', > ['matrix.c','matrix.i'], > include_dirs = [numpy.get_include(),'.'])]) > > > > So, this is not working. The compilation is fine but when I try to > launch it, Python asks for 4 arguments. However, I would just like > to put one... > > matrix.vecOut([1,2,3]) > Traceback (most recent call last): > TypeError: vecOut() takes exactly 4 arguments (1 given) > > I expected to get [1,2,3]... > > > Is it possible to do this with SWIG? Or should I use something else? > Any advice? > > I thank you in advance for your precious suggestions. > > Best regards, > > Antoine > Hotmail : un service de messagerie gratuit, fiable et complet > Profitez-en ** Bill Spotz ** ** Sandia National Laboratories Voice: (505)845-0170 ** ** P.O. Box 5800 Fax: (505)284-0154 ** ** Albuquerque, NM 87185-0370 Email: wfspotz at sandia.gov ** From ralf.gommers at googlemail.com Thu Apr 15 21:54:30 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Fri, 16 Apr 2010 09:54:30 +0800 Subject: [Numpy-discussion] rc2 for NumPy 1.4.1 and Scipy 0.7.2 In-Reply-To: References: <336574F6-63FD-497F-BB78-5EB7BEC4D62B@llnl.gov> Message-ID: 2010/4/16 Charles ???? Doutriaux > Thx Ralf, > > I was able to get the 64bit version going. But as I mentioned earlier this > doesn't help me since I have other externals dying... > > I'll try the 32bit one again with these hints. > > The dmg installer on sourceforge is 32-bit, so unless you're applying your own patches to numpy that may be the way to go? Cheers, Ralf > > 2010/4/16 Charles ???? Doutriaux > >> Hi There, >> >> 1.4.1rc2 >> gcc 4.2.1 >> >> And yes I'm trying to build a 32bit python because a lot of externals seem >> to be breaking with the 64bit stuff... >> >> But I'm giving it another shot building 64bit >> > > For 64-bit the command is: > $ LDFLAGS="-arch x86_64" FFLAGS="-arch x86_64" python setup.py install > or equivalent with numscons. > > For 32-bit > $ python setup.py install > should work, but the compiler should be gcc-4.0 (when building against > python.org python this happens automatically), even > though your default compiler is gcc-4.2. Also, you need to have the 10.4SDK > installed from XCode optional installs. > > Cheers, > Ralf > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://*mail.scipy.org/mailman/listinfo/numpy-discussion > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From Nikolaus at rath.org Fri Apr 16 00:08:10 2010 From: Nikolaus at rath.org (Nikolaus Rath) Date: Fri, 16 Apr 2010 00:08:10 -0400 Subject: [Numpy-discussion] Find indices of largest elements References: <874ojedpzh.fsf@inspiron.ap.columbia.edu> <8739yw7b53.fsf@inspiron.ap.columbia.edu> Message-ID: <87pr2082fp.fsf@vostro.rath.org> eat writes: > Nikolaus Rath rath.org> writes: > > [snip] >> Not quite, because I'm interested in the n largest values over all >> elements, not the largest element in each row or column. But Keith's >> solution seems to work fine, even though I'm still struggling to >> understand what's going on there . > > My bad. I just concentrated on your example, not the actual question. > > However, what's wrong with your above approach > "[ np.unravel_index(i, x.shape) for i in idx[-3:] ]" ? > > Especially if your n largest elements are just a small fraction of all > elements. The fact that it sorts the entire list. But since for my arrays it's not really an efficiency problem, I will use it anyway. Best, -Nikolaus -- ?Time flies like an arrow, fruit flies like a Banana.? PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C From ndbecker2 at gmail.com Fri Apr 16 07:09:29 2010 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 16 Apr 2010 07:09:29 -0400 Subject: [Numpy-discussion] Find indices of largest elements References: <874ojedpzh.fsf@inspiron.ap.columbia.edu> <8739yw7b53.fsf@inspiron.ap.columbia.edu> <87pr2082fp.fsf@vostro.rath.org> Message-ID: Nikolaus Rath wrote: > eat writes: >> Nikolaus Rath rath.org> writes: >> >> [snip] >>> Not quite, because I'm interested in the n largest values over all >>> elements, not the largest element in each row or column. But Keith's >>> solution seems to work fine, even though I'm still struggling to >>> understand what's going on there . >> >> My bad. I just concentrated on your example, not the actual question. >> >> However, what's wrong with your above approach >> "[ np.unravel_index(i, x.shape) for i in idx[-3:] ]" ? >> >> Especially if your n largest elements are just a small fraction of all >> elements. > > The fact that it sorts the entire list. But since for my arrays it's not > really an efficiency problem, I will use it anyway. > > > > Best, > > -Nikolaus > I use boost::python and pyublas to package a function that exposes the c++ std::partial_sort From reckoner at gmail.com Fri Apr 16 09:28:05 2010 From: reckoner at gmail.com (reckoner) Date: Fri, 16 Apr 2010 06:28:05 -0700 Subject: [Numpy-discussion] Very good series of numpy tutorial videos here Message-ID: <4BC865E5.9050100@gmail.com> FYI, there is a very good series of numpy tutorial videos here: http://showmedo.com/videotutorials/series?name=i9KuJuNcG It covers a wide range of topics, not just the basics. Cheers! From antoinedelmotte at hotmail.com Fri Apr 16 09:58:29 2010 From: antoinedelmotte at hotmail.com (Antoine Delmotte) Date: Fri, 16 Apr 2010 14:58:29 +0100 Subject: [Numpy-discussion] Numpy+SWIG with arrays in input and output of a function In-Reply-To: References: Message-ID: Thank you very much for your help, Bill. It works perfectly now! On 16/04/10 02:07, Bill Spotz wrote: > Antoine, > > You want a python function with two python array arguments, therefore > you are dealing with two different typemaps, each of which requires > its own %apply directive: > > %apply (double* IN_ARRAY1 , int DIM1) {(double* vec , int m)}; > %apply (double* ARGOUT_ARRAY1, int DIM1) {(double* vec2, int n)}; > > -Bill > > On Apr 15, 2010, at 7:33 PM, Antoine Delmotte wrote: > > >> Dear Numpy and SWIG users, >> >> I am currently trying to use SWIG and Numpy to launch C++ codes from >> python. >> >> My C++ code takes an array as an input (as well as integers, but >> this will be my next problem...) and returns a different array (with >> different dimensions). >> >> I have managed to make very simple codes taking an array in input OR >> returning an array in output. However, I have not managed to develop >> a code with an array in input and returning a different array in >> output. Here is one of my attempts, which simply copies a vector (I >> know I could use inplace for this example, but this is not what I >> will need later since I want to return a different kind of array >> than what I have as an input): >> >> /* matrix.c */ >> >> #include >> #include >> void vecOut(double* vec,int m,double* vec2,int n){ >> int i,j; >> double sum =0.0; >> >> for(i=0;i> vec2[i] = 2*vec[i]; >> } >> >> } >> >> >> /* matrix.h */ >> >> void vecOut(double* vec,int m,double* vec2,int n); >> >> >> /* matrix.i */ >> >> %module matrix >> %{ >> #define SWIG_FILE_WITH_INIT >> #include "matrix.h" >> %} >> >> >> %include "numpy.i" >> >> %init %{ >> import_array(); >> %} >> >> %apply (double* IN_ARRAY1, int DIM1, int* ARGOUT_ARRAY1, int DIM1) >> {(double* vec, int m, double* vec2, int n)}; >> %include "matrix.h" >> >> >> /* matrixSetup.py */ >> >> from distutils.core import setup, Extension >> import numpy >> >> setup(name='matrix', version='1.0', ext_modules =[Extension('_matrix', >> ['matrix.c','matrix.i'], >> include_dirs = [numpy.get_include(),'.'])]) >> >> >> >> So, this is not working. The compilation is fine but when I try to >> launch it, Python asks for 4 arguments. However, I would just like >> to put one... >> >> matrix.vecOut([1,2,3]) >> Traceback (most recent call last): >> TypeError: vecOut() takes exactly 4 arguments (1 given) >> >> I expected to get [1,2,3]... >> >> >> Is it possible to do this with SWIG? Or should I use something else? >> Any advice? >> >> I thank you in advance for your precious suggestions. >> >> Best regards, >> >> Antoine >> Hotmail : un service de messagerie gratuit, fiable et complet >> Profitez-en >> > ** Bill Spotz ** > ** Sandia National Laboratories Voice: (505)845-0170 ** > ** P.O. Box 5800 Fax: (505)284-0154 ** > ** Albuquerque, NM 87185-0370 Email: wfspotz at sandia.gov ** > > > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > From doutriaux1 at llnl.gov Fri Apr 16 10:35:11 2010 From: doutriaux1 at llnl.gov (=?utf-8?Q?Charles_=D8=B3=D9=85=D9=8A=D8=B1_Doutriaux?=) Date: Fri, 16 Apr 2010 07:35:11 -0700 Subject: [Numpy-discussion] rc2 for NumPy 1.4.1 and Scipy 0.7.2 In-Reply-To: References: <336574F6-63FD-497F-BB78-5EB7BEC4D62B@llnl.gov> Message-ID: Unfortunately it's not possible... We need to redistribute... And on many Oses... C. On Apr 15, 2010, at 6:54 PM, Ralf Gommers wrote: > > > 2010/4/16 Charles ???? Doutriaux > Thx Ralf, > > I was able to get the 64bit version going. But as I mentioned earlier this doesn't help me since I have other externals dying... > > I'll try the 32bit one again with these hints. > > The dmg installer on sourceforge is 32-bit, so unless you're applying your own patches to numpy that may be the way to go? > > Cheers, > Ralf > >> >> 2010/4/16 Charles ???? Doutriaux >> Hi There, >> >> 1.4.1rc2 >> gcc 4.2.1 >> >> And yes I'm trying to build a 32bit python because a lot of externals seem to be breaking with the 64bit stuff... >> >> But I'm giving it another shot building 64bit >> >> For 64-bit the command is: >> $ LDFLAGS="-arch x86_64" FFLAGS="-arch x86_64" python setup.py install >> or equivalent with numscons. >> >> For 32-bit >> $ python setup.py install >> should work, but the compiler should be gcc-4.0 (when building against python.org python this happens automatically), even though your default compiler is gcc-4.2. Also, you need to have the 10.4SDK installed from XCode optional installs. >> >> Cheers, >> Ralf >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://**mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://*mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://*mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From pnorthug at gmail.com Fri Apr 16 15:35:23 2010 From: pnorthug at gmail.com (Paul Northug) Date: Fri, 16 Apr 2010 12:35:23 -0700 Subject: [Numpy-discussion] Fortran ordering Message-ID: I'd like to use numpy fortran ordering in order to use some external libraries more easily, but it looks like I don't understand how it works and it is causing transposes that are confusing me. When I define an array as: a = np.array([[1.,2.],[3.,4.]], order='F', type=np.float32) how is it stored in memory, as [1, 2, 3, 4] or [1, 3, 2, 4]? a.strides == (4, 8), suggesting [1, 3, 2, 4]. I can also check by stepping through: ap = a.ctypes.data_as(POINTER(c_float)) ap[:4] == [1., 3., 2., 4.] But when I call external libraries through ctypes with a.ctypes.data as argument, the external function gets [1, 2, 3, 4] (I think). Eventually, I plan to use cython's buffer interface and a.data. Will it be [1, 2, 3, 4] there as well? Or is my external code broken. I am glad that numpy hides ordering issues to the extent that it does as it prevents my code from exploding. But I would like to get around this one particular case, so if someone can explain what is going on underneath, I would appreciate it. From jhapk at utias.utoronto.ca Fri Apr 16 18:59:41 2010 From: jhapk at utias.utoronto.ca (Pradeep Jha) Date: Fri, 16 Apr 2010 18:59:41 -0400 Subject: [Numpy-discussion] Numpy compilation error In-Reply-To: References: Message-ID: Hi, I am trying to install a software called Cantera which uses numpy. When I run the make command, during the installation it gives me the following error: */bin/rm -f _build (CXX="g++"; export CXX; CC="g++"; export CC; CFLAGS="-O3 -Wall -fPIC"; export CFLAGS; PURIFY=""; export PURIFY; /usr/bin/python2 setup.py build) Traceback (most recent call last): File "setup.py", line 3, in ? from distutils.core import setup, Extension File "/nfs/carv/d1/people/pradeep/src/numpy-1.3.0/numpy/distutils/__init__.py", line 6, in ? import ccompiler File "/nfs/carv/d1/people/pradeep/src/numpy-1.3.0/numpy/distutils/ccompiler.py", line 7, in ? from distutils import ccompiler ImportError: cannot import name ccompiler make[1]: *** [_build] Error 1 make[1]: Leaving directory `/nfs/carv/d1/people/pradeep/cantera/Cantera/python' make: *** [python] Error 2* My numpy is stored in the following directory */nfs/carv/d1/people/pradeep/src/numpy-1.3.0/ *I am running Linux (Fedora). When I do a sys.path in command line Python, the output is: *['', '/nfs/carv/d1/people/pradeep/src/numpy-1.3.0', '/nfs/carv/d1/people/pradeep/src/numpy-1.3.0/numpy', '/nfs/carv/d1/people/pradeep/cantera', '/nfs/carv/d1/people/pradeep/cantera/bin', '/nfs/carv/d1/people/pradeep/cantera/build/bin/i686-pc-linux-gnu', '/nfs/carv/d1/people/pradeep/cantera/Cantera/python', '/nfs/carv/d1/people/pradeep/cantera/Cantera/python/build/lib.linux-i686-2.4/Cantera', '/nfs/carv/d1/people/pradeep/cantera/Cantera/python/Cantera', '/usr/lib/python24.zip', '/usr/lib/python2.4', '/usr/lib/python2.4/plat-linux2', '/usr/lib/python2.4/lib-tk', '/usr/lib/python2.4/lib-dynload', '/usr/lib/python2.4/site-packages', '/usr/lib/python2.4/site-packages/Numeric', '/usr/lib/python2.4/site-packages/gtk-2.0'] * I have a lot of unwanted directories as I was trying out different things. Any clues whats going wrong? Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Fri Apr 16 19:11:09 2010 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 16 Apr 2010 18:11:09 -0500 Subject: [Numpy-discussion] Numpy compilation error In-Reply-To: References: Message-ID: On Fri, Apr 16, 2010 at 17:59, Pradeep Jha wrote: > Hi, > > I am trying to install a software called Cantera which uses numpy. When I > run the make command, during the installation it gives me the following > error: > > /bin/rm -f _build > (CXX="g++"; export CXX; CC="g++"; export CC; CFLAGS="-O3 -Wall???? -fPIC"; > export CFLAGS; PURIFY=""; export PURIFY; /usr/bin/python2 setup.py build) > Traceback (most recent call last): > ? File "setup.py", line 3, in ? > ??? from distutils.core import setup, Extension > ? File > "/nfs/carv/d1/people/pradeep/src/numpy-1.3.0/numpy/distutils/__init__.py", > line 6, in ? > ??? import ccompiler > ? File > "/nfs/carv/d1/people/pradeep/src/numpy-1.3.0/numpy/distutils/ccompiler.py", > line 7, in ? > ??? from distutils import ccompiler > ImportError: cannot import name ccompiler > make[1]: *** [_build] Error 1 > make[1]: Leaving directory > `/nfs/carv/d1/people/pradeep/cantera/Cantera/python' > make: *** [python] Error 2 > > My numpy is stored in the following directory > /nfs/carv/d1/people/pradeep/src/numpy-1.3.0/ > > I am running Linux (Fedora). When I do a sys.path in command line Python, > the output is: > ['', '/nfs/carv/d1/people/pradeep/src/numpy-1.3.0', > '/nfs/carv/d1/people/pradeep/src/numpy-1.3.0/numpy', You should not have these two entries in your sys.path. Did you add them yourself via PYTHONPATH? If so, remove them. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From jhapk at utias.utoronto.ca Fri Apr 16 19:53:42 2010 From: jhapk at utias.utoronto.ca (Pradeep Jha) Date: Fri, 16 Apr 2010 19:53:42 -0400 Subject: [Numpy-discussion] Numpy compilation error In-Reply-To: References: Message-ID: Hi, I changed it. I am not getting that error anymore, but I get the following error: *running build running build_py running build_ext building 'Cantera._cantera' extension g++ -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m32 -march=i386 -mtune=generic -fasynchronous-unwind-tables -D_GNU_SOURCE -fPIC -O3 -Wall -fPIC -fPIC -I../../build/include -Isrc -I../clib/src -I/nfs/carv/d1/people/pradeep/src/numpy-1.3.0//include/python -I/usr/include/python2.4 -c src/pycantera.cpp -o build/temp.linux-i686-2.4/src/pycantera.o src/pycantera.cpp:25:31: error: numpy/arrayobject.h: No such file or directory src/ctphase_methods.cpp: In function ?PyObject* phase_getarray(PyObject*, PyObject*)?: src/ctphase_methods.cpp:126: error: ?PyArrayObject? was not declared in this scope src/ctphase_methods.cpp:126: error: ?x? was not declared in this scope src/ctphase_methods.cpp:132: error: ?npy_intp? was not declared in this scope ************************** ************************** ************************** ************************** *The pycantera.cpp file is at* /nfs/carv/d1/people/pradeep/cantera/Cantera/python/src* In the file pycantera.cpp, line number 30-31-32 say:* #ifdef HAS_NUMPY #include "numpy/arrayobject.h" #else* So the error is still with NUMPY somewhere. My sys.path in Python now shows: * ['', '/nfs/carv/d1/people/pradeep/cantera/Cantera/python/src', '/nfs/carv/d1/people/pradeep/cantera/bin', '/nfs/carv/d1/people/pradeep/cantera/build/bin/i686-pc-linux-gnu', '/nfs/carv/d1/people/pradeep/cantera/Cantera/python', '/nfs/carv/d1/people/pradeep/cantera/Cantera/python/build/lib.linux-i686-2.4/Cantera', '/nfs/carv/d1/people/pradeep/cantera/Cantera/python/Cantera', '/usr/lib/python24.zip', '/usr/lib/python2.4', '/usr/lib/python2.4/plat-linux2', '/usr/lib/python2.4/lib-tk', '/usr/lib/python2.4/lib-dynload', '/usr/lib/python2.4/site-packages', '/usr/lib/python2.4/site-packages/Numeric', '/usr/lib/python2.4/site-packages/gtk-2.0'] *In my bashrc file, the PATH variables are declared as follows: *export CANTERA_ROOT=$HOME/cantera export PATH=$JAVA_HOME/bin/:$HOME/src/$PATH:$CANTERA_ROOT/build/bin/i686-pc-linux-gnu/:$HOME/src/pdfjam/scripts export PYTHONPATH=$PYTHONPATH:$CANTERA_ROOT/bin:$CANTERA_ROOT/build/bin/i686-pc-linux-gnu:$CANTERA_ROOT/Cantera/python:$CANTERA_ROOT/Cantera/python/build/lib.linux-i686-2.4/Cantera:$CANTERA_ROOT/Cantera/python/Cantera: export Cantera_Path=$CANTERA_ROOT * I am not sure whats going wrong.* *Thanks for your help. 2010/4/16 Robert Kern > On Fri, Apr 16, 2010 at 17:59, Pradeep Jha > wrote: > > Hi, > > > > I am trying to install a software called Cantera which uses numpy. When I > > run the make command, during the installation it gives me the following > > error: > > > > /bin/rm -f _build > > (CXX="g++"; export CXX; CC="g++"; export CC; CFLAGS="-O3 -Wall > -fPIC"; > > export CFLAGS; PURIFY=""; export PURIFY; /usr/bin/python2 setup.py build) > > Traceback (most recent call last): > > File "setup.py", line 3, in ? > > from distutils.core import setup, Extension > > File > > > "/nfs/carv/d1/people/pradeep/src/numpy-1.3.0/numpy/distutils/__init__.py", > > line 6, in ? > > import ccompiler > > File > > > "/nfs/carv/d1/people/pradeep/src/numpy-1.3.0/numpy/distutils/ccompiler.py", > > line 7, in ? > > from distutils import ccompiler > > ImportError: cannot import name ccompiler > > make[1]: *** [_build] Error 1 > > make[1]: Leaving directory > > `/nfs/carv/d1/people/pradeep/cantera/Cantera/python' > > make: *** [python] Error 2 > > > > My numpy is stored in the following directory > > /nfs/carv/d1/people/pradeep/src/numpy-1.3.0/ > > > > I am running Linux (Fedora). When I do a sys.path in command line Python, > > the output is: > > ['', '/nfs/carv/d1/people/pradeep/src/numpy-1.3.0', > > '/nfs/carv/d1/people/pradeep/src/numpy-1.3.0/numpy', > > You should not have these two entries in your sys.path. Did you add > them yourself via PYTHONPATH? If so, remove them. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > -- Umberto Eco > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Fri Apr 16 21:09:57 2010 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 16 Apr 2010 20:09:57 -0500 Subject: [Numpy-discussion] Numpy compilation error In-Reply-To: References: Message-ID: On Fri, Apr 16, 2010 at 18:53, Pradeep Jha wrote: > Hi, > > I changed it. I am not getting that error anymore, but I get the following > error: > running build > running build_py > running build_ext > building 'Cantera._cantera' extension > g++ -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 > -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m32 -march=i386 > -mtune=generic -fasynchronous-unwind-tables -D_GNU_SOURCE -fPIC -O3 -Wall > -fPIC -fPIC -I../../build/include -Isrc -I../clib/src > -I/nfs/carv/d1/people/pradeep/src/numpy-1.3.0//include/python > -I/usr/include/python2.4 -c src/pycantera.cpp -o > build/temp.linux-i686-2.4/src/pycantera.o > src/pycantera.cpp:25:31: error: numpy/arrayobject.h: No such file or > directory The error is with Cantera's setup.py, most likely. It is not providing the correct include path for numpy. Please show it. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From jhapk at utias.utoronto.ca Fri Apr 16 21:47:09 2010 From: jhapk at utias.utoronto.ca (Pradeep Jha) Date: Fri, 16 Apr 2010 21:47:09 -0400 Subject: [Numpy-discussion] Numpy compilation error In-Reply-To: References: Message-ID: Hi, before running the make command, it asks me to run the preconfig file. When I run preconfig, I get the following warning message during the run: *checking for python2... /usr/bin/python2 Python command set by configure to /usr/bin/python2 WARNING: NUMPY include dir /nfs/carv/d1/people/pradeep/src/numpy-1.3.0//include/python does not exist. setting NUMPY_INC_DIR to /nfs/carv/d1/people/pradeep/src/numpy-1.3.0//include/python checking for matlab... /usr2/local_net1/bin/matlab checking MATLAB (/usr2/local_net1/bin/matlab)... /usr2/local_net1/bin/matlab *In my preconfig file, there is a line where I am supposed to specify the numpy location, and I do it like this: * # If numpy was installed using the --home option, set this to the # home directory for numpy. This will be needed for all numpy installations # that don't put the include files into python's native include directory. NUMPY_HOME=${NUMPY_HOME:="/nfs/carv/d1/people/pradeep/src/numpy-1.3.0/"} * I have attached setup.py file with this email Thank you so much. 2010/4/16 Robert Kern > On Fri, Apr 16, 2010 at 18:53, Pradeep Jha > wrote: > > Hi, > > > > I changed it. I am not getting that error anymore, but I get the > following > > error: > > running build > > running build_py > > running build_ext > > building 'Cantera._cantera' extension > > g++ -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall > -Wp,-D_FORTIFY_SOURCE=2 > > -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m32 -march=i386 > > -mtune=generic -fasynchronous-unwind-tables -D_GNU_SOURCE -fPIC -O3 -Wall > > -fPIC -fPIC -I../../build/include -Isrc -I../clib/src > > -I/nfs/carv/d1/people/pradeep/src/numpy-1.3.0//include/python > > -I/usr/include/python2.4 -c src/pycantera.cpp -o > > build/temp.linux-i686-2.4/src/pycantera.o > > src/pycantera.cpp:25:31: error: numpy/arrayobject.h: No such file or > > directory > > The error is with Cantera's setup.py, most likely. It is not providing > the correct include path for numpy. Please show it. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > -- Umberto Eco > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: setup.py Type: application/octet-stream Size: 4470 bytes Desc: not available URL: From josef.pktd at gmail.com Fri Apr 16 21:56:33 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 16 Apr 2010 21:56:33 -0400 Subject: [Numpy-discussion] Numpy compilation error In-Reply-To: References: Message-ID: On Fri, Apr 16, 2010 at 9:47 PM, Pradeep Jha wrote: > Hi, > > before running the make command, it asks me to run the preconfig file. When > I run preconfig, I get the following warning message during the run: > checking for python2... /usr/bin/python2 > Python command set by configure to? /usr/bin/python2 > WARNING: NUMPY include dir > /nfs/carv/d1/people/pradeep/src/numpy-1.3.0//include/python does not exist. > setting NUMPY_INC_DIR to > /nfs/carv/d1/people/pradeep/src/numpy-1.3.0//include/python Is duplicate forward-slash legal in a Linux path? Josef > checking for matlab... /usr2/local_net1/bin/matlab > checking MATLAB (/usr2/local_net1/bin/matlab)... /usr2/local_net1/bin/matlab > > In my preconfig file, there is a line where I am supposed to specify the > numpy location, and I do it like this: > > # If numpy was installed using the --home option, set this to the > # home directory for numpy. This will be needed for all numpy installations > # that don't put the include files into python's native include directory. > NUMPY_HOME=${NUMPY_HOME:="/nfs/carv/d1/people/pradeep/src/numpy-1.3.0/"} > > I have attached setup.py file with this email > > Thank you so much. > > 2010/4/16 Robert Kern >> >> On Fri, Apr 16, 2010 at 18:53, Pradeep Jha >> wrote: >> > Hi, >> > >> > I changed it. I am not getting that error anymore, but I get the >> > following >> > error: >> > running build >> > running build_py >> > running build_ext >> > building 'Cantera._cantera' extension >> > g++ -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall >> > -Wp,-D_FORTIFY_SOURCE=2 >> > -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m32 >> > -march=i386 >> > -mtune=generic -fasynchronous-unwind-tables -D_GNU_SOURCE -fPIC -O3 >> > -Wall >> > -fPIC -fPIC -I../../build/include -Isrc -I../clib/src >> > -I/nfs/carv/d1/people/pradeep/src/numpy-1.3.0//include/python >> > -I/usr/include/python2.4 -c src/pycantera.cpp -o >> > build/temp.linux-i686-2.4/src/pycantera.o >> > src/pycantera.cpp:25:31: error: numpy/arrayobject.h: No such file or >> > directory >> >> The error is with Cantera's setup.py, most likely. It is not providing >> the correct include path for numpy. Please show it. >> >> -- >> Robert Kern >> >> "I have come to believe that the whole world is an enigma, a harmless >> enigma that is made terrible by our own mad attempt to interpret it as >> though it had an underlying truth." >> ?-- Umberto Eco >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From muzgash.lists at gmail.com Fri Apr 16 22:20:11 2010 From: muzgash.lists at gmail.com (Gerardo Gutierrez) Date: Fri, 16 Apr 2010 21:20:11 -0500 Subject: [Numpy-discussion] IPythonQt and IPythonZMQ projects In-Reply-To: References: Message-ID: Hi all. I'm writting to tell you about some projects we're developing for you and the entire python programmers community. Read some of the documentation of the proposals and please tell us what are your ideas and what do you want or expect to see of this. I'm going to leave you a summary and some links: *IPythonQt* We will implement a Qt-based Graphical User Interface (GUI) to execute Python code with an interpreter that runs in a separate process and the two systems (GUI frontend and interpreter kernel) communicating via the ZeroMQ Messaging library. The bulk of the implementation will be done without dependencies on IPython (only on Zmq). Once the key features are ready, IPython-specific features can be added using the IPython codebase. *links*: http://ipythonqt.blogspot.com/ https://code.launchpad.net/~ipython-contrib/ipython/qt-frontend *IPythonZMQ* IPython's execution in a command-line environment will be ported to a two process model using the ZeroMQ library for inter-process communication. This will: - prevent an interpreter crash from destroying the user session, - allow multiple clients to interact simultaneously with a single interpreter - allow IPython to reuse code for local execution and distributed computing (DC) - give us a path for Python3 support, since ZeroMQ supports Python3 while Twisted (what we use today for DC) does not. *links*: http://ipythonzmq.blogspot.com/ https://code.launchpad.net/~ipython-contrib/+junk/ipython-zmq Best regards. -- Gerardo Guti?rrez Guti?rrez Physics student Universidad de Antioquia Computational physics and astrophysics group (FACom ) Computational science and development branch(FACom-dev ) Usuario Linux #492295 -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Fri Apr 16 23:24:11 2010 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 16 Apr 2010 22:24:11 -0500 Subject: [Numpy-discussion] Numpy compilation error In-Reply-To: References: Message-ID: On Fri, Apr 16, 2010 at 20:56, wrote: > On Fri, Apr 16, 2010 at 9:47 PM, Pradeep Jha wrote: >> Hi, >> >> before running the make command, it asks me to run the preconfig file. When >> I run preconfig, I get the following warning message during the run: >> checking for python2... /usr/bin/python2 >> Python command set by configure to? /usr/bin/python2 >> WARNING: NUMPY include dir >> /nfs/carv/d1/people/pradeep/src/numpy-1.3.0//include/python does not exist. >> setting NUMPY_INC_DIR to >> /nfs/carv/d1/people/pradeep/src/numpy-1.3.0//include/python > > Is duplicate forward-slash legal in a Linux path? Yes. It is ignored. You need to set NUMPY_INC_DIR to $PYTHON_SITE_PACKAGES/numpy/core/include where $PYTHON_SITE_PACKAGES is the site-packages/ directory where your installation of numpy got installed to. Note that this is not /nfs/carv/d1/people/pradeep/src/numpy-1.3.0/. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From ademan555 at gmail.com Sat Apr 17 00:50:17 2010 From: ademan555 at gmail.com (Dan Roberts) Date: Fri, 16 Apr 2010 21:50:17 -0700 Subject: [Numpy-discussion] My GSoC Proposal to Implement a Subset of NumPy for PyPy In-Reply-To: References: Message-ID: Hello NumPy Users, Hi everybody, my name is Dan Roberts, and my Google Summer of Code proposal was categorized under NumPy rather than PyPy, so it will end up being reviewed by mentors for the NumPy project. I'd like to take this chance to introduce myself and my proposal. I hadn't prepared for review by the NumPy mentors, but this can make my proposal stronger than before. With a bit of help from all of you, I can dedicate my summer to creating more useful code than I would have previously. I realize that from the perspective of NumPy, my proposal might seem lacking, so I'd like to also invite the scrutiny of all of the readers of this list. Why should we bother reimplimenting anything? PyPy, for those who are unfamiliar, has the ability to Just-in-Time compile itself and programs that it's running. One of the major advantages of this is that code operating on NumPy arrays could potentially be written in pure-python, with normal looping constructs, and be nearly as fast as a ufunc painstakingly crafted in C. I'd love to see as much Python and as little C as possible, and I'm sure I'm not alone in that wish. A short introduction: I've been coding in Python for the past few years, and have increasingly become interested in speeding up what has become my favorite language. To that end I've become interested in both the PyPy project and the NumPy projects. I've spent a fair amount of time frustrating the PyPy developers with silly questions, written a bit of code for them, and now my GSoC proposal involves both them, and NumPy. Finally, I'd like to ask all of you: what features are most important to you? It's not practical, wise, or even possible for me to reimpliment more than a small portion of NumPy, but if I can address the most important parts, maybe I can make this project useful enough for some of you to use, and close enough for the rest of you that I can drum up some support for more development in the future. My proposal lives at http://codespeak.net/~dan/gsoc/micronumpy.htmlthanks for making it this far through my long winded introduction! I welcome all constructive criticism and thoughts. Thanks, Daniel Roberts -------------- next part -------------- An HTML attachment was scrubbed... URL: From dagss at student.matnat.uio.no Sat Apr 17 03:12:50 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 17 Apr 2010 09:12:50 +0200 Subject: [Numpy-discussion] My GSoC Proposal to Implement a Subset of NumPy for PyPy In-Reply-To: References: Message-ID: <4BC95F72.9020005@student.matnat.uio.no> Dan Roberts wrote: > Hello NumPy Users, > Hi everybody, my name is Dan Roberts, and my Google Summer of Code > proposal was categorized under NumPy rather than PyPy, so it will end up > being reviewed by mentors for the NumPy project. I'd like to take this > chance to introduce myself and my proposal. > I hadn't prepared for review by the NumPy mentors, but this can make > my proposal stronger than before. With a bit of help from all of you, I > can dedicate my summer to creating more useful code than I would have > previously. I realize that from the perspective of NumPy, my proposal > might seem lacking, so I'd like to also invite the scrutiny of all of > the readers of this list. > Why should we bother reimplimenting anything? PyPy, for those who > are unfamiliar, has the ability to Just-in-Time compile itself and > programs that it's running. One of the major advantages of this is that > code operating on NumPy arrays could potentially be written in > pure-python, with normal looping constructs, and be nearly as fast as a > ufunc painstakingly crafted in C. I'd love to see as much Python and as > little C as possible, and I'm sure I'm not alone in that wish. > A short introduction: I've been coding in Python for the past few > years, and have increasingly become interested in speeding up what has > become my favorite language. To that end I've become interested in both > the PyPy project and the NumPy projects. I've spent a fair amount of > time frustrating the PyPy developers with silly questions, written a bit > of code for them, and now my GSoC proposal involves both them, and > NumPy. > Finally, I'd like to ask all of you: what features are most > important to you? It's not practical, wise, or even possible for me to > reimpliment more than a small portion of NumPy, but if I can address the > most important parts, maybe I can make this project useful enough for > some of you to use, and close enough for the rest of you that I can drum > up some support for more development in the future. > My proposal lives at http://codespeak.net/~dan/gsoc/micronumpy.html > thanks for making it this far through my long winded introduction! I > welcome all constructive criticism and thoughts. I'm curious about what role natively compiled code in C would play in your project. Would you use BLAS, or would you reimplement e.g. matrix multiplication in RPython and hope that PyPy optimize it? (Hint: It stands no chance of even coming close. A BLAS implementation is easily 4-5 times faster (or more) than a simple hand-written C code for matrix multiplication, which I assume is the lower bound for any RPython code it is realistic to write. They use CPU-specific cache-aware algorithms which you really can't hope to implement yourself.) Eventually, for this to be at all useful for the NumPy crowd, one has to make available eigenvalue finders, FFTs, and so on as well. This is a massive amount of work unless one is willing to connect to existing C implementations. So even if all of this doesn't happen in the GSoC project, it would be useful to know whether it is possible long-term to connect with BLAS and LAPACK, or whether you intend everything to be done in RPython. In my opinion, the *primary* reason Python is used for scientific programming rather than some other language is how easy it is to connect with C, C++ and Fortran code in CPython. That's something to keep in mind. -- Dag Sverre From dagss at student.matnat.uio.no Sat Apr 17 03:15:08 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 17 Apr 2010 09:15:08 +0200 Subject: [Numpy-discussion] My GSoC Proposal to Implement a Subset of NumPy for PyPy In-Reply-To: <4BC95F72.9020005@student.matnat.uio.no> References: <4BC95F72.9020005@student.matnat.uio.no> Message-ID: <4BC95FFC.6010905@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > Dan Roberts wrote: >> Hello NumPy Users, >> Hi everybody, my name is Dan Roberts, and my Google Summer of Code >> proposal was categorized under NumPy rather than PyPy, so it will end up >> being reviewed by mentors for the NumPy project. I'd like to take this >> chance to introduce myself and my proposal. >> I hadn't prepared for review by the NumPy mentors, but this can make >> my proposal stronger than before. With a bit of help from all of you, I >> can dedicate my summer to creating more useful code than I would have >> previously. I realize that from the perspective of NumPy, my proposal >> might seem lacking, so I'd like to also invite the scrutiny of all of >> the readers of this list. >> Why should we bother reimplimenting anything? PyPy, for those who >> are unfamiliar, has the ability to Just-in-Time compile itself and >> programs that it's running. One of the major advantages of this is that >> code operating on NumPy arrays could potentially be written in >> pure-python, with normal looping constructs, and be nearly as fast as a >> ufunc painstakingly crafted in C. I'd love to see as much Python and as >> little C as possible, and I'm sure I'm not alone in that wish. >> A short introduction: I've been coding in Python for the past few >> years, and have increasingly become interested in speeding up what has >> become my favorite language. To that end I've become interested in both >> the PyPy project and the NumPy projects. I've spent a fair amount of >> time frustrating the PyPy developers with silly questions, written a bit >> of code for them, and now my GSoC proposal involves both them, and >> NumPy. >> Finally, I'd like to ask all of you: what features are most >> important to you? It's not practical, wise, or even possible for me to >> reimpliment more than a small portion of NumPy, but if I can address the >> most important parts, maybe I can make this project useful enough for >> some of you to use, and close enough for the rest of you that I can drum >> up some support for more development in the future. >> My proposal lives at http://codespeak.net/~dan/gsoc/micronumpy.html >> thanks for making it this far through my long winded introduction! I >> welcome all constructive criticism and thoughts. WHOOPS!! Looks like I'm making a fool of myself. I foolishly based my comments on an earlier reading of your proposal (I'm a PSF mentor), and didn't see (soon enough) that you had updated the proposal to answer just this question. So please just ignore everything I've written :-) Dag Sverre > > I'm curious about what role natively compiled code in C would play in > your project. Would you use BLAS, or would you reimplement e.g. matrix > multiplication in RPython and hope that PyPy optimize it? (Hint: It > stands no chance of even coming close. A BLAS implementation is easily > 4-5 times faster (or more) than a simple hand-written C code for matrix > multiplication, which I assume is the lower bound for any RPython code > it is realistic to write. They use CPU-specific cache-aware algorithms > which you really can't hope to implement yourself.) > > Eventually, for this to be at all useful for the NumPy crowd, one has to > make available eigenvalue finders, FFTs, and so on as well. This is a > massive amount of work unless one is willing to connect to existing C > implementations. > > So even if all of this doesn't happen in the GSoC project, it would be > useful to know whether it is possible long-term to connect with BLAS and > LAPACK, or whether you intend everything to be done in RPython. > > In my opinion, the *primary* reason Python is used for scientific > programming rather than some other language is how easy it is to connect > with C, C++ and Fortran code in CPython. That's something to keep in mind. > -- Dag Sverre From stefan at sun.ac.za Sat Apr 17 03:24:40 2010 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Sat, 17 Apr 2010 09:24:40 +0200 Subject: [Numpy-discussion] My GSoC Proposal to Implement a Subset of NumPy for PyPy In-Reply-To: References: Message-ID: Hi Dan On 17 April 2010 06:50, Dan Roberts wrote: > ??? Hi everybody, my name is Dan Roberts, and my Google Summer of Code > proposal was categorized under NumPy rather than PyPy, so it will end up > being reviewed by mentors for the NumPy project.? I'd like to take this > chance to introduce myself and my proposal. Thanks for the introduction, and welcome to NumPy! > ??? I hadn't prepared for review by the NumPy mentors, but this can make my > proposal stronger than before.? With a bit of help from all of you, I can > dedicate my summer to creating more useful code than I would have > previously. I realize that from the perspective of NumPy, my proposal might > seem lacking, so I'd like to also invite the scrutiny of all of the readers > of this list. This proposal builds a bridge between two projects, so even if it technically falls under the NumPy banner, we'll lean heavily on Maciej Fijalkowski from PyPy for guidance. > ??? Why should we bother reimplimenting anything?? PyPy, for those who are > unfamiliar, has the ability to Just-in-Time compile itself and programs that > it's running.? One of the major advantages of this is that code operating on > NumPy arrays could potentially be written in pure-python, with normal > looping constructs, and be nearly as fast as a ufunc painstakingly crafted > in C.? I'd love to see as much Python and as little C as possible, and I'm > sure I'm not alone in that wish. Your code has a fairly specialised application and it's worth discussing exactly where it would fit in. For example, from our perspective rewriting things such as zeros(), ones(), etc. is not of much interest. However, the ability to whip up fast ufuncs and generalised ufuncs is in great demand. Also, it is sometimes clearer to express an algorithm as for i in range(n): for j in range(m): x[i, j] = some_op(x[i, j]) instead of vectorising the code. Here, PyPy can provide a big speed improvement. I'm not sure, but it sounds like the "interface" you refer to would be things such as the [] operator on arrays, for example? Just an an aside, I think PyPy would be perfect for managing sparse matrices (such as scipy.sparse), where there are so many loops involved---in fact, an RPy implementation of scipy.sparse could be an interesting proposal for a next SoC!. I spoke briefly with Maciej the other day, and I realised that there is a lot of detail on how PyPy interacts with C modules that we are not aware of. It would be great if you could elaborate a bit on the way PyPy is able to access current C functionality. For example, can you use NumPy as is, and just replace functionality piece by piece, or would you need to rewrite a large part of the interface at a time? Regards St?fan From dagss at student.matnat.uio.no Sat Apr 17 04:41:17 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 17 Apr 2010 10:41:17 +0200 Subject: [Numpy-discussion] Fortran ordering In-Reply-To: References: Message-ID: <4BC9742D.3020007@student.matnat.uio.no> Paul Northug wrote: > I'd like to use numpy fortran ordering in order to use some external > libraries more easily, but it looks like I don't understand how it > works and it is causing transposes that are confusing me. > > When I define an array as: > > a = np.array([[1.,2.],[3.,4.]], order='F', type=np.float32) > > how is it stored in memory, as [1, 2, 3, 4] or [1, 3, 2, 4]? > > a.strides == (4, 8), suggesting [1, 3, 2, 4]. I can also check by > stepping through: > > ap = a.ctypes.data_as(POINTER(c_float)) > ap[:4] == [1., 3., 2., 4.] > > But when I call external libraries through ctypes with a.ctypes.data > as argument, the external function gets [1, 2, 3, 4] (I think). > Eventually, I plan to use cython's buffer interface and a.data. Will > it be [1, 2, 3, 4] there as well? Or is my external code broken. Using Cython's a.data, it will definitely be [1, 3, 2, 4]. Declare your array in Cython as np.ndarray[float, mode='fortran'], that way you'll get a ValueError unless the ordering is correct. -- Dag Sverre From stefan at sun.ac.za Sat Apr 17 13:50:22 2010 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Sat, 17 Apr 2010 19:50:22 +0200 Subject: [Numpy-discussion] Fortran ordering In-Reply-To: References: Message-ID: On 16 April 2010 21:35, Paul Northug wrote: > how is it stored in memory, as [1, 2, 3, 4] or [1, 3, 2, 4]? The latter: In [22]: np.fromstring(str(x.data)) Out[22]: array([ 1., 3., 2., 4.]) > But when I call external libraries through ctypes with a.ctypes.data > as argument, the external function gets [1, 2, 3, 4] (I think). That should not be the case. Please see the attached code as a demonstration, which has the following output: Printing values as in memory: 1 3 2 4 Let's assign values 0, 1, 2 and 3 to consecutive memory addresses. The result is: [[ 0. 2.] [ 1. 3.]] Regards St?fan -------------- next part -------------- A non-text attachment was scrubbed... Name: bytes_.c Type: application/octet-stream Size: 259 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: fortran_test.py Type: text/x-python-script Size: 775 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: SConstruct Type: application/octet-stream Size: 116 bytes Desc: not available URL: From efiring at hawaii.edu Sat Apr 17 16:01:10 2010 From: efiring at hawaii.edu (Eric Firing) Date: Sat, 17 Apr 2010 10:01:10 -1000 Subject: [Numpy-discussion] Bug in numpy.fix(): broken for scalar arguments Message-ID: <4BCA1386.9040609@hawaii.edu> np.fix() no longer works for scalar arguments: In [1]:import numpy as np In [2]:np.version.version Out[2]:'2.0.0.dev8334' In [3]:np.fix(3.14) --------------------------------------------------------------------------- TypeError Traceback (most recent call last) /home/efiring/ in () /usr/local/lib/python2.6/dist-packages/numpy/lib/ufunclike.pyc in fix(x, y) 46 if y is None: 47 y = y1 ---> 48 y[...] = nx.where(x >= 0, y1, y2) 49 return y 50 TypeError: 'numpy.float64' object does not support item assignment Eric From charlesr.harris at gmail.com Sat Apr 17 16:16:50 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 17 Apr 2010 14:16:50 -0600 Subject: [Numpy-discussion] Bug in numpy.fix(): broken for scalar arguments In-Reply-To: <4BCA1386.9040609@hawaii.edu> References: <4BCA1386.9040609@hawaii.edu> Message-ID: On Sat, Apr 17, 2010 at 2:01 PM, Eric Firing wrote: > np.fix() no longer works for scalar arguments: > > > In [1]:import numpy as np > > In [2]:np.version.version > Out[2]:'2.0.0.dev8334' > > In [3]:np.fix(3.14) > --------------------------------------------------------------------------- > TypeError Traceback (most recent call last) > > /home/efiring/ in () > > /usr/local/lib/python2.6/dist-packages/numpy/lib/ufunclike.pyc in fix(x, y) > 46 if y is None: > 47 y = y1 > ---> 48 y[...] = nx.where(x >= 0, y1, y2) > 49 return y > 50 > > TypeError: 'numpy.float64' object does not support item assignment > > > Looks like r8293. Darren? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From pnorthug at gmail.com Sat Apr 17 20:25:41 2010 From: pnorthug at gmail.com (Paul Northug) Date: Sun, 18 Apr 2010 00:25:41 +0000 (UTC) Subject: [Numpy-discussion] Fortran ordering References: Message-ID: St?fan van der Walt sun.ac.za> writes: > > On 16 April 2010 21:35, Paul Northug gmail.com> wrote: > > how is it stored in memory, as [1, 2, 3, 4] or [1, 3, 2, 4]? > > The latter: > > In [22]: np.fromstring(str(x.data)) > Out[22]: array([ 1., 3., 2., 4.]) > > > But when I call external libraries through ctypes with a.ctypes.data > > as argument, the external function gets [1, 2, 3, 4] (I think). > > That should not be the case. Please see the attached code as a > demonstration, which has the following output: > > Printing values as in memory: > 1 3 2 4 > Let's assign values 0, 1, 2 and 3 to consecutive memory > addresses. The result is: > [[ 0. 2.] > [ 1. 3.]] > > Regards > St?fan Thanks for the explanation and especially the code example. That clears it up for me. I also implemented it in cython as Dag suggested (wrapping cuda sgemm). The confusion was due to how pycuda maps arrays to and from cuda device memory using pycuda.gpuarray. From cournape at gmail.com Sun Apr 18 06:05:47 2010 From: cournape at gmail.com (David Cournapeau) Date: Sun, 18 Apr 2010 19:05:47 +0900 Subject: [Numpy-discussion] My GSoC Proposal to Implement a Subset of NumPy for PyPy In-Reply-To: References: Message-ID: Hi Dan, On Sat, Apr 17, 2010 at 1:50 PM, Dan Roberts wrote: > Hello NumPy Users, > ??? Hi everybody, my name is Dan Roberts, and my Google Summer of Code > proposal was categorized under NumPy rather than PyPy, so it will end up > being reviewed by mentors for the NumPy project.? I'd like to take this > chance to introduce myself and my proposal. Thanks for proposing a NumPy-related project ! > ??? Finally, I'd like to ask all of you: what features are most important to > you? It's not practical, wise, or even possible for me to reimpliment more > than a small portion of NumPy, but if I can address the most important > parts, maybe I can make this project useful enough for some of you to use, > and close enough for the rest of you that I can drum up some support for > more development in the future. First, an aside: with the recent announcement from pypy concerning the new way of interfacing with C, wouldn't it make more sense to go the other way around - i.e. starting from full numpy, and replace some parts in rpython ? Of course, this assumes that numpy can work with pypy using the new C API "emulation", but this may not be a lot of work. This would gives the advantage of having something useful from the start. I think this project is extremely interesting in nature, and the parts which are the most interesting to implement are (IMHO of course): - indexing, fancy indexing - broadcasting - basic ufunc support, although I am not sure how to plug this with the C math library for math functions (cos, log, etc...) Especially for indexing and broadcasting, even if your project "fails", having a pure, reference python implementation would be tremendously useful - in particular for the use in sparse arrays as mentioned by Stefan. cheers, David From pav at iki.fi Sun Apr 18 06:46:36 2010 From: pav at iki.fi (Pauli Virtanen) Date: Sun, 18 Apr 2010 12:46:36 +0200 Subject: [Numpy-discussion] My GSoC Proposal to Implement a Subset of NumPy for PyPy In-Reply-To: References: Message-ID: <1271587596.2105.7.camel@talisman> su, 2010-04-18 kello 19:05 +0900, David Cournapeau kirjoitti: [clip] > First, an aside: with the recent announcement from pypy concerning the > new way of interfacing with C, wouldn't it make more sense to go the > other way around - i.e. starting from full numpy, and replace some > parts in rpython ? Of course, this assumes that numpy can work with > pypy using the new C API "emulation", but this may not be a lot of > work. This would gives the advantage of having something useful from > the start. > > I think this project is extremely interesting in nature, and the parts > which are the most interesting to implement are (IMHO of course): > - indexing, fancy indexing > - broadcasting > - basic ufunc support, although I am not sure how to plug this with > the C math library for math functions (cos, log, etc...) I agree here: if it's possible to supplement the existing Numpy code base by interpreter-level (JIT-able) support for indexing, fancy indexing, and broadcasting, this would reduce the roadblock in writing effective algorithms without using vectorization. Also, if you can incrementally replace parts of Numpy, the code could be immediately usable in real world, being less of a proof-of-concept subset. -- Pauli Virtanen From lasagnadavide at gmail.com Sun Apr 18 08:16:00 2010 From: lasagnadavide at gmail.com (Davide Lasagna) Date: Sun, 18 Apr 2010 14:16:00 +0200 Subject: [Numpy-discussion] np.mean and np.std performances In-Reply-To: References: Message-ID: Hi all, I noticed some performance problems with np.mean and np.std functions. Here is the console output in ipython: # make some test data >>>: a = np.arange(80*64, dtype=np.float64).reshape(80, 64) >>>: c = np.tile( a, [10000, 1, 1]) >>>: timeit np.mean(c, axis=0) 1 loops, best of 3: 2.09 s per loop But using reduce is much faster: def mean_reduce(c): return reduce(lambda som, array: som+array, c) / c.shape[0] >>>:timeit mean_reduce(c) 1 loops, best of 3: 355 ms per loop The same applies to np.std(): # slighlty smaller c matrix (too much memory is used) >>>: c = np.tile( a, [7000, 1, 1]) >>>: timeit np.std(c, axis=0) 1 loops, best of 3: 3.73 s per loop With the reduce version: def std_reduce(c): c -= mean_reduce(c) return np.sqrt( reduce(lambda som, array: som + array**2, c ) / c.shape[0] ) >>>: timeit std_reduce(c) 1 loops, best of 3: 1.18 s per loop For the std function also look at the memory usage during the execution of the function. The functions i gave here can be easily modified to accept an axis option and other stuff needed. Is there any drawback of using them? Why np.mean and np.std are so slow? I'm sure I'm missing something. Cheers Davide -------------- next part -------------- An HTML attachment was scrubbed... URL: From dsdale24 at gmail.com Sun Apr 18 09:08:54 2010 From: dsdale24 at gmail.com (Darren Dale) Date: Sun, 18 Apr 2010 09:08:54 -0400 Subject: [Numpy-discussion] Bug in numpy.fix(): broken for scalar arguments In-Reply-To: References: <4BCA1386.9040609@hawaii.edu> Message-ID: On Sat, Apr 17, 2010 at 4:16 PM, Charles R Harris wrote: > > > On Sat, Apr 17, 2010 at 2:01 PM, Eric Firing wrote: >> >> np.fix() no longer works for scalar arguments: >> >> >> In [1]:import numpy as np >> >> In [2]:np.version.version >> Out[2]:'2.0.0.dev8334' >> >> In [3]:np.fix(3.14) >> >> --------------------------------------------------------------------------- >> TypeError ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Traceback (most recent call >> last) >> >> /home/efiring/ in () >> >> /usr/local/lib/python2.6/dist-packages/numpy/lib/ufunclike.pyc in fix(x, >> y) >> ? ? ?46 ? ? if y is None: >> ? ? ?47 ? ? ? ? y = y1 >> ---> 48 ? ? y[...] = nx.where(x >= 0, y1, y2) >> ? ? ?49 ? ? return y >> ? ? ?50 >> >> TypeError: 'numpy.float64' object does not support item assignment >> >> > > Looks like r8293. Darren? Thanks, I'm looking into it. From dsdale24 at gmail.com Sun Apr 18 09:28:02 2010 From: dsdale24 at gmail.com (Darren Dale) Date: Sun, 18 Apr 2010 09:28:02 -0400 Subject: [Numpy-discussion] Bug in numpy.fix(): broken for scalar arguments In-Reply-To: References: <4BCA1386.9040609@hawaii.edu> Message-ID: On Sun, Apr 18, 2010 at 9:08 AM, Darren Dale wrote: > On Sat, Apr 17, 2010 at 4:16 PM, Charles R Harris > wrote: >> >> >> On Sat, Apr 17, 2010 at 2:01 PM, Eric Firing wrote: >>> >>> np.fix() no longer works for scalar arguments: >>> >>> >>> In [1]:import numpy as np >>> >>> In [2]:np.version.version >>> Out[2]:'2.0.0.dev8334' >>> >>> In [3]:np.fix(3.14) >>> >>> --------------------------------------------------------------------------- >>> TypeError ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Traceback (most recent call >>> last) >>> >>> /home/efiring/ in () >>> >>> /usr/local/lib/python2.6/dist-packages/numpy/lib/ufunclike.pyc in fix(x, >>> y) >>> ? ? ?46 ? ? if y is None: >>> ? ? ?47 ? ? ? ? y = y1 >>> ---> 48 ? ? y[...] = nx.where(x >= 0, y1, y2) >>> ? ? ?49 ? ? return y >>> ? ? ?50 >>> >>> TypeError: 'numpy.float64' object does not support item assignment >>> >>> >> >> Looks like r8293. Darren? > > Thanks, I'm looking into it. The old np.fix behavior is different from np.floor and np.ceil. np.fix(3.14) would return array(3.0), while np.floor(3.14) would return 3.0. Shall I fix it to conform with the old but inconsistent behavior of fix? Darren From sebastian.walter at gmail.com Sun Apr 18 09:35:53 2010 From: sebastian.walter at gmail.com (Sebastian Walter) Date: Sun, 18 Apr 2010 15:35:53 +0200 Subject: [Numpy-discussion] Math Library In-Reply-To: References: <6DD1061A-60A3-4B84-99B0-793C0532321D@enthought.com> Message-ID: On Tue, Apr 13, 2010 at 12:29 AM, Charles R Harris wrote: > > > On Mon, Apr 12, 2010 at 4:19 PM, Travis Oliphant > wrote: >> >> On Apr 11, 2010, at 4:17 PM, Sebastian Walter wrote: >> >> > >> > Ermm, the reply above is quite poor, sorry about that. >> > What I meant to say is the following: >> > >> > If there is going to be a discussion about creating a pure C numpy >> > library I'd like to join ;) >> >> Great. ? I would really like to get the discussion going. ? In an >> ideal world we can finish any kind of significant re-factoring in time >> for SciPy this year. ? ?It actually feels like the kind of thing that >> can motivate NumPy 2.0 a bit better. >> >> It sounds to me like nobody will be opposed as long as there is >> continuity to the project and current code still works without >> disruption (i.e. the current C-API for Python extensions is available). >> >> I am interested in re-factoring in such a way to create minimal impact >> on current NumPy C-API users, but improve maintainability going >> forward and the ability for other projects to use NumPy. >> > > My own thoughts were to have a lowlevel 'loop' library that worked with > strided memory, and an intermediate level above that for buffer objects. > Numpy ufuncs would be a level above that and implement policy type things > like casting, kinds, etc. > > Then there is the lowlevel c-library for the functions. I don't think we > should aim at duplicating commonly available functions like sin and exp, but > rather that subset that are sometimes unavailable. In particular, I would > like to get away from having to use double versions of functions instead of > type specific versions. This sounds reasonable. However, I'm not sure that I understand exactly what the consequences would be. Maybe it would be a good idea that one writes down prototypical examples that should be supported by the new code? Sebastian > > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From dsdale24 at gmail.com Sun Apr 18 10:07:43 2010 From: dsdale24 at gmail.com (Darren Dale) Date: Sun, 18 Apr 2010 10:07:43 -0400 Subject: [Numpy-discussion] Bug in numpy.fix(): broken for scalar arguments In-Reply-To: References: <4BCA1386.9040609@hawaii.edu> Message-ID: On Sun, Apr 18, 2010 at 9:28 AM, Darren Dale wrote: > On Sun, Apr 18, 2010 at 9:08 AM, Darren Dale wrote: >> On Sat, Apr 17, 2010 at 4:16 PM, Charles R Harris >> wrote: >>> >>> >>> On Sat, Apr 17, 2010 at 2:01 PM, Eric Firing wrote: >>>> >>>> np.fix() no longer works for scalar arguments: >>>> >>>> >>>> In [1]:import numpy as np >>>> >>>> In [2]:np.version.version >>>> Out[2]:'2.0.0.dev8334' >>>> >>>> In [3]:np.fix(3.14) >>>> >>>> --------------------------------------------------------------------------- >>>> TypeError ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Traceback (most recent call >>>> last) >>>> >>>> /home/efiring/ in () >>>> >>>> /usr/local/lib/python2.6/dist-packages/numpy/lib/ufunclike.pyc in fix(x, >>>> y) >>>> ? ? ?46 ? ? if y is None: >>>> ? ? ?47 ? ? ? ? y = y1 >>>> ---> 48 ? ? y[...] = nx.where(x >= 0, y1, y2) >>>> ? ? ?49 ? ? return y >>>> ? ? ?50 >>>> >>>> TypeError: 'numpy.float64' object does not support item assignment >>>> >>>> >>> >>> Looks like r8293. Darren? >> >> Thanks, I'm looking into it. > > The old np.fix behavior is different from np.floor and np.ceil. > np.fix(3.14) would return array(3.0), while np.floor(3.14) would > return 3.0. Shall I fix it to conform with the old but inconsistent > behavior of fix? I think this is the underlying issue: np.floor(np.array(3.14)) returns 3.0, not array(3.14). The current implementation of fix had already taken care to ensure that it was working with an array for the input. What is numpy's policy here? np.fix returned a len-0 ndarray even for scalar input, floor and ceil return scalars even for len-0 ndarrays. This inconsistency makes it difficult to make even small modifications to the numpy codebase. r8351 includes a one-line change that addresses Eric's report and is commensurate with the previous behavior of fix. Darren From jhapk at utias.utoronto.ca Sun Apr 18 14:02:13 2010 From: jhapk at utias.utoronto.ca (Pradeep Jha) Date: Sun, 18 Apr 2010 14:02:13 -0400 Subject: [Numpy-discussion] Numpy compilation error In-Reply-To: References: Message-ID: Hi, how do I figure where did my numpy got installed to? I downloaded a tar.gz file and then unzipped it at /nfs/carv/d1/people/pradeep/src/numpy-1.3.0/. I didn't do any additional installation steps. Also, just fyi, I don't have root access to this system. Thanks 2010/4/16 Robert Kern > On Fri, Apr 16, 2010 at 20:56, wrote: > > On Fri, Apr 16, 2010 at 9:47 PM, Pradeep Jha > wrote: > >> Hi, > >> > >> before running the make command, it asks me to run the preconfig file. > When > >> I run preconfig, I get the following warning message during the run: > >> checking for python2... /usr/bin/python2 > >> Python command set by configure to /usr/bin/python2 > >> WARNING: NUMPY include dir > >> /nfs/carv/d1/people/pradeep/src/numpy-1.3.0//include/python does not > exist. > >> setting NUMPY_INC_DIR to > >> /nfs/carv/d1/people/pradeep/src/numpy-1.3.0//include/python > > > > Is duplicate forward-slash legal in a Linux path? > > Yes. It is ignored. > > You need to set NUMPY_INC_DIR to > $PYTHON_SITE_PACKAGES/numpy/core/include where $PYTHON_SITE_PACKAGES > is the site-packages/ directory where your installation of numpy got > installed to. Note that this is not > /nfs/carv/d1/people/pradeep/src/numpy-1.3.0/. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > -- Umberto Eco > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jhapk at utias.utoronto.ca Sun Apr 18 14:32:01 2010 From: jhapk at utias.utoronto.ca (Pradeep Jha) Date: Sun, 18 Apr 2010 14:32:01 -0400 Subject: [Numpy-discussion] Numpy compilation error In-Reply-To: References: Message-ID: Hi, I am running python 2.4.4. I installed numpy using the command* * *python setup.py install --prefix=$HOME/**bin* my site-packages directory got installed to the following address:* /nfs/carv/d1/people/pradeep/bin/lib/python2.4* in my preconfig file I changed the path to the following as you recommended * # If numpy was installed using the --home option, set this to the # home directory for numpy. This will be needed for all numpy installations # that don't put the include files into python's native include directory. NUMPY_HOME=${NUMPY_HOME:="/nfs/carv/d1/people/pradeep/bin/lib/python2.4/site-packages/numpy/core/include"} * I still get the following warning message when I run the preconfig file:* checking for python2... /usr/bin/python2 Python command set by configure to /usr/bin/python2 WARNING: NUMPY include dir /nfs/carv/d1/people/pradeep/bin/lib/python2.4/site-packages/numpy/core/include/include/python does not exist. setting NUMPY_INC_DIR to /nfs/carv/d1/people/pradeep/bin/lib/python2.4/site-packages/numpy/core/include/include/python * Whats going wrong? The warning is right, as there is no directory called *python *on that address. Just to give you the structure of the installed packages, I am attaching the output to the cd and ls commands: *[pradeep at scaramanga lib]$ls python2.4 [pradeep at scaramanga lib]$cd python2.4/ [pradeep at scaramanga python2.4]$ls site-packages [pradeep at scaramanga python2.4]$cd site-packages/ [pradeep at scaramanga site-packages]$ls numpy [pradeep at scaramanga site-packages]$cd numpy/ [pradeep at scaramanga numpy]$ls add_newdocs.py __config__.pyc DEV_README.txt dual.pyc _import_tools.pyc lib matlib.py random setupscons.py tests add_newdocs.pyc core distutils f2py __init__.py LICENSE.txt matlib.pyc README.txt setupscons.pyc THANKS.txt COMPATIBILITY ctypeslib.py doc fft __init__.pyc linalg numarray setup.py site.cfg.example version.py __config__.py ctypeslib.pyc dual.py _import_tools.py INSTALL.txt ma oldnumeric setup.pyc testing version.pyc [pradeep at scaramanga numpy]$cd core/ [pradeep at scaramanga core]$ls arrayprint.py defmatrix.pyc include _internal.py numeric.py records.pyc setup_common.pyc _sort.so arrayprint.pyc fromnumeric.py info.py _internal.pyc numeric.pyc scalarmath.so setup.py tests defchararray.py fromnumeric.pyc info.pyc memmap.py numerictypes.py scons_support.py setup.pyc umath.so defchararray.pyc generate_numpy_api.py __init__.py memmap.pyc numerictypes.pyc scons_support.pyc setupscons.py umath_tests.so defmatrix.py generate_numpy_api.pyc __init__.pyc multiarray.so records.py setup_common.py setupscons.pyc [pradeep at scaramanga core]$cd include/ [pradeep at scaramanga include]$ls numpy [pradeep at scaramanga include]$cd numpy/ [pradeep at scaramanga numpy]$ls arrayobject.h mingw_amd64_fenv.h multiarray_api.txt noprefix.h npy_cpu.h npy_interrupt.h numpyconfig.h oldnumeric.h ufunc_api.txt utils.h arrayscalars.h __multiarray_api.h ndarrayobject.h npy_common.h npy_endian.h npy_math.h old_defines.h __ufunc_api.h ufuncobject.h * Thanks 2010/4/18 Pradeep Jha > Hi, > > how do I figure where did my numpy got installed to? I downloaded a tar.gz > file and then unzipped it at > /nfs/carv/d1/people/pradeep/src/numpy-1.3.0/. > I didn't do any additional installation steps. Also, just fyi, I don't have > root access to this system. > Thanks > > > 2010/4/16 Robert Kern > >> On Fri, Apr 16, 2010 at 20:56, wrote: >> >> > On Fri, Apr 16, 2010 at 9:47 PM, Pradeep Jha >> wrote: >> >> Hi, >> >> >> >> before running the make command, it asks me to run the preconfig file. >> When >> >> I run preconfig, I get the following warning message during the run: >> >> checking for python2... /usr/bin/python2 >> >> Python command set by configure to /usr/bin/python2 >> >> WARNING: NUMPY include dir >> >> /nfs/carv/d1/people/pradeep/src/numpy-1.3.0//include/python does not >> exist. >> >> setting NUMPY_INC_DIR to >> >> /nfs/carv/d1/people/pradeep/src/numpy-1.3.0//include/python >> > >> > Is duplicate forward-slash legal in a Linux path? >> >> Yes. It is ignored. >> >> You need to set NUMPY_INC_DIR to >> $PYTHON_SITE_PACKAGES/numpy/core/include where $PYTHON_SITE_PACKAGES >> is the site-packages/ directory where your installation of numpy got >> installed to. Note that this is not >> /nfs/carv/d1/people/pradeep/src/numpy-1.3.0/. >> >> -- >> Robert Kern >> >> "I have come to believe that the whole world is an enigma, a harmless >> enigma that is made terrible by our own mad attempt to interpret it as >> though it had an underlying truth." >> -- Umberto Eco >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sun Apr 18 19:06:54 2010 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 18 Apr 2010 18:06:54 -0500 Subject: [Numpy-discussion] Numpy compilation error In-Reply-To: References: Message-ID: On Sun, Apr 18, 2010 at 13:32, Pradeep Jha wrote: > Hi, > > I am running python 2.4.4. I installed numpy using the command > > python setup.py install --prefix=$HOME/bin That's not a good prefix. --prefix=$HOME would be better. It will create, if necessary, bin/ and lib/ underneath that. > my site-packages directory got installed to the following address: > > /nfs/carv/d1/people/pradeep/bin/lib/python2.4 > > in my preconfig file I changed the path to the following as you recommended > > # If numpy was installed using the --home option, set this to the > # home directory for numpy. This will be needed for all numpy installations > # that don't put the include files into python's native include directory. > NUMPY_HOME=${NUMPY_HOME:="/nfs/carv/d1/people/pradeep/bin/lib/python2.4/site-packages/numpy/core/include"} > > I still get the following warning message when I run the preconfig file: > checking for python2... /usr/bin/python2 > Python command set by configure to? /usr/bin/python2 > WARNING: NUMPY include dir > /nfs/carv/d1/people/pradeep/bin/lib/python2.4/site-packages/numpy/core/include/include/python > does not exist. > setting NUMPY_INC_DIR to > /nfs/carv/d1/people/pradeep/bin/lib/python2.4/site-packages/numpy/core/include/include/python > > Whats going wrong? The warning is right, as there is no directory called > python? on that address. You need to set the variable NUMPY_INC_DIR to '/nfs/carv/d1/people/pradeep/bin/lib/python2.4/site-packages/numpy/core/include/' or the equivalent if you decide to fix the --prefix as above. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From jhapk at utias.utoronto.ca Sun Apr 18 20:49:19 2010 From: jhapk at utias.utoronto.ca (Pradeep Jha) Date: Sun, 18 Apr 2010 20:49:19 -0400 Subject: [Numpy-discussion] Numpy compilation error In-Reply-To: References: Message-ID: Hi, I am setting NUMPY_INC_DIR to what you said but still am getting the same error. Basically, there is no directory path like "*******/include/python/***" anywhere in the installed numpy directory. whatever I am setting in my NUMPY_INC_DIR, the preconfig file tries to add a /include/python to that string and tries to locate that path, but fails. 2010/4/18 Robert Kern > On Sun, Apr 18, 2010 at 13:32, Pradeep Jha > wrote: > > Hi, > > > > I am running python 2.4.4. I installed numpy using the command > > > > python setup.py install --prefix=$HOME/bin > > That's not a good prefix. --prefix=$HOME would be better. It will > create, if necessary, bin/ and lib/ underneath that. > > > my site-packages directory got installed to the following address: > > > > /nfs/carv/d1/people/pradeep/bin/lib/python2.4 > > > > in my preconfig file I changed the path to the following as you > recommended > > > > # If numpy was installed using the --home option, set this to the > > # home directory for numpy. This will be needed for all numpy > installations > > # that don't put the include files into python's native include > directory. > > > NUMPY_HOME=${NUMPY_HOME:="/nfs/carv/d1/people/pradeep/bin/lib/python2.4/site-packages/numpy/core/include"} > > > > I still get the following warning message when I run the preconfig file: > > checking for python2... /usr/bin/python2 > > Python command set by configure to /usr/bin/python2 > > WARNING: NUMPY include dir > > > /nfs/carv/d1/people/pradeep/bin/lib/python2.4/site-packages/numpy/core/include/include/python > > does not exist. > > setting NUMPY_INC_DIR to > > > /nfs/carv/d1/people/pradeep/bin/lib/python2.4/site-packages/numpy/core/include/include/python > > > > Whats going wrong? The warning is right, as there is no directory called > > python on that address. > > You need to set the variable NUMPY_INC_DIR to > > '/nfs/carv/d1/people/pradeep/bin/lib/python2.4/site-packages/numpy/core/include/' > or the equivalent if you decide to fix the --prefix as above. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > -- Umberto Eco > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sun Apr 18 21:20:59 2010 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 18 Apr 2010 20:20:59 -0500 Subject: [Numpy-discussion] Numpy compilation error In-Reply-To: References: Message-ID: On Sun, Apr 18, 2010 at 19:49, Pradeep Jha wrote: > Hi, > > I am setting NUMPY_INC_DIR to what you said but still am getting the same > error. Basically, > there is no directory path like "*******/include/python/***" anywhere in the > installed numpy directory. This is correct. No such directory should exist. > whatever I am setting in my NUMPY_INC_DIR, the > preconfig file tries to add a /include/python to that string and tries to > locate that path, but fails. I am not familiar with the Cantera build process. You will probably need to ask the Cantera developers. You could probably replace the line in setup.py.in (sorry if this is not exactly right; I only downloaded the 1.7.0 sources, which still uses numarray): numpy_incl = "@NUMPY_INC_DIR@" to: import numpy numpy_incl = numpy.get_include() Then the value of NUMPY_INC_DIR will not matter. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From ademan555 at gmail.com Sun Apr 18 21:57:27 2010 From: ademan555 at gmail.com (Dan Roberts) Date: Sun, 18 Apr 2010 18:57:27 -0700 Subject: [Numpy-discussion] My GSoC Proposal to Implement a Subset of NumPy for PyPy In-Reply-To: References: Message-ID: On Apr 18, 2010 6:46 PM, "Dan Roberts" wrote: I've been trying my best to take my time formulating my replies but I need to respond eventually. :-) This is embarassing, but I'm actually not sure where I talked about an interface specifically. I did rather nebulously talk about interfacing with C code and LAPACK and the interfaces there would be provided by the respective code and consumed by micronumpy, at least as I currently see it. I haven't consulted maciej about this yet, but I think working backwards from a complete C NumPy depends on a great deal of "ifs", many of which I think aren't satisfied. I need to look into this, but I assume NumPy operates on array structures directly, rather than through an interface. If it's through an interface, there's a real possibility that approach is possible, though it would require me to write some adaptors, I think it would be ok, and a low enough time investment. Like I said I'm currently speaking from ignorance so I need to look into it and get back to you. Cheers, Dan P.S. I agree about the sparse matrices, I've bugged fijal a small bit about that. P.P.S. Forgot to CC the mailing list... assumed this mail client would do it for me.. lol > > On Apr 17, 2010 12:25 AM, "St?fan van der Walt" wrote: > > Hi Dan > > > On 17 April 2010 06:50, Dan Roberts wrote: > Hi everybody, my name is Dan... > > Thanks for the introduction, and welcome to NumPy! > > > > I hadn't prepared for review by the NumPy mentors, but this can make my > proposal stronger t... > > This proposal builds a bridge between two projects, so even if it > technically falls under the... > > > > Why should we bother reimplimenting anything? PyPy, for those who are > unfamiliar, has the ... > > Your code has a fairly specialised application and it's worth > discussing exactly where it wou... -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Mon Apr 19 00:25:07 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Mon, 19 Apr 2010 12:25:07 +0800 Subject: [Numpy-discussion] Release candidate 3 for NumPy 1.4.1 and SciPy 0.7.2 Message-ID: Hi, I am pleased to announce the third release candidate of both Scipy 0.7.2 and NumPy 1.4.1. Please test, and report any problems on the NumPy or SciPy list. Binaries, sources and release notes can be found at https://sourceforge.net/projects/numpy/files/ https://sourceforge.net/projects/scipy/files/ Changes from RC2 ============== SciPy: warnings about possible binary incompatibilities with numpy have been suppressed NumPy: - fixed compatibility with Python 2.7b1 - marked test for complex log as a known failure NumPy 1.4.1 ========== The main change over 1.4.0 is that datetime support has been removed. This fixes the binary incompatibility issues between NumPy and other libraries like SciPy and Matplotlib. There are also a number of other bug fixes, and no new features. Binaries for Python 2.5 and 2.6 are available for both Windows and OS X. SciPy 0.7.2 ========= The only change compared to 0.7.1 is that the C sources for Cython code have been regenerated with Cython 0.12.1. This ensures that SciPy 0.7.2 will work with NumPy 1.4.1, while also retaining backwards compatibility with NumPy 1.3.0. Note that the 0.7.x branch was created in January 2009, so a lot of fixes and new functionality in current trunk is not present in this release. Binaries for Python 2.6 are available for both Windows and OS X. Due to the age of the code no binaries for Python 2.5 are available. On behalf of the NumPy and SciPy developers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From joschu at caltech.edu Mon Apr 19 02:16:35 2010 From: joschu at caltech.edu (John Schulman) Date: Sun, 18 Apr 2010 23:16:35 -0700 Subject: [Numpy-discussion] Infix operators for numpy arrays Message-ID: Hi all, I just posted a recipe (http://code.activestate.com/recipes/577201/) for infix operators that work with numpy arrays, so you can write X *dot* Y. It works by making the Infix class a subclass of ndarray. John From pnorthug at gmail.com Mon Apr 19 02:36:00 2010 From: pnorthug at gmail.com (Paul Northug) Date: Sun, 18 Apr 2010 23:36:00 -0700 Subject: [Numpy-discussion] reformulating a series of correlations as one fft, ifft pair Message-ID: I am having trouble reformulating a series of correlations as a single fft, ifft pair. I have a set of kernels phi : (M = channel, N = kernel, T = time) correlated with signal a : (N, P+T-1) yielding x : (M, T). The correlation, for now, is only in the last dimension, with the two other dimensions degenerate. Below are 4 implementations (x == y == z != w). I would like to reformulate the operation as a single pair of 3d ffts to compare how it fares for different problem dimensions but I am not able to do it. Implementation #4 is incorrect. Please note the .conj() to get correlations rather than convolutions. Any tips would be appreciated. Optimally, I would not have to pad with too many zeros as the matrices can be quite large. The challenge is to beat implementation 2 as P grows. M, N, P are typically on the order of 10^2 and T, 10^3. # import numpy as np from scipy.signal import fftn, ifftn, correlate M, N, P, T = 2, 3, 4, 5 phi = np.random.randn(M, N, P) a = np.random.randn(N, P+T-1) x = np.zeros((M, T)) # 1. correlate x = np.array([correlate(phi[i], a, 'valid').squeeze() for i in range(M)]) # 2. dot products y = np.zeros_like(x) for b in range(P): y += np.dot(phi[:,:,b], a[:,b:b+T]) # 3. a series of ffts (code adapted from scipy convolve) s1 = np.array(phi.shape[1:]) s2 = np.array(a.shape) size = s1+s2-1 z = np.empty_like(y) fsize = (2**np.ceil(np.log2(size))).astype(np.int) # power of 2 pad af = fftn(a, fsize) fslice = tuple([slice(0, sz) for sz in size]) for i in range(M): phif = fftn(phi[i], fsize).conj() z[i] = np.real(ifftn(phif * af)[fslice])[0,:T] # 4. single fft, ifft pair - some rows are correct s1 = np.array(phi.shape) s2 = np.array(a[np.newaxis].shape) size = s1+s2-1 fsize = (2**np.ceil(np.log2(size))).astype(np.int) af = fftn(a[np.newaxis], fsize) phif = fftn(phi, fsize).conj() fslice = tuple([slice(0, sz) for sz in size]) w = np.real(ifftn(phif * af)[fslice])[:,:,:T] From seb.haase at gmail.com Mon Apr 19 03:35:57 2010 From: seb.haase at gmail.com (Sebastian Haase) Date: Mon, 19 Apr 2010 09:35:57 +0200 Subject: [Numpy-discussion] Release candidate 3 for NumPy 1.4.1 and SciPy 0.7.2 In-Reply-To: References: Message-ID: Hi, Congratulations. I might be unnecessarily dense - but what SciPy am I supposed to use with the new numpy 1.4.1 for Python 2.5? I'm surprised that there are no SciPy 0.7.2 binaries for Python 2.5 - is that technically not possible ? Thanks, Sebastian Haase On Mon, Apr 19, 2010 at 6:25 AM, Ralf Gommers wrote: > Hi, > > I am pleased to announce the third release candidate of both Scipy 0.7.2 and > NumPy 1.4.1. Please test, and report any problems on the NumPy or SciPy > list. > > Binaries, sources and release notes can be found at > https://sourceforge.net/projects/numpy/files/ > https://sourceforge.net/projects/scipy/files/ > > > Changes from RC2 > ============== > SciPy: warnings about possible binary incompatibilities with numpy have been > suppressed > NumPy: - fixed compatibility with Python 2.7b1 > ?????? - marked test for complex log as a known failure > > > NumPy 1.4.1 > ========== > The main change over 1.4.0 is that datetime support has been removed. This > fixes the binary incompatibility issues between NumPy and other libraries > like SciPy and Matplotlib. > > There are also a number of other bug fixes, and no new features. > > Binaries for Python 2.5 and 2.6 are available for both Windows and OS X. > > > SciPy 0.7.2 > ========= > The only change compared to 0.7.1 is that the C sources for Cython code have > been regenerated with Cython 0.12.1. This ensures that SciPy 0.7.2 will work > with NumPy 1.4.1, while also retaining backwards compatibility with NumPy > 1.3.0. > > Note that the 0.7.x branch was created in January 2009, so a lot of fixes > and new functionality in current trunk is not present in this release. > > Binaries for Python 2.6 are available for both Windows and OS X. Due to the > age of the code no binaries for Python 2.5 are available. > > > On behalf of the NumPy and SciPy developers, > Ralf > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From matthieu.brucher at gmail.com Mon Apr 19 03:45:07 2010 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Mon, 19 Apr 2010 09:45:07 +0200 Subject: [Numpy-discussion] Release candidate 3 for NumPy 1.4.1 and SciPy 0.7.2 In-Reply-To: References: Message-ID: Hi, I'm trying to compile scipy with ICC (numpy got through correctly), but I have issue with infinites in cephes: icc: scipy/special/cephes/const.c scipy/special/cephes/const.c(94): error: floating-point operation result is out of range double INFINITY = 1.0/0.0; /* 99e999; */ ^ scipy/special/cephes/const.c(99): error: floating-point operation result is out of range double NAN = 1.0/0.0 - 1.0/0.0; ^ scipy/special/cephes/const.c(99): error: floating-point operation result is out of range double NAN = 1.0/0.0 - 1.0/0.0; ^ compilation aborted for scipy/special/cephes/const.c (code 2) scipy/special/cephes/const.c(94): error: floating-point operation result is out of range double INFINITY = 1.0/0.0; /* 99e999; */ ^ scipy/special/cephes/const.c(99): error: floating-point operation result is out of range double NAN = 1.0/0.0 - 1.0/0.0; ^ scipy/special/cephes/const.c(99): error: floating-point operation result is out of range double NAN = 1.0/0.0 - 1.0/0.0; ^ compilation aborted for scipy/special/cephes/const.c (code 2) Matthieu 2010/4/19 Sebastian Haase : > Hi, > Congratulations. I might be unnecessarily dense - but what SciPy am I > supposed to use with the new numpy 1.4.1 for Python 2.5? I'm surprised > that there are no SciPy 0.7.2 binaries for Python 2.5 - is that > technically not possible ? > > Thanks, > Sebastian Haase > > On Mon, Apr 19, 2010 at 6:25 AM, Ralf Gommers > wrote: >> Hi, >> >> I am pleased to announce the third release candidate of both Scipy 0.7.2 and >> NumPy 1.4.1. Please test, and report any problems on the NumPy or SciPy >> list. >> >> Binaries, sources and release notes can be found at >> https://sourceforge.net/projects/numpy/files/ >> https://sourceforge.net/projects/scipy/files/ >> >> >> Changes from RC2 >> ============== >> SciPy: warnings about possible binary incompatibilities with numpy have been >> suppressed >> NumPy: - fixed compatibility with Python 2.7b1 >> ?????? - marked test for complex log as a known failure >> >> >> NumPy 1.4.1 >> ========== >> The main change over 1.4.0 is that datetime support has been removed. This >> fixes the binary incompatibility issues between NumPy and other libraries >> like SciPy and Matplotlib. >> >> There are also a number of other bug fixes, and no new features. >> >> Binaries for Python 2.5 and 2.6 are available for both Windows and OS X. >> >> >> SciPy 0.7.2 >> ========= >> The only change compared to 0.7.1 is that the C sources for Cython code have >> been regenerated with Cython 0.12.1. This ensures that SciPy 0.7.2 will work >> with NumPy 1.4.1, while also retaining backwards compatibility with NumPy >> 1.3.0. >> >> Note that the 0.7.x branch was created in January 2009, so a lot of fixes >> and new functionality in current trunk is not present in this release. >> >> Binaries for Python 2.6 are available for both Windows and OS X. Due to the >> age of the code no binaries for Python 2.5 are available. >> >> >> On behalf of the NumPy and SciPy developers, >> Ralf >> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher From matthieu.brucher at gmail.com Mon Apr 19 04:10:20 2010 From: matthieu.brucher at gmail.com (Matthieu Brucher) Date: Mon, 19 Apr 2010 10:10:20 +0200 Subject: [Numpy-discussion] Release candidate 3 for NumPy 1.4.1 and SciPy 0.7.2 In-Reply-To: References: Message-ID: BTW, there still is an error with ifort, so scipy is still incompatible with the Intel compilers (which is at least very sad...) Matthieu 2010/4/19 Matthieu Brucher : > Hi, > > I'm trying to compile scipy with ICC (numpy got through correctly), > but I have issue with infinites in cephes: > > icc: scipy/special/cephes/const.c > scipy/special/cephes/const.c(94): error: floating-point operation > result is out of range > ?double INFINITY = 1.0/0.0; ?/* 99e999; */ > ? ? ? ? ? ? ? ? ? ? ? ^ > > scipy/special/cephes/const.c(99): error: floating-point operation > result is out of range > ?double NAN = 1.0/0.0 - 1.0/0.0; > ? ? ? ? ? ? ? ? ?^ > > scipy/special/cephes/const.c(99): error: floating-point operation > result is out of range > ?double NAN = 1.0/0.0 - 1.0/0.0; > ? ? ? ? ? ? ? ? ? ? ? ? ? ?^ > > compilation aborted for scipy/special/cephes/const.c (code 2) > scipy/special/cephes/const.c(94): error: floating-point operation > result is out of range > ?double INFINITY = 1.0/0.0; ?/* 99e999; */ > ? ? ? ? ? ? ? ? ? ? ? ^ > > scipy/special/cephes/const.c(99): error: floating-point operation > result is out of range > ?double NAN = 1.0/0.0 - 1.0/0.0; > ? ? ? ? ? ? ? ? ?^ > > scipy/special/cephes/const.c(99): error: floating-point operation > result is out of range > ?double NAN = 1.0/0.0 - 1.0/0.0; > ? ? ? ? ? ? ? ? ? ? ? ? ? ?^ > > compilation aborted for scipy/special/cephes/const.c (code 2) > > Matthieu > > 2010/4/19 Sebastian Haase : >> Hi, >> Congratulations. I might be unnecessarily dense - but what SciPy am I >> supposed to use with the new numpy 1.4.1 for Python 2.5? I'm surprised >> that there are no SciPy 0.7.2 binaries for Python 2.5 - is that >> technically not possible ? >> >> Thanks, >> Sebastian Haase >> >> On Mon, Apr 19, 2010 at 6:25 AM, Ralf Gommers >> wrote: >>> Hi, >>> >>> I am pleased to announce the third release candidate of both Scipy 0.7.2 and >>> NumPy 1.4.1. Please test, and report any problems on the NumPy or SciPy >>> list. >>> >>> Binaries, sources and release notes can be found at >>> https://sourceforge.net/projects/numpy/files/ >>> https://sourceforge.net/projects/scipy/files/ >>> >>> >>> Changes from RC2 >>> ============== >>> SciPy: warnings about possible binary incompatibilities with numpy have been >>> suppressed >>> NumPy: - fixed compatibility with Python 2.7b1 >>> ?????? - marked test for complex log as a known failure >>> >>> >>> NumPy 1.4.1 >>> ========== >>> The main change over 1.4.0 is that datetime support has been removed. This >>> fixes the binary incompatibility issues between NumPy and other libraries >>> like SciPy and Matplotlib. >>> >>> There are also a number of other bug fixes, and no new features. >>> >>> Binaries for Python 2.5 and 2.6 are available for both Windows and OS X. >>> >>> >>> SciPy 0.7.2 >>> ========= >>> The only change compared to 0.7.1 is that the C sources for Cython code have >>> been regenerated with Cython 0.12.1. This ensures that SciPy 0.7.2 will work >>> with NumPy 1.4.1, while also retaining backwards compatibility with NumPy >>> 1.3.0. >>> >>> Note that the 0.7.x branch was created in January 2009, so a lot of fixes >>> and new functionality in current trunk is not present in this release. >>> >>> Binaries for Python 2.6 are available for both Windows and OS X. Due to the >>> age of the code no binaries for Python 2.5 are available. >>> >>> >>> On behalf of the NumPy and SciPy developers, >>> Ralf >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion at scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > > > -- > Information System Engineer, Ph.D. > Blog: http://matt.eifelle.com > LinkedIn: http://www.linkedin.com/in/matthieubrucher > -- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher From ralf.gommers at googlemail.com Mon Apr 19 04:21:45 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Mon, 19 Apr 2010 16:21:45 +0800 Subject: [Numpy-discussion] Release candidate 3 for NumPy 1.4.1 and SciPy 0.7.2 In-Reply-To: References: Message-ID: On Mon, Apr 19, 2010 at 3:35 PM, Sebastian Haase wrote: > Hi, > Congratulations. I might be unnecessarily dense - but what SciPy am I > supposed to use with the new numpy 1.4.1 for Python 2.5? I'm surprised > that there are no SciPy 0.7.2 binaries for Python 2.5 - is that > technically not possible ? > > You're not being dense, there are no 2.5 scipy binaries. I did not succeed in building them. The scipy 0.7 branch is so old (from Jan 2009) that it has never been compiled on OS X 10.6, and I did not yet find a way to get it to work. For Windows I also had problems, there it should be compiled against numpy 1.2 while the 2.6 binaries are compiled against numpy 1.3. The variations in numpy, python and OS just added up to make it unworkable. I can give it another try after the final release, but first priority is to finally release. Cheers, Ralf > On Mon, Apr 19, 2010 at 6:25 AM, Ralf Gommers > wrote: > > Hi, > > > > I am pleased to announce the third release candidate of both Scipy 0.7.2 > and > > NumPy 1.4.1. Please test, and report any problems on the NumPy or SciPy > > list. > > > > Binaries, sources and release notes can be found at > > https://sourceforge.net/projects/numpy/files/ > > https://sourceforge.net/projects/scipy/files/ > > > > > > Changes from RC2 > > ============== > > SciPy: warnings about possible binary incompatibilities with numpy have > been > > suppressed > > NumPy: - fixed compatibility with Python 2.7b1 > > - marked test for complex log as a known failure > > > > > > NumPy 1.4.1 > > ========== > > The main change over 1.4.0 is that datetime support has been removed. > This > > fixes the binary incompatibility issues between NumPy and other libraries > > like SciPy and Matplotlib. > > > > There are also a number of other bug fixes, and no new features. > > > > Binaries for Python 2.5 and 2.6 are available for both Windows and OS X. > > > > > > SciPy 0.7.2 > > ========= > > The only change compared to 0.7.1 is that the C sources for Cython code > have > > been regenerated with Cython 0.12.1. This ensures that SciPy 0.7.2 will > work > > with NumPy 1.4.1, while also retaining backwards compatibility with NumPy > > 1.3.0. > > > > Note that the 0.7.x branch was created in January 2009, so a lot of fixes > > and new functionality in current trunk is not present in this release. > > > > Binaries for Python 2.6 are available for both Windows and OS X. Due to > the > > age of the code no binaries for Python 2.5 are available. > > > > > > On behalf of the NumPy and SciPy developers, > > Ralf > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion at scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Mon Apr 19 04:25:04 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Mon, 19 Apr 2010 16:25:04 +0800 Subject: [Numpy-discussion] Release candidate 3 for NumPy 1.4.1 and SciPy 0.7.2 In-Reply-To: References: Message-ID: On Mon, Apr 19, 2010 at 4:10 PM, Matthieu Brucher < matthieu.brucher at gmail.com> wrote: > BTW, there still is an error with ifort, so scipy is still > incompatible with the Intel compilers (which is at least very sad...) > > Is this true for trunk as well? Scipy 0.7 branch has hardly been touched for a year so it's not surprising that there's no progress on this. If you're compiling, I don't see much of a point in going for the 0.7 branch. Cheers, Ralf > Matthieu > > 2010/4/19 Matthieu Brucher : > > Hi, > > > > I'm trying to compile scipy with ICC (numpy got through correctly), > > but I have issue with infinites in cephes: > > > > icc: scipy/special/cephes/const.c > > scipy/special/cephes/const.c(94): error: floating-point operation > > result is out of range > > double INFINITY = 1.0/0.0; /* 99e999; */ > > ^ > > > > scipy/special/cephes/const.c(99): error: floating-point operation > > result is out of range > > double NAN = 1.0/0.0 - 1.0/0.0; > > ^ > > > > scipy/special/cephes/const.c(99): error: floating-point operation > > result is out of range > > double NAN = 1.0/0.0 - 1.0/0.0; > > ^ > > > > compilation aborted for scipy/special/cephes/const.c (code 2) > > scipy/special/cephes/const.c(94): error: floating-point operation > > result is out of range > > double INFINITY = 1.0/0.0; /* 99e999; */ > > ^ > > > > scipy/special/cephes/const.c(99): error: floating-point operation > > result is out of range > > double NAN = 1.0/0.0 - 1.0/0.0; > > ^ > > > > scipy/special/cephes/const.c(99): error: floating-point operation > > result is out of range > > double NAN = 1.0/0.0 - 1.0/0.0; > > ^ > > > > compilation aborted for scipy/special/cephes/const.c (code 2) > > > > Matthieu > > > > 2010/4/19 Sebastian Haase : > >> Hi, > >> Congratulations. I might be unnecessarily dense - but what SciPy am I > >> supposed to use with the new numpy 1.4.1 for Python 2.5? I'm surprised > >> that there are no SciPy 0.7.2 binaries for Python 2.5 - is that > >> technically not possible ? > >> > >> Thanks, > >> Sebastian Haase > >> > >> On Mon, Apr 19, 2010 at 6:25 AM, Ralf Gommers > >> wrote: > >>> Hi, > >>> > >>> I am pleased to announce the third release candidate of both Scipy > 0.7.2 and > >>> NumPy 1.4.1. Please test, and report any problems on the NumPy or SciPy > >>> list. > >>> > >>> Binaries, sources and release notes can be found at > >>> https://sourceforge.net/projects/numpy/files/ > >>> https://sourceforge.net/projects/scipy/files/ > >>> > >>> > >>> Changes from RC2 > >>> ============== > >>> SciPy: warnings about possible binary incompatibilities with numpy have > been > >>> suppressed > >>> NumPy: - fixed compatibility with Python 2.7b1 > >>> - marked test for complex log as a known failure > >>> > >>> > >>> NumPy 1.4.1 > >>> ========== > >>> The main change over 1.4.0 is that datetime support has been removed. > This > >>> fixes the binary incompatibility issues between NumPy and other > libraries > >>> like SciPy and Matplotlib. > >>> > >>> There are also a number of other bug fixes, and no new features. > >>> > >>> Binaries for Python 2.5 and 2.6 are available for both Windows and OS > X. > >>> > >>> > >>> SciPy 0.7.2 > >>> ========= > >>> The only change compared to 0.7.1 is that the C sources for Cython code > have > >>> been regenerated with Cython 0.12.1. This ensures that SciPy 0.7.2 will > work > >>> with NumPy 1.4.1, while also retaining backwards compatibility with > NumPy > >>> 1.3.0. > >>> > >>> Note that the 0.7.x branch was created in January 2009, so a lot of > fixes > >>> and new functionality in current trunk is not present in this release. > >>> > >>> Binaries for Python 2.6 are available for both Windows and OS X. Due to > the > >>> age of the code no binaries for Python 2.5 are available. > >>> > >>> > >>> On behalf of the NumPy and SciPy developers, > >>> Ralf > >>> > >>> _______________________________________________ > >>> NumPy-Discussion mailing list > >>> NumPy-Discussion at scipy.org > >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion > >>> > >>> > >> _______________________________________________ > >> NumPy-Discussion mailing list > >> NumPy-Discussion at scipy.org > >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > >> > > > > > > > > -- > > Information System Engineer, Ph.D. > > Blog: http://matt.eifelle.com > > LinkedIn: http://www.linkedin.com/in/matthieubrucher > > > > > > -- > Information System Engineer, Ph.D. > Blog: http://matt.eifelle.com > LinkedIn: http://www.linkedin.com/in/matthieubrucher > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From david at silveregg.co.jp Mon Apr 19 05:30:58 2010 From: david at silveregg.co.jp (David) Date: Mon, 19 Apr 2010 18:30:58 +0900 Subject: [Numpy-discussion] Release candidate 3 for NumPy 1.4.1 and SciPy 0.7.2 In-Reply-To: References: Message-ID: <4BCC22D2.3080705@silveregg.co.jp> On 04/19/2010 04:45 PM, Matthieu Brucher wrote: > Hi, > > I'm trying to compile scipy with ICC (numpy got through correctly), > but I have issue with infinites in cephes: > > icc: scipy/special/cephes/const.c > scipy/special/cephes/const.c(94): error: floating-point operation > result is out of range > double INFINITY = 1.0/0.0; /* 99e999; */ > ^ > > scipy/special/cephes/const.c(99): error: floating-point operation > result is out of range > double NAN = 1.0/0.0 - 1.0/0.0; > ^ > > scipy/special/cephes/const.c(99): error: floating-point operation > result is out of range > double NAN = 1.0/0.0 - 1.0/0.0; > ^ > > compilation aborted for scipy/special/cephes/const.c (code 2) > scipy/special/cephes/const.c(94): error: floating-point operation > result is out of range > double INFINITY = 1.0/0.0; /* 99e999; */ > ^ > > scipy/special/cephes/const.c(99): error: floating-point operation > result is out of range > double NAN = 1.0/0.0 - 1.0/0.0; > ^ > > scipy/special/cephes/const.c(99): error: floating-point operation > result is out of range > double NAN = 1.0/0.0 - 1.0/0.0; > ^ > > compilation aborted for scipy/special/cephes/const.c (code 2) All those have been fixed in scipy 0.8, and cannot be backported to scipy 0.7.x (because it requires the new math library as available from numpy 1.4.0). I know for sure that scipy trunk + numpy 1.4.x work with ifort + MSVC on windows 64 (more exactly it worked in december 2009), cheers, David From josef.pktd at gmail.com Mon Apr 19 09:01:27 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Mon, 19 Apr 2010 09:01:27 -0400 Subject: [Numpy-discussion] reformulating a series of correlations as one fft, ifft pair In-Reply-To: References: Message-ID: On Mon, Apr 19, 2010 at 2:36 AM, Paul Northug wrote: > I am having trouble reformulating a series of correlations as a single > fft, ifft pair. > > I have a set of kernels phi : (M = channel, N = kernel, T = time) > correlated with signal a : (N, P+T-1) yielding x : (M, T). > > The correlation, for now, is only in the last dimension, with the two > other dimensions degenerate. Below are 4 implementations (x == y == z > != w). I would like to reformulate the operation as a single pair of > 3d ffts to compare how it fares for different problem dimensions but I > am not able to do it. Implementation #4 is incorrect. Please note the > .conj() to get correlations rather than convolutions. > > Any tips would be appreciated. Optimally, I would not have to pad with > too many zeros as the matrices can be quite large. The challenge is to > beat implementation 2 as P grows. M, N, P are typically on the order > of 10^2 and T, 10^3. > > # > import numpy as np > from scipy.signal import fftn, ifftn, correlate > > M, N, P, T = 2, 3, 4, 5 > > phi = np.random.randn(M, N, P) > a = np.random.randn(N, P+T-1) > x = np.zeros((M, T)) > > # 1. correlate > > x = np.array([correlate(phi[i], a, 'valid').squeeze() for i in range(M)]) > > # 2. dot products > > y = np.zeros_like(x) > for b in range(P): y += np.dot(phi[:,:,b], a[:,b:b+T]) > > # 3. a series of ffts (code adapted from scipy convolve) > > s1 = np.array(phi.shape[1:]) > s2 = np.array(a.shape) > size = s1+s2-1 > z = np.empty_like(y) > fsize = (2**np.ceil(np.log2(size))).astype(np.int) # power of 2 pad > af = fftn(a, fsize) > fslice = tuple([slice(0, sz) for sz in size]) > for i in range(M): > ? ?phif = fftn(phi[i], fsize).conj() > ? ?z[i] = np.real(ifftn(phif * af)[fslice])[0,:T] > > # 4. single fft, ifft pair - some rows are correct > > s1 = np.array(phi.shape) > s2 = np.array(a[np.newaxis].shape) > size = s1+s2-1 > fsize = (2**np.ceil(np.log2(size))).astype(np.int) > af = fftn(a[np.newaxis], fsize) > phif = fftn(phi, fsize).conj() > fslice = tuple([slice(0, sz) for sz in size]) > w = np.real(ifftn(phif * af)[fslice])[:,:,:T] I only checked this in your example >>> np.squeeze(signal.fftconvolve(phi[:,::-1,::-1], a[None,...],mode='valid')) array([[ 0.42695078, 4.3433924 , -2.53084395, -4.22672281, -5.16705981], [-2.35108765, -2.15060513, -1.23528122, 1.40386599, 1.38189896]]) >>> y array([[ 0.42695078, 4.3433924 , -2.53084395, -4.22672281, -5.16705981], [-2.35108765, -2.15060513, -1.23528122, 1.40386599, 1.38189896]]) >>> x array([[ 0.42695078, 4.3433924 , -2.53084395, -4.22672281, -5.16705981], [-2.35108765, -2.15060513, -1.23528122, 1.40386599, 1.38189896]]) Josef > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From ralf.gommers at googlemail.com Mon Apr 19 09:19:21 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Mon, 19 Apr 2010 21:19:21 +0800 Subject: [Numpy-discussion] Release candidate 3 for NumPy 1.4.1 and SciPy 0.7.2 In-Reply-To: References: Message-ID: On Mon, Apr 19, 2010 at 4:21 PM, Ralf Gommers wrote: > > > On Mon, Apr 19, 2010 at 3:35 PM, Sebastian Haase wrote: > >> Hi, >> Congratulations. I might be unnecessarily dense - but what SciPy am I >> supposed to use with the new numpy 1.4.1 for Python 2.5? I'm surprised >> that there are no SciPy 0.7.2 binaries for Python 2.5 - is that >> technically not possible ? >> >> You're not being dense, there are no 2.5 scipy binaries. I did not succeed > in building them. The scipy 0.7 branch is so old (from Jan 2009) that it has > never been compiled on OS X 10.6, and I did not yet find a way to get it to > work. For Windows I also had problems, there it should be compiled against > numpy 1.2 while the 2.6 binaries are compiled against numpy 1.3. The > variations in numpy, python and OS just added up to make it unworkable. > > I can give it another try after the final release, but first priority is to > finally release. > To remind myself of the issue I tried building it again, and managed to build a 2.5 binary against numpy 1.3 on OS X at least. Can anyone tell me why 2.5 binaries are supposed to be built against numpy 1.2 and 2.6 binaries against numpy 1.3? Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From patricktmarsh at gmail.com Mon Apr 19 10:44:27 2010 From: patricktmarsh at gmail.com (Patrick Marsh) Date: Mon, 19 Apr 2010 09:44:27 -0500 Subject: [Numpy-discussion] Building DMG w/ Python 2.6 (could not access...no space left on device) Message-ID: Greetings, I'm trying to build DMG releases of numpy-1.4.1rc3 as part of the release process. I recently got my mac back from repairs and spent the weekend trying to get the build environment configured on it. I thought it was correct, but I've got errors when building with both Python 2.5.4 and Python 2.6.5. (I should add that, ironically, I have no problem building the Windows superpack on OS X [10.6].) Below is the end of the output from running 'paver dmg -p 2.6'. I have no idea why I'm told I'm out of space, nor do I know where to begin in trying to track that down. I'll send a separate email regarding my errors regarding Python 2.5 as it is an entirely different issue. Thanks in advance for any help... ============================================================ creating dist creating dist/numpy-1.4.1rc3-py2.6-macosx10.6.mpkg creating dist/numpy-1.4.1rc3-py2.6-macosx10.6.mpkg/Contents creating dist/numpy-1.4.1rc3-py2.6-macosx10.6.mpkg/Contents/Packages nothing to be installed for scheme purelib nothing to be installed for scheme headers creating dist/numpy-1.4.1rc3-py2.6-macosx10.6.mpkg/Contents/Packages/numpy-platlib-1.4.1rc3-py2.6-macosx10.6.pkg creating dist/numpy-1.4.1rc3-py2.6-macosx10.6.mpkg/Contents/Packages/numpy-platlib-1.4.1rc3-py2.6-macosx10.6.pkg/Contents creating dist/numpy-1.4.1rc3-py2.6-macosx10.6.mpkg/Contents/Packages/numpy-platlib-1.4.1rc3-py2.6-macosx10.6.pkg/Contents/Resources /usr/bin/mkbom build/bdist.macosx-10.3-fat/mpkg/platlib/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages dist/numpy-1.4.1rc3-py2.6-macosx10.6.mpkg/./Contents/Packages/numpy-platlib-1.4.1rc3-py2.6-macosx10.6.pkg/Contents/Archive.bom /bin/pax -w -f /Users/pmarsh/git/numpy.release/dist/numpy-1.4.1rc3-py2.6-macosx10.6.mpkg/Contents/Packages/numpy-platlib-1.4.1rc3-py2.6-macosx10.6.pkg/Contents/Archive.pax.gz -x cpio -z . creating dist/numpy-1.4.1rc3-py2.6-macosx10.6.mpkg/Contents/Packages/numpy-platlib-1.4.1rc3-py2.6-macosx10.6.pkg/Contents/Resources/English.lproj copying License.txt -> dist/numpy-1.4.1rc3-py2.6-macosx10.6.mpkg/./Contents/Packages/numpy-platlib-1.4.1rc3-py2.6-macosx10.6.pkg/Contents/Resources nothing to be installed for scheme data creating dist/numpy-1.4.1rc3-py2.6-macosx10.6.mpkg/Contents/Packages/numpy-scripts-1.4.1rc3-py2.6-macosx10.6.pkg creating dist/numpy-1.4.1rc3-py2.6-macosx10.6.mpkg/Contents/Packages/numpy-scripts-1.4.1rc3-py2.6-macosx10.6.pkg/Contents creating dist/numpy-1.4.1rc3-py2.6-macosx10.6.mpkg/Contents/Packages/numpy-scripts-1.4.1rc3-py2.6-macosx10.6.pkg/Contents/Resources /usr/bin/mkbom build/bdist.macosx-10.3-fat/mpkg/scripts/usr/local/bin dist/numpy-1.4.1rc3-py2.6-macosx10.6.mpkg/./Contents/Packages/numpy-scripts-1.4.1rc3-py2.6-macosx10.6.pkg/Contents/Archive.bom /bin/pax -w -f /Users/pmarsh/git/numpy.release/dist/numpy-1.4.1rc3-py2.6-macosx10.6.mpkg/Contents/Packages/numpy-scripts-1.4.1rc3-py2.6-macosx10.6.pkg/Contents/Archive.pax.gz -x cpio -z . creating dist/numpy-1.4.1rc3-py2.6-macosx10.6.mpkg/Contents/Packages/numpy-scripts-1.4.1rc3-py2.6-macosx10.6.pkg/Contents/Resources/English.lproj copying License.txt -> dist/numpy-1.4.1rc3-py2.6-macosx10.6.mpkg/./Contents/Packages/numpy-scripts-1.4.1rc3-py2.6-macosx10.6.pkg/Contents/Resources creating dist/numpy-1.4.1rc3-py2.6-macosx10.6.mpkg/Contents/Resources creating dist/numpy-1.4.1rc3-py2.6-macosx10.6.mpkg/Contents/Resources/English.lproj copying License.txt -> dist/numpy-1.4.1rc3-py2.6-macosx10.6.mpkg/Contents/Resources removing 'build/bdist.macosx-10.3-fat/mpkg' (and everything under it) Creating disk image... ....................................................................................................................... 2010-04-19 09:17:58.614 copy-helper[10435:903] copy error (canceling): /Volumes/numpy/numpy-1.4.1rc3-py2.6.mpkg/Contents/Packages/numpy-platlib-1.4.1rc3-py2.6-macosx10.6.pkg/Contents/Archive.pax.gz: No space left on device ............................................................................could not access /Volumes/numpy/numpy-1.4.1rc3-py2.6.mpkg/Contents/Packages/numpy-platlib-1.4.1rc3-py2.6-macosx10.6.pkg/Contents/Archive.pax.gz - No space left on device hdiutil: create failed - No space left on device hdiutil: attach failed - No such file or directory mkdir: /Volumes/numpy: No such file or directory cp: /Volumes/numpy/.background/dmgbackground.png: No such file or directory 63:67: execution error: Finder got an error: Can?t get disk "numpy". (-1728) chmod: /Volumes/numpy: No such file or directory hdiutil: detach: no device name specified Usage: hdiutil detach [options] hdiutil detach -help hdiutil: convert failed - No such file or directory Captured Task Output: --------------------- ---> pavement.dmg ---> pavement.pdf ---> pavement.latex Build pdf doc ---> pavement.clean LDFLAGS='-undefined dynamic_lookup -bundle -arch i386 -arch ppc -Wl,-search_paths_first -Lbuild' /Library/Frameworks/Python.framework/Versions/2.6/bin/python setupegg.py bdist_mpkg Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/paver/tasks.py", line 170, in _run_task return do_task() File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/paver/tasks.py", line 167, in do_task return func(**kw) File "pavement.py", line 470, in dmg shutil.copy(source, target) File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/shutil.py", line 88, in copy copyfile(src, dst) File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/shutil.py", line 52, in copyfile fsrc = open(src, 'rb') IOError: [Errno 2] No such file or directory: 'tools/numpy-macosx-installer/numpy-1.4.1rc3-py2.6-python.org.dmg' Patrick -- Patrick Marsh Ph.D. Student / NSSL Liaison to the HWT School of Meteorology / University of Oklahoma Cooperative Institute for Mesoscale Meteorological Studies National Severe Storms Laboratory http://www.patricktmarsh.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From patrickmarshwx at gmail.com Mon Apr 19 10:58:27 2010 From: patrickmarshwx at gmail.com (Patrick Marsh) Date: Mon, 19 Apr 2010 09:58:27 -0500 Subject: [Numpy-discussion] Building DMG w/ Python 2.6 (could not access...no space left on device) Message-ID: Greetings, (I apologize if this becomes a duplicate. I originally sent this email from the wrong email address.) I'm trying to build DMG releases of numpy-1.4.1rc3 as part of the release process. I recently got my mac back from repairs and spent the weekend trying to get the build environment configured on it. I thought it was correct, but I've got errors when building with both Python 2.5.4 and Python 2.6.5. (I should add that, ironically, I have no problem building the Windows superpack on OS X [10.6].) Below is the end of the output from running 'paver dmg -p 2.6'. I have no idea why I'm told I'm out of space, nor do I know where to begin in trying to track that down. I'll send a separate email regarding my errors regarding Python 2.5 as it is an entirely different issue. Thanks in advance for any help... ============================================================ creating dist creating dist/numpy-1.4.1rc3-py2.6-macosx10.6.mpkg creating dist/numpy-1.4.1rc3-py2.6-macosx10.6.mpkg/Contents creating dist/numpy-1.4.1rc3-py2.6-macosx10.6.mpkg/Contents/Packages nothing to be installed for scheme purelib nothing to be installed for scheme headers creating dist/numpy-1.4.1rc3-py2.6-macosx10.6.mpkg/Contents/Packages/numpy-platlib-1.4.1rc3-py2.6-macosx10.6.pkg creating dist/numpy-1.4.1rc3-py2.6-macosx10.6.mpkg/Contents/Packages/numpy-platlib-1.4.1rc3-py2.6-macosx10.6.pkg/Contents creating dist/numpy-1.4.1rc3-py2.6-macosx10.6.mpkg/Contents/Packages/numpy-platlib-1.4.1rc3-py2.6-macosx10.6.pkg/Contents/Resources /usr/bin/mkbom build/bdist.macosx-10.3-fat/mpkg/platlib/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages dist/numpy-1.4.1rc3-py2.6-macosx10.6.mpkg/./Contents/Packages/numpy-platlib-1.4.1rc3-py2.6-macosx10.6.pkg/Contents/Archive.bom /bin/pax -w -f /Users/pmarsh/git/numpy.release/dist/numpy-1.4.1rc3-py2.6-macosx10.6.mpkg/Contents/Packages/numpy-platlib-1.4.1rc3-py2.6-macosx10.6.pkg/Contents/Archive.pax.gz -x cpio -z . creating dist/numpy-1.4.1rc3-py2.6-macosx10.6.mpkg/Contents/Packages/numpy-platlib-1.4.1rc3-py2.6-macosx10.6.pkg/Contents/Resources/English.lproj copying License.txt -> dist/numpy-1.4.1rc3-py2.6-macosx10.6.mpkg/./Contents/Packages/numpy-platlib-1.4.1rc3-py2.6-macosx10.6.pkg/Contents/Resources nothing to be installed for scheme data creating dist/numpy-1.4.1rc3-py2.6-macosx10.6.mpkg/Contents/Packages/numpy-scripts-1.4.1rc3-py2.6-macosx10.6.pkg creating dist/numpy-1.4.1rc3-py2.6-macosx10.6.mpkg/Contents/Packages/numpy-scripts-1.4.1rc3-py2.6-macosx10.6.pkg/Contents creating dist/numpy-1.4.1rc3-py2.6-macosx10.6.mpkg/Contents/Packages/numpy-scripts-1.4.1rc3-py2.6-macosx10.6.pkg/Contents/Resources /usr/bin/mkbom build/bdist.macosx-10.3-fat/mpkg/scripts/usr/local/bin dist/numpy-1.4.1rc3-py2.6-macosx10.6.mpkg/./Contents/Packages/numpy-scripts-1.4.1rc3-py2.6-macosx10.6.pkg/Contents/Archive.bom /bin/pax -w -f /Users/pmarsh/git/numpy.release/dist/numpy-1.4.1rc3-py2.6-macosx10.6.mpkg/Contents/Packages/numpy-scripts-1.4.1rc3-py2.6-macosx10.6.pkg/Contents/Archive.pax.gz -x cpio -z . creating dist/numpy-1.4.1rc3-py2.6-macosx10.6.mpkg/Contents/Packages/numpy-scripts-1.4.1rc3-py2.6-macosx10.6.pkg/Contents/Resources/English.lproj copying License.txt -> dist/numpy-1.4.1rc3-py2.6-macosx10.6.mpkg/./Contents/Packages/numpy-scripts-1.4.1rc3-py2.6-macosx10.6.pkg/Contents/Resources creating dist/numpy-1.4.1rc3-py2.6-macosx10.6.mpkg/Contents/Resources creating dist/numpy-1.4.1rc3-py2.6-macosx10.6.mpkg/Contents/Resources/English.lproj copying License.txt -> dist/numpy-1.4.1rc3-py2.6-macosx10.6.mpkg/Contents/Resources removing 'build/bdist.macosx-10.3-fat/mpkg' (and everything under it) Creating disk image... ....................................................................................................................... 2010-04-19 09:17:58.614 copy-helper[10435:903] copy error (canceling): /Volumes/numpy/numpy-1.4.1rc3-py2.6.mpkg/Contents/Packages/numpy-platlib-1.4.1rc3-py2.6-macosx10.6.pkg/Contents/Archive.pax.gz: No space left on device ............................................................................could not access /Volumes/numpy/numpy-1.4.1rc3-py2.6.mpkg/Contents/Packages/numpy-platlib-1.4.1rc3-py2.6-macosx10.6.pkg/Contents/Archive.pax.gz - No space left on device hdiutil: create failed - No space left on device hdiutil: attach failed - No such file or directory mkdir: /Volumes/numpy: No such file or directory cp: /Volumes/numpy/.background/dmgbackground.png: No such file or directory 63:67: execution error: Finder got an error: Can?t get disk "numpy". (-1728) chmod: /Volumes/numpy: No such file or directory hdiutil: detach: no device name specified Usage: hdiutil detach [options] hdiutil detach -help hdiutil: convert failed - No such file or directory Captured Task Output: --------------------- ---> pavement.dmg ---> pavement.pdf ---> pavement.latex Build pdf doc ---> pavement.clean LDFLAGS='-undefined dynamic_lookup -bundle -arch i386 -arch ppc -Wl,-search_paths_first -Lbuild' /Library/Frameworks/Python.framework/Versions/2.6/bin/python setupegg.py bdist_mpkg Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/paver/tasks.py", line 170, in _run_task return do_task() File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/paver/tasks.py", line 167, in do_task return func(**kw) File "pavement.py", line 470, in dmg shutil.copy(source, target) File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/shutil.py", line 88, in copy copyfile(src, dst) File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/shutil.py", line 52, in copyfile fsrc = open(src, 'rb') IOError: [Errno 2] No such file or directory: 'tools/numpy-macosx-installer/numpy-1.4.1rc3-py2.6-python.org.dmg' Patrick -- Patrick Marsh Ph.D. Student / NSSL Liaison to the HWT School of Meteorology / University of Oklahoma Cooperative Institute for Mesoscale Meteorological Studies National Severe Storms Laboratory http://www.patricktmarsh.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From jhapk at utias.utoronto.ca Mon Apr 19 10:58:24 2010 From: jhapk at utias.utoronto.ca (Pradeep Jha) Date: Mon, 19 Apr 2010 10:58:24 -0400 Subject: [Numpy-discussion] Numpy compilation error In-Reply-To: References: Message-ID: Thank you so much Robert. You are awesome :) That totally the problem. One more question for you. Which are the things that you have to declare in PYTHONPATH manually? 2010/4/18 Robert Kern > On Sun, Apr 18, 2010 at 19:49, Pradeep Jha > wrote: > > Hi, > > > > I am setting NUMPY_INC_DIR to what you said but still am getting the same > > error. Basically, > > there is no directory path like "*******/include/python/***" anywhere in > the > > installed numpy directory. > > This is correct. No such directory should exist. > > > whatever I am setting in my NUMPY_INC_DIR, the > > preconfig file tries to add a /include/python to that string and tries to > > locate that path, but fails. > > I am not familiar with the Cantera build process. You will probably > need to ask the Cantera developers. > > You could probably replace the line in setup.py.in (sorry if this is > not exactly right; I only downloaded the 1.7.0 sources, which still > uses numarray): > > numpy_incl = "@NUMPY_INC_DIR@" > > to: > > import numpy > numpy_incl = numpy.get_include() > > Then the value of NUMPY_INC_DIR will not matter. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > -- Umberto Eco > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From patrickmarshwx at gmail.com Mon Apr 19 10:55:52 2010 From: patrickmarshwx at gmail.com (Patrick Marsh) Date: Mon, 19 Apr 2010 09:55:52 -0500 Subject: [Numpy-discussion] Brooken Toolchain (Mac 10.6) Message-ID: Greetings, Per my previous email, I'm trying to setup the release process for Numpy on my MacBook Pro. When trying to build Numpy 1.4.1r3 with Python 2.5.4 I get a broken toolchain error (below). I do not get this error when trying to build Numpy with Python 2.6.5 - and there is nothing fundamentally different (that I know of) between my Python 2.5.4 and Python 2.6.5 environments. To address a previous suggestion I received offline, I do have sufficient permissions (I've even tried using sudo, just to make sure) and I do have setuptools installed. Any help, would be appreciated. Patrick ======================================================================== compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/include -I/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5 -c' gcc: _configtest.c cc1: error: unrecognized command line option "-Wno-long-double" cc1: error: unrecognized command line option "-Wno-long-double" lipo: can't figure out the architecture type of: /var/tmp//ccjTUva4.out cc1: error: unrecognized command line option "-Wno-long-double" cc1: error: unrecognized command line option "-Wno-long-double" lipo: can't figure out the architecture type of: /var/tmp//ccjTUva4.out failure. removing: _configtest.c _configtest.o Traceback (most recent call last): File "setup.py", line 187, in setup_package() File "setup.py", line 180, in setup_package configuration=configuration ) File "/Users/pmarsh/git/numpy.release/numpy/distutils/core.py", line 186, in setup return old_setup(**new_attr) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/core.py", line 151, in setup dist.run_commands() File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/dist.py", line 974, in run_commands self.run_command(cmd) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/dist.py", line 994, in run_command cmd_obj.run() File "/Users/pmarsh/git/numpy.release/numpy/distutils/command/build.py", line 37, in run old_build.run(self) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/command/build.py", line 112, in run self.run_command(cmd_name) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/cmd.py", line 333, in run_command self.distribution.run_command(command) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/dist.py", line 994, in run_command cmd_obj.run() File "/Users/pmarsh/git/numpy.release/numpy/distutils/command/build_src.py", line 152, in run self.build_sources() File "/Users/pmarsh/git/numpy.release/numpy/distutils/command/build_src.py", line 163, in build_sources self.build_library_sources(*libname_info) File "/Users/pmarsh/git/numpy.release/numpy/distutils/command/build_src.py", line 298, in build_library_sources sources = self.generate_sources(sources, (lib_name, build_info)) File "/Users/pmarsh/git/numpy.release/numpy/distutils/command/build_src.py", line 385, in generate_sources source = func(extension, build_dir) File "numpy/core/setup.py", line 657, in get_mathlib_info raise RuntimeError("Broken toolchain: cannot link a simple C program") RuntimeError: Broken toolchain: cannot link a simple C program -- Patrick Marsh Ph.D. Student / NSSL Liaison to the HWT School of Meteorology / University of Oklahoma Cooperative Institute for Mesoscale Meteorological Studies National Severe Storms Laboratory http://www.patricktmarsh.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Mon Apr 19 12:49:29 2010 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 19 Apr 2010 09:49:29 -0700 Subject: [Numpy-discussion] My GSoC Proposal to Implement a Subset of NumPy for PyPy In-Reply-To: References: Message-ID: <4BCC8999.5070508@noaa.gov> Dan Roberts wrote: > Hi everybody, my name is Dan Roberts, and my Google Summer of Code cool stuff. A few years back, someone did a tiny micro-prototype of numpy implemented in Psyco, using the array.array data type as the base. It was actually pretty promising performance-wise. I don't know if there is anything there that would be useful, but I'd do a bit of googling and see if you can find it. Dag Sverre Seljebotn wrote: > Eventually, for this to be at all useful for the NumPy crowd, one has to > make available eigenvalue finders, FFTs, and so on as well. I'm not sure I agree -- here I find I rarely use any of that "standard" stuff -- not sure why, maybe I do weird work. > In my opinion, the *primary* reason Python is used for scientific > programming rather than some other language is how easy it is to connect > with C, C++ and Fortran code in CPython. That's something to keep in mind. Well, yes, this really is key -- performance aside, there are a lot of really useful libs out there -- I don't want to have to re-write them. Another note: I only glanced at your proposal, but it seems that one thing you may want to start with is getting PyPy to "understand" PEP 3118 buffers, from there, you can write numpy in python, and you should be able to use C numpy as well, passing the buffers back and forth (and interface with arbitrary C code) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From gberbeglia at gmail.com Mon Apr 19 13:32:51 2010 From: gberbeglia at gmail.com (gerardob) Date: Mon, 19 Apr 2010 10:32:51 -0700 (PDT) Subject: [Numpy-discussion] scipy error undefined symbol: lsame_ Message-ID: <28287720.post@talk.nabble.com> I installed scipy (and all the required libraries) and the following error appears when i tried run a simple example which uses the optimize package of scipy. I tried also numpy alone and it works ( at least for printing numpy.array([10,20,10])) error: Traceback (most recent call last): File "main_test.py", line 2, in from scipy import optimize File "/home/gberbeglia/python/Python-2.6.5/lib/python2.6/site-packages/scipy/optimize/__init__.py", line 11, in from lbfgsb import fmin_l_bfgs_b File "/home/gberbeglia/python/Python-2.6.5/lib/python2.6/site-packages/scipy/optimize/lbfgsb.py", line 30, in import _lbfgsb ImportError: /home/gberbeglia/python/Python-2.6.5/lib/python2.6/site-packages/scipy/optimize/_lbfgsb.so: undefined symbol: lsame_ gberbeglia at actarus:~/python/mycodes> Any ideas on how to solve this problem? Thanks. PS: the example is below: import numpy from scipy import optimize a = numpy.array([10,20,10]) print a def f_(x): return x*x x,f,d = optimize.fmin_l_bfgs_b(f_,[0.1],fprime=None, approx_grad = True, bounds = [(-10000,10000)], iprint=30, maxfun=150000) -- View this message in context: http://old.nabble.com/scipy-error-undefined-symbol%3A-lsame_-tp28287720p28287720.html Sent from the Numpy-discussion mailing list archive at Nabble.com. From bpederse at gmail.com Mon Apr 19 13:39:59 2010 From: bpederse at gmail.com (Brent Pedersen) Date: Mon, 19 Apr 2010 10:39:59 -0700 Subject: [Numpy-discussion] patch to pickle np.memmap In-Reply-To: References: Message-ID: On Tue, Apr 13, 2010 at 8:59 PM, Brent Pedersen wrote: > On Tue, Apr 13, 2010 at 8:52 PM, Brent Pedersen wrote: >> hi, i posted a patch to allow pickling of np.memmap objects. >> http://projects.scipy.org/numpy/ticket/1452 >> >> currently, it always returns 'r' for the mode. >> is that the best thing to do there? >> any other changes? >> -brent >> > > and i guess it should (but does not with that patch) correctly handle: > >>>> a = np.memmap(...) >>>> b = a[2:] >>>> cPickle.dumps(b) > any thoughts? it still always loads in mode='r', but i updated the patch to handle slicing so it works like this: >>> import numpy as np >>> a = np.memmap('t.bin', mode='w+', shape=(10,)) >>> a[:] = np.arange(10) >>> a memmap([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=uint8) >>> np.loads(a[1:4].dumps()) memmap([1, 2, 3], dtype=uint8) >>> np.loads(a[-2:].dumps()) memmap([8, 9], dtype=uint8) >>> np.loads(a[-4:-1].dumps()) memmap([6, 7, 8], dtype=uint8) http://projects.scipy.org/numpy/ticket/1452 -brent From stefan at sun.ac.za Mon Apr 19 16:03:51 2010 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 19 Apr 2010 22:03:51 +0200 Subject: [Numpy-discussion] reformulating a series of correlations as one fft, ifft pair In-Reply-To: References: Message-ID: Hi Paul On 19 April 2010 08:36, Paul Northug wrote: > I am having trouble reformulating a series of correlations as a single > fft, ifft pair. > > I have a set of kernels phi : (M = channel, N = kernel, T = time) > correlated with signal a : (N, P+T-1) yielding x : (M, T). > > The correlation, for now, is only in the last dimension, with the two > other dimensions degenerate. Below are 4 implementations (x == y == z > != w). I would like to reformulate the operation as a single pair of > 3d ffts to compare how it fares for different problem dimensions but I > am not able to do it. Implementation #4 is incorrect. Please note the > .conj() to get correlations rather than convolutions. > > Any tips would be appreciated. Optimally, I would not have to pad with > too many zeros as the matrices can be quite large. The challenge is to > beat implementation 2 as P grows. M, N, P are typically on the order > of 10^2 and T, 10^3. I haven't checked your code in detail, but I'll mention two common problems with this approach in case it fits: 1) When convolving a KxL with an MxN array, they both need to be zero padded to (K+M-1)x(L+N-1). 2) One of the signals needs to be reversed and conjugated. Also have a look at scipy.signal.fftconvolve. For real 2D images, I use: def fft_corr(A, B, *args, **kwargs): return ss.fftconvolve(A, B[::-1, ::-1, ...], *args, **kwargs) Regards St?fan From robert.kern at gmail.com Mon Apr 19 19:07:51 2010 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 19 Apr 2010 18:07:51 -0500 Subject: [Numpy-discussion] Numpy compilation error In-Reply-To: References: Message-ID: On Mon, Apr 19, 2010 at 09:58, Pradeep Jha wrote: > Thank you so much Robert. You are awesome :) That totally the problem. > One more question for you. Which? are the things that you have to declare in > PYTHONPATH manually? In your case, just the site-packages/ directory you installed to. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From ralf.gommers at googlemail.com Mon Apr 19 20:04:38 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Tue, 20 Apr 2010 08:04:38 +0800 Subject: [Numpy-discussion] Brooken Toolchain (Mac 10.6) In-Reply-To: References: Message-ID: On Mon, Apr 19, 2010 at 10:55 PM, Patrick Marsh wrote: > Greetings, > > Per my previous email, I'm trying to setup the release process for Numpy on > my MacBook Pro. When trying to build Numpy 1.4.1r3 with Python 2.5.4 I get > a broken toolchain error (below). I do not get this error when trying to > build Numpy with Python 2.6.5 - and there is nothing fundamentally different > (that I know of) between my Python 2.5.4 and Python 2.6.5 environments. To > address a previous suggestion I received offline, I do have sufficient > permissions (I've even tried using sudo, just to make sure) and I do have > setuptools installed. > > Any help, would be appreciated. > Try "export CC=/usr/bin/gcc-4.0" before running the build. I had a similar problem last night for scipy where it picks up the right gcc for many modules and then calls plain gcc, which defaults to 4.2 (and that's bad). Cheers, Ralf > > > Patrick > > > ======================================================================== > compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core > -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath > -Inumpy/core/include > -I/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5 -c' > gcc: _configtest.c > cc1: error: unrecognized command line option "-Wno-long-double" > cc1: error: unrecognized command line option "-Wno-long-double" > lipo: can't figure out the architecture type of: /var/tmp//ccjTUva4.out > cc1: error: unrecognized command line option "-Wno-long-double" > cc1: error: unrecognized command line option "-Wno-long-double" > lipo: can't figure out the architecture type of: /var/tmp//ccjTUva4.out > failure. > removing: _configtest.c _configtest.o > Traceback (most recent call last): > File "setup.py", line 187, in > setup_package() > File "setup.py", line 180, in setup_package > configuration=configuration ) > File "/Users/pmarsh/git/numpy.release/numpy/distutils/core.py", line 186, > in setup > return old_setup(**new_attr) > File > "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/core.py", > line 151, in setup > dist.run_commands() > File > "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/dist.py", > line 974, in run_commands > self.run_command(cmd) > File > "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/dist.py", > line 994, in run_command > cmd_obj.run() > File "/Users/pmarsh/git/numpy.release/numpy/distutils/command/build.py", > line 37, in run > old_build.run(self) > File > "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/command/build.py", > line 112, in run > self.run_command(cmd_name) > File > "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/cmd.py", > line 333, in run_command > self.distribution.run_command(command) > File > "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/dist.py", > line 994, in run_command > cmd_obj.run() > File > "/Users/pmarsh/git/numpy.release/numpy/distutils/command/build_src.py", line > 152, in run > self.build_sources() > File > "/Users/pmarsh/git/numpy.release/numpy/distutils/command/build_src.py", line > 163, in build_sources > self.build_library_sources(*libname_info) > File > "/Users/pmarsh/git/numpy.release/numpy/distutils/command/build_src.py", line > 298, in build_library_sources > sources = self.generate_sources(sources, (lib_name, build_info)) > File > "/Users/pmarsh/git/numpy.release/numpy/distutils/command/build_src.py", line > 385, in generate_sources > source = func(extension, build_dir) > File "numpy/core/setup.py", line 657, in get_mathlib_info > raise RuntimeError("Broken toolchain: cannot link a simple C program") > RuntimeError: Broken toolchain: cannot link a simple C program > > > > -- > Patrick Marsh > Ph.D. Student / NSSL Liaison to the HWT > School of Meteorology / University of Oklahoma > Cooperative Institute for Mesoscale Meteorological Studies > National Severe Storms Laboratory > http://www.patricktmarsh.com > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at googlemail.com Mon Apr 19 20:23:43 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Tue, 20 Apr 2010 08:23:43 +0800 Subject: [Numpy-discussion] Release candidate 3 for NumPy 1.4.1 and SciPy 0.7.2 In-Reply-To: References: Message-ID: On Mon, Apr 19, 2010 at 9:19 PM, Ralf Gommers wrote: > > > On Mon, Apr 19, 2010 at 4:21 PM, Ralf Gommers > wrote: > >> >> >> On Mon, Apr 19, 2010 at 3:35 PM, Sebastian Haase wrote: >> >>> Hi, >>> Congratulations. I might be unnecessarily dense - but what SciPy am I >>> supposed to use with the new numpy 1.4.1 for Python 2.5? I'm surprised >>> that there are no SciPy 0.7.2 binaries for Python 2.5 - is that >>> technically not possible ? >>> >>> You're not being dense, there are no 2.5 scipy binaries. I did not >> succeed in building them. The scipy 0.7 branch is so old (from Jan 2009) >> that it has never been compiled on OS X 10.6, and I did not yet find a way >> to get it to work. For Windows I also had problems, there it should be >> compiled against numpy 1.2 while the 2.6 binaries are compiled against numpy >> 1.3. The variations in numpy, python and OS just added up to make it >> unworkable. >> >> I can give it another try after the final release, but first priority is >> to finally release. >> > > To remind myself of the issue I tried building it again, and managed to > build a 2.5 binary against numpy 1.3 on OS X at least. Can anyone tell me > why 2.5 binaries are supposed to be built against numpy 1.2 and 2.6 binaries > against numpy 1.3? > > > It seems I've crawled a bit further up the learning curve since last time I tried. Scipy binaries for python 2.5 (built against numpy 1.2) are now on Sourceforge, please test them. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From pnorthug at gmail.com Tue Apr 20 00:51:09 2010 From: pnorthug at gmail.com (Paul Northug) Date: Mon, 19 Apr 2010 21:51:09 -0700 Subject: [Numpy-discussion] reformulating a series of correlations as one fft, ifft pair Message-ID: gmail.com> writes: > > On Mon, Apr 19, 2010 at 2:36 AM, Paul Northug gmail.com> wrote: > > > > # > > import numpy as np > > from scipy.signal import fftn, ifftn, correlate > > > > M, N, P, T = 2, 3, 4, 5 > > > > phi = np.random.randn(M, N, P) > > a = np.random.randn(N, P+T-1) > > x = np.zeros((M, T)) > > > > # 1. correlate > > > > x = np.array([correlate(phi[i], a, 'valid').squeeze() for i in range(M)]) > > > > # 2. dot products > > > > y = np.zeros_like(x) > > for b in range(P): y += np.dot(phi[:,:,b], a[:,b:b+T]) > > > > # 3. a series of ffts (code adapted from scipy convolve) > > > > s1 = np.array(phi.shape[1:]) > > s2 = np.array(a.shape) > > size = s1+s2-1 > > z = np.empty_like(y) > > fsize = (2**np.ceil(np.log2(size))).astype(np.int) # power of 2 pad > > af = fftn(a, fsize) > > fslice = tuple([slice(0, sz) for sz in size]) > > for i in range(M): > > phif = fftn(phi[i], fsize).conj() > > z[i] = np.real(ifftn(phif * af)[fslice])[0,:T] > > > > # 4. single fft, ifft pair - some rows are correct > > > > s1 = np.array(phi.shape) > > s2 = np.array(a[np.newaxis].shape) > > size = s1+s2-1 > > fsize = (2**np.ceil(np.log2(size))).astype(np.int) > > af = fftn(a[np.newaxis], fsize) > > phif = fftn(phi, fsize).conj() > > fslice = tuple([slice(0, sz) for sz in size]) > > w = np.real(ifftn(phif * af)[fslice])[:,:,:T] > > I only checked this in your example > > >>> np.squeeze(signal.fftconvolve(phi[:,::-1,::-1], a[None,...],mode='valid')) > array([[ 0.42695078, 4.3433924 , -2.53084395, -4.22672281, -5.16705981], > [-2.35108765, -2.15060513, -1.23528122, 1.40386599, 1.38189896]]) > > >>> y > array([[ 0.42695078, 4.3433924 , -2.53084395, -4.22672281, -5.16705981], > [-2.35108765, -2.15060513, -1.23528122, 1.40386599, 1.38189896]]) > >>> x > array([[ 0.42695078, 4.3433924 , -2.53084395, -4.22672281, -5.16705981], > [-2.35108765, -2.15060513, -1.23528122, 1.40386599, 1.38189896]]) > > Josef > Thanks Josef. You saved me a lot of work. I was missing the middle ::-1. The fft approach is orders of magnitude slower than method #2 in this case. Using CUDA fft3, it is about 15 times slower for the desired problem size. It is not a fair comparison as the convolution is degenerate along 2 dimensions but it is useful to know. From cournape at gmail.com Tue Apr 20 02:45:49 2010 From: cournape at gmail.com (David Cournapeau) Date: Tue, 20 Apr 2010 15:45:49 +0900 Subject: [Numpy-discussion] Brooken Toolchain (Mac 10.6) In-Reply-To: References: Message-ID: On Mon, Apr 19, 2010 at 11:55 PM, Patrick Marsh wrote: > Greetings, > Per my previous email, I'm trying to setup the release process for Numpy on > my MacBook Pro. ?When trying to build Numpy 1.4.1r3 with Python 2.5.4 I get > a broken toolchain error (below). ?I do not get this error when trying to > build Numpy with Python 2.6.5 - and there is nothing fundamentally different > (that I know of) between my Python 2.5.4 and Python 2.6.5 environments. Actually, there is, especially if you are on 10.6. Both python are installed from python.org ? > > ======================================================================== > compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core > -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath > -Inumpy/core/include > -I/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5 -c' > gcc: _configtest.c > cc1: error: unrecognized command line option "-Wno-long-double" The error is caused by using a different gcc than the one used to build python (the -Wno-long-double option is not valid anymore for gcc 4.2 I think). To check this, please compare gcc -v against the version reported by python (for example at the python prompt). cheers, David From cournape at gmail.com Tue Apr 20 02:49:31 2010 From: cournape at gmail.com (David Cournapeau) Date: Tue, 20 Apr 2010 15:49:31 +0900 Subject: [Numpy-discussion] Building DMG w/ Python 2.6 (could not access...no space left on device) In-Reply-To: References: Message-ID: On Mon, Apr 19, 2010 at 11:44 PM, Patrick Marsh wrote: > Greetings, > I'm trying to build DMG releases of numpy-1.4.1rc3 as part of the release > process. ?I recently got my mac back from repairs and spent the weekend > trying to get the build environment configured on it. ?I thought it was > correct, but I've got errors when building with both Python 2.5.4 and Python > 2.6.5. ?(I should add that, ironically, I have no problem building the > Windows superpack on OS X [10.6].) ?Below is the end of the output from > running 'paver dmg -p 2.6'. ?I have no idea why I'm told ?I'm out of space, > nor do I know where to begin in trying to track that down. The no space left refers to the space allocated to the dmg file, which is 10 Mb ATM. You could try to play with the DMG_SIZE parameter in tools/numpy-macosx-installer/new-create-dmg script, but I would like to know what became so big so that numpy is now > 10 Mb. David From robince at gmail.com Tue Apr 20 05:11:28 2010 From: robince at gmail.com (Robin) Date: Tue, 20 Apr 2010 10:11:28 +0100 Subject: [Numpy-discussion] Brooken Toolchain (Mac 10.6) In-Reply-To: References: Message-ID: To build against the python.org 2.5 you need to use the older gcc: export CC=/usr/bin/gcc-4.0 export CXX=/usr/bin/g++-4.0 should do it. By default snow leopard uses 4.2 now, which doesn't support the -Wno-long-double option used when building python. Cheers Robin On Mon, Apr 19, 2010 at 3:55 PM, Patrick Marsh wrote: > Greetings, > Per my previous email, I'm trying to setup the release process for Numpy on > my MacBook Pro. ?When trying to build Numpy 1.4.1r3 with Python 2.5.4 I get > a broken toolchain error (below). ?I do not get this error when trying to > build Numpy with Python 2.6.5 - and there is nothing fundamentally different > (that I know of) between my Python 2.5.4 and Python 2.6.5 environments. ?To > address a previous suggestion I received offline, I do have sufficient > permissions (I've even tried using sudo, just to make sure) and I do have > setuptools installed. > Any help, would be appreciated. > > Patrick > > ======================================================================== > compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core > -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath > -Inumpy/core/include > -I/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5 -c' > gcc: _configtest.c > cc1: error: unrecognized command line option "-Wno-long-double" > cc1: error: unrecognized command line option "-Wno-long-double" > lipo: can't figure out the architecture type of: /var/tmp//ccjTUva4.out > cc1: error: unrecognized command line option "-Wno-long-double" > cc1: error: unrecognized command line option "-Wno-long-double" > lipo: can't figure out the architecture type of: /var/tmp//ccjTUva4.out > failure. > removing: _configtest.c _configtest.o > Traceback (most recent call last): > ??File "setup.py", line 187, in > ?? ?setup_package() > ??File "setup.py", line 180, in setup_package > ?? ?configuration=configuration ) > ??File "/Users/pmarsh/git/numpy.release/numpy/distutils/core.py", line 186, > in setup > ?? ?return old_setup(**new_attr) > ??File > "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/core.py", > line 151, in setup > ?? ?dist.run_commands() > ??File > "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/dist.py", > line 974, in run_commands > ?? ?self.run_command(cmd) > ??File > "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/dist.py", > line 994, in run_command > ?? ?cmd_obj.run() > ??File "/Users/pmarsh/git/numpy.release/numpy/distutils/command/build.py", > line 37, in run > ?? ?old_build.run(self) > ??File > "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/command/build.py", > line 112, in run > ?? ?self.run_command(cmd_name) > ??File > "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/cmd.py", > line 333, in run_command > ?? ?self.distribution.run_command(command) > ??File > "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/dist.py", > line 994, in run_command > ?? ?cmd_obj.run() > ??File > "/Users/pmarsh/git/numpy.release/numpy/distutils/command/build_src.py", line > 152, in run > ?? ?self.build_sources() > ??File > "/Users/pmarsh/git/numpy.release/numpy/distutils/command/build_src.py", line > 163, in build_sources > ?? ?self.build_library_sources(*libname_info) > ??File > "/Users/pmarsh/git/numpy.release/numpy/distutils/command/build_src.py", line > 298, in build_library_sources > ?? ?sources = self.generate_sources(sources, (lib_name, build_info)) > ??File > "/Users/pmarsh/git/numpy.release/numpy/distutils/command/build_src.py", line > 385, in generate_sources > ?? ?source = func(extension, build_dir) > ??File "numpy/core/setup.py", line 657, in get_mathlib_info > ?? ?raise RuntimeError("Broken toolchain: cannot link a simple C program") > RuntimeError: Broken toolchain: cannot link a simple C program > > > -- > Patrick Marsh > Ph.D. Student / NSSL Liaison to the HWT > School of Meteorology / University of Oklahoma > Cooperative Institute for Mesoscale Meteorological Studies > National Severe Storms Laboratory > http://www.patricktmarsh.com > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From david at silveregg.co.jp Tue Apr 20 06:16:19 2010 From: david at silveregg.co.jp (David) Date: Tue, 20 Apr 2010 19:16:19 +0900 Subject: [Numpy-discussion] scipy error undefined symbol: lsame_ In-Reply-To: <28287720.post@talk.nabble.com> References: <28287720.post@talk.nabble.com> Message-ID: <4BCD7EF3.1020806@silveregg.co.jp> On 04/20/2010 02:32 AM, gerardob wrote: > > I installed scipy (and all the required libraries) and the following error > appears when i tried run a simple example which uses the optimize package of > scipy. I tried also numpy alone and it works ( at least for printing > numpy.array([10,20,10])) > > error: > > Traceback (most recent call last): > File "main_test.py", line 2, in > from scipy import optimize > File > "/home/gberbeglia/python/Python-2.6.5/lib/python2.6/site-packages/scipy/optimize/__init__.py", > line 11, in > from lbfgsb import fmin_l_bfgs_b > File > "/home/gberbeglia/python/Python-2.6.5/lib/python2.6/site-packages/scipy/optimize/lbfgsb.py", > line 30, in > import _lbfgsb > ImportError: > /home/gberbeglia/python/Python-2.6.5/lib/python2.6/site-packages/scipy/optimize/_lbfgsb.so: > undefined symbol: lsame_ > gberbeglia at actarus:~/python/mycodes> You did not build scipy properly: you need to make sure that everything is built with exactly the same fortran compiler. One way to check this is to do ldd on the .so files which fail: if you see g2c as a dependency, it is using g77, if you see libgfortran, it is using gfortran. cheers, David From lists at hilboll.de Tue Apr 20 09:03:47 2010 From: lists at hilboll.de (Andreas Hilboll) Date: Tue, 20 Apr 2010 15:03:47 +0200 (CEST) Subject: [Numpy-discussion] trim_zeros in more than one dimension? Message-ID: <8519e13fc8c2e059f4954b0aaabe1e9c.squirrel@srv2.hilboll.net> Hi there, is there an easy way to do something like trim_zeros() does, but for a n-dimensional array? I have a 2d array with only zeros in the first and last rows and columns, and would like to trim this array to only the non-zero part ... Thanks, Andreas. From oliphant at enthought.com Tue Apr 20 08:58:26 2010 From: oliphant at enthought.com (Travis Oliphant) Date: Tue, 20 Apr 2010 07:58:26 -0500 Subject: [Numpy-discussion] My GSoC Proposal to Implement a Subset of NumPy for PyPy In-Reply-To: References: Message-ID: On Apr 16, 2010, at 11:50 PM, Dan Roberts wrote: > Hello NumPy Users, > Hi everybody, my name is Dan Roberts, and my Google Summer of > Code proposal was categorized under NumPy rather than PyPy, so it > will end up being reviewed by mentors for the NumPy project. I'd > like to take this chance to introduce myself and my proposal. > I hadn't prepared for review by the NumPy mentors, but this can > make my proposal stronger than before. With a bit of help from all > of you, I can dedicate my summer to creating more useful code than I > would have previously. I realize that from the perspective of NumPy, > my proposal might seem lacking, so I'd like to also invite the > scrutiny of all of the readers of this list. > Why should we bother reimplimenting anything? PyPy, for those > who are unfamiliar, has the ability to Just-in-Time compile itself > and programs that it's running. One of the major advantages of this > is that code operating on NumPy arrays could potentially be written > in pure-python, with normal looping constructs, and be nearly as > fast as a ufunc painstakingly crafted in C. I'd love to see as much > Python and as little C as possible, and I'm sure I'm not alone in > that wish. > A short introduction: I've been coding in Python for the past > few years, and have increasingly become interested in speeding up > what has become my favorite language. To that end I've become > interested in both the PyPy project and the NumPy projects. I've > spent a fair amount of time frustrating the PyPy developers with > silly questions, written a bit of code for them, and now my GSoC > proposal involves both them, and NumPy. > Finally, I'd like to ask all of you: what features are most > important to you? It's not practical, wise, or even possible for me > to reimpliment more than a small portion of NumPy, but if I can > address the most important parts, maybe I can make this project > useful enough for some of you to use, and close enough for the rest > of you that I can drum up some support for more development in the > future. > My proposal lives at http://codespeak.net/~dan/gsoc/micronumpy.html > thanks for making it this far through my long winded introduction! > I welcome all constructive criticism and thoughts. > Hi Daniel, This sounds like a great project, and I think it has promise. I would especially pay attention to the requests to make it easy to write ufuncs and generalized ufuncs in RPython. That has the most possibility of being immediately useful. Your timing is also very good. I am going to be spending some time re-factoring NumPy to separate out the CPython interface from the underlying algorithms. I think this re-factoring should help you in your long-term goals. If you have any input or suggestions while the refactoring is taking place, we are always open to suggestions and criticisms. Thanks for writing a NumPy-related proposal. Best regards, -Travis > Thanks, > Daniel Roberts > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -- Travis Oliphant Enthought Inc. 1-512-536-1057 http://www.enthought.com oliphant at enthought.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Mon Apr 19 21:34:08 2010 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 19 Apr 2010 18:34:08 -0700 Subject: [Numpy-discussion] Numpy compilation error In-Reply-To: References: Message-ID: <4BCD0490.1010108@noaa.gov> Pradeep Jha wrote: > Thank you so much Robert. You are awesome :) That totally the problem. > One more question for you. Which are the things that you have to > declare in PYTHONPATH manually? I never put anything in PYTHONPATH -- if you install everything you need, you won't need to. When I'm using something under development, I use setuptools' "setup.py develop" -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From robert.kern at gmail.com Tue Apr 20 11:09:40 2010 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 20 Apr 2010 10:09:40 -0500 Subject: [Numpy-discussion] Numpy compilation error In-Reply-To: <4BCD0490.1010108@noaa.gov> References: <4BCD0490.1010108@noaa.gov> Message-ID: On Mon, Apr 19, 2010 at 20:34, Christopher Barker wrote: > > > Pradeep Jha wrote: >> Thank you so much Robert. You are awesome :) That totally the problem. >> One more question for you. Which ?are the things that you have to >> declare in PYTHONPATH manually? > > I never put anything in PYTHONPATH -- if you install everything you > need, you won't need to. When I'm using something under development, I > use setuptools' "setup.py develop" He does not have root. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From kwgoodman at gmail.com Tue Apr 20 11:32:12 2010 From: kwgoodman at gmail.com (Keith Goodman) Date: Tue, 20 Apr 2010 08:32:12 -0700 Subject: [Numpy-discussion] trim_zeros in more than one dimension? In-Reply-To: <8519e13fc8c2e059f4954b0aaabe1e9c.squirrel@srv2.hilboll.net> References: <8519e13fc8c2e059f4954b0aaabe1e9c.squirrel@srv2.hilboll.net> Message-ID: On Tue, Apr 20, 2010 at 6:03 AM, Andreas Hilboll wrote: > Hi there, > > is there an easy way to do something like trim_zeros() does, but for a > n-dimensional array? I have a 2d array with only zeros in the first and > last rows and columns, and would like to trim this array to only the > non-zero part ... > > Thanks, > > Andreas. I have some code that does something similar for NaNs. Create an array: >> x = np.arange(12, dtype=float).reshape(3,4) >> x[[0,-1],:] = np.nan >> x[:,[0,-1]] = np.nan >> x array([[ NaN, NaN, NaN, NaN], [ NaN, 5., 6., NaN], [ NaN, NaN, NaN, NaN]]) Use the la package to convert the array to a labeled array: >> y = la.larry(x) Then remove all rows and columns that contain only NaN: >> y.vacuum().A array([[ 5., 6.]]) or >> y.vacuum().squeeze().A array([ 5., 6.]) It should work for arbitrary number of dimensions. The code (Simplified BSD License) is here: http://bazaar.launchpad.net/~kwgoodman/larry/trunk/annotate/head%3A/la/deflarry.py From seb.haase at gmail.com Tue Apr 20 13:52:25 2010 From: seb.haase at gmail.com (Sebastian Haase) Date: Tue, 20 Apr 2010 19:52:25 +0200 Subject: [Numpy-discussion] Release candidate 3 for NumPy 1.4.1 and SciPy 0.7.2 In-Reply-To: References: Message-ID: On Tue, Apr 20, 2010 at 2:23 AM, Ralf Gommers wrote: > > > On Mon, Apr 19, 2010 at 9:19 PM, Ralf Gommers > wrote: >> >> >> On Mon, Apr 19, 2010 at 4:21 PM, Ralf Gommers >> wrote: >>> >>> >>> On Mon, Apr 19, 2010 at 3:35 PM, Sebastian Haase >>> wrote: >>>> >>>> Hi, >>>> Congratulations. I might be unnecessarily dense - but what SciPy am I >>>> supposed to use with the new numpy 1.4.1 for Python 2.5? I'm surprised >>>> that there are no SciPy 0.7.2 binaries for Python 2.5 - is that >>>> technically not possible ? >>>> >>> You're not being dense, there are no 2.5 scipy binaries. I did not >>> succeed in building them. The scipy 0.7 branch is so old (from Jan 2009) >>> that it has never been compiled on OS X 10.6, and I did not yet find a way >>> to get it to work. For Windows I also had problems, there it should be >>> compiled against numpy 1.2 while the 2.6 binaries are compiled against numpy >>> 1.3. The variations in numpy, python and OS just added up to make it >>> unworkable. >>> >>> I can give it another try after the final release, but first priority is >>> to finally release. >> >> To remind myself of the issue I tried building it again, and managed to >> build a 2.5 binary against numpy 1.3 on OS X at least. Can anyone tell me >> why 2.5 binaries are supposed to be built against numpy 1.2 and 2.6 binaries >> against numpy 1.3? >> >> > It seems I've crawled a bit further up the learning curve since last time I > tried. Scipy binaries for python 2.5 (built against numpy 1.2) are now on > Sourceforge, please test them. > What does "built against numpy 1.2" actually mean exactly ? Is that just a "building time" thing and they actually work together (at run time) with numpy 1.4.1 ? That would be all fine then ... what platform(s) are you talking about ? (What about Windows?) - Sebastian From charlesr.harris at gmail.com Tue Apr 20 19:45:32 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Tue, 20 Apr 2010 17:45:32 -0600 Subject: [Numpy-discussion] trim_zeros in more than one dimension? In-Reply-To: <8519e13fc8c2e059f4954b0aaabe1e9c.squirrel@srv2.hilboll.net> References: <8519e13fc8c2e059f4954b0aaabe1e9c.squirrel@srv2.hilboll.net> Message-ID: On Tue, Apr 20, 2010 at 7:03 AM, Andreas Hilboll wrote: > Hi there, > > is there an easy way to do something like trim_zeros() does, but for a > n-dimensional array? I have a 2d array with only zeros in the first and > last rows and columns, and would like to trim this array to only the > non-zero part ... > > I think for your application it would be easier to use a subarray, something like a[1:-1, 1:-1]. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From dagss at student.matnat.uio.no Tue Apr 20 20:30:28 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 21 Apr 2010 02:30:28 +0200 Subject: [Numpy-discussion] Release candidate 3 for NumPy 1.4.1 and SciPy 0.7.2 In-Reply-To: References: Message-ID: <4BCE4724.9030509@student.matnat.uio.no> Ralf Gommers wrote: > Hi, > > I am pleased to announce the third release candidate of both Scipy 0.7.2 > and NumPy 1.4.1. Please test, and report any problems on the NumPy or > SciPy list. I had a round of segfaults in the SciPy/NumPy interlink, which I eventually tracked down to a leftover _dotblas.so from an older NumPy version lying in site-packages (apparently that file is gone in the newer NumPy, but would still be imported under some circumstances, and I foolishly just installed over the old version and subsequently forgot that I had done so). This could have been avoided with a warning about removing any old numpy lying around first (though perhaps others are not as stupid as I was). -- Dag Sverre From dagss at student.matnat.uio.no Tue Apr 20 20:33:38 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 21 Apr 2010 02:33:38 +0200 Subject: [Numpy-discussion] Release candidate 3 for NumPy 1.4.1 and SciPy 0.7.2 In-Reply-To: <4BCE4724.9030509@student.matnat.uio.no> References: <4BCE4724.9030509@student.matnat.uio.no> Message-ID: <4BCE47E2.6070704@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > Ralf Gommers wrote: >> Hi, >> >> I am pleased to announce the third release candidate of both Scipy >> 0.7.2 and NumPy 1.4.1. Please test, and report any problems on the >> NumPy or SciPy list. > > I had a round of segfaults in the SciPy/NumPy interlink, which I > eventually tracked down to a leftover _dotblas.so from an older NumPy > version lying in site-packages (apparently that file is gone in the > newer NumPy, but would still be imported under some circumstances, and I > foolishly just installed over the old version and subsequently forgot > that I had done so). Sorry for the noise, the segfault was due to something else, I only thought I had fixed it. But it's my own MKL-specific mess causing it, so nothing to worry about. -- Dag Sverre From pnorthug at gmail.com Tue Apr 20 21:33:38 2010 From: pnorthug at gmail.com (Paul Northug) Date: Wed, 21 Apr 2010 01:33:38 +0000 (UTC) Subject: [Numpy-discussion] =?utf-8?q?reformulating_a_series_of_correlatio?= =?utf-8?q?ns_as=09one_fft=2C_ifft_pair?= References: Message-ID: St?fan van der Walt sun.ac.za> writes: > > I haven't checked your code in detail, but I'll mention two common > problems with this approach in case it fits: > > 1) When convolving a KxL with an MxN array, they both need to be zero > padded to (K+M-1)x(L+N-1). > 2) One of the signals needs to be reversed and conjugated. > > Also have a look at scipy.signal.fftconvolve. For real 2D images, I use: > > def fft_corr(A, B, *args, **kwargs): > return ss.fftconvolve(A, B[::-1, ::-1, ...], *args, **kwargs) Thanks Stefan. Do you mean either reverse or conjugate, not and? I am still a little confused. Suppose you had a 3d problem and you wanted to convolve on two axes and correlate on the third. I can see now how you could do so by using fftconvolve and reversing the axis you wanted to correlate as you've shown. But how would you implement it the other way by fft'ing and then conjugating one of the fft pair, multiplying and ifft'ing? You can't conjugate along an axis. Also I am not sure which one is faster, conjugating a list or reversing it. I would like to correct another mistake I made when I said that the fft method is much slower in this case. This is only true when the padding doesn't give you dimensions that are split radix method-friendly or just powers of 2. From ademan555 at gmail.com Tue Apr 20 22:38:28 2010 From: ademan555 at gmail.com (Dan Roberts) Date: Tue, 20 Apr 2010 19:38:28 -0700 Subject: [Numpy-discussion] My GSoC Proposal to Implement a Subset of NumPy for PyPy In-Reply-To: References: Message-ID: Thanks for the reply. You're certainly right that your work is extremely beneficial to mine. At present I'm afraid a great deal of NumPy C code isn't easily reusable and it's great you're addressing that. I may not have been thinking in line with Maciej, but I was thinking ufuncs would be written in pure Python and jit compiled to an efficient form. (We can make lots of nice assumptions about them) That said, I think being able to write generic ufuncs is a very good idea, and absolutely doable. On Apr 20, 2010 7:48 AM, "Travis Oliphant" wrote: On Apr 16, 2010, at 11:50 PM, Dan Roberts wrote: > Hello NumPy Users, > Hi everybody, my name i... Hi Daniel, This sounds like a great project, and I think it has promise. I would especially pay attention to the requests to make it easy to write ufuncs and generalized ufuncs in RPython. That has the most possibility of being immediately useful. Your timing is also very good. I am going to be spending some time re-factoring NumPy to separate out the CPython interface from the underlying algorithms. I think this re-factoring should help you in your long-term goals. If you have any input or suggestions while the refactoring is taking place, we are always open to suggestions and criticisms. Thanks for writing a NumPy-related proposal. Best regards, -Travis Thanks, Daniel Roberts > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discuss... -- Travis Oliphant Enthought Inc. 1-512-536-1057 http://www.enthought.com oliphant at enthought.com _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -------------- next part -------------- An HTML attachment was scrubbed... URL: From ademan555 at gmail.com Tue Apr 20 22:44:19 2010 From: ademan555 at gmail.com (Dan Roberts) Date: Tue, 20 Apr 2010 19:44:19 -0700 Subject: [Numpy-discussion] My GSoC Proposal to Implement a Subset of NumPy for PyPy In-Reply-To: References: Message-ID: Oops, I intended to dig more through the NumPy source before I sent the final version of that message, so I could be speaking from an informed standpoint. Thanks, Daniel Roberts > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discuss... -- Travis Oliphant Enthought Inc. 1-512-536-1057 http://www.enthought.com oliphant at enthought.com ... -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Wed Apr 21 01:30:31 2010 From: cournape at gmail.com (David Cournapeau) Date: Wed, 21 Apr 2010 14:30:31 +0900 Subject: [Numpy-discussion] Numpy compilation error In-Reply-To: <4BCD0490.1010108@noaa.gov> References: <4BCD0490.1010108@noaa.gov> Message-ID: On Tue, Apr 20, 2010 at 10:34 AM, Christopher Barker wrote: > > > Pradeep Jha wrote: >> Thank you so much Robert. You are awesome :) That totally the problem. >> One more question for you. Which ?are the things that you have to >> declare in PYTHONPATH manually? > > I never put anything in PYTHONPATH -- if you install everything you > need, you won't need to. When I'm using something under development, I > use setuptools' "setup.py develop" I don't think it is wise to advocate the use of develop for python newcomers. It comes with issues which are difficult to track down (stalled files for entry points which are not removed by uninstall -u, etc...). Those are non issues for the experienced users, but a pain in my experience for beginners. The easy and reliable solution for non root install is PYTHONPATH for python < 2.6 and install --user for python >= 2.6. cheers, David From dagss at student.matnat.uio.no Wed Apr 21 08:21:48 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 21 Apr 2010 14:21:48 +0200 Subject: [Numpy-discussion] My GSoC Proposal to Implement a Subset of NumPy for PyPy In-Reply-To: References: Message-ID: <4BCEEDDC.2010300@student.matnat.uio.no> Dan Roberts wrote: > Thanks for the reply. You're certainly right that your work is > extremely beneficial to mine. At present I'm afraid a great deal of > NumPy C code isn't easily reusable and it's great you're addressing > that. I may not have been thinking in line with Maciej, but I was > thinking ufuncs would be written in pure Python and jit compiled to an > efficient form. (We can make lots of nice assumptions about them) That > said, I think being able to write generic ufuncs is a very good idea, > and absolutely doable. This might be relevant? http://conference.scipy.org/proceedings/SciPy2008/paper_16/ -- Dag Sverre From ralf.gommers at googlemail.com Wed Apr 21 09:03:27 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Wed, 21 Apr 2010 21:03:27 +0800 Subject: [Numpy-discussion] Release candidate 3 for NumPy 1.4.1 and SciPy 0.7.2 In-Reply-To: References: Message-ID: On Wed, Apr 21, 2010 at 1:52 AM, Sebastian Haase wrote: > On Tue, Apr 20, 2010 at 2:23 AM, Ralf Gommers > wrote: > > It seems I've crawled a bit further up the learning curve since last time > I > > tried. Scipy binaries for python 2.5 (built against numpy 1.2) are now on > > Sourceforge, please test them. > > > What does "built against numpy 1.2" actually mean exactly ? Is that > just a "building time" thing and they actually work together (at run > time) with numpy 1.4.1 ? That would be all fine then ... what > platform(s) are you talking about ? (What about Windows?) > > Correct, and it's the same for Windows and OS X binaries. Some scipy modules include a numpy header file (mostly arrayobject.h). This is now forward-compatible, so scipy compiled against numpy 1.2 works with 1.3 and 1.4.1 as well. It won't work with numpy 2.0 though, and was the reason for the issues with numpy 1.4.0. In 1.4.0 the layout of the ndarray object in memory changed (hence this minor release to undo that change), causing segfaults when used with scipy or other extensions compiled against older numpy. Cheers, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Wed Apr 21 09:36:24 2010 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 21 Apr 2010 08:36:24 -0500 Subject: [Numpy-discussion] trim_zeros in more than one dimension? In-Reply-To: References: <8519e13fc8c2e059f4954b0aaabe1e9c.squirrel@srv2.hilboll.net> Message-ID: On Tue, Apr 20, 2010 at 18:45, Charles R Harris wrote: > > On Tue, Apr 20, 2010 at 7:03 AM, Andreas Hilboll wrote: >> >> Hi there, >> >> is there an easy way to do something like trim_zeros() does, but for a >> n-dimensional array? I have a 2d array with only zeros in the first and >> last rows and columns, and would like to trim this array to only the >> non-zero part ... > > I think for your application it would be easier to use a subarray, something > like a[1:-1, 1:-1]. Yes, but I think he's asking for a function that will find the appropriate slices for him, justl like trim_zeros() does for 1D arrays. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From aj.guillon at gmail.com Wed Apr 21 10:47:23 2010 From: aj.guillon at gmail.com (Adrien Guillon) Date: Wed, 21 Apr 2010 10:47:23 -0400 Subject: [Numpy-discussion] Disabling Extended Precision in NumPy (like -ffloat-store) Message-ID: Hello all, I've recently started to use NumPy to prototype some numerical algorithms, which will eventually find their way to a GPU (where I want to limit myself to single-precision operations for performance reasons). I have recently switched to the use of the "single" type in NumPy to ensure I use single-precision floating point operations. My understanding, however, is that Intel processors may use extended precision for some operations anyways unless this is explicitly disabled, which is done with gcc via the -ffloat-store operation. Since I am prototyping algorithms for a different processor architecture, where the extended precision registers simply do not exist, I would really like to force NumPy to limit itself to using single-precision operations throughout the calculation (no extended precision in registers). How can I do this? Thanks! AJ From pav at iki.fi Wed Apr 21 11:03:54 2010 From: pav at iki.fi (Pauli Virtanen) Date: Wed, 21 Apr 2010 17:03:54 +0200 Subject: [Numpy-discussion] Disabling Extended Precision in NumPy (like -ffloat-store) In-Reply-To: References: Message-ID: <1271862234.2346.39.camel@talisman> ke, 2010-04-21 kello 10:47 -0400, Adrien Guillon kirjoitti: [clip] > My understanding, however, is that Intel processors may use extended > precision for some operations anyways unless this is explicitly > disabled, which is done with gcc via the -ffloat-store operation. > Since I am prototyping algorithms for a different processor > architecture, where the extended precision registers simply do not > exist, I would really like to force NumPy to limit itself to using > single-precision operations throughout the calculation (no extended > precision in registers). Probably the only way to ensure this is to recompile Numpy (and possibly, also Python) with the -ffloat-store option turned on. When compiling Numpy, you should be able to insert the flag via OPT="-ffloat-store" python setup.py build -- Pauli Virtanen From charlesr.harris at gmail.com Wed Apr 21 11:07:59 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 21 Apr 2010 09:07:59 -0600 Subject: [Numpy-discussion] Disabling Extended Precision in NumPy (like -ffloat-store) In-Reply-To: References: Message-ID: On Wed, Apr 21, 2010 at 8:47 AM, Adrien Guillon wrote: > Hello all, > > I've recently started to use NumPy to prototype some numerical > algorithms, which will eventually find their way to a GPU (where I > want to limit myself to single-precision operations for performance > reasons). I have recently switched to the use of the "single" type in > NumPy to ensure I use single-precision floating point operations. > > My understanding, however, is that Intel processors may use extended > precision for some operations anyways unless this is explicitly > disabled, which is done with gcc via the -ffloat-store operation. > Since I am prototyping algorithms for a different processor > architecture, where the extended precision registers simply do not > exist, I would really like to force NumPy to limit itself to using > single-precision operations throughout the calculation (no extended > precision in registers). > > How can I do this? > > Interesting question. But what precisely do you mean about limiting? Do you want *all* floating calculations done in float32? Why does the extended precision cause you problems? How do the operations of the GPU conflict with those of the intel FPU? So on an so forth. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From peridot.faceted at gmail.com Wed Apr 21 11:19:22 2010 From: peridot.faceted at gmail.com (Anne Archibald) Date: Wed, 21 Apr 2010 11:19:22 -0400 Subject: [Numpy-discussion] Disabling Extended Precision in NumPy (like -ffloat-store) In-Reply-To: <1271862234.2346.39.camel@talisman> References: <1271862234.2346.39.camel@talisman> Message-ID: On 21 April 2010 11:03, Pauli Virtanen wrote: > ke, 2010-04-21 kello 10:47 -0400, Adrien Guillon kirjoitti: > [clip] >> My understanding, however, is that Intel processors may use extended >> precision for some operations anyways unless this is explicitly >> disabled, which is done with gcc via the -ffloat-store operation. >> Since I am prototyping algorithms for a different processor >> architecture, where the extended precision registers simply do not >> exist, I would really like to force NumPy to limit itself to using >> single-precision operations throughout the calculation (no extended >> precision in registers). > > Probably the only way to ensure this is to recompile Numpy (and > possibly, also Python) with the -ffloat-store option turned on. > > When compiling Numpy, you should be able to insert the flag via > > ? ? ? ?OPT="-ffloat-store" python setup.py build This will certainly work. But maybe you should be careful even so: you can't trust floating-point operations to behave identically across architectures (or even across compilations). Given that, it may be perfectly sufficient to simply do your calculations normally in numpy, without recompiling. Even if x86 FPUs store intermediate results in extended precision, the way numpy structures its computations normally forces all intermediate results out to main memory: r = a*b + c, when applied to arrays, carries out all the multiplications, then all the additions. So extended precision may not be an issue. Anne From bsouthey at gmail.com Wed Apr 21 11:34:40 2010 From: bsouthey at gmail.com (Bruce Southey) Date: Wed, 21 Apr 2010 10:34:40 -0500 Subject: [Numpy-discussion] trim_zeros in more than one dimension? In-Reply-To: References: <8519e13fc8c2e059f4954b0aaabe1e9c.squirrel@srv2.hilboll.net> Message-ID: <4BCF1B10.7040907@gmail.com> On 04/21/2010 08:36 AM, Robert Kern wrote: > On Tue, Apr 20, 2010 at 18:45, Charles R Harris > wrote: > >> On Tue, Apr 20, 2010 at 7:03 AM, Andreas Hilboll wrote: >> >>> Hi there, >>> >>> is there an easy way to do something like trim_zeros() does, but for a >>> n-dimensional array? I have a 2d array with only zeros in the first and >>> last rows and columns, and would like to trim this array to only the >>> non-zero part ... >>> >> I think for your application it would be easier to use a subarray, something >> like a[1:-1, 1:-1]. >> > Yes, but I think he's asking for a function that will find the > appropriate slices for him, justl like trim_zeros() does for 1D > arrays. > > If the sum of axis to be removed equals zero then you can conditionally remove that axis. For 2-d you can do: import numpy as np ba=np.array([[0,0,0], [1,2, 0], [3,4, 0], [0,0,0]]) ndx0=ba.sum(axis=0)>0 #indices of axis 0 where the sum is greater than zero ac=ba[:,ndx0] ndx1=ac.sum(axis=1)>0 #indices of axis 1 where the sum is greater than zero ad=ac[ndx1,:] >>> ba array([[0, 0, 0], [1, 2, 0], [3, 4, 0], [0, 0, 0]]) >>> ad array([[1, 2], [3, 4]]) If the sum of an dimension is also zero somewhere else in the array then you need to correct the appropriate index first (which is why I created a separate variable). Obviously higher dimensions are left untested. Bruce From kbasye1 at jhu.edu Wed Apr 21 11:47:48 2010 From: kbasye1 at jhu.edu (Ken Basye) Date: Wed, 21 Apr 2010 11:47:48 -0400 Subject: [Numpy-discussion] Is this a bug, and if so, who's? Message-ID: <4BCF1E24.4060203@jhu.edu> Folks, Apologies for asking here, but I ran across this problem yesterday and probably need to file a bug. The problem is I don't know if this is a Numpy bug, a Python bug, or both. Here's an illustration, platform information follows. TIA, Ken ############################################# import collections import numpy as np class A (collections.namedtuple('ANT', ('x', 'y'))): def __float__(self): return self.y # Same as A, but explicitly convert y to a float in __float__() - this works around the assert fail class B (collections.namedtuple('BNT', ('x', 'y'))): def __float__(self): return float(self.y) a0 = A(1.0, 2.0) f0 = np.float64(a0) print f0 a1 = A(float(1.0), float(2.0)) f1 = np.float64(a1) print f1 b1 = B(np.float64(1.0), np.float64(2.0)) f2 = np.float64(b1) print f2 a2 = A(np.float64(1.0), np.float64(2.0)) # On some platforms, the next line will trigger an assert: # python: Objects/floatobject.c:1674: float_subtype_new: Assertion `((((PyObject*)(tmp))->ob_type) == &PyFloat_Type)' failed. f3 = np.float64(a2) print f3 ############################################# Platform info: Python 2.6.5 (r265:79063, Apr 14 2010, 13:32:56) [GCC 4.1.2 20080704 (Red Hat 4.1.2-46)] on linux2 >>> numpy.__version__ '1.3.0' ~--$ uname -srvmpio Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux From Chris.Barker at noaa.gov Wed Apr 21 13:20:03 2010 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 21 Apr 2010 10:20:03 -0700 Subject: [Numpy-discussion] Disabling Extended Precision in NumPy (like -ffloat-store) In-Reply-To: References: Message-ID: <4BCF33C3.4020802@noaa.gov> Adrien Guillon wrote: > I use single-precision floating point operations. > > My understanding, however, is that Intel processors may use extended > precision for some operations anyways unless this is explicitly > disabled, which is done with gcc via the -ffloat-store operation. IIUC, that forces the FPU to not use the 80 bits in temps, but I think the fpu still uses 64 bits -- or does that flag actually force singles to stay single (32 bit) through the whole process as well? -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Chris.Barker at noaa.gov Wed Apr 21 13:23:29 2010 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 21 Apr 2010 10:23:29 -0700 Subject: [Numpy-discussion] Numpy compilation error In-Reply-To: References: <4BCD0490.1010108@noaa.gov> Message-ID: <4BCF3491.5020404@noaa.gov> David Cournapeau wrote: > I don't think it is wise to advocate the use of develop for python > newcomers. Fair enough. What I know is that every scheme I've come up with for working with my own under-development packages has been a pain in the #$@, and -develop has worked well for me. > The easy and reliable solution for non root install is PYTHONPATH for > python < 2.6 and install --user for python >= 2.6. I hadn't read the thread carefully enough to realize that the OP was asking about a non-root install, but in any case, I'd still encourage that folks set ONE standard place in their PYTHONPATH, rather than one for each package, and make sure you're careful about running more than one version of Python. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From matthewturk at gmail.com Wed Apr 21 15:41:02 2010 From: matthewturk at gmail.com (Matthew Turk) Date: Wed, 21 Apr 2010 12:41:02 -0700 Subject: [Numpy-discussion] Aggregate memmap Message-ID: Hi there, I've quite a bit of unformatted fortran data that I'd like to use as input to a memmap, as sort of a staging area for selection of subregions to be loaded into RAM. Unfortunately, what I'm running into is that the data was output as a set of "slices" through a 3D cube, instead of a single 3D cube -- the end result being that each slice also contains a record delimiter. I was wondering if there's a way to either specify that every traversal through the least-frequent dimension requires an additional offset or to calculate individual offsets for each slice myself and then aggregate these into a "super memmap." Thanks for any suggestions you might have! -Matt From robert.kern at gmail.com Wed Apr 21 15:45:17 2010 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 21 Apr 2010 14:45:17 -0500 Subject: [Numpy-discussion] trim_zeros in more than one dimension? In-Reply-To: <4BCF1B10.7040907@gmail.com> References: <8519e13fc8c2e059f4954b0aaabe1e9c.squirrel@srv2.hilboll.net> <4BCF1B10.7040907@gmail.com> Message-ID: On Wed, Apr 21, 2010 at 10:34, Bruce Southey wrote: > On 04/21/2010 08:36 AM, Robert Kern wrote: >> On Tue, Apr 20, 2010 at 18:45, Charles R Harris >> ?wrote: >> >>> On Tue, Apr 20, 2010 at 7:03 AM, Andreas Hilboll ?wrote: >>> >>>> Hi there, >>>> >>>> is there an easy way to do something like trim_zeros() does, but for a >>>> n-dimensional array? I have a 2d array with only zeros in the first and >>>> last rows and columns, and would like to trim this array to only the >>>> non-zero part ... >>>> >>> I think for your application it would be easier to use a subarray, something >>> like a[1:-1, 1:-1]. >>> >> Yes, but I think he's asking for a function that will find the >> appropriate slices for him, justl like trim_zeros() does for 1D >> arrays. >> >> > If the sum of axis to be removed equals zero then you can conditionally > remove that axis. No. Negative numbers can cancel out positive numbers. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From david at silveregg.co.jp Wed Apr 21 20:38:06 2010 From: david at silveregg.co.jp (David) Date: Thu, 22 Apr 2010 09:38:06 +0900 Subject: [Numpy-discussion] Disabling Extended Precision in NumPy (like -ffloat-store) In-Reply-To: References: Message-ID: <4BCF9A6E.5060006@silveregg.co.jp> On 04/21/2010 11:47 PM, Adrien Guillon wrote: > Hello all, > > I've recently started to use NumPy to prototype some numerical > algorithms, which will eventually find their way to a GPU (where I > want to limit myself to single-precision operations for performance > reasons). I have recently switched to the use of the "single" type in > NumPy to ensure I use single-precision floating point operations. > > My understanding, however, is that Intel processors may use extended > precision for some operations anyways unless this is explicitly > disabled, which is done with gcc via the -ffloat-store operation. > Since I am prototyping algorithms for a different processor > architecture, where the extended precision registers simply do not > exist, I would really like to force NumPy to limit itself to using > single-precision operations throughout the calculation (no extended > precision in registers). I don't think it is a good idea - even if you compile numpy itself with -ffloat-store, most runtime capabilities are built without this, so you will have differences wether the computation is done in the C library, in numpy, in the fortran runtime, or by the compiler (when computing constants). This sounds worse than what you can get from numpy by default, cheers, David From cournape at gmail.com Wed Apr 21 21:25:28 2010 From: cournape at gmail.com (David Cournapeau) Date: Thu, 22 Apr 2010 10:25:28 +0900 Subject: [Numpy-discussion] Numpy compilation error In-Reply-To: <4BCF3491.5020404@noaa.gov> References: <4BCD0490.1010108@noaa.gov> <4BCF3491.5020404@noaa.gov> Message-ID: On Thu, Apr 22, 2010 at 2:23 AM, Christopher Barker wrote: > I hadn't read the thread carefully enough to realize that the OP was > asking about a non-root install, but in any case, I'd still encourage > that folks set ONE standard place in their PYTHONPATH, rather than one > for each package, and make sure you're careful about running more than > one version of Python. The beauty of --user is that you don't need PYTHONPATH, and it is interpreter specific (at least if the package is correctly done). PYTHONPATH is becoming a pain with python 3.* cheers, David From aj.guillon at gmail.com Wed Apr 21 23:04:07 2010 From: aj.guillon at gmail.com (Adrien Guillon) Date: Wed, 21 Apr 2010 23:04:07 -0400 Subject: [Numpy-discussion] Disabling Extended Precision in NumPy (like -ffloat-store) In-Reply-To: <4BCF33C3.4020802@noaa.gov> References: <4BCF33C3.4020802@noaa.gov> Message-ID: Thank you for your questions... I'll answer them now. The motivation behind using Python and NumPy is to be able to "double check" that the numerical algorithms work okay in an engineer/scientist friendly language. We're basically prototyping a bunch of algorithms in Python, validating that they work right... so that when we move to the GPU we (hopefully) know that the algorithms are okay sequentially... any problems we come against therefore would have to be something else with the implementation (i.e. we don't want to be debugging too much at once). We are only targeting GPUs that support IEEE floating point... in theory the results should be similar on the GPU and CPU under certain assumptions (that the floating point is limited to single precision throughout... which Intel processors are a bit funny about). The main concern I have, and thus my motivation for wanting a restricted mode, is that parts of the algorithm may not work properly if done in extended precision. We are trying to ensure that theoretically there should be no problems, but realistically it's nice to have an extra layer of protection where we say "oh, wait, that's not right" when we look at the results. The idea here, is that if I can ensure there is never extended precision in the Python code... it should closely emulate the GPU which in fact has no extended precision in the hardware. Ordinarily, it's probably a silly thing to want to do (who would complain about extended precision for free)? But in this case I think it's the right decision. Hope this clarifies the reasons and intentions. AJ On Wed, Apr 21, 2010 at 1:20 PM, Christopher Barker wrote: > Adrien Guillon wrote: >> ?I use single-precision floating point operations. >> >> My understanding, however, is that Intel processors may use extended >> precision for some operations anyways unless this is explicitly >> disabled, which is done with gcc via the -ffloat-store operation. > > IIUC, that forces the FPU to not use the 80 bits in temps, but I think > the fpu still uses 64 bits -- or does that flag actually force singles > to stay single (32 bit) through the whole process as well? > > -CHB > > > > -- > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R ? ? ? ? ? ?(206) 526-6959 ? voice > 7600 Sand Point Way NE ? (206) 526-6329 ? fax > Seattle, WA ?98115 ? ? ? (206) 526-6317 ? main reception > > Chris.Barker at noaa.gov > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From oliphant at enthought.com Thu Apr 22 11:09:06 2010 From: oliphant at enthought.com (Travis Oliphant) Date: Thu, 22 Apr 2010 10:09:06 -0500 Subject: [Numpy-discussion] Is this a bug, and if so, who's? In-Reply-To: <4BCF1E24.4060203@jhu.edu> References: <4BCF1E24.4060203@jhu.edu> Message-ID: <59107FF9-5F41-40E3-A1D3-EA4AB3E92A57@enthought.com> On Apr 21, 2010, at 10:47 AM, Ken Basye wrote: > Folks, > Apologies for asking here, but I ran across this problem yesterday > and probably need to file a bug. The problem is I don't know if > this is > a Numpy bug, a Python bug, or both. Here's an illustration, platform > information follows. It's a bug in your implementation of class A. The __float__ method is supposed to return a Python float. In A, you are just returning the 'y' field which is initialized to whatever you passed in (which in the last example is an np.float64. An np.float64 is a subclass of Python's float, but it is not Python's float. So, the explicit conversion is proper. -Travis > TIA, > Ken > > > ############################################# > import collections > import numpy as np > > class A (collections.namedtuple('ANT', ('x', 'y'))): > def __float__(self): > return self.y > > # Same as A, but explicitly convert y to a float in __float__() - > this > works around the assert fail > class B (collections.namedtuple('BNT', ('x', 'y'))): > def __float__(self): > return float(self.y) > > a0 = A(1.0, 2.0) > f0 = np.float64(a0) > print f0 > > a1 = A(float(1.0), float(2.0)) > f1 = np.float64(a1) > print f1 > > b1 = B(np.float64(1.0), np.float64(2.0)) > f2 = np.float64(b1) > print f2 > > a2 = A(np.float64(1.0), np.float64(2.0)) > # On some platforms, the next line will trigger an > assert: > > # python: Objects/floatobject.c:1674: float_subtype_new: Assertion > `((((PyObject*)(tmp))->ob_type) == &PyFloat_Type)' failed. > f3 = np.float64(a2) > print f3 > ############################################# > > Platform info: > > Python 2.6.5 (r265:79063, Apr 14 2010, 13:32:56) > [GCC 4.1.2 20080704 (Red Hat 4.1.2-46)] on linux2 > >>>> numpy.__version__ > '1.3.0' > > ~--$ uname -srvmpio > Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009 x86_64 x86_64 > x86_64 GNU/Linux > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -- Travis Oliphant Enthought Inc. 1-512-536-1057 http://www.enthought.com oliphant at enthought.com From ralf.gommers at googlemail.com Thu Apr 22 11:52:39 2010 From: ralf.gommers at googlemail.com (Ralf Gommers) Date: Thu, 22 Apr 2010 23:52:39 +0800 Subject: [Numpy-discussion] [ANN] NumPy 1.4.1 Message-ID: Hi, I am pleased to announce the release of NumPy 1.4.1. This maintenance release removes datetime support, which fixes the binary incompatibility issues between NumPy 1.4.0 and SciPy and other extension packages. Several other bug fixes are also included. Binaries, sources and release notes can be found at https://sourceforge.net/projects/numpy/files/ Thank you to everyone who contributed to this release; developers, users who reported bugs, documentation writers, etc... Enjoy! The NumPy developers -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Thu Apr 22 17:16:50 2010 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 22 Apr 2010 16:16:50 -0500 Subject: [Numpy-discussion] Aggregate memmap In-Reply-To: References: Message-ID: On Wed, Apr 21, 2010 at 14:41, Matthew Turk wrote: > Hi there, > > I've quite a bit of unformatted fortran data that I'd like to use as > input to a memmap, as sort of a staging area for selection of > subregions to be loaded into RAM. ?Unfortunately, what I'm running > into is that the data was output as a set of "slices" through a 3D > cube, instead of a single 3D cube -- the end result being that each > slice also contains a record delimiter. ?I was wondering if there's a > way to either specify that every traversal through the least-frequent > dimension requires an additional offset or to calculate individual > offsets for each slice myself and then aggregate these into a "super > memmap." Is the record delimiter uniform in size? Like always 8 bytes or something similar? If so, you can make a record dtype that contains the delimiter and the slice array. np.dtype([('delimiter', '|V8'), ('slice', np.float32, (N, M))]) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From bsouthey at gmail.com Thu Apr 22 17:23:51 2010 From: bsouthey at gmail.com (Bruce Southey) Date: Thu, 22 Apr 2010 16:23:51 -0500 Subject: [Numpy-discussion] trim_zeros in more than one dimension? In-Reply-To: References: <8519e13fc8c2e059f4954b0aaabe1e9c.squirrel@srv2.hilboll.net> <4BCF1B10.7040907@gmail.com> Message-ID: <4BD0BE67.7@gmail.com> On 04/21/2010 02:45 PM, Robert Kern wrote: > On Wed, Apr 21, 2010 at 10:34, Bruce Southey wrote: > >> On 04/21/2010 08:36 AM, Robert Kern wrote: >> >>> On Tue, Apr 20, 2010 at 18:45, Charles R Harris >>> wrote: >>> >>> >>>> On Tue, Apr 20, 2010 at 7:03 AM, Andreas Hilboll wrote: >>>> >>>> >>>>> Hi there, >>>>> >>>>> is there an easy way to do something like trim_zeros() does, but for a >>>>> n-dimensional array? I have a 2d array with only zeros in the first and >>>>> last rows and columns, and would like to trim this array to only the >>>>> non-zero part ... >>>>> >>>>> >>>> I think for your application it would be easier to use a subarray, something >>>> like a[1:-1, 1:-1]. >>>> >>>> >>> Yes, but I think he's asking for a function that will find the >>> appropriate slices for him, justl like trim_zeros() does for 1D >>> arrays. >>> >>> >>> >> If the sum of axis to be removed equals zero then you can conditionally >> remove that axis. >> > No. Negative numbers can cancel out positive numbers. > > In such cases you can use the sum of the absolute value of the array or the sum of a suitability constructed boolean indicator for the selected axis. In the case of floating point dtypes, compare it to a value near zero rather than exactly zero. Bruce From peridot.faceted at gmail.com Thu Apr 22 18:20:56 2010 From: peridot.faceted at gmail.com (Anne Archibald) Date: Thu, 22 Apr 2010 18:20:56 -0400 Subject: [Numpy-discussion] Aggregate memmap In-Reply-To: References: Message-ID: On 21 April 2010 15:41, Matthew Turk wrote: > Hi there, > > I've quite a bit of unformatted fortran data that I'd like to use as > input to a memmap, as sort of a staging area for selection of > subregions to be loaded into RAM. Unfortunately, what I'm running > into is that the data was output as a set of "slices" through a 3D > cube, instead of a single 3D cube -- the end result being that each > slice also contains a record delimiter. I was wondering if there's a > way to either specify that every traversal through the least-frequent > dimension requires an additional offset or to calculate individual > offsets for each slice myself and then aggregate these into a "super > memmap." If you had a delimiter for each entry in a record array you could just make a dummy record field for the delimiter, but it sounds like you have one delimiter per slice instead. I think the way to go is to memmap it in as a flat uint8 array and then construct a view into that array with appropriate strides (which includes skipping the delimiter). This may well be what you meant by a "super memmap". > Thanks for any suggestions you might have! Good luck, Anne > -Matt > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From david.huard at gmail.com Thu Apr 22 22:57:39 2010 From: david.huard at gmail.com (David Huard) Date: Thu, 22 Apr 2010 22:57:39 -0400 Subject: [Numpy-discussion] Aggregate memmap In-Reply-To: References: Message-ID: Hi Matt, I don't think the memmap code support this. However, you can stack memmaps just as easily as arrays, so if you define individual memmaps for each slice and stack them (numpy.vstack), the resulting array will behave as a regular 3D array. HTH, David H. On Wed, Apr 21, 2010 at 3:41 PM, Matthew Turk wrote: > Hi there, > > I've quite a bit of unformatted fortran data that I'd like to use as > input to a memmap, as sort of a staging area for selection of > subregions to be loaded into RAM. Unfortunately, what I'm running > into is that the data was output as a set of "slices" through a 3D > cube, instead of a single 3D cube -- the end result being that each > slice also contains a record delimiter. I was wondering if there's a > way to either specify that every traversal through the least-frequent > dimension requires an additional offset or to calculate individual > offsets for each slice myself and then aggregate these into a "super > memmap." > > Thanks for any suggestions you might have! > > -Matt > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lists at hilboll.de Fri Apr 23 04:16:39 2010 From: lists at hilboll.de (Andreas) Date: Fri, 23 Apr 2010 10:16:39 +0200 Subject: [Numpy-discussion] Speeding up loadtxt / savetxt Message-ID: <4BD15767.6070600@hilboll.de> Hi there, I was wondering if there's a way to speedup loadtxt/savetxt for big arrays? So far, I'm plainly using something like this:: file = open("array.txt","w") a = np.loadtxt(file) file.close() However, since my files are pretty big (~200M), that's taking a long time... Perhaps there's a way to speed things up? Thanks a lot for your insight, Andreas. From lists at hilboll.de Fri Apr 23 04:16:47 2010 From: lists at hilboll.de (Andreas) Date: Fri, 23 Apr 2010 10:16:47 +0200 Subject: [Numpy-discussion] wrapping arrays with savetxt Message-ID: <4BD1576F.6000306@hilboll.de> Hi there, I would like to use savetxt() to write a 2d array to a txt file. It should be written row-wise, but there should only be 10 values per line. So, if I want to write an array a((2,12)), the file should read like this: X X X X X X X X X X X X X X X X X X X X X X X X Is there any way to accomplish this? I thought I could iterate over the rows (with for), and use savetxt on each row individually. But then there's still the problem of wrapping. Since in the end the files are going to be pretty big (1440x2880), it would be good to do this as efficiently as possible ... Thanks for your insight, Andreas From renesd at gmail.com Fri Apr 23 09:20:03 2010 From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=) Date: Fri, 23 Apr 2010 14:20:03 +0100 Subject: [Numpy-discussion] My GSoC Proposal to Implement a Subset of NumPy for PyPy In-Reply-To: <4BCEEDDC.2010300@student.matnat.uio.no> References: <4BCEEDDC.2010300@student.matnat.uio.no> Message-ID: On Wed, Apr 21, 2010 at 1:21 PM, Dag Sverre Seljebotn < dagss at student.matnat.uio.no> wrote: > Dan Roberts wrote: > > Thanks for the reply. You're certainly right that your work is > > extremely beneficial to mine. At present I'm afraid a great deal of > > NumPy C code isn't easily reusable and it's great you're addressing > > that. I may not have been thinking in line with Maciej, but I was > > thinking ufuncs would be written in pure Python and jit compiled to an > > efficient form. (We can make lots of nice assumptions about them) That > > said, I think being able to write generic ufuncs is a very good idea, > > and absolutely doable. > > This might be relevant? > > http://conference.scipy.org/proceedings/SciPy2008/paper_16/ > > -- > Dag Sverre > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > Also this: http://numcorepy.blogspot.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From dagss at student.matnat.uio.no Fri Apr 23 09:29:22 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 23 Apr 2010 15:29:22 +0200 Subject: [Numpy-discussion] toydist: Using manually after build system now? Message-ID: <4BD1A0B2.5050200@student.matnat.uio.no> (The toydist manual says to use this list, so here I go...) Is it possible to invoke toydist manually to install something built manually with a build system? How is the final .info specification supposed to look like when toysetup is coupled with a build system? I tried this: Library: Packages: healpix Extension: healpix.lib Sources: healpix/lib.so but it didn't work (invokes gcc on .so file). Will it perhaps be something like this: Library: Packages: healpix Extension: healpix.lib Built: build/something/lib.so in toydist 0.3 or 0.4? Dag Sverre From charlesr.harris at gmail.com Fri Apr 23 13:13:06 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Fri, 23 Apr 2010 11:13:06 -0600 Subject: [Numpy-discussion] Disabling Extended Precision in NumPy (like -ffloat-store) In-Reply-To: References: <4BCF33C3.4020802@noaa.gov> Message-ID: On Wed, Apr 21, 2010 at 9:04 PM, Adrien Guillon wrote: > Thank you for your questions... I'll answer them now. > > The motivation behind using Python and NumPy is to be able to "double > check" that the numerical algorithms work okay in an > engineer/scientist friendly language. We're basically prototyping a > bunch of algorithms in Python, validating that they work right... so > that when we move to the GPU we (hopefully) know that the algorithms > are okay sequentially... any problems we come against therefore would > have to be something else with the implementation (i.e. we don't want > to be debugging too much at once). We are only targeting GPUs that > support IEEE floating point... in theory the results should be similar > on the GPU and CPU under certain assumptions (that the floating point > is limited to single precision throughout... which Intel processors > are a bit funny about). > > The main concern I have, and thus my motivation for wanting a > restricted mode, is that parts of the algorithm may not work properly > if done in extended precision. We are trying to ensure that > theoretically there should be no problems, but realistically it's nice > to have an extra layer of protection where we say "oh, wait, that's > not right" when we look at the results. > > The idea here, is that if I can ensure there is never extended > precision in the Python code... it should closely emulate the GPU > which in fact has no extended precision in the hardware. Ordinarily, > it's probably a silly thing to want to do (who would complain about > extended precision for free)? But in this case I think it's the right > decision. > > Hope this clarifies the reasons and intentions. > > Thanks for the explanation. You might be able to do something with the variable precision float types in mpmath, they would allow you to select whatever precision you want. If your algorithms aren't too complicated you could then explore the effects of different precisions. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From peridot.faceted at gmail.com Fri Apr 23 13:36:40 2010 From: peridot.faceted at gmail.com (Anne Archibald) Date: Fri, 23 Apr 2010 13:36:40 -0400 Subject: [Numpy-discussion] Disabling Extended Precision in NumPy (like -ffloat-store) In-Reply-To: References: <4BCF33C3.4020802@noaa.gov> Message-ID: On 21 April 2010 23:04, Adrien Guillon wrote: > Thank you for your questions... I'll answer them now. > > The motivation behind using Python and NumPy is to be able to "double > check" that the numerical algorithms work okay in an > engineer/scientist friendly language. ?We're basically prototyping a > bunch of algorithms in Python, validating that they work right... so > that when we move to the GPU we (hopefully) know that the algorithms > are okay sequentially... any problems we come against therefore would > have to be something else with the implementation (i.e. we don't want > to be debugging too much at once). ?We are only targeting GPUs that > support IEEE floating point... in theory the results should be similar > on the GPU and CPU under certain assumptions (that the floating point > is limited to single precision throughout... which Intel processors > are a bit funny about). > > The main concern I have, and thus my motivation for wanting a > restricted mode, is that parts of the algorithm may not work properly > if done in extended precision. ?We are trying to ensure that > theoretically there should be no problems, but realistically it's nice > to have an extra layer of protection where we say "oh, wait, that's > not right" when we look at the results. > > The idea here, is that if I can ensure there is never extended > precision in the Python code... it should closely emulate the GPU > which in fact has no extended precision in the hardware. ?Ordinarily, > it's probably a silly thing to want to do (who would complain about > extended precision for free)? ?But in this case I think it's the right > decision. Funny you should ask: if you look in the mailing list archive you'll find that we were just recently bitten by this in the function log10_1p: the algorithm was one of those clever numerical analyst's hacks where the roundoff in one part was supposed to cancel the roundoff in another part, but extended precision was being used for one part and not the other. The ultimate solution is to just use the base e log1p, which we either get from the standard library or implement ourselves in a way that doesn't require this delicate cancellation. Anne > Hope this clarifies the reasons and intentions. > > AJ > > > > On Wed, Apr 21, 2010 at 1:20 PM, Christopher Barker > wrote: >> Adrien Guillon wrote: >>> ?I use single-precision floating point operations. >>> >>> My understanding, however, is that Intel processors may use extended >>> precision for some operations anyways unless this is explicitly >>> disabled, which is done with gcc via the -ffloat-store operation. >> >> IIUC, that forces the FPU to not use the 80 bits in temps, but I think >> the fpu still uses 64 bits -- or does that flag actually force singles >> to stay single (32 bit) through the whole process as well? >> >> -CHB >> >> >> >> -- >> Christopher Barker, Ph.D. >> Oceanographer >> >> Emergency Response Division >> NOAA/NOS/OR&R ? ? ? ? ? ?(206) 526-6959 ? voice >> 7600 Sand Point Way NE ? (206) 526-6329 ? fax >> Seattle, WA ?98115 ? ? ? (206) 526-6317 ? main reception >> >> Chris.Barker at noaa.gov >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion at scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From josef.pktd at gmail.com Fri Apr 23 13:45:10 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 23 Apr 2010 13:45:10 -0400 Subject: [Numpy-discussion] ma.std(ddof=1) bug? Message-ID: Is there a reason why ma.std(ddof=1) does not calculated the std if there are 2 valid values? example nan = np.nan x1 = np.array([[9.0, 3.0, nan, nan, 9.0, nan], [1.0, 1.0, 1.0, nan, nan, nan], [2.0, 2.0, 0.01, nan, 1.0, nan], [3.0, 9.0, 2.0, nan, nan, nan], [4.0, 4.0, 3.0, 9.0, 2.0, nan], [5.0, 5.0, 4.0, 4.0, nan, nan]]) >>> print np.ma.fix_invalid(x1).std(0, ddof=0) [2.58198889747 2.58198889747 1.41138796934 2.5 3.55902608401 --] >>> print np.ma.fix_invalid(x1).std(0, ddof=1) [2.82842712475 2.82842712475 1.57797972104 -- 4.35889894354 --] # invalid column 3 scipy stats (bias=True is default) >>> print stats.nanstd(x1,0) [ 2.82842712 2.82842712 1.57797972 3.53553391 4.35889894 NaN] numpy with valid values >>> np.array((9,4.)).std(ddof=1) 3.5355339059327378 Josef From jason-sage at creativetrax.com Fri Apr 23 14:53:13 2010 From: jason-sage at creativetrax.com (Jason Grout) Date: Fri, 23 Apr 2010 13:53:13 -0500 Subject: [Numpy-discussion] Disabling Extended Precision in NumPy (like -ffloat-store) In-Reply-To: References: Message-ID: <4BD1EC99.5010300@creativetrax.com> On 04/21/2010 09:47 AM, Adrien Guillon wrote: > Hello all, > > I've recently started to use NumPy to prototype some numerical > algorithms, which will eventually find their way to a GPU (where I > want to limit myself to single-precision operations for performance > reasons). I have recently switched to the use of the "single" type in > NumPy to ensure I use single-precision floating point operations. > > My understanding, however, is that Intel processors may use extended > precision for some operations anyways unless this is explicitly > disabled, which is done with gcc via the -ffloat-store operation. > Since I am prototyping algorithms for a different processor > architecture, where the extended precision registers simply do not > exist, I would really like to force NumPy to limit itself to using > single-precision operations throughout the calculation (no extended > precision in registers). > > How can I do this? > If you want to avoid any assumptions the hardware might make that you aren't aware of, you might look at software libraries which emulate single precision numbers. For example, in Sage [1], we have a floating point type that is a python object wrapped around an MPFR [2] number (look at the RealField class or email sage-support for details). You might also look at minifloat libraries or related libraries. For example, a google search turns up http://groups.google.com/group/comp.lang.python/browse_thread/thread/ba2b69a8fd8d45f6. The cool thing is that the numpy arrays can use these objects as naturally as other python objects. Thanks, Jason [1] http://www.sagemath.org [2] http://www.mpfr.org From ademan555 at gmail.com Fri Apr 23 15:56:22 2010 From: ademan555 at gmail.com (Dan Roberts) Date: Fri, 23 Apr 2010 12:56:22 -0700 Subject: [Numpy-discussion] My GSoC Proposal to Implement a Subset of NumPy for PyPy In-Reply-To: References: <4BCEEDDC.2010300@student.matnat.uio.no> Message-ID: Wow, that's a very cool idea. I think that's an excellent approach to allowing user RPython functions. Maciej expressed concern this could create a support burden for RPython for the core PyPy developers (There aren't many of them). I think, handled correctly, this could help create a community knowledgable about RPython that could support it, but initially I would be the only one supporting this use. I think the best approach would be to provide a similar fast_vectorize() in addition to accelerating normal python looping constructs with the JIT compiler, that way we can have faster code without even trying. I'm going to be contacting the author of that paper to see what its implementation status is, as the paper mentioned he was trying to get it into SciPy. Sorry about the latency, I tend to make multiple drafts of my emails before I send them... On Apr 21, 2010 5:20 AM, "Dag Sverre Seljebotn" wrote: Dan Roberts wrote: > Thanks for the reply. You're certainly right that your work is > extremely be... This might be relevant? http://conference.scipy.org/proceedings/SciPy2008/paper_16/ -- Dag Sverre _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion at scipy... -------------- next part -------------- An HTML attachment was scrubbed... URL: From dwf at cs.toronto.edu Sat Apr 24 01:07:32 2010 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Sat, 24 Apr 2010 01:07:32 -0400 Subject: [Numpy-discussion] Disabling Extended Precision in NumPy (like -ffloat-store) In-Reply-To: References: <4BCF33C3.4020802@noaa.gov> Message-ID: <4BD27C94.6070307@cs.toronto.edu> Adrien Guillon wrote: > Thank you for your questions... I'll answer them now. > > The motivation behind using Python and NumPy is to be able to "double > check" that the numerical algorithms work okay in an > engineer/scientist friendly language. We're basically prototyping a > bunch of algorithms in Python, validating that they work right... so > that when we move to the GPU we (hopefully) know that the algorithms > are okay sequentially... any problems we come against therefore would > have to be something else with the implementation (i.e. we don't want > to be debugging too much at once). We are only targeting GPUs that > support IEEE floating point... in theory the results should be similar > on the GPU and CPU under certain assumptions (that the floating point > is limited to single precision throughout... which Intel processors > are a bit funny about). > > The main concern I have, and thus my motivation for wanting a > restricted mode, is that parts of the algorithm may not work properly > if done in extended precision. We are trying to ensure that > theoretically there should be no problems, but realistically it's nice > to have an extra layer of protection where we say "oh, wait, that's > not right" when we look at the results. > > The idea here, is that if I can ensure there is never extended > precision in the Python code... it should closely emulate the GPU > which in fact has no extended precision in the hardware. Ordinarily, > it's probably a silly thing to want to do (who would complain about > extended precision for free)? But in this case I think it's the right > decision. > > Hope this clarifies the reasons and intentions. If you are mainly targeting Nvidia GPUs you should check out theano, which allows you to prototype algorithms and have theano generate and run CUDA GPU code for you. Translating a NumPy-using algorithm into Theano calls is straightforward and fairly quick, and if you're running your prototypes on a GPU in the first place, you can be sure that the hardware limitations are not being violated. It will also give your prototype a speed boost over the CPU version :) http://deeplearning.net/software/theano/ David From charlesr.harris at gmail.com Sat Apr 24 17:35:00 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Sat, 24 Apr 2010 15:35:00 -0600 Subject: [Numpy-discussion] Patches for review cleanup. Message-ID: Hi folks, There are a lot of patches sitting around waiting for review. I think most can be taken care of pretty quickly and closed. There are, however, a half dozen or so small patches relating to distutils that someone familiar with that part of numpy should go over (David?). Anyway, lets get that queue cleaned out. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Sun Apr 25 08:01:30 2010 From: cournape at gmail.com (David Cournapeau) Date: Sun, 25 Apr 2010 21:01:30 +0900 Subject: [Numpy-discussion] toydist: Using manually after build system now? In-Reply-To: <4BD1A0B2.5050200@student.matnat.uio.no> References: <4BD1A0B2.5050200@student.matnat.uio.no> Message-ID: On Fri, Apr 23, 2010 at 10:29 PM, Dag Sverre Seljebotn wrote: > (The toydist manual says to use this list, so here I go...) > > Is it possible to invoke toydist manually to install something built > manually with a build system? This is not yet supported, but is basically the main feature for 0.0.3. > How is the final .info specification > supposed to look like when toysetup is coupled with a build system? You will still only add sources (i.e. no modification will be done to the .info file), like you would do in automake if you have some autotools experience. Instead, for customized builds, you would use an additional python file which contain hooks: ... @override def build(build_context): # Gives you a PackageDescription instance pkg = get_configured_state().pkg You can see some examples here: http://github.com/cournape/toydist/blob/hooks/examples/hook Note that the branch is "private", though, that is the code may not end up as is in master. Instead of designing something up front, I decided to implement the hooks mechanism by adding what I need to build something like numpy.core with waf within toydist. Nothing there is set in stone yet, so comments are welcomed there, cheers, David From cournape at gmail.com Sun Apr 25 08:07:22 2010 From: cournape at gmail.com (David Cournapeau) Date: Sun, 25 Apr 2010 21:07:22 +0900 Subject: [Numpy-discussion] Disabling Extended Precision in NumPy (like -ffloat-store) In-Reply-To: References: <4BCF33C3.4020802@noaa.gov> Message-ID: On Thu, Apr 22, 2010 at 12:04 PM, Adrien Guillon wrote: > > The idea here, is that if I can ensure there is never extended > precision in the Python code... This is totally out of reach with numpy is you use the float32 dtype, for the reasons I have given before. The only solutions I could see are: using an enviroment geared toward IEEE 754 compliance (CPU emulation), simply use one of the existing package to run code on GPU, or use software-implemented FPU. The latter meaning that you cannot use linear algebra and so on, at least not with just numpy/scipy, David From josef.pktd at gmail.com Sun Apr 25 09:16:49 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 25 Apr 2010 09:16:49 -0400 Subject: [Numpy-discussion] floats as axis Message-ID: (some) numpy functions take floats as valid axis argument. Is this a feature? >>> np.ones((2,3)).sum(1.2) array([ 3., 3.]) >>> np.ones((2,3)).sum(1.99) array([ 3., 3.]) >>> np.mean((1.5,0.5)) 1.0 >>> np.mean(1.5,0.5) 1.5 Keith pointed out that scipy.stats.nanmean has a different behavior >>> stats.nanmean(np.ones((2,3)), axis= 1.2) Traceback (most recent call last): <...> TypeError: tuple indices must be integers see: http://projects.scipy.org/scipy/ticket/1165 What's the preferred behavior? Josef From kwgoodman at gmail.com Sun Apr 25 10:45:38 2010 From: kwgoodman at gmail.com (Keith Goodman) Date: Sun, 25 Apr 2010 07:45:38 -0700 Subject: [Numpy-discussion] floats as axis In-Reply-To: References: Message-ID: On Sun, Apr 25, 2010 at 6:16 AM, wrote: > (some) numpy functions take floats as valid axis argument. Is this a feature? > >>>> np.ones((2,3)).sum(1.2) > array([ 3., ?3.]) >>>> np.ones((2,3)).sum(1.99) > array([ 3., ?3.]) > >>>> np.mean((1.5,0.5)) > 1.0 >>>> np.mean(1.5,0.5) > 1.5 > > Keith pointed out that scipy.stats.nanmean has a different behavior > >>>> stats.nanmean(np.ones((2,3)), axis= 1.2) > Traceback (most recent call last): > <...> > TypeError: tuple indices must be integers It fails on axis=0.0 and axis=1.0 too. Those are probably more common values and less likely to be an error. > see: http://projects.scipy.org/scipy/ticket/1165 > > What's the preferred behavior? > > Josef From josef.pktd at gmail.com Sun Apr 25 10:54:26 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Sun, 25 Apr 2010 10:54:26 -0400 Subject: [Numpy-discussion] floats as axis In-Reply-To: References: Message-ID: On Sun, Apr 25, 2010 at 10:45 AM, Keith Goodman wrote: > On Sun, Apr 25, 2010 at 6:16 AM, ? wrote: >> (some) numpy functions take floats as valid axis argument. Is this a feature? >> >>>>> np.ones((2,3)).sum(1.2) >> array([ 3., ?3.]) >>>>> np.ones((2,3)).sum(1.99) >> array([ 3., ?3.]) >> >>>>> np.mean((1.5,0.5)) >> 1.0 >>>>> np.mean(1.5,0.5) >> 1.5 >> >> Keith pointed out that scipy.stats.nanmean has a different behavior >> >>>>> stats.nanmean(np.ones((2,3)), axis= 1.2) >> Traceback (most recent call last): >> <...> >> TypeError: tuple indices must be integers > > It fails on axis=0.0 and axis=1.0 too. Those are probably more common > values and less likely to be an error. I wouldn't trust floating 1.0 or 0.0 either >>> np.ones((2,3)).sum(1.+1e-16) array([ 3., 3.]) >>> np.ones((2,3)).sum(1.-1e-16) array([ 2., 2., 2.]) Josef > >> see: http://projects.scipy.org/scipy/ticket/1165 >> >> What's the preferred behavior? >> >> Josef > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From robert.kern at gmail.com Sun Apr 25 10:57:05 2010 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 25 Apr 2010 09:57:05 -0500 Subject: [Numpy-discussion] Aggregate memmap In-Reply-To: References: Message-ID: On Thu, Apr 22, 2010 at 21:57, David Huard wrote: > Hi Matt, > I don't think the memmap code support this. However, you can stack memmaps > just as easily as arrays, so if you define individual memmaps for each slice > and stack them (numpy.vstack), the resulting array will behave as a regular > 3D array. However, that will make a main-memory copy of everything, presumably what he is trying to avoid. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From Chris.Barker at noaa.gov Sun Apr 25 12:20:38 2010 From: Chris.Barker at noaa.gov (Chris Barker) Date: Sun, 25 Apr 2010 09:20:38 -0700 Subject: [Numpy-discussion] Speeding up loadtxt / savetxt In-Reply-To: <4BD15767.6070600@hilboll.de> References: <4BD15767.6070600@hilboll.de> Message-ID: <4BD46BD6.6020002@noaa.gov> Andreas wrote: > I was wondering if there's a way to speedup loadtxt/savetxt for big > arrays? So far, I'm plainly using something like this:: > > file = open("array.txt","w") > a = np.loadtxt(file) > file.close() > However, since my files are pretty big (~200M), that's taking a long > time... Perhaps there's a way to speed things up? for the simplest of cases, you can use fromfile() with a delimiter -- if it works, it's much faster. See the archives of this list and the tickets for discussion of limitations and improvements it could use. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From threexk at hotmail.com Sun Apr 25 13:42:17 2010 From: threexk at hotmail.com (threexk threexk) Date: Sun, 25 Apr 2010 13:42:17 -0400 Subject: [Numpy-discussion] Incomplete uninstall of 1.4.0 superpack Message-ID: Hello, I recently uninstalled the NumPy 1.4.0 superpack for Python 2.6 on Windows 7, and afterward a dialog popped up that said 1 file or directory could not be removed. Does anyone have any idea which file/directory this is? The dialog gave no indication. Is an uninstall log with details generated anywhere? Perhaps it is some shared DLL, but I have no idea which! Thanks, threexk _________________________________________________________________ The New Busy is not the too busy. Combine all your e-mail accounts with Hotmail. http://www.windowslive.com/campaign/thenewbusy?tile=multiaccount&ocid=PID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_4 -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthewturk at gmail.com Sun Apr 25 16:20:35 2010 From: matthewturk at gmail.com (Matthew Turk) Date: Sun, 25 Apr 2010 13:20:35 -0700 Subject: [Numpy-discussion] Aggregate memmap In-Reply-To: References: Message-ID: Hi Everyone, Thanks for your suggestions and replies. I initially tried what Anne suggested, modifying the strides in the third dimension to account for the 8-byte delimiters between slabs, but I couldn't control the performance as much as I'd like, and I wasn't entirely sure when and where "real" memory was being allocated or deallocated, or how it was striding over the disk. (Memory-mapping also seems to require allocating the full size of the array as virtual memory.) In the end I ended up writing a very short Cython routine hand-tuned for the problem that accepted the specific 3D slab to read off disk and return that to the user, which also helped me coordinate better between multiple MPI processes, as well. Thanks very much for your suggestions! -Matt On Sun, Apr 25, 2010 at 7:57 AM, Robert Kern wrote: > On Thu, Apr 22, 2010 at 21:57, David Huard wrote: >> Hi Matt, >> I don't think the memmap code support this. However, you can stack memmaps >> just as easily as arrays, so if you define individual memmaps for each slice >> and stack them (numpy.vstack), the resulting array will behave as a regular >> 3D array. > > However, that will make a main-memory copy of everything, presumably > what he is trying to avoid. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > ?-- Umberto Eco > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From robert.kern at gmail.com Sun Apr 25 17:17:11 2010 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 25 Apr 2010 16:17:11 -0500 Subject: [Numpy-discussion] trim_zeros in more than one dimension? In-Reply-To: <4BD0BE67.7@gmail.com> References: <8519e13fc8c2e059f4954b0aaabe1e9c.squirrel@srv2.hilboll.net> <4BCF1B10.7040907@gmail.com> <4BD0BE67.7@gmail.com> Message-ID: On Thu, Apr 22, 2010 at 16:23, Bruce Southey wrote: > On 04/21/2010 02:45 PM, Robert Kern wrote: >> On Wed, Apr 21, 2010 at 10:34, Bruce Southey ?wrote: >>> If the sum of axis to be removed equals zero then you can conditionally >>> remove that axis. >>> >> No. Negative numbers can cancel out positive numbers. >> > In such cases you can use the sum of the absolute value of the array or > the sum of a suitability constructed boolean indicator for the selected > axis. In the case of floating point dtypes, compare it to a value near > zero rather than exactly zero. Or you can use numpy.nonzero(...).any(). -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Sun Apr 25 17:18:31 2010 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 25 Apr 2010 16:18:31 -0500 Subject: [Numpy-discussion] trim_zeros in more than one dimension? In-Reply-To: References: <8519e13fc8c2e059f4954b0aaabe1e9c.squirrel@srv2.hilboll.net> <4BCF1B10.7040907@gmail.com> <4BD0BE67.7@gmail.com> Message-ID: On Sun, Apr 25, 2010 at 16:17, Robert Kern wrote: > On Thu, Apr 22, 2010 at 16:23, Bruce Southey wrote: >> On 04/21/2010 02:45 PM, Robert Kern wrote: >>> On Wed, Apr 21, 2010 at 10:34, Bruce Southey ?wrote: > >>>> If the sum of axis to be removed equals zero then you can conditionally >>>> remove that axis. >>>> >>> No. Negative numbers can cancel out positive numbers. >>> >> In such cases you can use the sum of the absolute value of the array or >> the sum of a suitability constructed boolean indicator for the selected >> axis. In the case of floating point dtypes, compare it to a value near >> zero rather than exactly zero. > > Or you can use numpy.nonzero(...).any(). Or really just .any(). -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From cournape at gmail.com Mon Apr 26 01:57:22 2010 From: cournape at gmail.com (David Cournapeau) Date: Mon, 26 Apr 2010 14:57:22 +0900 Subject: [Numpy-discussion] Incomplete uninstall of 1.4.0 superpack In-Reply-To: References: Message-ID: On Mon, Apr 26, 2010 at 2:42 AM, threexk threexk wrote: > Hello, > > I recently uninstalled the NumPy 1.4.0 superpack for Python 2.6 on Windows > 7, and afterward a dialog popped up that said 1 file or directory could not > be removed.? Does anyone have any idea which file/directory this is?? The > dialog gave no indication.? Is an uninstall log with details generated > anywhere? There should be one in C:\Python*, something like numpy-*-wininst.log > Perhaps it is some shared DLL, but I have no idea which! The numpy installer does not have any shared DLL, cheers, David From dagss at student.matnat.uio.no Mon Apr 26 06:57:00 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 26 Apr 2010 12:57:00 +0200 Subject: [Numpy-discussion] Memory profiling NumPy code? Message-ID: <4BD5717C.6030301@student.matnat.uio.no> I'd like to profile the memory usage of my application using tools like e.g. Heapy. However since NumPy arrays are allocated in C they are not tracked by Python memory profiling. Does anybody have a workaround to share? I really just want to track a few arrays in a friendly way from within Python (I am aware of the existance of C-level profilers). (It should be possible to work around by using a custom function to allocate arrays using the Python memory allocater instead, but I'm hoping somebody else has done it already...) Dag Sverre From cournape at gmail.com Mon Apr 26 07:16:25 2010 From: cournape at gmail.com (David Cournapeau) Date: Mon, 26 Apr 2010 20:16:25 +0900 Subject: [Numpy-discussion] Memory profiling NumPy code? In-Reply-To: <4BD5717C.6030301@student.matnat.uio.no> References: <4BD5717C.6030301@student.matnat.uio.no> Message-ID: On Mon, Apr 26, 2010 at 7:57 PM, Dag Sverre Seljebotn wrote: > I'd like to profile the memory usage of my application using tools like > e.g. Heapy. However since NumPy arrays are allocated in C they are not > tracked by Python memory profiling. > > Does anybody have a workaround to share? I really just want to track a > few arrays in a friendly way from within Python (I am aware of the > existance of C-level profilers). I think heapy has some hooks so that you can add support for extensions. Maybe we could provide a C API in numpy to make this easy, David From renesd at gmail.com Mon Apr 26 11:59:59 2010 From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=) Date: Mon, 26 Apr 2010 16:59:59 +0100 Subject: [Numpy-discussion] Memory profiling NumPy code? In-Reply-To: References: <4BD5717C.6030301@student.matnat.uio.no> Message-ID: numpy could implement __sizeof__ On Mon, Apr 26, 2010 at 12:16 PM, David Cournapeau wrote: > On Mon, Apr 26, 2010 at 7:57 PM, Dag Sverre Seljebotn > wrote: > > I'd like to profile the memory usage of my application using tools like > > e.g. Heapy. However since NumPy arrays are allocated in C they are not > > tracked by Python memory profiling. > > > > Does anybody have a workaround to share? I really just want to track a > > few arrays in a friendly way from within Python (I am aware of the > > existance of C-level profilers). > > I think heapy has some hooks so that you can add support for > extensions. Maybe we could provide a C API in numpy to make this easy, > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From threexk at hotmail.com Mon Apr 26 12:08:09 2010 From: threexk at hotmail.com (threexk threexk) Date: Mon, 26 Apr 2010 12:08:09 -0400 Subject: [Numpy-discussion] Incomplete uninstall of 1.4.0 superpack In-Reply-To: References: , Message-ID: David Cournapeau wrote: > On Mon, Apr 26, 2010 at 2:42 AM, threexk threexk wrote: > > Hello, > > > > I recently uninstalled the NumPy 1.4.0 superpack for Python 2.6 on Windows > > 7, and afterward a dialog popped up that said 1 file or directory could not > > be removed. Does anyone have any idea which file/directory this is? The > > dialog gave no indication. Is an uninstall log with details generated > > anywhere? > > There should be one in C:\Python*, something like numpy-*-wininst.log Looks like that log gets deleted after uninstallation (as it probably should), so I still could not figure out which file/directory was not deleted. I found that \Python26\Lib\site-packages\numpy and many files/directories under it have remained after uninstall. So, I tried reinstalling 1.4.0 and uninstalling again. This time, the uninstaller did not report not being able to remove files/directories, but it still did not delete the aforementioned numpy directory. I believe this is a bug with the uninstaller? I have most everything closed, so I don't believe there could be any sort of sharing violations. Thanks, threexk _________________________________________________________________ Hotmail has tools for the New Busy. Search, chat and e-mail from your inbox. http://www.windowslive.com/campaign/thenewbusy?ocid=PID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_1 -------------- next part -------------- An HTML attachment was scrubbed... URL: From temesgen123 at gmail.com Mon Apr 26 12:04:12 2010 From: temesgen123 at gmail.com (Thomas) Date: Mon, 26 Apr 2010 16:04:12 +0000 (UTC) Subject: [Numpy-discussion] passing non-flat array to interpolator Message-ID: I have some problem with interpolators in Scipy does anyone knows if there is a way to pass a non-flat array variables to Rbf, or other Scipy interpolator eg. for my case of 17 x 1 problems of 500 data size x1.shape = (500,) x2.shape = (500,) ... X17.shape =(500,) b = Rbf(x1,x2,x3,...,x17,y) i would rather create a non-flat variables x.shape =(500,17) and pass it to Rbf, or even for all the interpolator in Scipy as bf = Rbf(X, Y) How can i do this ? Thank you for your time. Thomas From charlesr.harris at gmail.com Mon Apr 26 12:55:51 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 26 Apr 2010 10:55:51 -0600 Subject: [Numpy-discussion] What should be the value of nansum of nan's? Message-ID: Hi All, We need to make a decision for ticket #1123regarding what nansum should return when all values are nan. At some earlier point it was zero, but currently it is nan, in fact it is nan whatever the operation is. That is consistent, simple and serves to mark the array or axis as containing all nans. I would like to close the ticket and am a bit inclined to go with the current behaviour although there is an argument to be made for returning 0 for the nansum case. Thoughts? Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Mon Apr 26 13:03:38 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Mon, 26 Apr 2010 11:03:38 -0600 Subject: [Numpy-discussion] What should be the value of nansum of nan's? In-Reply-To: References: Message-ID: On Mon, Apr 26, 2010 at 10:55 AM, Charles R Harris < charlesr.harris at gmail.com> wrote: > Hi All, > > We need to make a decision for ticket #1123regarding what nansum should return when all values are nan. At some earlier > point it was zero, but currently it is nan, in fact it is nan whatever the > operation is. That is consistent, simple and serves to mark the array or > axis as containing all nans. I would like to close the ticket and am a bit > inclined to go with the current behaviour although there is an argument to > be made for returning 0 for the nansum case. Thoughts? > > To add a bit of context, one could argue that the results should be consistent with the equivalent operations on empty arrays and always be non-nan. In [1]: nansum([]) Out[1]: nan In [2]: sum([]) Out[2]: 0.0 Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From jkington at wisc.edu Mon Apr 26 17:22:15 2010 From: jkington at wisc.edu (Joe Kington) Date: Mon, 26 Apr 2010 16:22:15 -0500 Subject: [Numpy-discussion] Memory profiling NumPy code? In-Reply-To: References: <4BD5717C.6030301@student.matnat.uio.no> Message-ID: I know you're looking for something with much more fine-grained control, (which I can't help much with) but I often find it useful to just plot the overall memory of the program over time. There may be an slicker way to do it, but here's the script I use, anyway... (saved as ~/bin/quick_profile, usage quick_profile (whatever), e.g. "quick_profile python script.py") # /bin/sh # Setup datfile=$(mktemp) echo "ElapsedTime MemUsed" > $datfile starttime=$(date +%s.%N) # Run the specified command in the background $@ & # While the last process is still going while [ -n "`ps --no-headers $!`" ] do bytes=$(ps -o rss -C $1 --no-headers | awk '{SUM += $1} END {print SUM}') elapsed=$(echo $(date +%s.%N) - $starttime | bc) echo $elapsed $bytes >> $datfile sleep 0.1 done # Plot up the results with matplotlib cat <wrote: > On Mon, Apr 26, 2010 at 7:57 PM, Dag Sverre Seljebotn > wrote: > > I'd like to profile the memory usage of my application using tools like > > e.g. Heapy. However since NumPy arrays are allocated in C they are not > > tracked by Python memory profiling. > > > > Does anybody have a workaround to share? I really just want to track a > > few arrays in a friendly way from within Python (I am aware of the > > existance of C-level profilers). > > I think heapy has some hooks so that you can add support for > extensions. Maybe we could provide a C API in numpy to make this easy, > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dwf at cs.toronto.edu Mon Apr 26 20:19:25 2010 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Mon, 26 Apr 2010 20:19:25 -0400 Subject: [Numpy-discussion] proposing a "beware of [as]matrix()" warning Message-ID: <4BD62D8D.8060401@cs.toronto.edu> Trying to debug code written by an undergrad working for a colleague of mine who ported code over from MATLAB, I am seeing an ugly melange of matrix objects and ndarrays that are interacting poorly with each other and various functions in SciPy/other libraries. In particular there was a custom minimizer function that contained a line "a * b", that was receiving an Nx1 "matrix" and a N-length array and computing an outer product. Hence the unexpected 6 GB of memory usage and weird results... We've had this discussion before and it seems that the matrix class isn't going anywhere (I *really* wish it would at least be banished from the top-level namespace), but it has its adherents for pedagogical reasons. Could we at least consider putting a gigantic warning on all the functions for creating matrix objects (matrix, mat, asmatrix, etc.) that they may not behave quite so predictably in some situations and should be avoided when writing nontrivial code? There are already such warnings scattered about on SciPy.org but the situation is so bad, in my opinion (bad from a programming perspective and bad from a new user perspective, asking "why doesn't this work? why doesn't that work? why is this language/library/etc. so stupid, inconsistent, etc.?") that the situation warrants steering people still further away from the matrix object. I apologize for ranting, but it pains me when people give up on Python/NumPy because they can't figure out inconsistencies that aren't really there for a good reason. IMHO, of course. David David From seb.haase at gmail.com Tue Apr 27 04:27:31 2010 From: seb.haase at gmail.com (Sebastian Haase) Date: Tue, 27 Apr 2010 10:27:31 +0200 Subject: [Numpy-discussion] ndimage.label - howto force SWIG to use int32 - even on 64bit Linux ? Message-ID: Hi, I wanted to write some C code to accept labels as they come from ndimage.label. For some reason ndimage.label produces its output as an int32 array - even on my 64bit system . BTW, could this be considered a bug ? Now, if I use the typemaps of numpy.i I can choose between NPY_LONG and NPY_INT. But those are sometimes 32 sometimes 64 bit, depending on the system. Any ideas ... ? Thanks, Sebastian Haase From stefan at sun.ac.za Tue Apr 27 10:14:01 2010 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Tue, 27 Apr 2010 16:14:01 +0200 Subject: [Numpy-discussion] Speeding up loadtxt / savetxt In-Reply-To: <4BD15767.6070600@hilboll.de> References: <4BD15767.6070600@hilboll.de> Message-ID: Hi Andreas On 23 April 2010 10:16, Andreas wrote: > I was wondering if there's a way to speedup loadtxt/savetxt for big > arrays? So far, I'm plainly using something like this:: Do you specifically need to store text files? NumPy's binary storage functions (numpy.load and save) are faster. Also, an efficient reader for very simply formatted text is provided by numpy.fromfile. Regards St?fan From pgmdevlist at gmail.com Tue Apr 27 12:56:42 2010 From: pgmdevlist at gmail.com (Pierre GM) Date: Tue, 27 Apr 2010 09:56:42 -0700 Subject: [Numpy-discussion] ma.std(ddof=1) bug? In-Reply-To: References: Message-ID: On Apr 23, 2010, at 12:45 PM, josef.pktd at gmail.com wrote: > Is there a reason why ma.std(ddof=1) does not calculated the std if > there are 2 valid values? Bug! Good call... Should be fixed in SVN r8370. From kwgoodman at gmail.com Tue Apr 27 17:05:04 2010 From: kwgoodman at gmail.com (Keith Goodman) Date: Tue, 27 Apr 2010 14:05:04 -0700 Subject: [Numpy-discussion] [ANN] la 0.2, the labeled array Message-ID: I am pleased to announce the second release of the la package, version 0.2. The main class of the la package is a labeled array, larry. A larry consists of a data array and a label list. The data array is stored as a NumPy array and the label list as a list of lists. larry has built-in methods such as movingsum, ranking, merge, shuffle, zscore, demean, lag as well as typical Numpy methods like sum, max, std, sign, clip. NaNs are treated as missing data. Alignment by label is automatic when you add (or subtract, multiply, divide) two larrys. larry adds the convenience of labels, provides many built-in methods, and let's you use many of your existing array functions. Download: https://launchpad.net/larry/+download docs ?http://larry.sourceforge.net code ?https://launchpad.net/larry list ?http://groups.google.ca/group/pystatsmodels ============= Release Notes ============= la 0.2 (avocado) ================ *Release date: 2010-04-27* New larry methods ----------------- - lix : Index into a larry using labels or index numbers or both - swapaxes : Swap the two specified axes - sortaxis : Sort data (and label) according to label along specified axis - flipaxis : Reverse the order of the elements along the specified axis - tocsv : Save larry to a csv file - fromcsv : Load a larry from a csv file - insertaxis : Insert a new axis at the specified position - invert : Element by element inverting of True to False and False to True Enhancements ------------ - All larry methods can now take nd input arrays (some previously 2d only) - Added ability to save larrys with datetime.date labels to HDF5 - New function (panel) to convert larry of shape (n, m, k) to shape (m*k, n) - Expanded documentation - Over 280 new unit tests; testing easier with new assert_larry_equal function Bug fixes --------- - #517912: larry([]) == larry([]) raised IndexError - #518096: larry.fromdict failed due to missing import - #518106: la.larry.fromdict({}) failed - #518114: fromlist([]) and fromtuples([]) failed - #518135: keep_label crashed when there was nothing to keep - #518210: sum, std, var returned NaN for empty larrys; now return 0.0 - #518215: unflatten crashed on an empty larry - #518442: sum, std, var returned NaN for shapes that contain zero: (2, 0, 3) - #568175: larry.std(axis=-1) and var crashed on negative axis input - #569622: Negative axis input gave wrong output for several larry methods From matthew.brett at gmail.com Tue Apr 27 20:09:50 2010 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 27 Apr 2010 17:09:50 -0700 Subject: [Numpy-discussion] Recommended way to add Cython extension using numpy.distutils? Message-ID: Hi, We (neuroimaging.scipy.org) are using numpy.distutils, and we have .pyx files that we build with Cython. I wanted to add these in our current setup.py scripts, with something like: def configuration(parent_package='',top_path=None): from numpy.distutils.misc_util import Configuration config = Configuration('statistics', parent_package, top_path) config.add_extension('intvol', ['intvol.pyx'], include_dirs = [np.get_include()]) return config but of course numpy only knows about Pyrex, and returns: error: Pyrex required for compiling 'nipy/algorithms/statistics/intvol.pyx' but notavailable Is there a recommended way to plumb Cython into the numpy build machinery? Should I try and patch numpy distutils to use Cython if present? Best, Matthew From josef.pktd at gmail.com Tue Apr 27 23:37:45 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 27 Apr 2010 23:37:45 -0400 Subject: [Numpy-discussion] passing non-flat array to interpolator In-Reply-To: References: Message-ID: On Mon, Apr 26, 2010 at 12:04 PM, Thomas wrote: > > > I have some problem with interpolators in Scipy > does anyone knows if there is a way to pass a non-flat > array variables to Rbf, or other Scipy interpolator > eg. for my case of 17 x 1 problems of 500 data size > > x1.shape = (500,) > x2.shape = (500,) > ... > X17.shape =(500,) > b = Rbf(x1,x2,x3,...,x17,y) > > i would rather create a non-flat variables > x.shape =(500,17) and pass it to Rbf, or even for all the > interpolator in Scipy as > ?bf = Rbf(X, Y) > How can i do this ? ?Thank you for your time. > Thomas Rbf(*np.c_[X,Y].T) or Rbf(*(list(X.T)+[Y])) I think the second version does not make a copy of the data when building the list. It would be easier if the xs and y were reversed in the signature, Rbf(y, *X.T) Josef > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From david at silveregg.co.jp Wed Apr 28 01:56:32 2010 From: david at silveregg.co.jp (David) Date: Wed, 28 Apr 2010 14:56:32 +0900 Subject: [Numpy-discussion] Incomplete uninstall of 1.4.0 superpack In-Reply-To: References: , Message-ID: <4BD7CE10.1080200@silveregg.co.jp> On 04/27/2010 01:08 AM, threexk threexk wrote: > David Cournapeau wrote: > > On Mon, Apr 26, 2010 at 2:42 AM, threexk threexk > wrote: > > > Hello, > > > > > > I recently uninstalled the NumPy 1.4.0 superpack for Python 2.6 on > Windows > > > 7, and afterward a dialog popped up that said 1 file or directory > could not > > > be removed. Does anyone have any idea which file/directory this is? The > > > dialog gave no indication. Is an uninstall log with details generated > > > anywhere? > > > > There should be one in C:\Python*, something like numpy-*-wininst.log > > Looks like that log gets deleted after uninstallation (as it probably > should), so I still could not figure out which file/directory was not > deleted. I found that \Python26\Lib\site-packages\numpy and many > files/directories under it have remained after uninstall. So, I tried > reinstalling 1.4.0 and uninstalling again. This time, the uninstaller > did not report not being able to remove files/directories, but it still > did not delete the aforementioned numpy directory. I believe this is a > bug with the uninstaller? Could you maybe post the log (before uninstalling) and list the remaining files ? Note though that we most likely won't be able to do much - we do not have much control over the generated installers, cheers, David From seb.haase at gmail.com Wed Apr 28 03:16:13 2010 From: seb.haase at gmail.com (Sebastian Haase) Date: Wed, 28 Apr 2010 09:16:13 +0200 Subject: [Numpy-discussion] ndimage.label - howto force SWIG to use int32 - even on 64bit Linux ? In-Reply-To: References: Message-ID: (2nd try to get this post into the mailing list archive...) Hi, I wanted to write some C code to accept labels as they come from ndimage.label. For some reason ndimage.label produces its output as an int32 array - even on my 64bit system . BTW, could this be considered a bug ? Now, if I use the typemaps of numpy.i I can choose between NPY_LONG and NPY_INT. But those are sometimes 32 sometimes 64 bit, depending on the system. Any ideas ... ? Thanks, Sebastian Haase From oliphant at enthought.com Wed Apr 28 10:44:26 2010 From: oliphant at enthought.com (Travis Oliphant) Date: Wed, 28 Apr 2010 09:44:26 -0500 Subject: [Numpy-discussion] floats as axis In-Reply-To: References: Message-ID: <13F436D5-744B-4A4F-B499-223BFEAAB805@enthought.com> On Apr 25, 2010, at 8:16 AM, josef.pktd at gmail.com wrote: > (some) numpy functions take floats as valid axis argument. Is this a > feature? > >>>> np.ones((2,3)).sum(1.2) > array([ 3., 3.]) >>>> np.ones((2,3)).sum(1.99) > array([ 3., 3.]) > >>>> np.mean((1.5,0.5)) > 1.0 >>>> np.mean(1.5,0.5) > 1.5 > > Keith pointed out that scipy.stats.nanmean has a different behavior I think we should make float inputs raise an error for NumPy 2.0 -Travis From oliphant at enthought.com Wed Apr 28 12:05:26 2010 From: oliphant at enthought.com (Travis Oliphant) Date: Wed, 28 Apr 2010 11:05:26 -0500 Subject: [Numpy-discussion] proposing a "beware of [as]matrix()" warning In-Reply-To: <4BD62D8D.8060401@cs.toronto.edu> References: <4BD62D8D.8060401@cs.toronto.edu> Message-ID: <55C0F57B-A246-4D16-B6EF-5C6F58CEEA37@enthought.com> On Apr 26, 2010, at 7:19 PM, David Warde-Farley wrote: > Trying to debug code written by an undergrad working for a colleague > of > mine who ported code over from MATLAB, I am seeing an ugly melange of > matrix objects and ndarrays that are interacting poorly with each > other > and various functions in SciPy/other libraries. In particular there > was > a custom minimizer function that contained a line "a * b", that was > receiving an Nx1 "matrix" and a N-length array and computing an outer > product. Hence the unexpected 6 GB of memory usage and weird > results... Overloading '*' and '**' while convenient does have consequences. It would be nice if we could have a few more infix operators in Python to allow separation of element-by-element calculations and "dot-product" calculations. A proposal was made to allow "calling a NumPy array" to infer dot product: a(b) is equivalent to dot(a,b) a(b)(c) would be equivalent to dot(dot(a,b),c) This seems rather reasonable. While I don't have any spare cycles to push it forward and we are already far along on the NumPy to 3.0, I had wondered if we couldn't use the leverage of Python core developers wanting NumPy to be ported to Python 3 to actually add a few more infix operators to the language. One of the problems of moving to Python 3.0 for many people is that there are not "new features" to outweigh the hassle of moving. Having a few more infix operators would be a huge incentive to the NumPy community to move to Python 3. Anybody willing to lead the charge with the Python developers? -Travis From lists at hilboll.de Wed Apr 28 12:52:29 2010 From: lists at hilboll.de (Andreas Hilboll) Date: Wed, 28 Apr 2010 18:52:29 +0200 (CEST) Subject: [Numpy-discussion] Speeding up loadtxt / savetxt In-Reply-To: References: <4BD15767.6070600@hilboll.de> Message-ID: Hi St?fan, > Do you specifically need to store text files? NumPy's binary storage > functions (numpy.load and save) are faster. Yes, I know. But the files I create must be readable by an application developed in-house at our institude, and that only supports a) ASCII files or b) some home-grown binary format, which I hate. > Also, an efficient reader for very simply formatted text is provided > by numpy.fromfile. Yes, I heard about it. But the files I have to read have comments in them, and I didn't find a way to exclude these easily. Time needed to read a 100M file is ~13 seconds, and to write ~5 seconds. Which is not too bad, but also still too much ... Thanks, Andreas From dagss at student.matnat.uio.no Wed Apr 28 12:08:03 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 28 Apr 2010 18:08:03 +0200 Subject: [Numpy-discussion] proposing a "beware of [as]matrix()" warning In-Reply-To: <4BD62D8D.8060401@cs.toronto.edu> References: <4BD62D8D.8060401@cs.toronto.edu> Message-ID: <4BD85D63.5080808@student.matnat.uio.no> David Warde-Farley wrote: > Trying to debug code written by an undergrad working for a colleague of > mine who ported code over from MATLAB, I am seeing an ugly melange of > matrix objects and ndarrays that are interacting poorly with each other > and various functions in SciPy/other libraries. In particular there was > a custom minimizer function that contained a line "a * b", that was > receiving an Nx1 "matrix" and a N-length array and computing an outer > product. Hence the unexpected 6 GB of memory usage and weird results... If this was in a library function of some sort, I think they should always call np.asarray on the input arguments. That converts matrices to normal arrays. It could have been Python lists-of-lists, other PEP 3118 objects -- in Python an object can be everything in general, and I think it is very proper for most reusable functions to either validate the type of their arguments or take some steps to convert. That said, I second that it would be good to deprecate the matrix class from NumPy. The problem for me is not the existance of a matrix class as such, but the fact that it subclasses np.ndarray and is so similar with it, breaking a lot of rules for OO programming in the process. (Example: I happen to have my own oomatrix.py which allows me to do P, L = (A * A.H).cholesky() y = L.solve_right(x) This works fine because the matrices don't support any NumPy operations, and so I don't confuse them. But it helps to have to habit to do np.asarray in reusable functions so that errors are caught early. I do this so that A above can be either sparse, dense, triangular, diagonal, etc. -- i.e. polymorphic linear algebra. On the other hand, they don't even support single-element lookups, although that's just because I've been to lazy to implement it. Iteration is out of the question, it's just not the level of abstraction I'd like a "matrix" to work at.) Dag Sverre > > We've had this discussion before and it seems that the matrix class > isn't going anywhere (I *really* wish it would at least be banished from > the top-level namespace), but it has its adherents for pedagogical > reasons. Could we at least consider putting a gigantic warning on all > the functions for creating matrix objects (matrix, mat, asmatrix, etc.) > that they may not behave quite so predictably in some situations and > should be avoided when writing nontrivial code? > > There are already such warnings scattered about on SciPy.org but the > situation is so bad, in my opinion (bad from a programming perspective > and bad from a new user perspective, asking "why doesn't this work? why > doesn't that work? why is this language/library/etc. so stupid, > inconsistent, etc.?") that the situation warrants steering people still > further away from the matrix object. > > I apologize for ranting, but it pains me when people give up on > Python/NumPy because they can't figure out inconsistencies that aren't > really there for a good reason. IMHO, of course. > > David > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion -- Dag Sverre From oliphant at enthought.com Wed Apr 28 12:15:41 2010 From: oliphant at enthought.com (Travis Oliphant) Date: Wed, 28 Apr 2010 11:15:41 -0500 Subject: [Numpy-discussion] What should be the value of nansum of nan's? In-Reply-To: References: Message-ID: On Apr 26, 2010, at 12:03 PM, Charles R Harris wrote: > > > On Mon, Apr 26, 2010 at 10:55 AM, Charles R Harris > wrote: > Hi All, > > We need to make a decision for ticket #1123 regarding what nansum > should return when all values are nan. At some earlier point it was > zero, but currently it is nan, in fact it is nan whatever the > operation is. That is consistent, simple and serves to mark the > array or axis as containing all nans. I would like to close the > ticket and am a bit inclined to go with the current behaviour > although there is an argument to be made for returning 0 for the > nansum case. Thoughts? > > > To add a bit of context, one could argue that the results should be > consistent with the equivalent operations on empty arrays and always > be non-nan. > > In [1]: nansum([]) > Out[1]: nan > > In [2]: sum([]) > Out[2]: 0.0 I favor nansum([]) returning 0.0 which implies returning 0.0 when all the elements are nan. -Travis -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Wed Apr 28 12:19:26 2010 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 28 Apr 2010 11:19:26 -0500 Subject: [Numpy-discussion] proposing a "beware of [as]matrix()" warning In-Reply-To: <55C0F57B-A246-4D16-B6EF-5C6F58CEEA37@enthought.com> References: <4BD62D8D.8060401@cs.toronto.edu> <55C0F57B-A246-4D16-B6EF-5C6F58CEEA37@enthought.com> Message-ID: On Wed, Apr 28, 2010 at 11:05, Travis Oliphant wrote: > > On Apr 26, 2010, at 7:19 PM, David Warde-Farley wrote: > >> Trying to debug code written by an undergrad working for a colleague >> of >> mine who ported code over from MATLAB, I am seeing an ugly melange of >> matrix objects and ndarrays that are interacting poorly with each >> other >> and various functions in SciPy/other libraries. In particular there >> was >> a custom minimizer function that contained a line "a * b", that was >> receiving an Nx1 "matrix" and a N-length array and computing an outer >> product. Hence the unexpected 6 GB of memory usage and weird >> results... > > Overloading '*' and '**' while convenient does have consequences. ? It > would be nice if we could have a few more infix operators in Python to > allow separation of ?element-by-element calculations and "dot-product" > calculations. > > A proposal was made to allow "calling a NumPy array" to infer dot > product: > > a(b) is equivalent to dot(a,b) > > a(b)(c) would be equivalent to dot(dot(a,b),c) > > This seems rather reasonable. > > > While I don't have any spare cycles to push it forward and we are > already far along on the NumPy to 3.0, I had wondered if we couldn't > use the leverage of Python core developers wanting NumPy to be ported > to Python 3 to actually add a few more infix operators to the language. > > One of the problems of moving to Python 3.0 for many people is that > there are not ?"new features" to outweigh the hassle of moving. > Having a few more infix operators would be a huge incentive to the > NumPy community to move to Python 3. > > Anybody willing to lead the charge with the Python developers? There is currently a moratorium on language changes. This will have to wait. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From warren.weckesser at enthought.com Wed Apr 28 12:22:16 2010 From: warren.weckesser at enthought.com (Warren Weckesser) Date: Wed, 28 Apr 2010 11:22:16 -0500 Subject: [Numpy-discussion] What should be the value of nansum of nan's? In-Reply-To: References: Message-ID: <4BD860B8.7050903@enthought.com> Travis Oliphant wrote: > > On Apr 26, 2010, at 12:03 PM, Charles R Harris wrote: > >> >> >> On Mon, Apr 26, 2010 at 10:55 AM, Charles R Harris >> > wrote: >> >> Hi All, >> >> We need to make a decision for ticket #1123 >> >> regarding what nansum should return when all values are nan. At >> some earlier point it was zero, but currently it is nan, in fact >> it is nan whatever the operation is. That is consistent, simple >> and serves to mark the array or axis as containing all nans. I >> would like to close the ticket and am a bit inclined to go with >> the current behaviour although there is an argument to be made >> for returning 0 for the nansum case. Thoughts? >> >> >> To add a bit of context, one could argue that the results should be >> consistent with the equivalent operations on empty arrays and always >> be non-nan. >> >> In [1]: nansum([]) >> Out[1]: nan >> >> In [2]: sum([]) >> Out[2]: 0.0 > > > I favor nansum([]) returning 0.0 which implies returning 0.0 when all > the elements are nan. > +1 > -Travis > > > > > > ------------------------------------------------------------------------ > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From kwgoodman at gmail.com Wed Apr 28 12:28:39 2010 From: kwgoodman at gmail.com (Keith Goodman) Date: Wed, 28 Apr 2010 09:28:39 -0700 Subject: [Numpy-discussion] What should be the value of nansum of nan's? In-Reply-To: References: Message-ID: On Mon, Apr 26, 2010 at 9:55 AM, Charles R Harris wrote: > Hi All, > > We need to make a decision for ticket #1123 regarding what nansum should > return when all values are nan. At some earlier point it was zero, but > currently it is nan, in fact it is nan whatever the operation is. That is > consistent, simple and serves to mark the array or axis as containing all > nans. I would like to close the ticket and am a bit inclined to go with the > current behaviour although there is an argument to be made for returning 0 > for the nansum case. Thoughts? I use nansum a lot because I treat NaNs as missing data. I think a lot of people use NaNs as missing data but few admit it. My packages have grown to depend on nansum([nan, nan]) returning NaN. I vote to keep the current behavior. Changing nansum([]) to return zero, however, has no impact on me. From amenity at enthought.com Wed Apr 28 12:28:51 2010 From: amenity at enthought.com (Amenity Applewhite) Date: Wed, 28 Apr 2010 11:28:51 -0500 Subject: [Numpy-discussion] SciPy 2010: Talks Announced References: Message-ID: <920397F1-6129-4054-B4FD-CF3EEA39B01E@enthought.com> Email not displaying correctly? View it in your browser. Hello, As summer approaches, plans for SciPy 2010 are coming into focus! Here are a few quick updates: General conference talks announced Thanks so much to everyone who submitted an abstract. We received a number of fantastic proposals and are indebted to St?fan, Ond?ej, and our awesome Program Committee for reviewing them so quickly. We're really excited to see these presentations, and definitely encourage you to take a look at what is lined up. Keep in mind that this schedule is tentative, and that we will be announcing papers for our Biomedical and Parallel Processing & Cloud Computing tracks next week. Stay tuned... Less than two weeks left for early registration Yikes, time really does fly! Remember, early registration prices only apply if you register before May 10th. Best, The SciPy 2010 Team @SciPy2010 on Twitter You are receiving this email because you have registered for the SciPy 2010 conference in Austin, TX. Unsubscribe amenity at enthought.com from this list | Forward to a friend | Update your profile Our mailing address is: Enthought, Inc. 515 Congress Ave. Austin, TX 78701 Add us to your address book Copyright (C) 2010 Enthought, Inc. All rights reserved. -------------- next part -------------- An HTML attachment was scrubbed... URL: From philip.cooper at openvest.com Wed Apr 28 12:33:16 2010 From: philip.cooper at openvest.com (Philip Cooper) Date: Wed, 28 Apr 2010 10:33:16 -0600 Subject: [Numpy-discussion] Bug in MaskedArray min/max functions References: (sfid-20100425_040210_386333_A1745B509DCFACC1) Message-ID: if a masked array is created and has no masked values it can have a mask of just "False" scalar false that is. This causes an error when getting the max or min on axis=1 I don't have a fix but do offer a workaround. If you reset the mask to the "expanded"mask all works ok (but it's a bug that pops up all over my code) Below "a" and "m1" work but "m5" only works on axis = 0. anyway you can look at: (see http://www.openvest.com/trac/wiki/MaskedArrayMinMax if the formatting doesn't survive the mail posting process) ################################# >>> import numpy as np >>> a = np.array([np.arange(5)]) >>> a array([[0, 1, 2, 3, 4]]) >>> m1 = np.ma.masked_values(a,1) >>> m5 = np.ma.masked_values(a,5) >>> m1 masked_array(data = [[0 -- 2 3 4]], mask = [[False True False False False]], fill_value = 1) >>> m5 masked_array(data = [[0 1 2 3 4]], mask = False, fill_value = 5) >>> a.min(axis=0) array([0, 1, 2, 3, 4]) >>> m5.min(axis=0) masked_array(data = [0 1 2 3 4], mask = False, fill_value = 999999) >>> m1.min(axis=0) masked_array(data = [0 -- 2 3 4], mask = [False True False False False], fill_value = 999999) >>> a.min(axis=1) array([0]) >>> m1.min(axis=1) masked_array(data = [0], mask = [False], fill_value = 999999) >>> m5.min(axis=1) Traceback (most recent call last): File "", line 1, in File "/Library/Python/2.6/site-packages/numpy/ma/core.py", line 5020, in min newmask = _mask.all(axis=axis) ValueError: axis(=1) out of bounds ### workaround >>> m5.mask = np.ma.getmaskarray(m5) >>> m1.min(axis=1) masked_array(data = [0], mask = [False], fill_value = 999999) -- Philip J. Cooper (CFA) -------------- next part -------------- An HTML attachment was scrubbed... URL: From Chris.Barker at noaa.gov Wed Apr 28 12:36:53 2010 From: Chris.Barker at noaa.gov (Chris Barker) Date: Wed, 28 Apr 2010 09:36:53 -0700 Subject: [Numpy-discussion] Speeding up loadtxt / savetxt In-Reply-To: References: <4BD15767.6070600@hilboll.de> Message-ID: <4BD86425.6020606@noaa.gov> Andreas Hilboll wrote: > Yes, I know. But the files I create must be readable by an application > developed in-house at our institude, and that only supports a) ASCII files > or b) some home-grown binary format, which I hate. > >> Also, an efficient reader for very simply formatted text is provided >> by numpy.fromfile. > > Yes, I heard about it. But the files I have to read have comments in them, > and I didn't find a way to exclude these easily. you can't do i with fromfile -- I think it would be Very useful to have a fromfile() like functionality with a few more features: comments lines and allowing non-whitespace delimiters while reading multiple lines. See my posts about this in the past. I did spend a non-trivial amount of time looking into how to add these features, and fix some bugs in the process -- again, see my posts in the past. It turns out that the fromfile code is some pretty ugly C--a result of supporting all numpy data types, and compatibility with tradition C functions--so it's a bit of a chore, at least for a lame C programmer like me. I'm still not sure what I'll do when I get some time to look at this again -- I may simply start from scratch with Cython. It would be great if someone wanted to take it on > Time needed to read a 100M file is ~13 seconds, and to write ~5 seconds. > Which is not too bad, but also still too much ... You might try running fromfile() on a file with no comments, and you could see from that how much speed gain is possible -- at some point, you're waiting on the disk anyway. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From charlesr.harris at gmail.com Wed Apr 28 12:52:00 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 28 Apr 2010 10:52:00 -0600 Subject: [Numpy-discussion] Recommended way to add Cython extension using numpy.distutils? In-Reply-To: References: Message-ID: On Tue, Apr 27, 2010 at 6:09 PM, Matthew Brett wrote: > Hi, > > We (neuroimaging.scipy.org) are using numpy.distutils, and we have > .pyx files that we build with Cython. > > I wanted to add these in our current setup.py scripts, with something like: > > def configuration(parent_package='',top_path=None): > from numpy.distutils.misc_util import Configuration > config = Configuration('statistics', parent_package, top_path) > config.add_extension('intvol', > ['intvol.pyx'], include_dirs = [np.get_include()]) > return config > > but of course numpy only knows about Pyrex, and returns: > > error: Pyrex required for compiling > 'nipy/algorithms/statistics/intvol.pyx' but notavailable > > Is there a recommended way to plumb Cython into the numpy build > machinery? Should I try and patch numpy distutils to use Cython if > present? > > Patching distutils might be the way to go. We use Cython for the random build now because Pyrex couldn't handle long strings in a way suitable for Windows. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Apr 28 12:58:24 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 28 Apr 2010 10:58:24 -0600 Subject: [Numpy-discussion] ndimage.label - howto force SWIG to use int32 - even on 64bit Linux ? In-Reply-To: References: Message-ID: On Tue, Apr 27, 2010 at 2:27 AM, Sebastian Haase wrote: > Hi, > I wanted to write some C code to accept labels as they come from > ndimage.label. > For some reason ndimage.label produces its output as an int32 array - > even on my 64bit system . > > BTW, could this be considered a bug ? > > Likely. > Now, if I use the typemaps of numpy.i I can choose between NPY_LONG and > NPY_INT. > But those are sometimes 32 sometimes 64 bit, depending on the system. > > Any ideas ... ? > npy_intp. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Apr 28 12:59:04 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 28 Apr 2010 10:59:04 -0600 Subject: [Numpy-discussion] floats as axis In-Reply-To: <13F436D5-744B-4A4F-B499-223BFEAAB805@enthought.com> References: <13F436D5-744B-4A4F-B499-223BFEAAB805@enthought.com> Message-ID: On Wed, Apr 28, 2010 at 8:44 AM, Travis Oliphant wrote: > > On Apr 25, 2010, at 8:16 AM, josef.pktd at gmail.com wrote: > > > (some) numpy functions take floats as valid axis argument. Is this a > > feature? > > > >>>> np.ones((2,3)).sum(1.2) > > array([ 3., 3.]) > >>>> np.ones((2,3)).sum(1.99) > > array([ 3., 3.]) > > > >>>> np.mean((1.5,0.5)) > > 1.0 > >>>> np.mean(1.5,0.5) > > 1.5 > > > > > > > Keith pointed out that scipy.stats.nanmean has a different behavior > > > I think we should make float inputs raise an error for NumPy 2.0 > > Agree... Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Apr 28 13:08:09 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 28 Apr 2010 11:08:09 -0600 Subject: [Numpy-discussion] proposing a "beware of [as]matrix()" warning In-Reply-To: <4BD85D63.5080808@student.matnat.uio.no> References: <4BD62D8D.8060401@cs.toronto.edu> <4BD85D63.5080808@student.matnat.uio.no> Message-ID: On Wed, Apr 28, 2010 at 10:08 AM, Dag Sverre Seljebotn < dagss at student.matnat.uio.no> wrote: > David Warde-Farley wrote: > > Trying to debug code written by an undergrad working for a colleague of > > mine who ported code over from MATLAB, I am seeing an ugly melange of > > matrix objects and ndarrays that are interacting poorly with each other > > and various functions in SciPy/other libraries. In particular there was > > a custom minimizer function that contained a line "a * b", that was > > receiving an Nx1 "matrix" and a N-length array and computing an outer > > product. Hence the unexpected 6 GB of memory usage and weird results... > > If this was in a library function of some sort, I think they should > always call np.asarray on the input arguments. That converts matrices to > normal arrays. > > It could have been Python lists-of-lists, other PEP 3118 objects -- in > Python an object can be everything in general, and I think it is very > proper for most reusable functions to either validate the type of their > arguments or take some steps to convert. > > That said, I second that it would be good to deprecate the matrix class > from NumPy. The problem for me is not the existance of a matrix class as > such, but the fact that it subclasses np.ndarray and is so similar with > it, breaking a lot of rules for OO programming in the process. > > Yeah. Masked arrays have similar problems. Pierre has done so much work to have masked versions of the various functions that it might as well be a standalone class. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Wed Apr 28 13:13:35 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Wed, 28 Apr 2010 11:13:35 -0600 Subject: [Numpy-discussion] proposing a "beware of [as]matrix()" warning In-Reply-To: <55C0F57B-A246-4D16-B6EF-5C6F58CEEA37@enthought.com> References: <4BD62D8D.8060401@cs.toronto.edu> <55C0F57B-A246-4D16-B6EF-5C6F58CEEA37@enthought.com> Message-ID: On Wed, Apr 28, 2010 at 10:05 AM, Travis Oliphant wrote: > > On Apr 26, 2010, at 7:19 PM, David Warde-Farley wrote: > > > Trying to debug code written by an undergrad working for a colleague > > of > > mine who ported code over from MATLAB, I am seeing an ugly melange of > > matrix objects and ndarrays that are interacting poorly with each > > other > > and various functions in SciPy/other libraries. In particular there > > was > > a custom minimizer function that contained a line "a * b", that was > > receiving an Nx1 "matrix" and a N-length array and computing an outer > > product. Hence the unexpected 6 GB of memory usage and weird > > results... > > Overloading '*' and '**' while convenient does have consequences. It > would be nice if we could have a few more infix operators in Python to > allow separation of element-by-element calculations and "dot-product" > calculations. > > A proposal was made to allow "calling a NumPy array" to infer dot > product: > > a(b) is equivalent to dot(a,b) > > a(b)(c) would be equivalent to dot(dot(a,b),c) > > This seems rather reasonable. > > I like this too. A similar proposal that recently showed up on the list was to add a dot method to ndarrays so that a(b)(c) would be written a.dot(b).dot(c). > While I don't have any spare cycles to push it forward and we are > already far along on the NumPy to 3.0, I had wondered if we couldn't > use the leverage of Python core developers wanting NumPy to be ported > to Python 3 to actually add a few more infix operators to the language. > > One of the problems of moving to Python 3.0 for many people is that > there are not "new features" to outweigh the hassle of moving. > Having a few more infix operators would be a huge incentive to the > NumPy community to move to Python 3. > > Anybody willing to lead the charge with the Python developers? > > Problem is that we couldn't decide on an appropriate operator. Adding a keyword that functioned like "and" would likely break all sorts of code, so it needs to be something that is not currently seen in the wild. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From threexk at hotmail.com Wed Apr 28 13:17:50 2010 From: threexk at hotmail.com (threexk threexk) Date: Wed, 28 Apr 2010 13:17:50 -0400 Subject: [Numpy-discussion] Incomplete uninstall of 1.4.0 superpack In-Reply-To: <4BD7CE10.1080200@silveregg.co.jp> References: , , , , <4BD7CE10.1080200@silveregg.co.jp> Message-ID: David wrote: > On 04/27/2010 01:08 AM, threexk threexk wrote: > > David Cournapeau wrote: > > > On Mon, Apr 26, 2010 at 2:42 AM, threexk threexk > > wrote: > > > > Hello, > > > > > > > > I recently uninstalled the NumPy 1.4.0 superpack for Python 2.6 on > > Windows > > > > 7, and afterward a dialog popped up that said 1 file or directory > > could not > > > > be removed. Does anyone have any idea which file/directory this is? The > > > > dialog gave no indication. Is an uninstall log with details generated > > > > anywhere? > > > > > > There should be one in C:\Python*, something like numpy-*-wininst.log > > > > Looks like that log gets deleted after uninstallation (as it probably > > should), so I still could not figure out which file/directory was not > > deleted. I found that \Python26\Lib\site-packages\numpy and many > > files/directories under it have remained after uninstall. So, I tried > > reinstalling 1.4.0 and uninstalling again. This time, the uninstaller > > did not report not being able to remove files/directories, but it still > > did not delete the aforementioned numpy directory. I believe this is a > > bug with the uninstaller? > > Could you maybe post the log (before uninstalling) and list the > remaining files ? I attached it to the following issue. I also attached a listing of the files that are remaining after uninstall. http://projects.scipy.org/numpy/ticket/1466 _________________________________________________________________ The New Busy is not the old busy. Search, chat and e-mail from your inbox. http://www.windowslive.com/campaign/thenewbusy?ocid=PID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_3 -------------- next part -------------- An HTML attachment was scrubbed... URL: From bioinformed at gmail.com Wed Apr 28 13:20:00 2010 From: bioinformed at gmail.com (Kevin Jacobs ) Date: Wed, 28 Apr 2010 13:20:00 -0400 Subject: [Numpy-discussion] Recommended way to add Cython extension using numpy.distutils? In-Reply-To: References: Message-ID: On Tue, Apr 27, 2010 at 8:09 PM, Matthew Brett wrote: > Hi, > > We (neuroimaging.scipy.org) are using numpy.distutils, and we have > .pyx files that we build with Cython. > > I wanted to add these in our current setup.py scripts, with something like: > > def configuration(parent_package='',top_path=None): > from numpy.distutils.misc_util import Configuration > config = Configuration('statistics', parent_package, top_path) > config.add_extension('intvol', > ['intvol.pyx'], include_dirs = [np.get_include()]) > return config > > but of course numpy only knows about Pyrex, and returns: > > error: Pyrex required for compiling > 'nipy/algorithms/statistics/intvol.pyx' but notavailable > > Is there a recommended way to plumb Cython into the numpy build > machinery? Should I try and patch numpy distutils to use Cython if > present? > > Here is the monkey-patch I'm using in my project: def evil_numpy_monkey_patch(): from numpy.distutils.command import build_src import Cython import Cython.Compiler.Main build_src.Pyrex = Cython build_src.have_pyrex = True -------------- next part -------------- An HTML attachment was scrubbed... URL: From Nikolaus at rath.org Wed Apr 28 12:50:00 2010 From: Nikolaus at rath.org (Nikolaus Rath) Date: Wed, 28 Apr 2010 12:50:00 -0400 Subject: [Numpy-discussion] proposing a "beware of [as]matrix()" warning References: <4BD62D8D.8060401@cs.toronto.edu> <55C0F57B-A246-4D16-B6EF-5C6F58CEEA37@enthought.com> Message-ID: <87pr1jh687.fsf@inspiron.ap.columbia.edu> Robert Kern writes: >> Overloading '*' and '**' while convenient does have consequences. ? It >> would be nice if we could have a few more infix operators in Python to >> allow separation of ?element-by-element calculations and "dot-product" >> calculations. http://www.python.org/dev/peps/pep-0225/ was considered and rejected. But that was in 2000... >> While I don't have any spare cycles to push it forward and we are >> already far along on the NumPy to 3.0, I had wondered if we couldn't >> use the leverage of Python core developers wanting NumPy to be ported >> to Python 3 to actually add a few more infix operators to the language. I don't think that stands a chance: http://www.python.org/dev/peps/pep-3003/ Best, -Nikolaus -- ?Time flies like an arrow, fruit flies like a Banana.? PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C From dwf at cs.toronto.edu Wed Apr 28 13:30:00 2010 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Wed, 28 Apr 2010 13:30:00 -0400 Subject: [Numpy-discussion] proposing a "beware of [as]matrix()" warning In-Reply-To: <55C0F57B-A246-4D16-B6EF-5C6F58CEEA37@enthought.com> References: <4BD62D8D.8060401@cs.toronto.edu> <55C0F57B-A246-4D16-B6EF-5C6F58CEEA37@enthought.com> Message-ID: On 2010-04-28, at 12:05 PM, Travis Oliphant wrote: > a(b) is equivalent to dot(a,b) > > a(b)(c) would be equivalent to dot(dot(a,b),c) > > This seems rather reasonable. Indeed, and it leads to a rather pleasant way of permuting syntax to change the order of operations, i.e. a(b(c)) vs. a(b)(c). David From josef.pktd at gmail.com Wed Apr 28 13:43:21 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Wed, 28 Apr 2010 13:43:21 -0400 Subject: [Numpy-discussion] proposing a "beware of [as]matrix()" warning In-Reply-To: References: <4BD62D8D.8060401@cs.toronto.edu> <55C0F57B-A246-4D16-B6EF-5C6F58CEEA37@enthought.com> Message-ID: On Wed, Apr 28, 2010 at 1:30 PM, David Warde-Farley wrote: > > On 2010-04-28, at 12:05 PM, Travis Oliphant wrote: > >> a(b) is equivalent to dot(a,b) >> >> a(b)(c) would be equivalent to dot(dot(a,b),c) >> >> This seems rather reasonable. > > Indeed, and it leads to a rather pleasant way of permuting syntax to change the order of operations, i.e. a(b(c)) vs. a(b)(c). I like the explicit dot method much better, __call__ (parentheses) can mean anything, and reading the code will be more difficult. (especially when switching from matlab) Josef > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From dwf at cs.toronto.edu Wed Apr 28 13:53:51 2010 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Wed, 28 Apr 2010 13:53:51 -0400 Subject: [Numpy-discussion] proposing a "beware of [as]matrix()" warning In-Reply-To: <87pr1jh687.fsf@inspiron.ap.columbia.edu> References: <4BD62D8D.8060401@cs.toronto.edu> <55C0F57B-A246-4D16-B6EF-5C6F58CEEA37@enthought.com> <87pr1jh687.fsf@inspiron.ap.columbia.edu> Message-ID: <4BD8762F.1030901@cs.toronto.edu> Nikolaus Rath wrote: > Robert Kern writes: > >>> Overloading '*' and '**' while convenient does have consequences. It >>> would be nice if we could have a few more infix operators in Python to >>> allow separation of element-by-element calculations and "dot-product" >>> calculations. >>> > > http://www.python.org/dev/peps/pep-0225/ was considered and rejected. > But that was in 2000... > See also: http://www.python.org/dev/peps/pep-0211/ http://mail.python.org/pipermail/python-dev/2008-November/083493.html (The .html link Fernando gives is dead but the reST source is below in that message) From tjhnson at gmail.com Wed Apr 28 13:56:37 2010 From: tjhnson at gmail.com (T J) Date: Wed, 28 Apr 2010 10:56:37 -0700 Subject: [Numpy-discussion] What should be the value of nansum of nan's? In-Reply-To: References: Message-ID: On Mon, Apr 26, 2010 at 10:03 AM, Charles R Harris wrote: > > > On Mon, Apr 26, 2010 at 10:55 AM, Charles R Harris > wrote: >> >> Hi All, >> >> We need to make a decision for ticket #1123 regarding what nansum should >> return when all values are nan. At some earlier point it was zero, but >> currently it is nan, in fact it is nan whatever the operation is. That is >> consistent, simple and serves to mark the array or axis as containing all >> nans. I would like to close the ticket and am a bit inclined to go with the >> current behaviour although there is an argument to be made for returning 0 >> for the nansum case. Thoughts? >> > > To add a bit of context, one could argue that the results should be > consistent with the equivalent operations on empty arrays and always be > non-nan. > > In [1]: nansum([]) > Out[1]: nan > > In [2]: sum([]) > Out[2]: 0.0 > This seems like an obvious one to me. What is the spirit of nansum? """ Return the sum of array elements over a given axis treating Not a Numbers (NaNs) as zero. """ Okay. So NaNs in an array are treated as zeros and the sum is performed as one normally would perform it starting with an initial sum of zero. So if all values are NaN, then we add nothing to our original sum and still return 0. I'm not sure I understand the argument that it should return NaN. It is counter to the *purpose* of nansum. Also, if one wants to determine if all values in an array are NaN, isn't there another way? Let's keep (or make) those distinct operations, as they are definitely distinct concepts. From aisaac at american.edu Wed Apr 28 14:12:07 2010 From: aisaac at american.edu (Alan G Isaac) Date: Wed, 28 Apr 2010 14:12:07 -0400 Subject: [Numpy-discussion] proposing a "beware of [as]matrix()" warning In-Reply-To: <55C0F57B-A246-4D16-B6EF-5C6F58CEEA37@enthought.com> References: <4BD62D8D.8060401@cs.toronto.edu> <55C0F57B-A246-4D16-B6EF-5C6F58CEEA37@enthought.com> Message-ID: <4BD87A77.9030108@american.edu> On 4/28/2010 12:05 PM, Travis Oliphant wrote: > A proposal was made to allow "calling a NumPy array" to infer dot > product: > > a(b) is equivalent to dot(a,b) > > a(b)(c) would be equivalent to dot(dot(a,b),c) Here is a related ticket that proposes a more explicit alternative: adding a ``dot`` method to ndarray. http://projects.scipy.org/numpy/ticket/1456 fwiw, Alan From matthew.brett at gmail.com Wed Apr 28 15:01:00 2010 From: matthew.brett at gmail.com (Matthew Brett) Date: Wed, 28 Apr 2010 12:01:00 -0700 Subject: [Numpy-discussion] Recommended way to add Cython extension using numpy.distutils? In-Reply-To: References: Message-ID: Hi, Thanks a lot for the suggestion - I appreciate it. >> Is there a recommended way to plumb Cython into the numpy build >> machinery? ?Should I try and patch numpy distutils to use Cython if >> present? >> > > Here is the monkey-patch I'm using in my project: > def evil_numpy_monkey_patch(): > ??from ? numpy.distutils.command import build_src > ??import Cython > ??import Cython.Compiler.Main > ??build_src.Pyrex = Cython > ??build_src.have_pyrex = True I think this patch does not work for current numpy trunk; I've put a minimal test case here: http://github.com/matthew-brett/du-cy-numpy If you run the setup.py there (python setup.py build) then all works fine for - say - numpy 1.1. For current trunk you get an error ending in: File "/Users/mb312/usr/local/lib/python2.6/site-packages/numpy/distutils/command/build_src.py", line 466, in generate_a_pyrex_source if self.inplace or not have_pyrex(): TypeError: 'bool' object is not callable which is easily fixable of course ('build_src.have_pyrex = lambda : True') - leading to: File "/Users/mb312/usr/local/lib/python2.6/site-packages/numpy/distutils/command/build_src.py", line 474, in generate_a_pyrex_source import Pyrex.Compiler.Main ImportError: No module named Pyrex.Compiler.Main I'm afraid I did a rather crude monkey-patch to replace the 'generate_a_pyrex_source' function. It seems to work for numpy 1.1 and current trunk. The patching process is here: http://github.com/matthew-brett/du-cy-numpy/blob/master/matthew_monkey.py Best, Matthew From oliphant at enthought.com Wed Apr 28 16:56:18 2010 From: oliphant at enthought.com (Travis Oliphant) Date: Wed, 28 Apr 2010 15:56:18 -0500 Subject: [Numpy-discussion] proposing a "beware of [as]matrix()" warning In-Reply-To: <87pr1jh687.fsf@inspiron.ap.columbia.edu> References: <4BD62D8D.8060401@cs.toronto.edu> <55C0F57B-A246-4D16-B6EF-5C6F58CEEA37@enthought.com> <87pr1jh687.fsf@inspiron.ap.columbia.edu> Message-ID: <8DA1E741-C711-4EF8-AF50-4F818FF93512@enthought.com> On Apr 28, 2010, at 11:50 AM, Nikolaus Rath wrote: > Robert Kern writes: >>> Overloading '*' and '**' while convenient does have >>> consequences. It >>> would be nice if we could have a few more infix operators in >>> Python to >>> allow separation of element-by-element calculations and "dot- >>> product" >>> calculations. > > http://www.python.org/dev/peps/pep-0225/ was considered and rejected. > But that was in 2000... > >>> While I don't have any spare cycles to push it forward and we are >>> already far along on the NumPy to 3.0, I had wondered if we couldn't >>> use the leverage of Python core developers wanting NumPy to be >>> ported >>> to Python 3 to actually add a few more infix operators to the >>> language. > > I don't think that stands a chance: http://www.python.org/dev/peps/pep-3003/ Frankly, I still think we should move forward. It will take us as long as the moratorium is in effect to figure out what operators we want anyway and we can do things like put attributes on arrays in the meantime to implement the infix operators we think we need. It's too bad we don't have more of a voice with the Python core team. This is our fault of course (we don't have people with spare cycles to spend the time interfacing), but it's still too bad. -Travis From dwf at cs.toronto.edu Wed Apr 28 14:15:20 2010 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Wed, 28 Apr 2010 14:15:20 -0400 Subject: [Numpy-discussion] proposing a "beware of [as]matrix()" warning In-Reply-To: References: <4BD62D8D.8060401@cs.toronto.edu> <55C0F57B-A246-4D16-B6EF-5C6F58CEEA37@enthought.com> Message-ID: <4BD87B38.3070808@cs.toronto.edu> josef.pktd at gmail.com wrote: > On Wed, Apr 28, 2010 at 1:30 PM, David Warde-Farley wrote: > >> On 2010-04-28, at 12:05 PM, Travis Oliphant wrote: >> >> >>> a(b) is equivalent to dot(a,b) >>> >>> a(b)(c) would be equivalent to dot(dot(a,b),c) >>> >>> This seems rather reasonable. >>> >> Indeed, and it leads to a rather pleasant way of permuting syntax to change the order of operations, i.e. a(b(c)) vs. a(b)(c). >> > > I like the explicit dot method much better, __call__ (parentheses) can > mean anything, and reading the code will be more difficult. > (especially when switching from matlab) > Yes, you have a point, though I do think it would read easier as a method, a.dot(b.dot(c)) or a.dot(b).dot(c) rather than as a prefix/function. (Down the line, another concern is the fact that dot() always incurs a memory allocation for the result, which matters sometimes when you have a huge amount of data and limited RAM. As I currently understand the situation, it shouldn't be hard to add an optional 'out' argument or a gaxpy primitive that maps directly onto the BLAS call; I just haven't had time yet to investigate it further.) David David From aisaac at american.edu Wed Apr 28 14:30:11 2010 From: aisaac at american.edu (Alan G Isaac) Date: Wed, 28 Apr 2010 14:30:11 -0400 Subject: [Numpy-discussion] proposing a "beware of [as]matrix()" warning In-Reply-To: <4BD85D63.5080808@student.matnat.uio.no> References: <4BD62D8D.8060401@cs.toronto.edu> <4BD85D63.5080808@student.matnat.uio.no> Message-ID: <4BD87EB3.6010606@american.edu> On 4/28/2010 12:08 PM, Dag Sverre Seljebotn wrote: > it would be good to deprecate the matrix class > from NumPy Please let us not have this discussion all over again. The matrix class is very useful for teaching. In economics for example, the use of matrix algebra is widespread, while algebra with arrays that are not matrices is very rare. I can (and do) use NumPy matrices even in undergraduate courses. If you do not like them, do not use them. If you want `matrix` replaced with a better matrix object, offer a replacement for community consideration. Thank you, Alan Isaac PS There is one change I would not mind: let A * M be undefined if A is an ndarray and M is a NumPy matrix. From tom.denniston at alum.dartmouth.org Wed Apr 28 14:44:13 2010 From: tom.denniston at alum.dartmouth.org (Tom Denniston) Date: Wed, 28 Apr 2010 13:44:13 -0500 Subject: [Numpy-discussion] Fwd: Advice for grouping recarrays In-Reply-To: References: <44BCCD22.7060607@stevesimmons.com> Message-ID: Someone inquired about this one today and I wanted to clarify there is now a better way to do this that I didn't know about when I posted the original: >>> ind = numpy.array([0,0,0,0,1,1,1,2,2,2,]) >>> data = numpy.arange(10) >>> borders = numpy.arange(len(ind)).compress(numpy.hstack([[1], ind[1:]!=ind[:-1]])) >>> numpy.add.reduceat(data, borders) array([ 6, 15, 24]) On Tue, Jul 18, 2006 at 8:49 AM, Tom Denniston wrote: > I suggest > > lexsort > itertools.groupby of the indices > take > > I think it would be really great if numpy had the first two as a > function or something like that. ?It is really useful to be able to > take an array and bucket it and apply further numpy operations like > accumulation functions. > > On 7/18/06, Stephen Simmons wrote: >> >> Hi, >> >> Does anyone have any suggestions for summarising data in numpy? >> >> The quick description is that I want to do something like the SQL >> statement: >> ? SELECT sum(field1), sum(field2) FROM ?table GROUP BY field3; >> >> The more accurate description is that my data is stored in PyTables HDF >> format, with 24 monthly files, each with 4m records describing how >> customers performed that month. Each record looks something like this: >> ('200604', 651404500000L, '800', 'K', 12L, 162.0, 2000.0, 0.054581, 0.0, >> 0.0, 0.0, 0.0, 0.0, 0.0, 2.0, 0.0, 2.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, >> 8.80, 0.86, 7.80 17.46, 0.0, 70.0, 0.0, 70.0, -142.93, 0.0, 2000.0, >> 2063.93, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, -9.71, 7.75, >> 87.46, 77.75, -3.45, 0.22, -0.45, -0.57, 73.95) >> The first 5 fields are status fields (month_string, account_number, >> product_code, account_status, months_since_customer_joined). The >> remaining 48 fields represent different aspects of the customer's >> performance during that month. I read 100,000 of these records at a time >> and turn them into a numpy recarray with: >> ? dat = hdf_table.read(start=pos, stop=pos+block_size) >> ? dat = numpy.asarray(dat._flatArray, dtype=dat.array_descr) >> >> I'd like to reduce these 96m records x 53 fields down to monthly >> averages for each tuple (month_string, months_since_customer_joined) >> which in the case above is ('200604', 12L). This will let me compare the >> performance of newly acquired customers at the same point in their >> lifecycle as customers acquired 1 or 2 years ago. >> >> The end result should be a dataset something like >> ? res[month_index, months_since_customer_joined] >> ? = array([ num_records, sum_field_5, sum_field_6, sum_field_7, ... >> sum_field_52 ]) >> with a shape of (24, 24, 49). >> >> I've played around with lexsort(), take(), sum(), etc, but get very >> confused and end up feeling that I'm making things more complicated than >> they need to be. So any advice from numpy veterans on how best to >> proceed would be very welcome! >> >> Cheers >> >> Stephen >> >> ------------------------------------------------------------------------- >> Take Surveys. Earn Cash. Influence the Future of IT >> Join SourceForge.net's Techsay panel and you'll get the chance to share >> your >> opinions on IT & business topics through brief surveys -- and earn cash >> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV >> _______________________________________________ >> Numpy-discussion mailing list >> Numpy-discussion at lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/numpy-discussion >> > From oliphant at enthought.com Wed Apr 28 16:50:52 2010 From: oliphant at enthought.com (Travis Oliphant) Date: Wed, 28 Apr 2010 15:50:52 -0500 Subject: [Numpy-discussion] proposing a "beware of [as]matrix()" warning In-Reply-To: References: <4BD62D8D.8060401@cs.toronto.edu> <55C0F57B-A246-4D16-B6EF-5C6F58CEEA37@enthought.com> Message-ID: <324B4999-1091-49F9-885B-CDD40DF83D16@enthought.com> On Apr 28, 2010, at 11:19 AM, Robert Kern wrote: > On Wed, Apr 28, 2010 at 11:05, Travis Oliphant > wrote: >> >> On Apr 26, 2010, at 7:19 PM, David Warde-Farley wrote: >> >>> Trying to debug code written by an undergrad working for a colleague >>> of >>> mine who ported code over from MATLAB, I am seeing an ugly melange >>> of >>> matrix objects and ndarrays that are interacting poorly with each >>> other >>> and various functions in SciPy/other libraries. In particular there >>> was >>> a custom minimizer function that contained a line "a * b", that was >>> receiving an Nx1 "matrix" and a N-length array and computing an >>> outer >>> product. Hence the unexpected 6 GB of memory usage and weird >>> results... >> >> Overloading '*' and '**' while convenient does have consequences. >> It >> would be nice if we could have a few more infix operators in Python >> to >> allow separation of element-by-element calculations and "dot- >> product" >> calculations. >> >> A proposal was made to allow "calling a NumPy array" to infer dot >> product: >> >> a(b) is equivalent to dot(a,b) >> >> a(b)(c) would be equivalent to dot(dot(a,b),c) >> >> This seems rather reasonable. >> >> >> While I don't have any spare cycles to push it forward and we are >> already far along on the NumPy to 3.0, I had wondered if we couldn't >> use the leverage of Python core developers wanting NumPy to be ported >> to Python 3 to actually add a few more infix operators to the >> language. >> >> One of the problems of moving to Python 3.0 for many people is that >> there are not "new features" to outweigh the hassle of moving. >> Having a few more infix operators would be a huge incentive to the >> NumPy community to move to Python 3. >> >> Anybody willing to lead the charge with the Python developers? > > There is currently a moratorium on language changes. This will have > to wait. Exceptions can always be made for the right reasons. I don't think this particular question has received sufficient audience with Python core developers. The reason they want the moratorium is for stability, but they also want Python 3k to be adopted. -Travis From peridot.faceted at gmail.com Wed Apr 28 17:09:52 2010 From: peridot.faceted at gmail.com (Anne Archibald) Date: Wed, 28 Apr 2010 17:09:52 -0400 Subject: [Numpy-discussion] proposing a "beware of [as]matrix()" warning In-Reply-To: <4BD87EB3.6010606@american.edu> References: <4BD62D8D.8060401@cs.toronto.edu> <4BD85D63.5080808@student.matnat.uio.no> <4BD87EB3.6010606@american.edu> Message-ID: On 28 April 2010 14:30, Alan G Isaac wrote: > On 4/28/2010 12:08 PM, Dag Sverre Seljebotn wrote: >> it would be good to deprecate the matrix class >> from NumPy > > Please let us not have this discussion all over again. I think you may be too late on this, but it's worth a try. > The matrix class is very useful for teaching. > In economics for example, the use of matrix algebra > is widespread, while algebra with arrays that are > not matrices is very rare. ?I can (and do) use NumPy > matrices even in undergraduate courses. > > If you do not like them, do not use them. This is the problem: lots of people start using numpy and think "hmm, I want to store two-dimensional data so I'll use a matrix", and have no idea that "matrix" means anything different from "two-dimensional array". It was this that inspired David's original post, and it's this that we're trying to find a solution for. > If you want `matrix` replaced with a better matrix > object, offer a replacement for community consideration. > > Thank you, > Alan Isaac > > PS There is one change I would not mind: let > A * M be undefined if A is an ndarray and > M is a NumPy matrix. I can definitely vote for this, in the interest of catching as many inadvertent matrix users as possible. Anne From robert.kern at gmail.com Wed Apr 28 17:26:06 2010 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 28 Apr 2010 16:26:06 -0500 Subject: [Numpy-discussion] proposing a "beware of [as]matrix()" warning In-Reply-To: <324B4999-1091-49F9-885B-CDD40DF83D16@enthought.com> References: <4BD62D8D.8060401@cs.toronto.edu> <55C0F57B-A246-4D16-B6EF-5C6F58CEEA37@enthought.com> <324B4999-1091-49F9-885B-CDD40DF83D16@enthought.com> Message-ID: On Wed, Apr 28, 2010 at 15:50, Travis Oliphant wrote: > > On Apr 28, 2010, at 11:19 AM, Robert Kern wrote: > >> On Wed, Apr 28, 2010 at 11:05, Travis Oliphant >> wrote: >>> >>> On Apr 26, 2010, at 7:19 PM, David Warde-Farley wrote: >>> >>>> Trying to debug code written by an undergrad working for a colleague >>>> of >>>> mine who ported code over from MATLAB, I am seeing an ugly melange >>>> of >>>> matrix objects and ndarrays that are interacting poorly with each >>>> other >>>> and various functions in SciPy/other libraries. In particular there >>>> was >>>> a custom minimizer function that contained a line "a * b", that was >>>> receiving an Nx1 "matrix" and a N-length array and computing an >>>> outer >>>> product. Hence the unexpected 6 GB of memory usage and weird >>>> results... >>> >>> Overloading '*' and '**' while convenient does have consequences. >>> It >>> would be nice if we could have a few more infix operators in Python >>> to >>> allow separation of ?element-by-element calculations and "dot- >>> product" >>> calculations. >>> >>> A proposal was made to allow "calling a NumPy array" to infer dot >>> product: >>> >>> a(b) is equivalent to dot(a,b) >>> >>> a(b)(c) would be equivalent to dot(dot(a,b),c) >>> >>> This seems rather reasonable. >>> >>> >>> While I don't have any spare cycles to push it forward and we are >>> already far along on the NumPy to 3.0, I had wondered if we couldn't >>> use the leverage of Python core developers wanting NumPy to be ported >>> to Python 3 to actually add a few more infix operators to the >>> language. >>> >>> One of the problems of moving to Python 3.0 for many people is that >>> there are not ?"new features" to outweigh the hassle of moving. >>> Having a few more infix operators would be a huge incentive to the >>> NumPy community to move to Python 3. >>> >>> Anybody willing to lead the charge with the Python developers? >> >> There is currently a moratorium on language changes. This will have >> to wait. > > Exceptions can always be made for the right reasons. ? ?I don't think > this particular question has received sufficient audience with Python > core developers. It received plenty of audience on python-dev in 2008. But no one from our community cared enough to actually implement it. http://fperez.org/py4science/numpy-pep225/numpy-pep225.html >?The reason they want the moratorium is for > stability, but they also want Python 3k to be adopted. This is not something that will justify an exception. Things like "oh crap, this old feature has a lurking flaw that we've never noticed before and needs a language change to fix" are possible exceptions to the moratorium, not something like this. PEP 3003 quite clearly lays out the possible exceptions: """ Case-by-Case Exemptions New methods on built-ins The case for adding a method to a built-in object can be made. Incorrect language semantics If the language semantics turn out to be ambiguous or improperly implemented based on the intention of the original design then the semantics may change. Language semantics that are difficult to implement Because other VMs have not begun implementing Python 3.x semantics there is a possibility that certain semantics are too difficult to replicate. In those cases they can be changed to ease adoption of Python 3.x by the other VMs. """ This feature falls into none of these categories. It does fall into this one: """ Cannot Change ... Language syntax The grammar file essentially becomes immutable apart from ambiguity fixes. """ Guido is taking a hard line on this. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From jsseabold at gmail.com Wed Apr 28 17:28:51 2010 From: jsseabold at gmail.com (Skipper Seabold) Date: Wed, 28 Apr 2010 17:28:51 -0400 Subject: [Numpy-discussion] proposing a "beware of [as]matrix()" warning In-Reply-To: <4BD87A77.9030108@american.edu> References: <4BD62D8D.8060401@cs.toronto.edu> <55C0F57B-A246-4D16-B6EF-5C6F58CEEA37@enthought.com> <4BD87A77.9030108@american.edu> Message-ID: On Wed, Apr 28, 2010 at 2:12 PM, Alan G Isaac wrote: > On 4/28/2010 12:05 PM, Travis Oliphant wrote: >> A proposal was made to allow "calling a NumPy array" to infer dot >> product: >> >> a(b) is equivalent to dot(a,b) >> >> a(b)(c) would be equivalent to dot(dot(a,b),c) > > > Here is a related ticket that proposes a more > explicit alternative: adding a ``dot`` method > to ndarray. > http://projects.scipy.org/numpy/ticket/1456 > FWIW, I have borrowed a convenience function chain_dot originally from pandas that works for me as a stop gap for more readable code. def chain_dot(*arrs): """ Returns the dot product of the given matrices. Parameters ---------- arrs: argument list of ndarrays Returns ------- Dot product of all arguments. Example ------- >>> import numpy as np >>> from scikits.statsmodels.tools import chain_dot >>> A = np.arange(1,13).reshape(3,4) >>> B = np.arange(3,15).reshape(4,3) >>> C = np.arange(5,8).reshape(3,1) >>> chain_dot(A,B,C) array([[1820], [4300], [6780]]) """ return reduce(lambda x, y: np.dot(y, x), arrs[::-1]) Skipper From dwf at cs.toronto.edu Wed Apr 28 17:46:43 2010 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Wed, 28 Apr 2010 17:46:43 -0400 Subject: [Numpy-discussion] proposing a "beware of [as]matrix()" warning In-Reply-To: <4BD87EB3.6010606@american.edu> References: <4BD62D8D.8060401@cs.toronto.edu> <4BD85D63.5080808@student.matnat.uio.no> <4BD87EB3.6010606@american.edu> Message-ID: <6A5E3F22-85FF-48ED-ADF6-B02B260F294D@cs.toronto.edu> On 2010-04-28, at 2:30 PM, Alan G Isaac wrote: > Please let us not have this discussion all over again. Agreed. See my preface to this discussion. My main objection is that it's not easy to explain to a newcomer what the difference precisely is, how they interact, why two of them exist, how they are sort-of-compatible-but-not... > The matrix class is very useful for teaching. > In economics for example, the use of matrix algebra > is widespread, while algebra with arrays that are > not matrices is very rare. I can (and do) use NumPy > matrices even in undergraduate courses. Would it be acceptable to retain the matrix class but not have it imported in the default namespace, and have to import e.g. numpy.matlib to get at them? > If you do not like them, do not use them. The problem isn't really with seasoned users of NumPy not liking them, but rather new users being confused by the presence of (what seems to be) two primitives, array and matrix. Two things tend to happen: a) Example code that expects arrays instead receives matrices. If these aren't "cast" with asarray(), mayhem ensues at the first sight of *. b) Users of class matrix use a proper function correctly coerces input to ndarray, but returns an ndarray. Users are thus confused that, thinking of the function as a black box, putting matrices 'in' doesn't result in getting matrices 'out'. It doesn't take long to get the hang of if you really sit down and work it through, but it also "doesn't take long" to go back to MATLAB or whatever else. My interest is in having as few conceptual stumbling stones as possible. c) Complicating the situation further, people try to use functions e.g. from scipy.optimize which expect a 1d array by passing in column or row matrices. Even when coerced to array, these have the wrong rank and you get unexpected results (you could argue that we should instead use asarray(..).squeeze() on all incoming arguments, but this may not generalize well). > PS There is one change I would not mind: let > A * M be undefined if A is an ndarray and > M is a NumPy matrix. What about the other binary ops? I would say, matrix goes with matrix, array with array, never the two shall meet unless you explicitly coerce. The ability to mix the two in a single expression does more harm than good, IMHO. David From wfspotz at sandia.gov Wed Apr 28 17:51:52 2010 From: wfspotz at sandia.gov (Bill Spotz) Date: Wed, 28 Apr 2010 17:51:52 -0400 Subject: [Numpy-discussion] ndimage.label - howto force SWIG to use int32 - even on 64bit Linux ? In-Reply-To: References: Message-ID: Both types of typemaps are enabled, so you just need to do you %apply directives correctly: %apply (npy_intp* IN_ARRAY1, int DIM1) {(npy_intp* seq, int n)}; etc.... SWIG should be able to figure it out from there. On Apr 28, 2010, at 12:58 PM, Charles R Harris wrote: > On Tue, Apr 27, 2010 at 2:27 AM, Sebastian Haase > wrote: > Hi, > I wanted to write some C code to accept labels as they come from > ndimage.label. > For some reason ndimage.label produces its output as an int32 array - > even on my 64bit system . > > BTW, could this be considered a bug ? > > > Likely. > > Now, if I use the typemaps of numpy.i I can choose between NPY_LONG > and NPY_INT. > But those are sometimes 32 sometimes 64 bit, depending on the system. > > Any ideas ... ? > > npy_intp. > > Chuck ** Bill Spotz ** ** Sandia National Laboratories Voice: (505)845-0170 ** ** P.O. Box 5800 Fax: (505)284-0154 ** ** Albuquerque, NM 87185-0370 Email: wfspotz at sandia.gov ** From oliphant at enthought.com Wed Apr 28 18:18:38 2010 From: oliphant at enthought.com (Travis Oliphant) Date: Wed, 28 Apr 2010 17:18:38 -0500 Subject: [Numpy-discussion] proposing a "beware of [as]matrix()" warning In-Reply-To: <6A5E3F22-85FF-48ED-ADF6-B02B260F294D@cs.toronto.edu> References: <4BD62D8D.8060401@cs.toronto.edu> <4BD85D63.5080808@student.matnat.uio.no> <4BD87EB3.6010606@american.edu> <6A5E3F22-85FF-48ED-ADF6-B02B260F294D@cs.toronto.edu> Message-ID: <77DB9948-CC01-4B7A-A7E9-73BE460D3AA3@enthought.com> On Apr 28, 2010, at 4:46 PM, David Warde-Farley wrote: > On 2010-04-28, at 2:30 PM, Alan G Isaac wrote: > >> Please let us not have this discussion all over again. > > Agreed. See my preface to this discussion. > > My main objection is that it's not easy to explain to a newcomer > what the difference precisely is, how they interact, why two of them > exist, how they are sort-of-compatible-but-not... > >> The matrix class is very useful for teaching. >> In economics for example, the use of matrix algebra >> is widespread, while algebra with arrays that are >> not matrices is very rare. I can (and do) use NumPy >> matrices even in undergraduate courses. > > Would it be acceptable to retain the matrix class but not have it > imported in the default namespace, and have to import e.g. > numpy.matlib to get at them? > >> If you do not like them, do not use them. > > The problem isn't really with seasoned users of NumPy not liking > them, but rather new users being confused by the presence of (what > seems to be) two primitives, array and matrix. > > Two things tend to happen: > > a) Example code that expects arrays instead receives matrices. If > these aren't "cast" with asarray(), mayhem ensues at the first sight > of *. > > b) Users of class matrix use a proper function correctly coerces > input to ndarray, but returns an ndarray. Users are thus confused > that, thinking of the function as a black box, putting matrices 'in' > doesn't result in getting matrices 'out'. It doesn't take long to > get the hang of if you really sit down and work it through, but it > also "doesn't take long" to go back to MATLAB or whatever else. My > interest is in having as few conceptual stumbling stones as possible. > > c) Complicating the situation further, people try to use functions > e.g. from scipy.optimize which expect a 1d array by passing in > column or row matrices. Even when coerced to array, these have the > wrong rank and you get unexpected results (you could argue that we > should instead use asarray(..).squeeze() on all incoming arguments, > but this may not generalize well). > >> PS There is one change I would not mind: let >> A * M be undefined if A is an ndarray and >> M is a NumPy matrix. > > What about the other binary ops? I would say, matrix goes with > matrix, array with array, never the two shall meet unless you > explicitly coerce. The ability to mix the two in a single expression > does more harm than good, IMHO. This could be done in NumPy 2.0. I'm +1 on not allowing a mix of the two. -Travis -------------- next part -------------- An HTML attachment was scrubbed... URL: From aisaac at american.edu Wed Apr 28 20:43:15 2010 From: aisaac at american.edu (Alan G Isaac) Date: Wed, 28 Apr 2010 20:43:15 -0400 Subject: [Numpy-discussion] proposing a "beware of [as]matrix()" warning In-Reply-To: <6A5E3F22-85FF-48ED-ADF6-B02B260F294D@cs.toronto.edu> References: <4BD62D8D.8060401@cs.toronto.edu> <4BD85D63.5080808@student.matnat.uio.no> <4BD87EB3.6010606@american.edu> <6A5E3F22-85FF-48ED-ADF6-B02B260F294D@cs.toronto.edu> Message-ID: <4BD8D623.10502@american.edu> > Alan wrote: >> There is one change I would not mind: let >> A * M be undefined if A is an ndarray and >> M is a NumPy matrix. On 4/28/2010 5:46 PM, David Warde-Farley wrote: > What about the other binary ops? I would say, matrix goes with matrix, > array with array, never the two shall meet unless you explicitly > coerce. The ability to mix the two in a single expression does more > harm than good, IMHO. I would be fine with this. I mentioned ``*`` explicitly, because it is the dangerous one (most likely to cause surprises). Since M.A gives the array representation of the matrix M, explicit "coercion" is almost costless. Alan Isaac PS Just to be clear, I'm just a user ... From aisaac at american.edu Wed Apr 28 20:46:31 2010 From: aisaac at american.edu (Alan G Isaac) Date: Wed, 28 Apr 2010 20:46:31 -0400 Subject: [Numpy-discussion] proposing a "beware of [as]matrix()" warning In-Reply-To: <6A5E3F22-85FF-48ED-ADF6-B02B260F294D@cs.toronto.edu> References: <4BD62D8D.8060401@cs.toronto.edu> <4BD85D63.5080808@student.matnat.uio.no> <4BD87EB3.6010606@american.edu> <6A5E3F22-85FF-48ED-ADF6-B02B260F294D@cs.toronto.edu> Message-ID: <4BD8D6E7.5000701@american.edu> On 4/28/2010 5:46 PM, David Warde-Farley wrote: > Would it be acceptable to retain the matrix class but not have it > imported in the default namespace, and have to import e.g. > numpy.matlib to get at them? If we can have A * M undefined, then I do not think this is a needed addition. But I do not have a strong opinion. It does not create a significant barrier in teaching (and might even have some advantages). Alan From seb.haase at gmail.com Thu Apr 29 03:52:35 2010 From: seb.haase at gmail.com (Sebastian Haase) Date: Thu, 29 Apr 2010 09:52:35 +0200 Subject: [Numpy-discussion] ndimage.label - howto force SWIG to use int32 - even on 64bit Linux ? In-Reply-To: References: Message-ID: Thanks for those replies. But isn't npy_intp about pointers ? I would need something like npy_int32 . But does that exist ? Where is the list of types that numpy.i supports ? Also, BTW, is there code duplication if numpy.i supports (let's say) both npy_int and npy_int32 on a machine, where "int" is already 32 bits by default !? (I'm talking about the size of the resulting object file, I guess...) Thanks again for your help, Sebastian On Wed, Apr 28, 2010 at 11:51 PM, Bill Spotz wrote: > Both types of typemaps are enabled, so you just need to do you %apply > directives correctly: > > ? %apply (npy_intp* IN_ARRAY1, int DIM1) {(npy_intp* seq, int n)}; > ? etc.... > > SWIG should be able to figure it out from there. > > On Apr 28, 2010, at 12:58 PM, Charles R Harris wrote: > >> On Tue, Apr 27, 2010 at 2:27 AM, Sebastian Haase >> wrote: >> Hi, >> I wanted to write some C code to accept labels as they come from >> ndimage.label. >> For some reason ndimage.label produces its output as an int32 array - >> even on my 64bit system . >> >> BTW, could this be considered a bug ? >> >> >> Likely. >> >> Now, if I use the typemaps of numpy.i I can choose between NPY_LONG >> and NPY_INT. >> But those are sometimes 32 sometimes 64 bit, depending on the system. >> >> Any ideas ... ? >> >> npy_intp. >> >> Chuck > > ** Bill Spotz ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?** > ** Sandia National Laboratories ?Voice: (505)845-0170 ? ? ?** > ** P.O. Box 5800 ? ? ? ? ? ? ? ? Fax: ? (505)284-0154 ? ? ?** > ** Albuquerque, NM 87185-0370 ? ?Email: wfspotz at sandia.gov ** > > > > > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > From dagss at student.matnat.uio.no Thu Apr 29 03:58:12 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 29 Apr 2010 09:58:12 +0200 Subject: [Numpy-discussion] proposing a "beware of [as]matrix()" warning In-Reply-To: <4BD87EB3.6010606@american.edu> References: <4BD62D8D.8060401@cs.toronto.edu> <4BD85D63.5080808@student.matnat.uio.no> <4BD87EB3.6010606@american.edu> Message-ID: <4BD93C14.5090006@student.matnat.uio.no> Alan G Isaac wrote: > On 4/28/2010 12:08 PM, Dag Sverre Seljebotn wrote: > >> it would be good to deprecate the matrix class >> from NumPy >> > > > Please let us not have this discussion all over again. > > The matrix class is very useful for teaching. > In economics for example, the use of matrix algebra > is widespread, while algebra with arrays that are > not matrices is very rare. I can (and do) use NumPy > matrices even in undergraduate courses. > > If you do not like them, do not use them. > > If you want `matrix` replaced with a better matrix > object, offer a replacement for community consideration. > Point taken, I'm sorry. (If or when my own matrix linear algebra code matures I'll open source it for sure, but it's not there yet, and I don't believe it will fit within NumPy.) Dag Sverre From wright at esrf.fr Thu Apr 29 04:28:19 2010 From: wright at esrf.fr (Jon Wright) Date: Thu, 29 Apr 2010 10:28:19 +0200 Subject: [Numpy-discussion] put with increment ? In-Reply-To: References: Message-ID: <4BD94323.7000702@esrf.fr> Hello everyone, Is there a 'numpy' efficient way to do the following loop: for i, v in indices, values: total[ i ] += v The behaviour is like numpy.put, but instead of overwriting the array element, it is incremented. Currently I have a little C extension which does the job, but I'm lazy to fix it up for all the various dtypes etc. Am I missing a more general way? I have repeated values in indices, so that: total[indices]+=values ... does not give the result I'm after. Thanks Jon From robert.kern at gmail.com Thu Apr 29 12:20:39 2010 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 29 Apr 2010 11:20:39 -0500 Subject: [Numpy-discussion] put with increment ? In-Reply-To: <4BD94323.7000702@esrf.fr> References: <4BD94323.7000702@esrf.fr> Message-ID: On Thu, Apr 29, 2010 at 03:28, Jon Wright wrote: > Hello everyone, > > Is there a 'numpy' efficient way to do the following loop: > > for i, v in indices, values: > ? ? total[ i ] += v > > > The behaviour is like numpy.put, but instead of overwriting the array > element, it is incremented. Currently I have a little C extension which > does the job, but I'm lazy to fix it up for all the various dtypes etc. > Am I missing a more general way? total = np.bincount(indices, values) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From charlesr.harris at gmail.com Thu Apr 29 12:56:48 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 29 Apr 2010 10:56:48 -0600 Subject: [Numpy-discussion] What should be the value of nansum of nan's? In-Reply-To: References: Message-ID: On Wed, Apr 28, 2010 at 11:56 AM, T J wrote: > On Mon, Apr 26, 2010 at 10:03 AM, Charles R Harris > wrote: > > > > > > On Mon, Apr 26, 2010 at 10:55 AM, Charles R Harris > > wrote: > >> > >> Hi All, > >> > >> We need to make a decision for ticket #1123 regarding what nansum should > >> return when all values are nan. At some earlier point it was zero, but > >> currently it is nan, in fact it is nan whatever the operation is. That > is > >> consistent, simple and serves to mark the array or axis as containing > all > >> nans. I would like to close the ticket and am a bit inclined to go with > the > >> current behaviour although there is an argument to be made for returning > 0 > >> for the nansum case. Thoughts? > >> > > > > To add a bit of context, one could argue that the results should be > > consistent with the equivalent operations on empty arrays and always be > > non-nan. > > > > In [1]: nansum([]) > > Out[1]: nan > > > > In [2]: sum([]) > > Out[2]: 0.0 > > > > This seems like an obvious one to me. What is the spirit of nansum? > > """ > Return the sum of array elements over a given axis treating > Not a Numbers (NaNs) as zero. > """ > > Okay. So NaNs in an array are treated as zeros and the sum is > performed as one normally would perform it starting with an initial > sum of zero. So if all values are NaN, then we add nothing to our > original sum and still return 0. > > I'm not sure I understand the argument that it should return NaN. It > is counter to the *purpose* of nansum. Also, if one wants to > determine if all values in an array are NaN, isn't there another way? > Let's keep (or make) those distinct operations, as they are definitely > distinct concepts. > __ > It looks like the consensus is that zero should be returned. This is a change from current behaviour and that bothers me a bit. Here are some other oddities In [6]: nanmax([nan]) Out[6]: nan In [7]: nanargmax([nan]) Out[7]: nan In [8]: nanargmax([1]) Out[8]: 0 So it looks like the current behaviour is very much tilted towards nans as missing data flags. I think we should just leave that as is with perhaps a note in the docs to that effect. The decision here should probably accommodate the current users of these functions, of which I am not one. If we leave the current behaviour as is then I think the rest of the nan functions need fixes to return nan for empty sequences as nansum is the only one that currently does that. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From pav at iki.fi Thu Apr 29 15:30:31 2010 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 29 Apr 2010 19:30:31 +0000 (UTC) Subject: [Numpy-discussion] Adding an ndarray.dot method References: <4BD62D8D.8060401@cs.toronto.edu> <55C0F57B-A246-4D16-B6EF-5C6F58CEEA37@enthought.com> <4BD87A77.9030108@american.edu> Message-ID: Wed, 28 Apr 2010 14:12:07 -0400, Alan G Isaac wrote: [clip] > Here is a related ticket that proposes a more explicit alternative: > adding a ``dot`` method to ndarray. > http://projects.scipy.org/numpy/ticket/1456 I kind of like this idea. Simple, obvious, and leads to clear code: a.dot(b).dot(c) or in another multiplication order, a.dot(b.dot(c)) And here's an implementation: http://github.com/pv/numpy-work/commit/414429ce0bb0c4b7e780c4078c5ff71c113050b6 I think I'm going to apply this, unless someone complains, as I don't see any downsides (except maybe adding one more to the huge list of methods ndarray already has). Cheers, Pauli From jhapk at utias.utoronto.ca Thu Apr 29 16:23:39 2010 From: jhapk at utias.utoronto.ca (Pradeep Jha) Date: Thu, 29 Apr 2010 16:23:39 -0400 Subject: [Numpy-discussion] Numpy compilation error In-Reply-To: References: <4BCD0490.1010108@noaa.gov> <4BCF3491.5020404@noaa.gov> Message-ID: Hi, I have a few questions: 1) I downloaded numpy1.3.0 and installed it in a directory using the command *python setup.py install --prefix=$HOME/src/numpy *and I see that numpy files have been generated in that directory. Now when I tried to install matplotlib, it complained that my numpy version is 1.0.3. When I go in python mode on command line I get the following output: *[pradeep at scaramanga ~]$python Python 2.4.4 (#1, Oct 23 2006, 13:58:00) [GCC 4.1.1 20061011 (Red Hat 4.1.1-30)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> print numpy.__version__ 1.0.3 * 2) I donwloaded Scipy.0.7.2 from http://sourceforge.net/projects/scipy/files/ I installed it using *python setup.py install --prefix=$HOME/src/scipy * Again when I goto command line python, I get the following output *[pradeep at scaramanga ~]$python Python 2.4.4 (#1, Oct 23 2006, 13:58:00) [GCC 4.1.1 20061011 (Red Hat 4.1.1-30)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import scipy /usr/lib/python2.4/site-packages/scipy/misc/__init__.py:25: DeprecationWarning: ScipyTest is now called NumpyTest; please update your code test = ScipyTest().test * Why isn't my python reading things from the newer installed versions? Thanks PRadeep 2010/4/21 David Cournapeau > On Thu, Apr 22, 2010 at 2:23 AM, Christopher Barker > wrote: > > > I hadn't read the thread carefully enough to realize that the OP was > > asking about a non-root install, but in any case, I'd still encourage > > that folks set ONE standard place in their PYTHONPATH, rather than one > > for each package, and make sure you're careful about running more than > > one version of Python. > > The beauty of --user is that you don't need PYTHONPATH, and it is > interpreter specific (at least if the package is correctly done). > PYTHONPATH is becoming a pain with python 3.* > > cheers, > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Apr 29 17:20:30 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 29 Apr 2010 15:20:30 -0600 Subject: [Numpy-discussion] ndimage.label - howto force SWIG to use int32 - even on 64bit Linux ? In-Reply-To: References: Message-ID: On Thu, Apr 29, 2010 at 1:52 AM, Sebastian Haase wrote: > Thanks for those replies. > But isn't npy_intp about pointers ? > At one point perhaps, but how it is used is to get 32 bits on 32 OS's and 64 bits on 64 bit OS's, For the common architectures int will always be 32 bits regardless, it is the long type that moves around depending on both bits and os. > I would need something like npy_int32 . > But does that exist ? Where is the list of types that numpy.i supports ? > Also, BTW, is there code duplication if numpy.i supports (let's say) > both npy_int and npy_int32 on a machine, where "int" is already 32 > bits by default !? (I'm talking about the size of the resulting object > file, I guess...) > > Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Thu Apr 29 19:46:18 2010 From: matthew.brett at gmail.com (Matthew Brett) Date: Thu, 29 Apr 2010 16:46:18 -0700 Subject: [Numpy-discussion] Adding an ndarray.dot method In-Reply-To: References: <4BD62D8D.8060401@cs.toronto.edu> <55C0F57B-A246-4D16-B6EF-5C6F58CEEA37@enthought.com> <4BD87A77.9030108@american.edu> Message-ID: Hi, > I kind of like this idea. Simple, obvious, and leads > to clear code: > > ? ? ? ?a.dot(b).dot(c) > > or in another multiplication order, > > ? ? ? ?a.dot(b.dot(c)) > > And here's an implementation: > > ? ? ? ?http://github.com/pv/numpy-work/commit/414429ce0bb0c4b7e780c4078c5ff71c113050b6 > > I think I'm going to apply this, unless someone complains, as I > don't see any downsides (except maybe adding one more to the > huge list of methods ndarray already has). Excellent excellent excellent. Once again, I owe you a beverage of your choice. Matthew From oliphant at enthought.com Thu Apr 29 19:48:31 2010 From: oliphant at enthought.com (Travis Oliphant) Date: Thu, 29 Apr 2010 18:48:31 -0500 Subject: [Numpy-discussion] Adding an ndarray.dot method In-Reply-To: References: <4BD62D8D.8060401@cs.toronto.edu> <55C0F57B-A246-4D16-B6EF-5C6F58CEEA37@enthought.com> <4BD87A77.9030108@american.edu> Message-ID: <146A520B-E61B-4031-BC54-704E4CF0B5CB@enthought.com> On Apr 29, 2010, at 2:30 PM, Pauli Virtanen wrote: > Wed, 28 Apr 2010 14:12:07 -0400, Alan G Isaac wrote: > [clip] >> Here is a related ticket that proposes a more explicit alternative: >> adding a ``dot`` method to ndarray. >> http://projects.scipy.org/numpy/ticket/1456 > > I kind of like this idea. Simple, obvious, and leads > to clear code: > > a.dot(b).dot(c) > > or in another multiplication order, > > a.dot(b.dot(c)) > > And here's an implementation: > > http://github.com/pv/numpy-work/commit/414429ce0bb0c4b7e780c4078c5ff71c113050b6 > > I think I'm going to apply this, unless someone complains, as I > don't see any downsides (except maybe adding one more to the > huge list of methods ndarray already has). > +1 -Travis -------------- next part -------------- An HTML attachment was scrubbed... URL: From josef.pktd at gmail.com Thu Apr 29 19:52:58 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 29 Apr 2010 19:52:58 -0400 Subject: [Numpy-discussion] What should be the value of nansum of nan's? In-Reply-To: References: Message-ID: On Thu, Apr 29, 2010 at 12:56 PM, Charles R Harris wrote: > > > On Wed, Apr 28, 2010 at 11:56 AM, T J wrote: >> >> On Mon, Apr 26, 2010 at 10:03 AM, Charles R Harris >> wrote: >> > >> > >> > On Mon, Apr 26, 2010 at 10:55 AM, Charles R Harris >> > wrote: >> >> >> >> Hi All, >> >> >> >> We need to make a decision for ticket #1123 regarding what nansum >> >> should >> >> return when all values are nan. At some earlier point it was zero, but >> >> currently it is nan, in fact it is nan whatever the operation is. That >> >> is >> >> consistent, simple and serves to mark the array or axis as containing >> >> all >> >> nans. I would like to close the ticket and am a bit inclined to go with >> >> the >> >> current behaviour although there is an argument to be made for >> >> returning 0 >> >> for the nansum case. Thoughts? >> >> >> > >> > To add a bit of context, one could argue that the results should be >> > consistent with the equivalent operations on empty arrays and always be >> > non-nan. >> > >> > In [1]: nansum([]) >> > Out[1]: nan >> > >> > In [2]: sum([]) >> > Out[2]: 0.0 >> > >> >> This seems like an obvious one to me. ?What is the spirit of nansum? >> >> """ >> ? ?Return the sum of array elements over a given axis treating >> ? ?Not a Numbers (NaNs) as zero. >> """ >> >> Okay. ?So NaNs in an array are treated as zeros and the sum is >> performed as one normally would perform it starting with an initial >> sum of zero. ?So if all values are NaN, then we add nothing to our >> original sum and still return 0. >> >> I'm not sure I understand the argument that it should return NaN. ?It >> is counter to the *purpose* of nansum. ? Also, if one wants to >> determine if all values in an array are NaN, isn't there another way? >> Let's keep (or make) those distinct operations, as they are definitely >> distinct concepts. >> __ > > It looks like the consensus is that zero should be returned. This is a > change from current behaviour and that bothers me a bit. Here are some other > oddities > > In [6]: nanmax([nan]) > Out[6]: nan > > In [7]: nanargmax([nan]) > Out[7]: nan > > In [8]: nanargmax([1]) > Out[8]: 0 > > So it looks like the current behaviour is very much tilted towards nans as > missing data flags. I think we should just leave that as is with perhaps a > note in the docs to that effect. The decision here should probably > accommodate the current users of these functions, of which I am not one. If > we leave the current behaviour as is then I think the rest of the nan > functions need fixes to return nan for empty sequences as nansum is the only > one that currently does that. I disagree, I really would like to get nansum([Nan]) to be zero max, min don't have a neutral element, only sum and prod have it (and cumsum and cumprod and maybe others), so in this case nan is the obvious answer (besides an exception): >>> np.max([]) Traceback (most recent call last): File "", line 1, in np.max([]) File "C:\Programs\Python25\lib\site-packages\numpy\core\fromnumeric.py", line 1765, in amax return _wrapit(a, 'max', axis, out) File "C:\Programs\Python25\lib\site-packages\numpy\core\fromnumeric.py", line 37, in _wrapit result = getattr(asarray(obj),method)(*args, **kwds) ValueError: zero-size array to ufunc.reduce without identity >>> np.argmax([]) Traceback (most recent call last): File "", line 1, in np.argmax([]) File "C:\Programs\Python25\lib\site-packages\numpy\core\fromnumeric.py", line 717, in argmax return _wrapit(a, 'argmax', axis) File "C:\Programs\Python25\lib\site-packages\numpy\core\fromnumeric.py", line 37, in _wrapit result = getattr(asarray(obj),method)(*args, **kwds) ValueError: attempt to get argmax/argmin of an empty sequence >>> >>> min([]) Traceback (most recent call last): File "", line 1, in min([]) ValueError: min() arg is an empty sequence >>> max([]) Traceback (most recent call last): File "", line 1, in max([]) ValueError: max() arg is an empty sequence Josef > > Chuck > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From kwgoodman at gmail.com Thu Apr 29 20:00:52 2010 From: kwgoodman at gmail.com (Keith Goodman) Date: Thu, 29 Apr 2010 17:00:52 -0700 Subject: [Numpy-discussion] What should be the value of nansum of nan's? In-Reply-To: References: Message-ID: On Thu, Apr 29, 2010 at 9:56 AM, Charles R Harris wrote: > It looks like the consensus is that zero should be returned. This is a > change from current behaviour and that bothers me a bit. Here are some other > oddities > > In [6]: nanmax([nan]) > Out[6]: nan > > In [7]: nanargmax([nan]) > Out[7]: nan > > In [8]: nanargmax([1]) > Out[8]: 0 > > So it looks like the current behaviour is very much tilted towards nans as > missing data flags. I think we should just leave that as is with perhaps a > note in the docs to that effect. The decision here should probably > accommodate the current users of these functions, of which I am not one. That works well for me. > If > we leave the current behaviour as is then I think the rest of the nan > functions need fixes to return nan for empty sequences as nansum is the only > one that currently does that. From charlesr.harris at gmail.com Thu Apr 29 20:14:06 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 29 Apr 2010 18:14:06 -0600 Subject: [Numpy-discussion] Adding an ndarray.dot method In-Reply-To: References: <4BD62D8D.8060401@cs.toronto.edu> <55C0F57B-A246-4D16-B6EF-5C6F58CEEA37@enthought.com> <4BD87A77.9030108@american.edu> Message-ID: On Thu, Apr 29, 2010 at 1:30 PM, Pauli Virtanen wrote: > Wed, 28 Apr 2010 14:12:07 -0400, Alan G Isaac wrote: > [clip] > > Here is a related ticket that proposes a more explicit alternative: > > adding a ``dot`` method to ndarray. > > http://projects.scipy.org/numpy/ticket/1456 > > I kind of like this idea. Simple, obvious, and leads > to clear code: > > a.dot(b).dot(c) > > or in another multiplication order, > > a.dot(b.dot(c)) > > And here's an implementation: > > > http://github.com/pv/numpy-work/commit/414429ce0bb0c4b7e780c4078c5ff71c113050b6 > > I think I'm going to apply this, unless someone complains, as I > don't see any downsides (except maybe adding one more to the > huge list of methods ndarray already has). > > Hey, that was my weekend project 8-) But obviously I think it is a good idea. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From charlesr.harris at gmail.com Thu Apr 29 20:52:45 2010 From: charlesr.harris at gmail.com (Charles R Harris) Date: Thu, 29 Apr 2010 18:52:45 -0600 Subject: [Numpy-discussion] Adding an ndarray.dot method In-Reply-To: References: <4BD62D8D.8060401@cs.toronto.edu> <55C0F57B-A246-4D16-B6EF-5C6F58CEEA37@enthought.com> <4BD87A77.9030108@american.edu> Message-ID: On Thu, Apr 29, 2010 at 1:30 PM, Pauli Virtanen wrote: > Wed, 28 Apr 2010 14:12:07 -0400, Alan G Isaac wrote: > [clip] > > Here is a related ticket that proposes a more explicit alternative: > > adding a ``dot`` method to ndarray. > > http://projects.scipy.org/numpy/ticket/1456 > > I kind of like this idea. Simple, obvious, and leads > to clear code: > > a.dot(b).dot(c) > > or in another multiplication order, > > a.dot(b.dot(c)) > > And here's an implementation: > > > http://github.com/pv/numpy-work/commit/414429ce0bb0c4b7e780c4078c5ff71c113050b6 > > I think I'm going to apply this, unless someone complains, as I > don't see any downsides (except maybe adding one more to the > huge list of methods ndarray already has). > > That should do it. I was going to link directly to the code in multiarraymodule, maybe break it out into a separate dot.c file, but the call to the python function gets to the goal with less effort. Chuck -------------- next part -------------- An HTML attachment was scrubbed... URL: From dwf at cs.toronto.edu Thu Apr 29 21:06:15 2010 From: dwf at cs.toronto.edu (David Warde-Farley) Date: Thu, 29 Apr 2010 21:06:15 -0400 Subject: [Numpy-discussion] Adding an ndarray.dot method In-Reply-To: References: <4BD62D8D.8060401@cs.toronto.edu> <55C0F57B-A246-4D16-B6EF-5C6F58CEEA37@enthought.com> <4BD87A77.9030108@american.edu> Message-ID: <4BDA2D07.8050208@cs.toronto.edu> Pauli Virtanen wrote: > Wed, 28 Apr 2010 14:12:07 -0400, Alan G Isaac wrote: > [clip] > >> Here is a related ticket that proposes a more explicit alternative: >> adding a ``dot`` method to ndarray. >> http://projects.scipy.org/numpy/ticket/1456 >> > > I kind of like this idea. Simple, obvious, and leads > to clear code: > > a.dot(b).dot(c) > > or in another multiplication order, > > a.dot(b.dot(c)) > > And here's an implementation: > > http://github.com/pv/numpy-work/commit/414429ce0bb0c4b7e780c4078c5ff71c113050b6 > > I think I'm going to apply this, unless someone complains, as I > don't see any downsides (except maybe adding one more to the > huge list of methods ndarray already has). > Wonderful. Thanks for jumping on it. David From cournape at gmail.com Thu Apr 29 23:04:51 2010 From: cournape at gmail.com (David Cournapeau) Date: Fri, 30 Apr 2010 12:04:51 +0900 Subject: [Numpy-discussion] Numpy compilation error In-Reply-To: References: <4BCD0490.1010108@noaa.gov> <4BCF3491.5020404@noaa.gov> Message-ID: On Fri, Apr 30, 2010 at 5:23 AM, Pradeep Jha wrote: > Hi, > > I have a few questions: > > 1) > I downloaded numpy1.3.0 and installed it in a directory using the command > python setup.py install --prefix=$HOME/src/numpy > and I see that numpy files have been generated in that directory. > > Now when I tried to install matplotlib, it complained that my numpy version > is 1.0.3 That's because by default, python does not look into $HOME/src/numpy/.... You have several ways of controlling that, but the easy one is PYTHONPATH. This environment variable should contain your site-package, which is the $prefix/lib/python2.4/site-packages here where $prefix is what you gave to --prefix. PYTHONPATH is only on of the way to modify sys.path, which is what is used by python in the end: # python will only look for packages which are sys.path python -c "import sys; print sys.path" You can check the path of the package after import: python -c "import numpy; print numpy.__file__" David From gael.varoquaux at normalesup.org Fri Apr 30 01:00:01 2010 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Fri, 30 Apr 2010 07:00:01 +0200 Subject: [Numpy-discussion] Adding an ndarray.dot method In-Reply-To: References: <4BD62D8D.8060401@cs.toronto.edu> <55C0F57B-A246-4D16-B6EF-5C6F58CEEA37@enthought.com> <4BD87A77.9030108@american.edu> Message-ID: <20100430050001.GA29637@phare.normalesup.org> On Thu, Apr 29, 2010 at 07:30:31PM +0000, Pauli Virtanen wrote: > a.dot(b.dot(c)) > And here's an implementation: > http://github.com/pv/numpy-work/commit/414429ce0bb0c4b7e780c4078c5ff71c113050b6 /me very happy. Ga?l From jhapk at utias.utoronto.ca Fri Apr 30 10:01:36 2010 From: jhapk at utias.utoronto.ca (Pradeep Jha) Date: Fri, 30 Apr 2010 10:01:36 -0400 Subject: [Numpy-discussion] Numpy compilation error In-Reply-To: References: <4BCD0490.1010108@noaa.gov> <4BCF3491.5020404@noaa.gov> Message-ID: Thanks. That works. 2010/4/29 David Cournapeau > On Fri, Apr 30, 2010 at 5:23 AM, Pradeep Jha > wrote: > > Hi, > > > > I have a few questions: > > > > 1) > > I downloaded numpy1.3.0 and installed it in a directory using the command > > python setup.py install --prefix=$HOME/src/numpy > > and I see that numpy files have been generated in that directory. > > > > Now when I tried to install matplotlib, it complained that my numpy > version > > is 1.0.3 > > That's because by default, python does not look into > $HOME/src/numpy/.... You have several ways of controlling that, but > the easy one is PYTHONPATH. This environment variable should contain > your site-package, which is the $prefix/lib/python2.4/site-packages > here where $prefix is what you gave to --prefix. > > PYTHONPATH is only on of the way to modify sys.path, which is what is > used by python in the end: > > # python will only look for packages which are sys.path > python -c "import sys; print sys.path" > > You can check the path of the package after import: > python -c "import numpy; print numpy.__file__" > > David > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fadhley.salim at ca-cib.com Fri Apr 30 12:18:39 2010 From: fadhley.salim at ca-cib.com (Salim, Fadhley (CA-CIB)) Date: Fri, 30 Apr 2010 17:18:39 +0100 Subject: [Numpy-discussion] How to make scipy.interpolate give a an extrapolated result beyond the input range? Message-ID: I'm trying to port a program which currently uses a hand-rolled C++ interpolator (developed by a mathematician colleage) over to use the interpolators provided by scipy. I'd like to use or wrap the scipy interpolator so that it's behavior is as close as possible behavior to our old interpolator. A key difference between the two functions is that in our original interpolator - if the input value is above or below the input range, our original interpolator will extrapolate the result. If you try this with the scipy interpolator it raises a ValueError. Consider this program as an example: # EXAMPLE import numpy as np from scipy import interpolate x = np.arange(0,10) y = np.exp(-x/3.0) f = interpolate.interp1d(x, y) print f(9) print f(11) ##### Causes ValueError, because it's greater than max(x) # END OF EXAMPLE In the example above, I'd like the last line not to raise a ValueError, but to return a value calculated from the gradient of the line between f(x[-2]) and f(x[-1]). Is there a sensible way to make it so that instead of crashing, the final line will simply do a linear extrapolate, continuing the gradients defined by the first and last pairs of input data-points to infinity? I know that this is a simple enough function to write myself, however I'd rather not re-invent the wheel, especially as if I wanted to introduce new basic math functions into our library they would need to be validated by a number of gate-keepers before they were permitted into our library! I'm on Python 2.4, scipy 0.7 on Windows XP, 32bit Incidentally, I have seen this tutorial which has a "left" and "right" argument on the interpolator. This does not seem to exist on any version of the interp1d function which I can use on Windows Python 2.4 - can anybody speculate which version of Scipy this tutorial is intended for? http://projects.scipy.org/scipy/browser/branches/Interpolate1D/docs/tuto rial.rst?rev=4591 Sal -------------- next part -------------- This email does not create a legal relationship between any member of the Cr?dit Agricole group and the recipient or constitute investment advice. The content of this email should not be copied or disclosed (in whole or part) to any other person. It may contain information which is confidential, privileged or otherwise protected from disclosure. If you are not the intended recipient, you should notify us and delete it from your system. Emails may be monitored, are not secure and may be amended, destroyed or contain viruses and in communicating with us such conditions are accepted. Any content which does not relate to business matters is not endorsed by us. Cr?dit Agricole Corporate and Investment Bank is authorised by the Comit? des Etablissements de Cr?dit et des Entreprises d'Investissement (CECEI) and supervised by the Commission Bancaire in France and subject to limited regulation by the Financial Services Authority. Details about the extent of our regulation by the Financial Services Authority are available from us on request. Cr?dit Agricole Corporate and Investment Bank is incorporated in France with limited liability and registered in England & Wales. Registration number: FC008194. Registered office: Broadwalk House, 5 Appold Street, London, EC2A 2DA. From josef.pktd at gmail.com Fri Apr 30 13:03:13 2010 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 30 Apr 2010 13:03:13 -0400 Subject: [Numpy-discussion] How to make scipy.interpolate give a an extrapolated result beyond the input range? In-Reply-To: References: Message-ID: On Fri, Apr 30, 2010 at 12:18 PM, Salim, Fadhley (CA-CIB) wrote: > I'm trying to port a program which currently uses a hand-rolled C++ > interpolator (developed by a mathematician colleage) over to use the > interpolators provided by scipy. I'd like to use or wrap the scipy > interpolator so that it's behavior is as close as possible behavior to > our old interpolator. > > A key difference between the two functions is that in our original > interpolator - if the input value is above or below the input range, our > original interpolator will extrapolate the result. If you try this with > the scipy interpolator it raises a ValueError. ?Consider this program as > an example: > > # EXAMPLE > import numpy as np > from scipy import interpolate > > x = np.arange(0,10) > y = np.exp(-x/3.0) > f = interpolate.interp1d(x, y) > > print f(9) > print f(11) ##### Causes ValueError, because it's greater than max(x) # > END OF EXAMPLE > > In the example above, I'd like the last line not to raise a ValueError, > but to return a value calculated from the gradient of the line between > f(x[-2]) and f(x[-1]). > > Is there a sensible way to make it so that instead of crashing, the > final line will simply do a linear extrapolate, continuing the gradients > defined by the first and last pairs of input data-points to infinity? > > I know that this is a simple enough function to write myself, however > I'd rather not re-invent the wheel, especially as if I wanted to > introduce new basic math functions into our library they would need to > be validated by a number of gate-keepers before they were permitted into > our library! > > I'm on Python 2.4, scipy 0.7 on Windows XP, 32bit > > Incidentally, I have seen this tutorial which has a "left" and "right" > argument on the interpolator. This does not seem to exist on any version > of the interp1d function which I can use on Windows Python 2.4 - can > anybody speculate which version of Scipy this tutorial is intended for? > http://projects.scipy.org/scipy/browser/branches/Interpolate1D/docs/tuto > rial.rst?rev=4591 Interesting question, after renaming, the branch is at http://projects.scipy.org/scipy/browser/branches/interpolate?rev= Does anyone know what the purpose and status of this branch is? Josef > > Sal > > This email does not create a legal relationship between any member of the Cr?dit Agricole group and the recipient or constitute investment advice. > The content of this email should not be copied or disclosed (in whole or part) to any other person. It may contain information which is > confidential, privileged or otherwise protected from disclosure. If you are not the intended recipient, you should notify us and delete it from your system. Emails may be monitored, are not secure and may be amended, destroyed or contain viruses and in communicating with us such conditions are accepted. Any content which does not relate to business matters is not endorsed by us. > > Cr?dit Agricole Corporate and Investment Bank is authorised by the Comit? des Etablissements de Cr?dit et des Entreprises d'Investissement (CECEI) and supervised by the Commission Bancaire in France and subject to limited regulation by the Financial Services Authority. > Details about the extent of our regulation by the Financial Services Authority are available from us on request. > Cr?dit Agricole Corporate and Investment Bank is incorporated in France with limited liability and registered in England & Wales. Registration number: FC008194. > Registered office: Broadwalk House, 5 Appold Street, London, EC2A 2DA. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion at scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > From fperez.net at gmail.com Fri Apr 30 19:39:26 2010 From: fperez.net at gmail.com (Fernando Perez) Date: Fri, 30 Apr 2010 16:39:26 -0700 Subject: [Numpy-discussion] Adding an ndarray.dot method In-Reply-To: References: <4BD62D8D.8060401@cs.toronto.edu> <55C0F57B-A246-4D16-B6EF-5C6F58CEEA37@enthought.com> <4BD87A77.9030108@american.edu> Message-ID: On Thu, Apr 29, 2010 at 12:30 PM, Pauli Virtanen wrote: > > I think I'm going to apply this, unless someone complains, as I > don't see any downsides (except maybe adding one more to the > huge list of methods ndarray already has). But one of the most badly needed ones, so run, don't walk to commit it :) Thanks! f From doutriaux1 at llnl.gov Thu Apr 29 12:17:38 2010 From: doutriaux1 at llnl.gov (=?utf-8?Q?Charles_=D8=B3=D9=85=D9=8A=D8=B1_Doutriaux?=) Date: Thu, 29 Apr 2010 09:17:38 -0700 Subject: [Numpy-discussion] numpy 1.4.1 f2py on Snow Leopard. Message-ID: Hello, I jsut built numpy 1.4.1 on my mac (64bit only no universal) Now f2py fails building any fortran code... my python: 2.6.5 meryem /svn/cdat/trunk >gcc --version i686-apple-darwin10-gcc-4.2.1 (GCC) 4.2.1 (Apple Inc. build 5646) (dot 1) Copyright (C) 2007 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. meryem /svn/cdat/trunk >gfortran --version GNU Fortran (GCC) 4.5.0 20100107 (experimental) Copyright (C) 2009 Free Software Foundation, Inc. GNU Fortran comes with NO WARRANTY, to the extent permitted by law. You may redistribute copies of GNU Fortran under the terms of the GNU General Public License. For more information about these matters, see the file named COPYING I'm also attaching bzipped logs of my numpy and scipy (as an example that would be familiar to you guys) installs. scipy obviously fails because the f2py doesn't work. Thx, C. -------------- next part -------------- A non-text attachment was scrubbed... Name: scipy.LOG.bz2 Type: application/x-bzip2 Size: 24220 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: numpy-1.4.1.tar.gz.LOG.bz2 Type: application/x-bzip2 Size: 10228 bytes Desc: not available URL: From doutriaux1 at llnl.gov Fri Apr 30 12:11:15 2010 From: doutriaux1 at llnl.gov (=?UTF-8?Q?Charles_=D8=B3=D9=85=D9=8A=D8=B1_Doutriaux?=) Date: Fri, 30 Apr 2010 09:11:15 -0700 Subject: [Numpy-discussion] Fwd: [SciPy-Dev] numpy 1.4.1 f2py on Snow Leopard. References: <5215080B-91A3-4E71-A075-B4DF828E5D29@llnl.gov> Message-ID: <29431524-C0EA-4DEE-A519-7A29C8D45074@llnl.gov> I'm forwarding this to the numpy list as well since it seems to be f2py related therefore belonging here really. C. Begin forwarded message: > From: Charles ???? Doutriaux > Date: April 30, 2010 9:10:08 AM PDT > To: SciPy Developers List > Subject: Re: [SciPy-Dev] numpy 1.4.1 f2py on Snow Leopard. > > Ok, > > It seems to be totally unrelated. Even using your suggested > gfortran. On a 32bit machine. I still get the same error. > > Since scipy is HUGE I'm sending a REALLY simple example that fails > for me. > > omar /git/cdat >gfortran --version > GNU Fortran (GCC) 4.2.3 > Copyright (C) 2007 Free Software Foundation, Inc. > > GNU Fortran comes with NO WARRANTY, to the extent permitted by law. > You may redistribute copies of GNU Fortran > under the terms of the GNU General Public License. > For more information about these matters, see the file named COPYING > > omar /git/cdat >gcc --version > i686-apple-darwin9-gcc-4.2.1 (GCC) 4.2.1 (Apple Inc. build 5574) > Copyright (C) 2007 Free Software Foundation, Inc. > This is free software; see the source for copying conditions. There > is NO > warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR > PURPOSE. > > omar /git/cdat >uname -a > Darwin omar 9.8.0 Darwin Kernel Version 9.8.0: Wed Jul 15 16:55:01 > PDT 2009; root:xnu-1228.15.4~1/RELEASE_I386 i386 > > > > On Apr 29, 2010, at 4:52 PM, Jonathan Guyer wrote: > >> On Apr 29, 2010, at 12:17 PM, Charles ???? Doutriaux wrote: >> >>> I jsut built numpy 1.4.1 on my mac (64bit only no universal) >>> Now f2py fails building any fortran code... >> >> >>> meryem /svn/cdat/trunk >gfortran --version >>> GNU Fortran (GCC) 4.5.0 20100107 (experimental) >>> Copyright (C) 2009 Free Software Foundation, Inc. >> >> I've recently been writing up instructions for building FiPy's >> prerequisites on Snow Leopard http://*matforge.org/fipy/wiki/ >> SnowLeopardSourceBuild, largely based on this blog posting: http:// >> *blog.hyperjeff.net/?p=160 >> >> There seem to be issues with gfortran 4.5.0, whereas gfortran 4.2.3 >> has worked for me and others. >> >> >> >> _______________________________________________ >> SciPy-Dev mailing list >> SciPy-Dev at scipy.org >> http://*mail.scipy.org/mailman/listinfo/scipy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ort.LOG Type: application/octet-stream Size: 19469 bytes Desc: not available URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: setup.py Type: text/x-python-script Size: 927 bytes Desc: not available URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: reado.f Type: application/octet-stream Size: 5263 bytes Desc: not available URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ortreader.pyf Type: application/octet-stream Size: 649 bytes Desc: not available URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: